python 2.7 - Selecting nodes with non-ASCII characters in Scrapy -


i have following simple web scraper written in scrapy:

#!/usr/bin/env python # -*- coding: latin-1 -*-  scrapy.http import request scrapy.spider import basespider scrapy.selector import htmlxpathselector  class myspidertest(basespider):     name = 'myspidertest'     allowed_domains = ["boliga.dk"]     start_urls = ["http://www.boliga.dk/bbrinfo/3b71489c-aea0-44ca-a0b2-7bd909b35618",]      def parse(self, response):         hxs = htmlxpathselector(response)         item = bbritem()         print hxs.select("id('unitcontrol')/div[2]/table/tbody/tr[td//text()[contains(.,'antal badeværelser')]]/td[2]/text()").extract()

but when run spider following syntax error:

syntaxerror: non-ascii character '\xe6' in file... on line 32, no encoding declared

because of æ in xpath. xpath working in xpath checker firefox. tried url-encoding æ, didn't work. missing?

thanks!

update: have added encoding declaration in beginning of code (latin-1 should support danish characters)

use unicode string xpath expression

hxs.select(u"id('unitcontrol')/div[2]/table/tbody/tr[td//text()[contains(.,'antal badeværelser')]]/td[2]/text()").extract()

or

hxs.select(u"id('unitcontrol')/div[2]/table/tbody/tr[td//text()[contains(.,'antal badev\u00e6relser')]]/td[2]/text()").extract()

see unicode literals in python source code


Comments

Post a Comment

Popular posts from this blog

c++ - CryptStringToBinary API behavior -

i seeing strange behavior cryptstringtobinary cryptography api. please see below code (config: x64 debug): #include "stdafx.h" #include <windows.h> #include <strsafe.h> #include <iostream> #include <exception> void main() { dword dwskip; dword dwflags; dword dwdatalen; //lpcwstr pszinput = l"jaaaaaecaaadzgaaakqaagdnnl1l56bwgfjdgr3rpxtqqqn6daw3usv2emkjym4t"; //this works fine lpcwstr pszinput = l"mytest"; //doesnt work, api returns false,error code 0x0000000d // determine size of byte array , allocate memory. if(! cryptstringtobinary( pszinput, _tcslen( pszinput ) + 1, crypt_string_base64, null, &dwdatalen, &dwskip, &dwflags ) ) { dword dw = getlasterror(); //0x0000000d: data invalid throw std::exception( "error computing byte length." ); } byte *pbytebyte = null; try { ...
Read more

c++ - Correct method for redrawing a layered window -

i have window created ws_ex_layered window style. drawing onto memory bitmap using gdi+, , using updatelayeredwindow update graphical content of layered window. intend use window main window of application, require redraw frequently. seeing layered windows not receive wm_paint windows message [?] , need come appropriate method re-drawing window. optimisation not essential , it's nice have cake , eat too. therefore, in search of "correct" method use. here thoughts far: i guess it's idea render onto off-screen bitmap before bitblt ing or similar. 60 frames rendered per second should (more than?) enough (but how compare other applications' frame rates?). possible solutions: use settimer send wm_timer message on regular basis. useful because through specifying time-out value, can achieve desired frames per second, without requirement measure duration "frame" takes rendered. would cause input or other lags due frequency , speed of m...
Read more

java.util.scanner - How to read and add only numbers to array from a text file -

i'm learning java , have question regarding reading file want read numbers file contains strings well. here example of file: 66.56 "3 java 3-43 5-42 2.1 1 and here coding: public class test { public static void main (string [] args){ if (0 < args.length) { file x = new file(args[0]); try{ scanner in = new scanner( new fileinputstream(x)); arraylist<double> test = new arraylist<>(); while(in.hasnext()){ if(in.hasnextdouble()){ double f=in.nextdouble(); test.add(f);} else {in.next();} } catch(ioexception e) { system.err.println("exception during reading: " + e); } } my problem add 66.56,2.1 , 1 doesn't add 3 after "3 or ignores 3-43 , 5-42 can tell me how skip strings , add doubles here? thanks all said 3 ie; "3, 3-43 , 4-42 strings either u read string , split , check number @ " , - or put in space between characters , integers. jvm after compilation tre...
Read more