python - 2.2GB JSON file parses inconsistently -


i trying decode large utf-8 json file (2.2 gb). load file so:

f = codecs.open('output.json', encoding='utf-8') data = f.read()

if try of: json.load, json.loads or json.jsondecoder().raw_decode error: 

--------------------------------------------------------------------------- valueerror                                traceback (most recent call last) <ipython-input-40-fc2255017b19> in <module>() ----> 1 j = jd.decode(data)  /usr/lib/python2.7/json/decoder.pyc in decode(self, s, _w)     367         end = _w(s, end).end()     368         if end != len(s): --> 369             raise valueerror(errmsg("extra data", s, end, len(s)))     370         return obj     371  valueerror: data: line 1 column -2065998994 - line 1 column 2228968302     (char -2065998994 - 2228968302)


uname -m shows x86_64 , 

> python -c 'import sys;print("%x" % sys.maxsize, sys.maxsize > 2**32)' ('7fffffffffffffff', true)`

so should on 64 bit , integer size shouldn't problem.

however, if run:

jd = json.jsondecoder() len(data) # 2228968302 j = jd.raw_decode(data) j[1] # 2228968302

the second value in tuple returned raw_decode end of string, raw_decode seems parse entire file seemingly no garbage @ end.

so, there should doing differently json? raw_decode decoding entire file? why json.load(s) failing?

i'd add comment, formatting capabilities in comments limited.

staring @ source code,

raise valueerror(errmsg("extra data", s, end, len(s)))

calls function:

def errmsg(msg, doc, pos, end=none):     ...     fmt = '{0}: line {1} column {2} - line {3} column {4} (char {5} - {6})'     return fmt.format(msg, lineno, colno, endlineno, endcolno, pos, end)

the (char {5} - {6}) part of format part of error message showed:

(char -2065998994 - 2228968302)

so, in errmsg(), pos -2065998994 , end 2228968302. behold! ;-):

>>> pos = -2065998994 >>> end = 2228968302 >>> 2**32 + pos 2228968302l >>> 2**32 + pos == end true

that is, pos , end "really" same. errmsg() called, means end , len(s) same - end being viewed 32-bit signed integer. end in turn comes regular expression match object's end() method.

so real problem here appears 32-bit limitation/assumption in regexp engine. encourage open bug report!

later: answer questions, yes, raw_decode() decoding entire file. other methods call raw_decode(), add (failing!) sanity checks afterwards.


Comments

Post a Comment

Popular posts from this blog

php - Add the correct number of days for each month -

using following function can display years , months between 2 dates, how can add correct days each month array within each month? can't add days manually need account leap years etc. function yearmonth($start_date, $end_date) { $begin = new datetime( $start_date ); $end = new datetime( $end_date); $interval = new dateinterval('p1m'); // 1 month interval $period = new dateperiod($begin, $interval, $end); foreach ( $period $dt ) $years[$dt->format( "y" )][] = $dt->format( "f" ); return $years; } $list = yearmonth("2007-03-24", "2009-06-26"); var_dump($list); since nobody else answered: function yearmonth($start_date, $end_date) { $begin = new datetime( $start_date ); $end = new datetime( $end_date); $interval = new dateinterval('p1d'); // 1 month interval $period = new dateperiod($begin, $interval, $end); $lastmonth = null; $lastyear = null; $aresult = array(); forea...
Read more

java.util.scanner - How to read and add only numbers to array from a text file -

i'm learning java , have question regarding reading file want read numbers file contains strings well. here example of file: 66.56 "3 java 3-43 5-42 2.1 1 and here coding: public class test { public static void main (string [] args){ if (0 < args.length) { file x = new file(args[0]); try{ scanner in = new scanner( new fileinputstream(x)); arraylist<double> test = new arraylist<>(); while(in.hasnext()){ if(in.hasnextdouble()){ double f=in.nextdouble(); test.add(f);} else {in.next();} } catch(ioexception e) { system.err.println("exception during reading: " + e); } } my problem add 66.56,2.1 , 1 doesn't add 3 after "3 or ignores 3-43 , 5-42 can tell me how skip strings , add doubles here? thanks all said 3 ie; "3, 3-43 , 4-42 strings either u read string , split , check number @ " , - or put in space between characters , integers. jvm after compilation tre...
Read more