hadoop - HIVE delimiter \n ^M issue -


i have file columns delimited ^a , rows delimited '\n' new line character.

i first uploaded hdfs , create table in hive using command this:

create external table  if not exists  html_sample  ( ts string,    url string,    html string)  row format delimited  fields terminated '\001'  lines terminated '\n'  location '/tmp/directoryname/';

however, when select statement table. turned out mess.

the table looks this:

ts              url                    html 10082013        http://url.com/01      <doctype>.....style="padding-top: 10px; text-align...   null                   null  text-align...   null                   null text-align...   null                   null 10092013        http://url.com/02      <doctype>.....style="padding-top: 10px; text-align...   null                   null  text-align...   null                   null text-align...   null                   null

then went text file , found there exist several ^m characters in file, makes hive treat ^m new line character.

when first created file, intentionally removed new line character html guarantee each record 1 line. however, cannot understand how on earth hive treat ^m newline character. how can around without modifying file.

(i know might possible global substitution in vi or sed... doesn't make sense me how hive treat ^m \n)

^m way in vim displays windows line endings. here's more on this: what ^m character mean in vim?

and hive in turn uses textinputformat happens treat valid line terminator.

depending on versions of hadoop , hive you're using there can different ways overcome this(from changing property in config custom inputformat implementation).

just find way specify separator explicitly.

and yeah, lines terminated '\n' not looks like. i'm using hive 0.11 , possible value '\n' not promoted textinputformat


Comments

Post a Comment

Popular posts from this blog

java.util.scanner - How to read and add only numbers to array from a text file -

i'm learning java , have question regarding reading file want read numbers file contains strings well. here example of file: 66.56 "3 java 3-43 5-42 2.1 1 and here coding: public class test { public static void main (string [] args){ if (0 < args.length) { file x = new file(args[0]); try{ scanner in = new scanner( new fileinputstream(x)); arraylist<double> test = new arraylist<>(); while(in.hasnext()){ if(in.hasnextdouble()){ double f=in.nextdouble(); test.add(f);} else {in.next();} } catch(ioexception e) { system.err.println("exception during reading: " + e); } } my problem add 66.56,2.1 , 1 doesn't add 3 after "3 or ignores 3-43 , 5-42 can tell me how skip strings , add doubles here? thanks all said 3 ie; "3, 3-43 , 4-42 strings either u read string , split , check number @ " , - or put in space between characters , integers. jvm after compilation tre...
Read more

rewrite - Trouble with Wordpress multiple custom querystrings -

i'm working on adding multiple querystring , doesn't seem work. i've found codes seems work others might i'm missing something. i want like: http://mydomain.com/board/?getyear=2013&getsometext2=sometext to http://mydomain.com/board/2013/sometext/ i added code below functions.php file. doesn't seem work should though. this works. echoing querystring gets both variable without problem. http://mydomain.com/board/2013/sometext/ but doesn't. tested querystring vis isset() seems nothing being set/get http://mydomain.com/board/2013/ function add_query_vars($vars) { $vars[] = "getyear"; $vars[] = "getsometext2"; return $vars; } add_filter('query_vars', 'add_query_vars'); function add_rewrite_rules($arules) { $anewrules = array('board/([^/]+)/([^/]+)/?$' => 'index.php?pagename=board&getyear=$matches[1]&getsometext2=$matches[2]'); $arules = $a...
Read more