curl - solr does not insert the first line in the csv file -


when csv file uploaded on curl command below

c:\>curl "http://localhost:8983/solr/update/csv?commit=true&stream.file=c:\dev\tools\solr-4.7.2\data.txt&stream.contenttype=text/csv&header=false&fieldnames=id,cat,pubyear_i,title,author, series_s,sequence_i&skiplines=0" 

and data.txt content below

book1,fantasy,2000,a storm of swords,george r.r. martin,a song of ice , fire,3 book2,fantasy,2005,a feast crows,george r.r. martin,a song of ice , fire,4 book3,fantasy,2011,a dance dragons,george r.r. martin,a song of ice , fire,5 book4,sci-fi,1987,consider phlebas,iain m. banks,the culture,1 book5,sci-fi,1988,the player of games,iain m. banks,the culture,2 book6,sci-fi,1990,use of weapons,iain m. banks,the culture,3 book7,fantasy,1984,shadows linger,glen cook,the black company,2 book8,fantasy,1984,the white rose,glen cook,the black company,3 book9,fantasy,1989,shadow games,glen cook,the black company,4 book10,sci-fi,2001,gridlinked,neal asher,ian cormac,1 book11,sci-fi,2003,the line of polity,neal asher,ian cormac,2 book12,sci-fi,2005,brass man,neal asher,ian cormac,3 

first data in data.txt file not being inserted solr id "book1". can please tell why?

http://localhost:8983/solr/query?q=id:book1 {   "responseheader":{     "status":0,     "qtime":1,     "params":{       "q":"id:book1"}},   "response":{"numfound":0,"start":0,"docs":[]   }} 

solr logs tells book1 being added.

15440876 [searcherexecutor-5-thread-1] info  org.apache.solr.core.solrcore  û [collection1] registered new searcher searcher@177fcdf1[collection1] main{standarddirectoryreader(segments_1g:124:nrt _z(4.7):c12)} 15440877 [qtp84034882-11] info  org.apache.solr.update.processor.logupdateprocessor  û [collection1] webapp=/solr path=/update params={fieldnames=id,cat,pubyear_i,title,author,series_s,sequence_i&skiplines=0&commit=true&stream.con tenttype=text/csv&header=false&stream.file=c:\dev\tools\solr-4.7.2\data.txt} {add=[?book1 (1480070032327180288), book2 (1480070032332423168), book3 (1480070032335568896), book4 (1480070032337666048), book5 (1480070032339763200), b ook6 (1480070032341860352), book7 (1480070032343957504), book8 (1480070032347103232), book9 (1480070032349200384), book10 (1480070032351297536), ... (12 adds)],commit=} 0 92 

if ask data below can see book1 still missing

http://localhost:8983/solr/query?q=id:book*&sort=pubyear_i+desc&fl=id,title,pubyear_i&rows=15 {   "responseheader":{     "status":0,     "qtime":1,     "params":{       "fl":"id,title,pubyear_i",       "sort":"pubyear_i desc",       "q":"id:book*",       "rows":"15"}},   "response":{"numfound":11,"start":0,"docs":[       {         "id":"book3",         "pubyear_i":2011,         "title":["a dance dragons"]},       {         "id":"book2",         "pubyear_i":2005,         "title":["a feast crows"]},       {         "id":"book12",         "pubyear_i":2005,         "title":["brass man"]},       {         "id":"book11",         "pubyear_i":2003,         "title":["the line of polity"]},       {         "id":"book10",         "pubyear_i":2001,         "title":["gridlinked"]},       {         "id":"book6",         "pubyear_i":1990,         "title":["use of weapons"]},       {         "id":"book9",         "pubyear_i":1989,         "title":["shadow games"]},       {         "id":"book5",         "pubyear_i":1988,         "title":["the player of games"]},       {         "id":"book4",         "pubyear_i":1987,         "title":["consider phlebas"]},       {         "id":"book7",         "pubyear_i":1984,         "title":["shadows linger"]},       {         "id":"book8",         "pubyear_i":1984,         "title":["the white rose"]}]   }} 

data.txt context hex data

0000000 ef bb bf 62 6f 6f 6b 31 2c 66 61 6e 74 61 73 79 0000020 2c 32 30 30 30 2c 41 20 53 74 6f 72 6d 20 6f 66 0000040 20 53 77 6f 72 64 73 2c 47 65 6f 72 67 65 20 52 0000060 2e 52 2e 20 4d 61 72 74 69 6e 2c 41 20 53 6f 6e 0000100 67 20 6f 66 20 49 63 65 20 61 6e 64 20 46 69 72 0000120 65 2c 33 0d 0a 62 6f 6f 6b 32 2c 66 61 6e 74 61 0000140 73 79 2c 32 30 30 35 2c 41 20 46 65 61 73 74 20 0000160 66 6f 72 20 43 72 6f 77 73 2c 47 65 6f 72 67 65 0000200 20 52 2e 52 2e 20 4d 61 72 74 69 6e 2c 41 20 53 0000220 6f 6e 67 20 6f 66 20 49 63 65 20 61 6e 64 20 46 0000240 69 72 65 2c 34 0d 0a 62 6f 6f 6b 33 2c 66 61 6e 0000260 74 61 73 79 2c 32 30 31 31 2c 41 20 44 61 6e 63 0000300 65 20 77 69 74 68 20 44 72 61 67 6f 6e 73 2c 47 0000320 65 6f 72 67 65 20 52 2e 52 2e 20 4d 61 72 74 69 0000340 6e 2c 41 20 53 6f 6e 67 20 6f 66 20 49 63 65 20 0000360 61 6e 64 20 46 69 72 65 2c 35 0d 0a 62 6f 6f 6b 0000400 34 2c 73 63 69 2d 66 69 2c 31 39 38 37 2c 43 6f 0000420 6e 73 69 64 65 72 20 50 68 6c 65 62 61 73 2c 49 0000440 61 69 6e 20 4d 2e 20 42 61 6e 6b 73 2c 54 68 65 0000460 20 43 75 6c 74 75 72 65 2c 31 0d 0a 62 6f 6f 6b 0000500 35 2c 73 63 69 2d 66 69 2c 31 39 38 38 2c 54 68 0000520 65 20 50 6c 61 79 65 72 20 6f 66 20 47 61 6d 65 0000540 73 2c 49 61 69 6e 20 4d 2e 20 42 61 6e 6b 73 2c 0000560 54 68 65 20 43 75 6c 74 75 72 65 2c 32 0d 0a 62 0000600 6f 6f 6b 36 2c 73 63 69 2d 66 69 2c 31 39 39 30 0000620 2c 55 73 65 20 6f 66 20 57 65 61 70 6f 6e 73 2c 0000640 49 61 69 6e 20 4d 2e 20 42 61 6e 6b 73 2c 54 68 0000660 65 20 43 75 6c 74 75 72 65 2c 33 0d 0a 62 6f 6f 0000700 6b 37 2c 66 61 6e 74 61 73 79 2c 31 39 38 34 2c 0000720 53 68 61 64 6f 77 73 20 4c 69 6e 67 65 72 2c 47 0000740 6c 65 6e 20 43 6f 6f 6b 2c 54 68 65 20 42 6c 61 0000760 63 6b 20 43 6f 6d 70 61 6e 79 2c 32 0d 0a 62 6f 0001000 6f 6b 38 2c 66 61 6e 74 61 73 79 2c 31 39 38 34 0001020 2c 54 68 65 20 57 68 69 74 65 20 52 6f 73 65 2c 0001040 47 6c 65 6e 20 43 6f 6f 6b 2c 54 68 65 20 42 6c 0001060 61 63 6b 20 43 6f 6d 70 61 6e 79 2c 33 0d 0a 62 0001100 6f 6f 6b 39 2c 66 61 6e 74 61 73 79 2c 31 39 38 0001120 39 2c 53 68 61 64 6f 77 20 47 61 6d 65 73 2c 47 0001140 6c 65 6e 20 43 6f 6f 6b 2c 54 68 65 20 42 6c 61 0001160 63 6b 20 43 6f 6d 70 61 6e 79 2c 34 0d 0a 62 6f 0001200 6f 6b 31 30 2c 73 63 69 2d 66 69 2c 32 30 30 31 0001220 2c 47 72 69 64 6c 69 6e 6b 65 64 2c 4e 65 61 6c 0001240 20 41 73 68 65 72 2c 49 61 6e 20 43 6f 72 6d 61 0001260 63 2c 31 0d 0a 62 6f 6f 6b 31 31 2c 73 63 69 2d 0001300 66 69 2c 32 30 30 33 2c 54 68 65 20 4c 69 6e 65 0001320 20 6f 66 20 50 6f 6c 69 74 79 2c 4e 65 61 6c 20 0001340 41 73 68 65 72 2c 49 61 6e 20 43 6f 72 6d 61 63 0001360 2c 32 0d 0a 62 6f 6f 6b 31 32 2c 73 63 69 2d 66 0001400 69 2c 32 30 30 35 2c 42 72 61 73 73 20 4d 61 6e 0001420 2c 4e 65 61 6c 20 41 73 68 65 72 2c 49 61 6e 20 0001440 43 6f 72 6d 61 63 2c 33 0d 0a 0001452 

look closely @ log... says "?book1" added (notice question mark in id). best guess there funny characters @ start of file become part of id. perhaps bom (i know text editors annoyingly add that). http://en.wikipedia.org/wiki/byte_order_mark

you verify there using "hexdump data.txt" or "od -tx1 data.txt" try different text editor allow delete that.


Comments

Popular posts from this blog

php - Submit Form Data without Reloading page -

linux - Rails running on virtual machine in Windows -

php - $params->set Array between square bracket -