Scraping data from multiple html tables within one website in python -


i trying timeseries website python: http://www.boerse-frankfurt.de/en/etfs/db+x+trackers+msci+world+information+technology+trn+index+ucits+etf+lu0540980496/price+turnover+history/historical+data#page=1

i've gotten pretty far, don't know how data , not first 50 rows can see on page. view them online, have click through results @ bottom of table. able specify start , end date in python , corresponding dates , prices in list. here have far:

 bs4 import beautifulsoup  import requests  import lxml  import re   url = 'http://www.boerse-frankfurt.de/en/etfs/db+x+trackers+msci+world+information+technology+trn+index+ucits+etf+lu0540980496/price+turnover+history/historical+data'  soup = beautifulsoup(requests.get(url).text)   dates  = soup.findall('td', class_='column-date')  dates  = [re.sub('[\\nt\s]','',d.string) d in dates]  prices = soup.findall('td', class_='column-price')  prices = [re.sub('[\\nt\s]','',p.string) p in prices]

you need loop through rest of pages. can use post request that. server expects receive structure in each post request. structure defined below in values. page number parameter 'page' of structure. structure has several parameters have not tested interesting try, items_per_page, max_time , min_time. here below example code: 

from bs4 import beautifulsoup import urllib import urllib2 import re  url = 'http://www.boerse-frankfurt.de/en/parts/boxes/history/_histdata_full.m' values = {'component_id':'preeb7da7a4f4654f818494b6189b755e76',      'ag':'103708549',      'boerse_id': '12',     'include_url': '/parts/boxes/history/_histdata_full.m',     'item_count': '96',     'items_per_page': '50',     'lang': 'en',     'link_id': '',     'max_time': '2014-09-20',     'min_time': '2014-05-09',     'page': 1,     'page_size': '50',     'pages_total': '2',     'secu': '103708549',     'template': '0',     'titel': '',     'title': '',     'title_link': '',     'use_external_secu': '1'}  dates = [] prices = [] while true:     data = urllib.urlencode(values)     request = urllib.urlopen(url, data)     soup = beautifulsoup(request.read())     temp_dates  = soup.findall('td', class_='column-date')     temp_dates  = [re.sub('[\\nt\s]','',d.string) d in temp_dates]     temp_prices = soup.findall('td', class_='column-price')     temp_prices = [re.sub('[\\nt\s]','',p.string) p in temp_prices]     if not temp_prices:         break     else:         dates = dates + temp_dates         prices = prices + temp_prices         values['page'] += 1

Comments

Post a Comment

Popular posts from this blog

php - Submit Form Data without Reloading page -

i know question has been asked , answered bunch of times life of me can not solution work on code. believe if redo code, work properly, , in retrospect time i've spent on issue should have redone code...but beyond deadline , have working prototype up... can advise how update multiple forms on single page without reloading page? horrible @ php , complete noob js/jquery/ajax @ best. html/php in single file , believe causing me problems/is largest obstacle @ getting ajax form processing work. basically, want users to: select teams multiple dropdowns in form group 1 (league champs) , on update update db newly selected team without reloadding page. edit team scores per match , on score submit/update, update db new scores. for each requirement there multiple forms (1 each leagues, 1 each per match) have set class on each form id. everything works via php, form submission reloads page each time submitted , not ideal. i have tried ajax code here http://scotch.io/tutoria...
Read more

linux - Rails running on virtual machine in Windows -

the reason why want switch ubuntu people said gems works better on linux on windows. i don't want macbook pro rails development since love windows , cannot stand lack of apps mac. moreover, people saying ubuntu "better" mac: http://www.reddit.com/r/ubuntu/comments/1bgj1s/are_there_advantages_of_ubuntu_over_os_x_for/ does virtual box work enough doesn't have lag @ all? using i7-3517u cpu 8gb memory. need program in environment lag-free , not sluggish. or there alternatives recommend? i unsure permission settings or complicated stuffs regarding terminal. i'll using terminal develop. i'm still quite new here development in linux. please kind me >< thanks. @holyxiaoxin pretty sure should asking on superuser instead. nevertheless, suggest install ubuntu in partition instead of running virtual machine. here's guide https://help.ubuntu.com/community/windowsdualboot also, yes. developing in virtual machine accurate, @ end experiment w...
Read more