python - BeautifulSoup4 - All links within 1 div on multiple pages -


for schoolproject need scrape 'job-finding' website , store in db, , later match these profiles companies searching people.

on particular site, url's pages need scrape in 1 div (with 10 links per page) div called 'primaryresults' has 10 in it.

with beautifulsoup wish first scrape links in array looping through page number in url until 404 or similar pops up.

then go through each of these pages, , store information need each page array , lastly send db.

now i'm getting stuck @ part collect 10 links id = 'primaryresults' div.

how go , put python make store 10 url's array? far have tried this:

import urllib2 beautifulsoup import beautifulsoup  opener = urllib2.build_opener() opener.addheaders = [("user-agent", "mozilla/5.0")]  url = ("http://jobsearch.monsterboard.nl/browse/")  content = opener.open(url).read() soup = beautifulsoup(content)  soup.find(id="primaryresults") print soup.find_all('a')

but gives error:

traceback (most recent call last):  print soup.find_all('a') typeerror: 'nonetype' object not callable

could please me out? :)

here answer links in url have mentioned

from bs4 import beautifulsoup import urllib2 url="http://jobsearch.monsterboard.nl/browse/" page=urllib2.urlopen(url) soup = beautifulsoup(page.read()) jobs=soup.findall('a',{'class':'sljobtitle'}) eachjob in jobs:  print eachjob['href']

hope clear , helpful.


Comments

Post a Comment

Popular posts from this blog

java.util.scanner - How to read and add only numbers to array from a text file -

i'm learning java , have question regarding reading file want read numbers file contains strings well. here example of file: 66.56 "3 java 3-43 5-42 2.1 1 and here coding: public class test { public static void main (string [] args){ if (0 < args.length) { file x = new file(args[0]); try{ scanner in = new scanner( new fileinputstream(x)); arraylist<double> test = new arraylist<>(); while(in.hasnext()){ if(in.hasnextdouble()){ double f=in.nextdouble(); test.add(f);} else {in.next();} } catch(ioexception e) { system.err.println("exception during reading: " + e); } } my problem add 66.56,2.1 , 1 doesn't add 3 after "3 or ignores 3-43 , 5-42 can tell me how skip strings , add doubles here? thanks all said 3 ie; "3, 3-43 , 4-42 strings either u read string , split , check number @ " , - or put in space between characters , integers. jvm after compilation tre...
Read more

rewrite - Trouble with Wordpress multiple custom querystrings -

i'm working on adding multiple querystring , doesn't seem work. i've found codes seems work others might i'm missing something. i want like: http://mydomain.com/board/?getyear=2013&getsometext2=sometext to http://mydomain.com/board/2013/sometext/ i added code below functions.php file. doesn't seem work should though. this works. echoing querystring gets both variable without problem. http://mydomain.com/board/2013/sometext/ but doesn't. tested querystring vis isset() seems nothing being set/get http://mydomain.com/board/2013/ function add_query_vars($vars) { $vars[] = "getyear"; $vars[] = "getsometext2"; return $vars; } add_filter('query_vars', 'add_query_vars'); function add_rewrite_rules($arules) { $anewrules = array('board/([^/]+)/([^/]+)/?$' => 'index.php?pagename=board&getyear=$matches[1]&getsometext2=$matches[2]'); $arules = $a...
Read more