java - Google App Engine: Backend getting stuck after fetching a large number of URLs -


i'm having problem gae backends , taskqueues. happens is, after backend several url fetch calls few minutes, tasks start getting stuck without starting. enforced rate drops 0.10/s , queue hardly moves. starts move if restart backend instance reach 0.10/s enforced rate again.

i'm working on gae project requires app traverse around 70000 urls, retrieve html, check values in html, , update records in datastore based on values in html.

the implementation involves cron job takes around 300 urls every minute, splits them 10's, , assigns them different tasks in task queue. each task goes through 10 urls, processing contents.

i'm running b4 static backend instance. task queue rate @ 5/s. max concurrent requests 8. tried adding task aging didn't help.

---- october 19, 2013 ----

edit: tried commenting out lot of code , narrowed down problem url fetching. apparently when remove url fetching, things run smoothly. still, i'm not sure how fix since i'm pretty sure closed connection related resources.

you may reach the quota limit url fetch 3000 api calls/minute or 657,000 api calls/day.


Comments

Post a Comment

Popular posts from this blog

java.util.scanner - How to read and add only numbers to array from a text file -

i'm learning java , have question regarding reading file want read numbers file contains strings well. here example of file: 66.56 "3 java 3-43 5-42 2.1 1 and here coding: public class test { public static void main (string [] args){ if (0 < args.length) { file x = new file(args[0]); try{ scanner in = new scanner( new fileinputstream(x)); arraylist<double> test = new arraylist<>(); while(in.hasnext()){ if(in.hasnextdouble()){ double f=in.nextdouble(); test.add(f);} else {in.next();} } catch(ioexception e) { system.err.println("exception during reading: " + e); } } my problem add 66.56,2.1 , 1 doesn't add 3 after "3 or ignores 3-43 , 5-42 can tell me how skip strings , add doubles here? thanks all said 3 ie; "3, 3-43 , 4-42 strings either u read string , split , check number @ " , - or put in space between characters , integers. jvm after compilation tre...
Read more

rewrite - Trouble with Wordpress multiple custom querystrings -

i'm working on adding multiple querystring , doesn't seem work. i've found codes seems work others might i'm missing something. i want like: http://mydomain.com/board/?getyear=2013&getsometext2=sometext to http://mydomain.com/board/2013/sometext/ i added code below functions.php file. doesn't seem work should though. this works. echoing querystring gets both variable without problem. http://mydomain.com/board/2013/sometext/ but doesn't. tested querystring vis isset() seems nothing being set/get http://mydomain.com/board/2013/ function add_query_vars($vars) { $vars[] = "getyear"; $vars[] = "getsometext2"; return $vars; } add_filter('query_vars', 'add_query_vars'); function add_rewrite_rules($arules) { $anewrules = array('board/([^/]+)/([^/]+)/?$' => 'index.php?pagename=board&getyear=$matches[1]&getsometext2=$matches[2]'); $arules = $a...
Read more