Mimicking an ajax call with Curl PHP -

i'm scraping site using curl (via php) , information want list of products default showing first few ones. rest passed user when click button full list of products, triggers ajax call return list.

here in nutshell js use:

headers['__requestverificationtoken'] = token; $.ajax({ type: "post", url: "/ajax/getproductlist", datatype: 'html', data: json.stringify({ historypageindex: 1, displayperiod: 0, productstype: }), contenttype: 'application/json; charset=utf-8', success: function (result) {     $(target).html("");     $(target).html(result); }, beforesend: function (xmlhttprequest) {     if (headers['__requestverificationtoken']) {         xmlhttprequest.setrequestheader("__requestverificationtoken", headers['__requestverificationtoken']);     } } }); 

here php script:

curl_setopt($ch, curlopt_useragent, $useragent); curl_setopt($ch, curlopt_returntransfer, true); curl_setopt($ch, curlopt_followlocation, true); curl_setopt($ch, curlopt_maxredirs, 10); curl_setopt($ch, curlopt_cookiefile, $cookielocation); curl_setopt($ch, curlopt_cookiejar, $cookielocation); curl_setopt($ch, curlopt_post, false); curl_setopt($ch, curlopt_url, 'https://www.domain.com/applications/viewproducts'); curl_setopt($ch, curlopt_referer, 'https://www.domain.com/'); $webpage = curl_exec($ch); $productstype = trim(find_by_pattren($webpage, '<input id="productstype" name="productstype" type="hidden" value="(.*?)"')); $token = trim(find_by_pattren($webpage, '<input name="__requestverificationtoken" type="hidden" value="(.*?)"'));  $postvariables = 'productstype='.$productstype. '&historypageindex=1 &displayperiod=0 &__requestverificationtoken='.$token; curl_setopt($ch, curlopt_post, true); curl_setopt($ch, curlopt_postfields, $postvariables); curl_setopt($ch, curlopt_url, 'https://www.domain.com/ajax/getproductlist'); curl_setopt($ch, curlopt_referer, 'https://www.domain.com/applications/viewproducts'); $webpage = curl_exec($ch); 

this produces error page site. think main reasons that:

• they check whether it's ajax request (no clue how fix that)

• the token needs in header , not in post variables

any idea?

edit: here working code:

curl_setopt($ch, curlopt_useragent, $useragent); curl_setopt($ch, curlopt_returntransfer, true); curl_setopt($ch, curlopt_followlocation, true); curl_setopt($ch, curlopt_maxredirs, 10); curl_setopt($ch, curlopt_cookiefile, $cookielocation); curl_setopt($ch, curlopt_cookiejar, $cookielocation); curl_setopt($ch, curlopt_url, 'https://www.domain.com/applications/viewproducts'); curl_setopt($ch, curlopt_referer, 'https://www.domain.com/'); $webpage = curl_exec($ch); $productstype = trim(find_by_pattren($webpage, '<input id="productstype" name="productstype" type="hidden" value="(.*?)"')); $token = trim(find_by_pattren($webpage, '<input name="__requestverificationtoken" type="hidden" value="(.*?)"'));  $postvariables = json_encode(array('productstype' => $productstype, 'historypageindex' => 1, 'displayperiod' => 0)); curl_setopt($ch, curlopt_post, true); curl_setopt($ch, curlopt_httpheader, array("x-requested-with: xmlhttprequest", "content-type: application/json; charset=utf-8", "__requestverificationtoken: $token")); curl_setopt($ch, curlopt_postfields, $postvariables); curl_setopt($ch, curlopt_url, 'https://www.domain.com/ajax/getproductlist'); curl_setopt($ch, curlopt_referer, 'https://www.domain.com/applications/viewproducts'); $webpage = curl_exec($ch); 

to set request verification token header, more closely mimic ajax request, , set content-type json, use curlopt_header.

curl_setopt($ch, curlopt_httpheader, array("x-requested-with: xmlhttprequest", "content-type: application/json; charset=utf-8", "__requestverificationtoken: $token")); 

i notice you're superfluously setting curlopt_post false on line 7 of code, , post data you're sending isn't in json format. should have:

$postvariables = '{"historypageindex":1,"displayperiod":0,"productstype":"all"}';