Still no go with this, Im sure my POST variables are correct, including a JSESSIONID that looks like it is needed.<br><br>POST array looks like this:<br> <br> $cmd="cat /tmp/cookie | tail -1 | awk '{print $7;}'"; // From a prior curl_exec I get the jsessionid<br>
$sid=exec($cmd);<br><br> $post_data['username'] = 'USERNAME';<br> $post_data['password'] = 'PASSWORD';<br> $post_data['JSESSIONID'] = $sid;<br> $post_data['smauthreason'] = 0;<br>
$post_data['TARGET'] = '';<br> $post_data['error_target'] = 'http%3A%2F%<a href="http://2Fwww.telstrabusiness.com">2Fwww.telstrabusiness.com</a>%2Fbusiness%2Flogin.jsp%3Fmode%3Derror';<br>
$post_data['final_target'] = 'http%3A%2F%<a href="http://2Fwww.telstrabusiness.com">2Fwww.telstrabusiness.com</a>%2Fbusiness%2Fsecure%2FssoLoginProtected';<br> $post_data['postpreservationdata'] = '';<br>
$post_data['generallogondata'] = 'SMLogonVersion%3D1.0%3BDebug%3Dfalse%3BReferer%3Dhttp%253A%2F%<a href="http://2Fwww.telstrabusiness.com">2Fwww.telstrabusiness.com</a>%2Fbusiness%2Flogin.jsp%3B';<br><br>
foreach ( $post_data as $key => $value) {<br> $post_items[] = $key . '=' . $value;<br> }<br> <br> //create the final string to be posted using implode()<br> $post_string = implode ('&', $post_items);<br>
<br> $options = array( <br> CURLOPT_COOKIEJAR => "/tmp/cookie",<br> CURLOPT_COOKIEFILE => "/tmp/cookie",<br> CURLOPT_COOKIESESSION => true,<br> CURLOPT_RETURNTRANSFER => true, // return web page rather than printing it<br>
CURLOPT_HEADER => true, // return headers <br> CURLOPT_VERBOSE => true,<br> CURLOPT_FOLLOWLOCATION => true, // follow redirects <br> CURLOPT_SSL_VERIFYPEER => false, // accept any server certificate<br>
CURLOPT_ENCODING => "", // handle all encodings <br> CURLOPT_USERAGENT => "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)", // who am i <br> CURLOPT_AUTOREFERER => true, // set referer on redirect <br>
CURLOPT_POST => true, // Enable POST<br> CURLOPT_POSTFIELDS => $post_string //POST data<br> );<br><br>I notice that Im not using CURLOPT_HTTPHEADER to return any headers. Should I be?<br>
<br>thanks<br>Shanon<br><br><br><br><div class="gmail_quote">On Thu, Dec 2, 2010 at 2:12 PM, Lucas van Staden <span dir="ltr"><<a href="mailto:lvs@dedmeet.com">lvs@dedmeet.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
HA<br>
<br>
and then even still dumber, I again sent it as the non member email! <br>
I think I need to stay away from email today!<br>
<br>
On 02/12/10 14:09, Lucas van Staden wrote:
<blockquote type="cite">
Ok, being really dumb today.<br>
.<br>
First sent reply as non member email, then sent it to wrong list (sorry
for any duplicated posts to those belonging to off-topic list as well)<br>
<br>
-Lucas<br>
<br>
-------- Original Message --------
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap>Subject: </th>
<td>Re: [plug] Anyone know PHP + cURL?</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap>Date: </th>
<td>Thu, 02 Dec 2010 14:02:18 +0800</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap>From: </th>
<td>Lucas van Staden <a href="mailto:lvs@dedmeet.com" target="_blank"><lvs@dedmeet.com></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap>Reply-To: </th>
<td><a href="mailto:lvs@dedmeet.com" target="_blank">lvs@dedmeet.com</a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap>To: </th>
<td>plug off-topic <a href="mailto:off-topic@plug.org.au" target="_blank"><off-topic@plug.org.au></a></td>
</tr>
</tbody>
</table>
<br>
<br>
Hi,<br>
<br>
Not sure what you have tried, but here are some ideas to bounce.<br>
<br>
Are you 100% your post cars are correctly formatted?<br>
Are you building the postvars yourself that is used in curl?<br>
<br>
My original attempt was to build the actual post vars in the code, but
that failed. <br>
Just passing a cut and pasted string from tamper data plugin worked for
me.<br>
<br>
Install Firefox plug Tamper Data.<br>
Start it.<br>
Then login to the page normally<br>
Tamper Data will give a lot of results.<br>
The one you are interested should be the first or second, holding the
actual post vars as they are formatted by the form.<br>
Cut and paste that to your curl (remember to make any @ signs back to
be @ (they will be url encoded)<br>
<br>
See if you can then post/login using what you got from Tamper Data.<br>
<br>
Also, as mentioned before, I could not login, unless I had once loaded
the homepage (and saved the cookie from there first)<br>
Not sure why, but it is most likely just that site I scrape's manner in
detecting cookies is available.<br>
<br>
As to what comes back, just dump the result into a file (or use a
debugger like xdebug) to read the vars values in realtime.<br>
Solving this type of issue is very handy with a debugger and step
through teh code in 'real-time'<br>
<br>
(untested)<br>
<br>
$data = curl_exec($ch);<br>
$fp = fopen("/tmp/test",'w+');<br>
fputs($fp, $data);<br>
close($fp);<br>
<br>
If the result is False (bool) then it is most likely not even getting
to post to the page, or you are being diverted to another page, and
curl has been set not to follow.<br>
<br>
I am not sure on curl verbose level but the verbose option in the curel
I posted previously will place all headers to stdout.<br>
Read through them carefully, as they will most likely give you a hint.<br>
<br>
Also make sure you set a CURLOPT_USERAGENT and a CURLOPT_REFERER (they
could be detecing the scripting, and blocking you, or testing the
referer (make that the same as the login url, so the post looks like it
came from that page.)<br>
<br>
no further ideas.<br>
<br>
-Lucas<br>
<br>
<br>
<br>
-Lucas<div><div></div><div class="h5"><br>
<br>
<br>
On 02/12/10 10:09, Shanon Loughton wrote:
<blockquote type="cite">Thanks Daniel for the suggestion of Python, and I would
like to try that eventually. But there must be a way with cURL because
all examples online are about logging into sites.<br>
<br>
Lucas, your code makes sense. I was essentially doing the same thing.
However Im unsure that my post variables are actually being 'posted' or
rather what the server reply is, apart from redirect to the login page
again. Where do I look for curl logs? the Apache ones dont say much,
can I increase curl verbosity? Is all the info I need in the header
response for each curl exec?<br>
<br>
thanks<br>
Shanon<br>
<br>
<br>
<div class="gmail_quote">On Wed, Dec 1, 2010 at 9:42 PM, <span dir="ltr"><<a href="mailto:home@oranges.id.au" target="_blank">home@oranges.id.au</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div>On 1 December 2010 13:45, Shanon Loughton <<a href="mailto:autobot@iinet.net.au" target="_blank">autobot@iinet.net.au</a>>
wrote:<br>
> Im trying to extract our monthly telstra bill from the hideous
portal<br>
> <a href="https://www.telstrabusiness.com/business/login.jsp" target="_blank">https://www.telstrabusiness.com/business/login.jsp</a><br>
<br>
</div>
Once you get some success, and once I manage to get a login to said<br>
hideous portal, I'd love to get a snapshot of how you do it.<br>
<br>
Greg.<br>
<div>
<div>_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank">plug@plug.org.au</a><br>
<a href="http://www.plug.org.au/mailman/listinfo/plug" target="_blank">http://www.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.linux.org.au" target="_blank">committee@plug.linux.org.au</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<pre><fieldset></fieldset>
_______________________________________________
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank">plug@plug.org.au</a>
<a href="http://www.plug.org.au/mailman/listinfo/plug" target="_blank">http://www.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a href="mailto:committee@plug.linux.org.au" target="_blank">committee@plug.linux.org.au</a>
</pre>
</blockquote>
<br>
<br>
</div></div><div class="im"><pre cols="72">--
Regards
Lucas van Staden
<a href="http://www.proxiblue.com.au/" target="_blank">http://www.proxiblue.com.au/</a> (Consumer Electronics Retail Shop)
<a href="http://www.vanstaden.com.au/" target="_blank">http://www.vanstaden.com.au/</a> (Linux)
</pre>
</div></blockquote><div class="im">
<br>
<br>
<pre cols="72">--
Regards
Lucas van Staden
<a href="http://www.proxiblue.com.au/" target="_blank">http://www.proxiblue.com.au/</a> (Consumer Electronics Retail Shop)
<a href="http://www.vanstaden.com.au/" target="_blank">http://www.vanstaden.com.au/</a> (Linux)
</pre>
</div></div>
<br>_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au">plug@plug.org.au</a><br>
<a href="http://www.plug.org.au/mailman/listinfo/plug" target="_blank">http://www.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.linux.org.au">committee@plug.linux.org.au</a><br>
<br></blockquote></div><br>