Scraping Utility for Those New to Scraping

[rpjd99]

this may help point you in the right direction

Thanks, Kev. Your timing is impeccable, as you are aware! Haven't tried it yet, but will be doing so in the next few days.

Ray

 

[hedgehog]

Hello leftinthestalls
I've given the code a go but it's not ininteracting with the code of my scraper. Basically I'm a sad proceduralprogrammer and iI've coded using client.download String so it's a matter I suspect of different threads if not completely different objects

Thanks for the help

 

[rpjd99]

I've given the code a go but it's not ininteracting with the code of my scraper.

That's the problem with visual basic: there are too many varieties. And the VB in Visual Studio is different to the VB in Visual Basic for Applications (VBA). I was never happy in Visual Studio and I recently ported the excellent tutorial of MrDamon over to VBA (Excel). That allowed me to trace and squash some bugs that I couldn't track down in Visual Studio. What surprised me the most, though, was that the Excel VBA version was actually a bit quicker than the Visual Studio version. It seems to be the accepted wisdom that VBA is painfully slow. My experience shows the opposite. And then the P-T completely changed their website on the 27th of August...

If you need any help in porting it to VBA, give me a shout.

Ray

 

[rpjd99]

leftinthestalls

Hi Kev,

The code loops at both instances of: "Do Until ie.readyState = READYSTATE_COMPLETE: DoEvents: Loop". When I changed both instances of "READYSTATE_COMPLETE" to "4", it worked fine. (Using IE 11.)

Ray

 

[leftinthestalls]

cheers Ray!

schoolboy error - I had "READYSTATE_COMPLETE" set as a constant elsewhere in my code.

 

[hedgehog]

Hello rpjd99
I'm ququite happy running my scraper from visual studio, I've knocked most of the bugs out and if I add something I use a separate function based on one that I know is pretty bug free.
Can I ask, when you run your scraper do you have a session of internet explorer open? I don't as I access the html via client. Download string as in MrDamon s excellent example.

Thanks for the offer of help, I may call on that in the near future

 

[rpjd99]

Can I ask, when you run your scraper do you have a session of internet explorer open?

Hi hedgehog ,

No, I never run it with IE visible, as there's no point in doing so. I generate all progress I need to know about on Excel's status bar. This is the function I use in place of the client in VS-VB:

Function GetSource(sURL As String) As String

'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
'By Tim Williams
'Downloads an http page with an address "sURL" and stores it in "GetSource" as a string.
'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Dim rpt As Integer
Dim oXHTTP As Object
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")

'Counter for the number of attempts made if repeated errors occur with the "oXHTTP.send" instruction
rpt = 0

GetDoc:
On Error GoTo ErrorTrap
oXHTTP.Open "GET", sURL, False
oXHTTP.send
On Error GoTo 0

GetSource = oXHTTP.responseText
Set oXHTTP = Nothing

Exit Function

ErrorTrap:
'Increment the try counter
rpt = rpt + 1
'Allow three attempts
If rpt < 3 Then
    'Wait 2 seconds and try again
    If Application.Wait(Now + TimeValue("0:00:02")) Then Resume GetDoc
'Three failed attempts have been made so return to the line ("oXHTTP.send") to halt the macro with an error
Else: Resume
End If

End Function

Ray

 

[hedgehog]

Hello rpjd99,

I tried leftinthestalls code again and got a weird error when I ran my scraper.

I didn't explain very well the code I use is as a global

Dim client as New System.Net.WebClient

There are the uses of

Dim HTML as string = client. DownloadString (url)

Is the code you've offered vb or vba please? If it's vb I'll try and give it a go

Thanks for the help

 

[rpjd99]

Is the code you've offered vb or vba please?

Hi hedgehog ,

It's VBA, I'm afraid.

Ray

 

[hedgehog]

Hello rpjd99,

I'm currently trying a cookie aware WebClient class but no joy yet, will let you know if I crack it

 

[hedgehog]

Hello all,

Well I've a cookie aware client, overridden the web request and web response, saved the cookie container from the response to ue on the next request and it still bloody doesn't work. The other thing that is possible is the rp site is using JavaScript to change something.

Help!?!?!?

 

[Chesham]

I know nothing about coding but perhaps this may be of use to those that are

https://markatsmartersig.wordpress.com/category/betfair-api-ng/

 

[rpjd99]

Well I've a cookie aware client, overridden the web request and web response, saved the cookie container from the response to ue on the next request and it still bloody doesn't work. The other thing that is possible is the rp site is using JavaScript to change something.

Hi hedgehog ,

If you log in normally and then close your browser, are you automatically logged in when you go to the RP site again? I don't have a subscription to the RP site, so I can't test it, but I notice that there is no option to have your log-in details memorised on the log-in dialogue box. It may be that the RP is forcing a log-in for every visit. If that's the case, maybe this might help: http://www.howtogeek.com/62980/how-to-force-your-browser-to-remember-passwords/

Other possibilities: https://www.google.fr/search?source...orise log-in data&aqs=chrome..69i57.15784j0j7

Ray

 

[hedgehog]

Thanks for the help Chesham

rpjd99, I think I have to login each time I want the limited stuff. Most of the time I don't need to login but I'd like to be able to access more.

I probably did it wrong but I put a login before the access that gets the forecast and it didn't find it this evening.

I'll read your post at the weekend, at this point I'm naffed off with it.

Thanks for the help

 

[mlmrob]

leftinthestalls

Hi Kev

A while ago you wrote a scraper that pulled the RPR and Topspeed ratings from the Totesport site. They don't carry the ratings anymore but Coral do.
Is it possible to scrape the data from the Coral site.

The last five ratings for RPR and Topspeed are available from a drop down menu called Form and Replays.

Thanks

Rob

 

[leftinthestalls]

mlmrob

I have this in my database - I'll try to post up an Excel version this evening/tomorrow.

atb
Kev

 

[mlmrob]

mlmrob

I have this in my database - I'll try to post up an Excel version this evening/tomorrow.

atb
Kev

Awesome, leftinthestalls thanks.

 

[leftinthestalls]

mlmrob

looks like they've forgotten to load this data today (never loaded to my database last night either).

Will check again later. Have the code ready but not going to post it until I've done some basic testing.


Is Excel 2010 format ok for you?

 

[mlmrob]

No probs leftinthestalls thanks for the update.

2010 is fine.

 

[leftinthestalls]

mlmrob

posted in the IS