• Hi Guest Just in case you were not aware I wanted to highlight that you can now get a free 7 day trial of Horseracebase here.
    We have a lot of members who are existing users of Horseracebase so help is always available if needed, as well as dedicated section of the fourm here.
    Best Wishes
    AR

Preferred language for scraping

agourbet

agourbet

Newbie
Hello everyone, i would like to ask you about the preferred language for scraping. I use Python, it is ok for small scrapers but when things are getting bigger it is difficult to keep them organised in my opinion. What do you think? Thank you
 
Hi,
yes I use Python, and although I tend to use just the built in commands for extracting data from web pages I would think that for many the plentiful supply of modules like 'Beautiful soup' that people have produced, specifically for making web scraping easier, makes it a popular language to work in.

As long as your language has a way to download web pages and access the text contained on them, and string slicing commands, virtually any language would do though I'd think. If you can find a language that includes routines/commands to extract what you want is the main thing I'd have thought.

To my mind the biggest issue with web scraping is having the desire to do it and - more importantly - the determination to see it through, as writing code that then doesn't do what you intended is not uncommon when scraping! I remember pounding away for 2 or 3 days getting absolutely nowhere - I couldn't even grab a page and store it on my PC - before a tip on here helped me out and after that it was all plain sailing. (As in quite a lot of days of equally hard and frustrating work). That's why one of the few things that gets my blood starting to boil quickly is somebody saying 'could you just knock up a scraper that downloads this that and the other, and does all sorts of stuff as a sideline'....:D

Dave
 
:rofl::rofl: Thank you for your input davejb davejb. I also worked with Python for a scraping project,it is an easy language to use. I just found it very chaotic in respect of code formality. I prefer stricter languages, but maybe it is just me.
 
There's no reason why you can't write 'strict' code in python. You don't have to be sloppy just because the language allows you to do so.
 
Hi he did not ask what software , but i use outwit hub which scrapped most websites with ease if you spend a little time with it. mine is set to run from a schedule daily . you don't need to learn any coding to use it but if you are serious about getting data from websites using this software then it helps if you have a basic understanding of how regular expressions work. And no i am not affiliated in anyway shape or form withe software its just my personal preference.:D.
 
Does anyone actually build scraping tools to buy on here? My brain is way to old now to understand what is required these day's I just want it out of the box...bit like my pizza
 
Hello Everyone!
I am using only PHP CURL, ptoxy rotation, MultiCURL. Also I am using third-party APIs to get protected data.
 
You must log in or register to reply here.
Back
Top