Talk:Web scraping

Choice of page to scrape

I hope I've chosen a page that will be around for some time and in the same form. If not I could switch to extracting the last modified time of a Rosetta page I suppose, but we wouldn't be guaranteed a page that is simple to parse and that you can extract slightly changing answers from. --Paddy3118 20:53, 20 August 2008 (UTC)

Criticism

The task, as described and the examples so far are extremely weak by comparison to one's normal expectations of what "web scraping" means. The examples just pull a page and extract a line of text using simple regular expressions.

When developers talk about "web scraping" they are usually talking about much more than simply fetching the page and doing trivial extraction of a simple regular expression. Usually the task implies more sophisticated parsing of the page's HTML and frequently involves encoding the request into a query string (ReSTful sites) or an HTTP POST-able form.

Thus I would expect the task to describe the fetching and parsing of a web page, in HTML form ... with a subsequent encoding of selected results into a new query (and/or posted form). This would give a far more realistic example of what "web scraping" means to most people who would employ the phrase. JimD 23:00, 8 September 2008 (UTC)

Hi Jim,

I have read your criticism, and looked at the introduction of the definition of web scraping here. It seems we disagree on the size of example appropriate to R.C. but not really on what web scraping is. If you have a larger example then the central idea of extracting data from a live web page may be lost in the details of how the data is extracted from HTML, or what is done subsequently with that data. A lot of the tasks on R.C. are small and I thought this would fit that mould.

You could always add a separate task involving extracting data from HTML files? --Paddy3118 12:56, 9 September 2008 (UTC)

Whatever the task calls for, It's perfectly acceptable to break a particular language's code example out into a separate page (e.g. Web Scraping/x86 Macro Assembler) if the code size gets too large. If the task itself calls for something beyond a function or two, the task falls under a "project" scope as in RCBF and RCSNUSP, where the task itself becomes a category, and each example gets its own page there.

If the task's intent is tightly scoped, but the task's examples grow too far, then consider breaking up the task into multiple pieces that each illustrate some different function. --Short Circuit 05:59, 10 September 2008 (UTC)