Web scraping: Difference between revisions

 

;Task:

 

<!-- As of March 2014, the page is available

 

=={{header|8th}}==

: read-url \ -- s

"http://tycho.usno.navy.mil/cgi-bin/timer.pl" net:get

 

get-time bye

{{output}}

<pre>14:08:20</pre>

=={{header|Ada}}==

{{libheader|AWS}}

with Ada.Text_IO, Ada.Strings.Fixed;

use Ada, AWS, AWS.Resources, AWS.Messages;

end if;

end loop;

 

=={{header|ALGOL 68}}==

{{works with|ALGOL 68G|Any - tested with release [http://sourceforge.net/projects/algol68/files/algol68g/algol68g-1.18.0/algol68g-1.18.0-9h.tiny.el5.centos.fc11.i386.rpm/download 1.18.0-9h.tiny]}}

{{wont work with|ELLA ALGOL 68|Any (with appropriate job cards) - tested with release [http://sourceforge.net/projects/algol68/files/algol68toc/algol68toc-1.8.8d/algol68toc-1.8-8d.fc9.i386.rpm/download 1.8-8d] - due to extensive use of ''grep in string'' and ''http content''}}

domain="tycho.usno.navy.mil",

page="cgi-bin/timer.pl";

done: SKIP

ELSE raise error (strerror (rc))

{{out}} Sample

<pre>

It also has powerful text and list processing language blocks that simplify text scraping.<br>

This is how the code would look if it could be typed:<br>

when ScrapeButton.Click do

set ScrapeWeb.Url to SourceTextBox.Text

set Right to select list item (list: get Left, index: 2)

set ResultLabel.Text to select list item (list: split at first (text:get Right, at: PostTextBox.Text), index: 1)

 

[https://lh5.googleusercontent.com/-lFRRPzsi5N0/UvBM9E_gZMI/AAAAAAAAKBQ/AuqDPGlwXNg/s1600/composite.png A picture of the graphical program]/

 

=={{header|BBC BASIC}}==

{{works with|BBC BASIC for Windows}}

Note that the URL cache is cleared so the code works correctly if run more than once.

SYS "GetProcAddress", urlmon%, "URLDownloadToFileA" TO UDTF%

SYS "LoadLibrary", "WININET.DLL" TO wininet%

IF INSTR(text$, "UTC") PRINT MID$(text$, 5)

UNTIL EOF#file%

 

=={{header|C}}==

There is no proper error handling.

 

#include <string.h>

#include <curl/curl.h>

regfree(&cregex);

return 0;

 

=={{header|C++}}==

{{works with|Visual Studio| 2010 Express Edition with boost-1.46.1 from boostpro.com}}

{{works with|gcc|4.5.2 with boost-1.46.1, compiled with -lboost_regex -lboost_system -lboost_thread}}

#include <string>

#include <boost/asio.hpp>

}

}

 

=={{header|Caché ObjectScript}}==

 

Class Utils.Net [ Abstract ]

{

 

}

{{out|Examples}}

<pre>

=={{header|Ceylon}}==

Don't forget to import ceylon.uri and ceylon.http.client in your module.ceylon file.

parse

}

.filter((String element) => element.contains("UTC"))

.first

 

=={{header|Clojure}}==

Clojure 1.2:

 

(second (re-find #" (\d{1,2}:\d{1,2}:\d{1,2}) UTC" (slurp "http://tycho.usno.navy.mil/cgi-bin/timer.pl")))

 

=={{header|CoffeeScript}}==

{{works with|node.js}}

 

http = require 'http'

 

wget CONFIG.host, CONFIG.path, (data) ->

scrape_tycho_ust_time data

{{out}}

<pre>

{{libheader|DRAKMA}}

 

(regexp (load-time-value

(cl-ppcre:create-scanner "(?m)^.{4}(.+? UTC)")))

(when start

(subseq data (aref start-regs 0) (aref end-regs 0)))))

 

Another Common Lisp solution

(m ".*<BR>(.*)UTC.*"

(drakma:http-request "http://tycho.usno.navy.mil/cgi-bin/timer.pl"))

(print (cl-ppcre:regex-replace "<BR>(.*UTC).*" m "\\1")))

 

=={{header|D}}==

import std.stdio, std.string, std.net.curl, std.algorithm;

 

if (line.canFind(" UTC"))

line[4 .. $].writeln;

 

=={{header|Delphi}}==

There are a number of ways to do this with Delphi using any one of a number of free/open source TCP/IP component suites such as, for example, ICS, Synapse and Indy (which ships with Delphi anyway). However, I thought it would be interesting to do this using the Winsock API direct.

 

 

program WebScrape;

end.

 

 

 

Example using Indy's IdHTTP component.

 

 

{$APPTYPE CONSOLE}

lReader.Free;

end;

 

=={{header|E}}==

 

def rx`(?s).*>(@time.*? UTC).*` := html

println(time)

 

=={{header|Erlang}}==

 

Using regular expressions:

-export([main/0]).

-define(Url, "http://tycho.usno.navy.mil/cgi-bin/timer.pl").

{ok, {_Status, _Header, HTML}} = httpc:request(?Url),

{match, [Time]} = re:run(HTML, ?Match, [{capture, all_but_first, binary}]),

 

=={{header|F_Sharp|F#}}==

This code is asynchronous - it will not block any threads while it waits on a response from the remote server.

open System

open System.Net

|> Async.RunSynchronously

|> printfn "%s"

 

=={{header|Factor}}==

 

"http://tycho.usno.navy.mil/cgi-bin/timer.pl" http-get nip

 

=={{header|Forth}}==

{{works with|GNU Forth|0.7.0}}

 

: extract-time ( addr len type len -- time len )

s\" \r\n\r\n" search 0= abort" can't find headers!" \ skip headers

s" UTC" extract-time type cr

 

=={{header|FunL}}==

 

case Source.fromURL( 'http://tycho.usno.navy.mil/cgi-bin/timer.pl', 'UTF-8' ).getLines().find( ('Eastern' in) ) of

Some( time ) -> println( time.substring(4) )

 

{{out}}

 

=={{header|Gambas}}==

Dim sWeb, sTemp, sOutput As String 'Variables

 

Print Mid(sOutput, 5) 'Print the result without the '<BR>' tag

 

Output:

<pre>

 

=={{header|Go}}==

 

import (

// there is a human readable time in there somewhere.

fmt.Println(us)

{{out}}

<pre>

 

=={{header|Groovy}}==

def text = new URL('http://tycho.usno.navy.mil/cgi-bin/timer.pl').eachLine { line ->

def matcher = (line =~ "<BR>(.+) UTC")

}

}

{{out}}

<pre>UTC Time was 'Feb. 26, 11:02:30'</pre>

=={{header|Haskell}}==

Using package HTTP-4000.0.8 from [http://hackage.haskell.org/packages/hackage.html HackgageDB]

import Network.HTTP (simpleHTTP, getResponseBody, getRequest)

 

 

readUTC = simpleHTTP (getRequest tyd)>>=

Usage in GHCi:

 

== Icon and Unicon ==

Icon has capability to read web pages using the external function cfunc. The Unicon messaging extensions are more succinct.

==={{header|Unicon}}===

m := open(url := "http://tycho.usno.navy.mil/cgi-bin/timer.pl","m") | stop("Unable to open ",url)

every (p := "") ||:= |read(m) # read the page into a single string

 

map(p) ? ( tab(find("<br>")), ="<br>", write("UTC time=",p[&pos:find(" utc")])) # scrape and show

 

=={{header|J}}==

 

_8{. ' UTC' taketo gethttp 'http://tycho.usno.navy.mil/cgi-bin/timer.pl'

 

The <code>[[J:Addons/web/gethttp|web/gethttp]]</code> addon uses Wget on Linux or Windows (J ships with Wget on Windows) and cURL on the Mac.

 

=={{header|Java}}==

import java.io.IOException;

import java.io.InputStreamReader;

import java.net.URL;

 

 

=={{header|JavaScript}}==

Due to browser cross-origin restrictions, this script will probably not work in other domains.

 

req.onload = function () {

var re = /[JFMASOND].+ UTC/; //beginning of month name to 'UTC'

};

req.open('GET', 'http://tycho.usno.navy.mil/cgi-bin/timer.pl', true);

 

=={{header|jq}}==

 

Currently jq does not have built-in curl support, but jq is intended to work seamlessly with other command-line utilities, so we present a simple solution to the problem in the form of a three-line script:

curl -Ss 'http://tycho.usno.navy.mil/cgi-bin/timer.pl' |\

{{out}}

 

=={{header|Julia}}==

I'm using the <code>Requests.jl</code> package for this solution. Note, I used a slightly different URL after finding that the one specified in the task description is deprecated (though it still works).

function getusnotime()

println("Failed to fetch UNSO time:\n", t)

end

 

{{out}}

 

By breaking the regular expression (using <code>&lt;BR&gt;(.*UTd)</code>)

Failed to fetch UNSO time:

raw html:

 

</body></html>

 

=={{header|Kotlin}}==

 

import java.net.URL

}

sc.close()

 

Sample output:

 

=={{header|Lasso}}==

local(raw_htmlstring = '<TITLE>What time is it?</TITLE>

<H2> US Naval Observatory Master Clock Time</H2> <H3><PRE>

// added bonus showing how parsed string can be converted to date object

local(mydate = date(#datepart_txt, -format = `MMM'.' dd',' HH:mm:ss`))

Result:

Jul. 27, 22:57:22

 

=={{header|Liberty BASIC}}==

open DefaultDir$ + "\timer.htm" for input as #f

html$ = lower$(input$(#f, LOF(#f)))

DownloadToFile as ulong '0=success

close #url

 

Because Liberty has to do web operations in ways like this, calling Windows DLLs, there is the [[#Run BASIC|Run BASIC]] variant of LB, in which the task becomes a one-liner.

{{libheader|LuaSocket}}

The web page is split on the HTML line break tags. Each line is checked for the required time zone code. Once it is found, we return the instance within that line of three numbers separated by colons - I.E. the time.

local http = require("socket.http") -- Debian package is 'lua-socket'

 

print(scrapeTime(url, "UTC"))

 

The task description states "just the UTC time" but of course we could return the whole line including the zone name and date if required.

 

=={{header|Maple}}==

{{Out|Output}}

<pre>May. 16, 20:17:28 UTC Universal Time</pre>

 

 

=={{header|MATLAB}} / {{header|Octave}}==

ix = [findstr(s,'<BR>'), length(s)+1];

for k = 2:length(ix)

disp(tok);

end;

 

=={{header|mIRC Scripting Language}}==

sockclose UTC

sockopen UTC tycho.usno.navy.mil 80

sockread %UTC

}

 

=={{header|NetRexx}}==

options replace format comments java crossref symbols binary

 

method isFalse() public constant returns boolean

return \isTrue()

 

{{out}}

 

=={{header|Nim}}==

 

 

=={{header|Objeck}}==

use Net;

use IO;

}

}

 

=={{header|OCaml}}==

 

let _,_, page_content = make_request ~url:Sys.argv.(1) ~kind:GET () in

 

let str = Str.global_replace (Str.regexp "<BR>") "" str in

print_endline str;

 

There are libraries for this, but it's rather interesting to see how to use a socket to achieve this, so see the implementation of the above function <tt>make_request</tt> on [[Web_Scraping/OCaml|this page]].

As an alternative the supplied rxSock socket library could be used

 

/* load the RexxcURL library */

Call RxFuncAdd 'CurlLoadFuncs', 'rexxcurl', 'CurlLoadFuncs'

utcTime = content~substr(content~lastpos('<BR>') + 4)

say utcTime

 

=={{header|Oz}}==

[Regex] = {Module.link ['x-oz://contrib/regex']}

 

Url = "http://tycho.usno.navy.mil/cgi-bin/timer.pl"

in

 

=={{header|Peloton}}==

English dialect, short form, using integrated Rexx pattern matcher:

<@ DEFPRSLIT>Rexx Parse|'<BR>' UTCtime 'UTC'</@>

<@ LETVARURL>timer|http://tycho.usno.navy.mil/cgi-bin/timer.pl</@>

<@ ACTRPNPRSVAR>Rexx Parse|timer</@>

 

English dialect, padded variable-length form:

<# DEFINE PARSEVALUES LITERAL>Rexx Parse|'<BR>' UTCtime 'UTC'</#>

<# LET VARIABLE URLSOURCE>timer|http://tycho.usno.navy.mil/cgi-bin/timer.pl</#>

<# ACT REPLACEBYPATTERN PARSEVALUES VARIABLE>Rexx Parse|timer</#>

 

English dialect, padded short form, using string functions AFT and BEF:

 

=={{header|Perl}}==

{{libheader|LWP}}

 

my $url = 'http://tycho.usno.navy.mil/cgi-bin/timer.pl';

get($url) =~ /<BR>(.+? UTC)/

 

 

=={{header|PHP}}==

By iterating over each line:

 

 

$contents = file('http://tycho.usno.navy.mil/cgi-bin/timer.pl');

echo subStr($line, 4, $pos - 4); //Prints something like "Dec. 06, 16:18:03"

break;

 

By [[regular expressions]] ({{works with|PHP|4.3.0}}):

 

 

echo preg_replace(

file_get_contents('http://tycho.usno.navy.mil/cgi-bin/timer.pl')

);

 

=={{header|PicoLisp}}==

 

(client "tycho.usno.navy.mil" 80 "cgi-bin/timer.pl"

(when (from "<BR>")

{{out}}

<pre>-> "Feb. 19, 18:11:37"</pre>

 

=={{header|PowerShell}}==

$html = $wc.DownloadString('http://tycho.usno.navy.mil/cgi-bin/timer.pl')

$html -match ', (.*) UTC' | Out-Null

 

===fyi===

.NET provides a property named '''UtcNow''':

[System.DateTime]::UtcNow

{{Out}}

<pre>

</pre>

I am currently in the Pacific timezone:

[System.DateTime]::Now

{{Out}}

<pre>

 

=={{header|PureBasic}}==

ReadFile(0, "timer.htm")

While Not Eof(0) : Text$ + ReadString(0) : Wend

 

=={{header|Python}}==

page = urllib.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl')

for line in page:

print line.strip()[4:]

break

{{out}}

<pre>Aug. 12, 15:22:08 UTC Universal Time</pre>

Read the page as lines, find the line containing the string "UTC", then extract the portion of that string that is the date.

 

all_lines <- readLines("http://tycho.usno.navy.mil/cgi-bin/timer.pl")

utc_line <- grep("UTC", all_lines, value = TRUE)

matched <- regexpr("(\\w{3}.*UTC)", utc_line)

utc_time_str <- substring(line, matched, matched + attr(matched, "match.length") - 1L)

 

The last three lines can be made simpler by using {{libheader|stringr}}

 

library(stringr)

utc_line <- all_lines[str_detect(all_lines, "UTC")]

utc_time_str <- str_extract(utc_line, "\\w{3}.*UTC")

 

Finally, the date and time must be parsed and printed in the desired format.

 

utc_time <- strptime(utc_time_str, "%b. %d, %H:%M:%S UTC")

strftime(utc_time, "%A, %d %B %Y, %H:%M:%S")

 

Friday, 13 May 2011, 15:12:20

First, retrieve the web page. See [[HTTP_Request]] for more options with this.

 

library(RCurl)

web_page <- getURL("http://tycho.usno.navy.mil/cgi-bin/timer.pl")

 

Now parse the html code into a tree and retrieve the pre node that contains interesting bit. Without xPath, the syntax is quite clunky.

 

library(XML)

page_tree <- htmlTreeParse(webpage)

times_node <- times_node[names(times_node) == "text"]

time_lines <- sapply(times_node, function(x) x$value)

 

Here, xPath simplifies things a little bit.

 

page_tree <- htmlTreeParse(web_page, useInternalNodes = TRUE)

times_node <- xpathSApply(page_tree, "//pre")[[1]]

times_node <- times_node[names(times_node) == "text"]

time_lines <- sapply(times_node, function(x) as(x, "character"))

 

Either way, the solution proceeds from here as in the regex method.

utc_line <- time_lines[str_detect(time_lines, "UTC")]

#etc.

 

=={{header|Racket}}==

 

#lang racket

(require net/url)

((compose1 car (curry regexp-match #rx"[^ <>][^<>]+ UTC")

port->string get-pure-port string->url)

 

=={{header|REBOL}}==

Title: "Web Scraping"

URL: http://rosettacode.org/wiki/Web_Scraping

parse html [thru <br> copy current thru "UTC" to end]

 

 

=={{header|Ruby}}==

A verbose example for comparison

 

 

open('http://tycho.usno.navy.mil/cgi-bin/timer.pl') do |p|

end

end

 

A more concise example

 

puts URI.parse('http://tycho.usno.navy.mil/cgi-bin/timer.pl').read.match(/ (\d{1,2}:\d{1,2}:\d{1,2}) UTC/)[1]

 

=={{header|Scala}}==

 

import scala.io.Source

 

}

}

 

=={{header|Scheme}}==

{{works with|Guile}}

(use-modules (ice-9 regex) (ice-9 rdelim))

 

; Display result

(display time)

 

=={{header|Seed7}}==

include "gethttp.s7i";

 

end if;

end if;

 

=={{header|Sidef}}==

var url = 'http://tycho.usno.navy.mil/cgi-bin/timer.pl';

var match = /<BR>(.+? UTC)/.match(ua.get(url));

{{out}}

<pre>

</pre>

 

 

=={{header|Tcl}}==

===<tt>http</tt> and regular expressions===

 

set request [http::geturl "http://tycho.usno.navy.mil/cgi-bin/timer.pl"]

if {[regexp -line {<BR>(.* UTC)} [http::data $request] --> utc]} {

puts $utc

 

===curl(1) and list operations===

Considering the web resource returns tabular data wrapped in a <tt>&lt;PRE&gt;</tt> tag, you can use Tcl's list processing commands to process its contents.

 

=={{header|TUSCRIPT}}==

$$ MODE TUSCRIPT

SET time = REQUEST ("http://tycho.usno.navy.mil/cgi-bin/timer.pl")

SET utc = FILTER (time,":*UTC*:",-)

 

=={{header|TXR}}==

If the web page changes too much, the query will fail to match. TXR will print the word "false" and terminate with a failed exit status. This is preferrable to finding a false positive match and printing a wrong result. (E.g. any random garbage that happened to be in a line of HTML accidentally containing the string UTC).

 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final"//EN>

<html>

@MO-@DD @HH:@MM:@SS @PM @TZ

@ (end)

 

Sample run:<pre>$ txr navytime.txr

Skip stuff until a line beginning with <code>&lt;BR&gt;</code> has some stuff before "UTC", and capture that stuff:

 

@(skip)

<BR>@time@\ UTC@(skip)

@(output)

@time

 

=={{header|UNIX Shell}}==

This solution uses 'curl' and the standard POSIX command 'sed'.

 

curl -s http://tycho.usno.navy.mil/cgi-bin/timer.pl |

 

This solution uses tcsh, wget and awk

 

set page = `wget -q -O- "http://tycho.usno.navy.mil/cgi-bin/timer.pl"`

 

=={{header|Ursala}}==

This works by launching the wget command in a separate process and capturing its output.

The program is compiled to an executable command.

#import cli

 

<.file$[contents: --<''>]>+ -+

@hm skip/*4+ ~=(9%cOi&)-~l*+ *~ ~&K3/'UTC',

Here is a bash session.

<pre>$ whatime

=={{header|VBA}}==

Note For this example I altered the VBScript

 

Function GetUTC() As String

Debug.Print ReturnValue

MsgBox (ReturnValue)

 

 

 

=={{header|VBScript}}==

Url = "http://tycho.usno.navy.mil/cgi-bin/timer.pl"

With CreateObject("MSXML2.XMLHTTP.6.0")

 

WScript.StdOut.Write GetUTC

 

{{Out}}

Apr. 21, 21:02:03 UTC Universal Time

</pre>

 

=={{header|Visual Basic .NET}}==

New, .NET way with StringReader:

Imports System.IO

Dim client As WebClient = New WebClient()

Console.WriteLine(time(0))

End If

 

Alternative, old fashioned way using VB "Split" function:

Dim client As WebClient = New WebClient()

Dim content As String = client.DownloadString("http://tycho.usno.navy.mil/cgi-bin/timer.pl")

Console.WriteLine(time(0))

End If

 

=={{header|zkl}}==

get:="GET %s HTTP/1.0\r\nHost: %s:%s\r\n\r\n".fmt(dir,HOST,PORT);

server:=Network.TCPClientSocket.connectTo(HOST,PORT);

re:=RegExp(0'|.*(\d\d:\d\d:\d\d)|); // get time

re.search(line);

{{out}}

<pre>