Text processing/1: Difference between revisions

Often data is produced by one program, in the wrong format for later use by another program or person. In these situations another program can be written to parse and transform the original data into a format useful to the other. The term "Data Munging" is [http://www.google.co.uk/search?q=%22data+munging%22 often] used in programming circles for this task.

 

<pre>I have to analyse data files that have the following format:

Each row corresponds to 1 day and the field logic is: $1 is the date,

 

The data is [http://www.eea.europa.eu/help/eea-help-centre/faqs/how-do-i-obtain-eea-reports free to download and use] and is of this format:

1991-03-30 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1

1991-03-31 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 20.000 1 20.000 1 20.000 1 35.000 1 50.000 1 60.000 1 40.000 1 30.000 1 30.000 1 30.000 1 25.000 1 20.000 1 20.000 1 20.000 1 20.000 1 20.000 1 35.000 1

 

Structure your program to show statistics for each line of the file, (similar to the original Python, Perl, and AWK examples below), followed by summary statistics for the file. When showing example output just show a few line statistics and the full end summary.

 

=={{header|Ada}}==

{{libheader|Simple components for Ada}}

with Strings_Edit; use Strings_Edit;

with Strings_Edit.Floats; use Strings_Edit.Floats;

Close (File);

Put_Line ("Syntax error at " & Image (Current.Line) & ':' & Image (Max.Pointer));

<pre>

Average 10.47915 over 129628

Max. 589 false readings start at 1136:20 stamped 1993-2-9

</pre>

=={{header|ALGOL 68}}==

{{trans|python}}

<!--{{does not work with|ALGOL 68|Standard - argc and argv are extensions}} -->

{{works with|ALGOL 68G|Any - tested with release mk15-0.8b.fc9.i386}}

<!--{{does not work with|ELLA ALGOL 68|Any (with appropriate job cards) - argc and argv are extensions}} -->

INT no data max := -1; # Max consecutive flags<0 in lines of file #

FLEX[0]STRING no data max line; # ... and line number(s) where it occurs #

upb list := UPB no data max line;

printf(($l"Maximum run"f(p)" of "g(-0)" consecutive false reading"f(p)" ends at line starting with date"f(p)": "$,

Command:

$ a68g ./Data_Munging.a68 - data

<pre>

Line: 1991-03-30 Reject: 0 Accept: 24 Line tot: 240.000 Line avg: 10.000

</pre>

 

 

nodata = 0; # Current run of consecutive flags<0 in lines of file

nodata_max=-1; # Max consecutive flags<0 in lines of file

 

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n", nodata_max, nodata_maxline

<pre>bash$ awk -f readings.awk readings.txt | tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05

bash$ </pre>

 

=={{header|C}}==

#include <stdlib.h>

#include <string.h>

}

 

{

FILE *infile, *outfile;

int lineNo = 0;

char line[512];

outfile = fopen("V0.txt", "w");

 

fclose(outfile);

return 0;

<pre>1990-01-01 Reject: 2 Accept: 22 Average: 26.818

1990-01-02 Reject: 0 Accept: 24 Average: 17.083

Ends on date 1993-03-05</pre>

 

 

 

 

 

 

 

 

 

 

 

 

 

 

=={{header|D}}==

 

 

string[] noDataMaxLine;

 

 

double lineTotal = 0.0;

 

 

 

continue;

}

 

noDataMax = noData;

noDataMaxLine.length = 1;

}

 

noData = 0;

 

lineTotal += value;

}

 

fileTotal += lineTotal;

fileValues += lineValues;

 

}

}

 

writefln("Total = %10.3f", fileTotal);

writefln("Readings = %6d", fileValues);

 

 

 

=={{header|Forth}}==

{{works with|GNU Forth}}

 

\ 1991-03-30[\t10.000\t[-]1]*24

total-sum f@ total-n @ .mean cr ;

 

 

=={{header|Haskell}}==

import Numeric

import Control.Arrow

(\(t,n) -> printf totalFmt n t (t/fromIntegral n)) $ fst summ

mapM_ ((\(l, d1,d2) -> printf maxFmt l d1 d2)

<pre>Some lines:

 

 

Maximum of 589 consecutive false readings, starting on line /1993-02-09/ and ending on line /1993-03-05/</pre>

 

=={{header|J}}==

'''Solution:'''

parseLine=: 10&({. ,&< (_99&".;._1)@:}.) NB. custom parser

summarize=: # , +/ , +/ % # NB. count,sum,mean

589

]StartDates=: Dates {~ (>:@I.@e.&MaxRun (24 <.@%~ +/)@{. ]) RunLengths

'''Formatting Output'''<br>

Define report formatting verbs:

labels=. , ];.2 'Line: Accept: Line_tot: Line_avg: '

labels , x ,. 7j0 10j3 10j3 ": y

'maxrun dates'=. x

out=. out,LF,'Maximum run(s) of ',(": maxrun),' consecutive false readings ends at line(s) starting with date(s): ',dates

Line: Accept: Line_tot: Line_avg:

2004-12-28 23 77.800 3.383

Average: 10.497

 

 

=={{header|JavaScript}}==

{{works with|JScript}}

var show_lines = 5;

var file_stats = {

function dec3(value) {

return Math.round(value * 1e3) / 1e3;

<pre>Line: 1990-01-01 Reject: 2 Accept: 22 Line_tot: 590 Line_avg: 26.818

Line: 1990-01-02 Reject: 0 Accept: 24 Line_tot: 410 Line_avg: 17.083

 

Maximum run of 589 consecutive false readings ends at 1993-03-05</pre>

=={{header|OCaml}}==

try Some(input_line ic)

with End_of_file -> None

Printf.printf "Maximum run(s) of %d consecutive false readings \

ends at line starting with date(s): %s\n"

 

=={{header|Perl}}==

 

 

 

# extract field info, skipping initial date field

$nodata++;

}

}

# totals for the file so far

$tot_file += $tot_line;

$num_file += $num_line;

printf "Line: %11s Reject: %2i Accept: %2i Line_tot: %10.3f Line_avg: %10.3f\n",

}

printf "\n";

printf "File(s) = %s\n", $infiles;

printf "Readings = %6i\n", $num_file;

printf "Average = %10.3f\n", $tot_file / $num_file;

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n",

<pre>bash$ perl -f readings.pl readings.txt | tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

bash$</pre>

 

 

import sys

 

 

print "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s" % (

<pre>bash$ /cygdrive/c/Python26/python readings.py readings.txt|tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

 

=={{header|R}}==

dfr <- read.delim("readings.txt")

#Calculate daily means

#Calculate time between good measurements

times <- strptime(dfr[1,1], "%Y-%m-%d", tz="GMT") + 3600*seq(1,24*nrow(dfr),1)

 

=={{header|Ruby}}==

total = { "num_readings" => 0, "num_good_readings" => 0, "sum_readings" => 0.0 }

invalid_count = 0

printf "Average = %.3f\n", total['sum_readings']/total['num_good_readings']

puts ""

 

=={{header|Scala}}==

import scala.io.Source

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, invalidCount, startDate))

}

A quick&dirty solution:

def main(args: Array[String]) {

var totalSum = 0.0

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, maxInvalidCount, maxInvalidDate))

}

Last few lines of the sample output (either version):

<pre>

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

Maximum run(s) of 589 consecutive false readings began at 1993-02-09

</pre>

 

 

=={{header|Tcl}}==

set max_invalid_run_end ""

set tot_file 0

puts "Average = [format %.3f [expr {$tot_file / $num_file}]]"

puts ""

 

=={{header|Vedit macro language}}==

{{trans|AWK}}

Vedit does not have floating point data type, so fixed point calculations are used here.

File_Open("output.txt")

#51 = Buf_Num // Edit buffer for output file

IT("Maximum run(s) of ") Num_Ins(#13, LEFT+NOCR)

IT(" consecutive false readings ends at line starting with date(s): ") Reg_Ins(15)

<pre>

Line: 2004-12-28 Reject: 1 Accept: 23 Line tot: 77.800 Line avg: 3.383

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05

</pre>