Text processing/1: Difference between revisions

Often data is produced by one program, in the wrong format for later use by another program or person. In these situations another program can be written to parse and transform the original data into a format useful to the other. The term "Data Munging" is [http://www.google.co.uk/search?q=%22data+munging%22 often] used in programming circles for this task.

 

<pre>I have to analyse data files that have the following format:

Each row corresponds to 1 day and the field logic is: $1 is the date,

 

The data is [http://www.eea.europa.eu/help/eea-help-centre/faqs/how-do-i-obtain-eea-reports free to download and use] and is of this format:

1991-03-30 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1

1991-03-31 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 20.000 1 20.000 1 20.000 1 35.000 1 50.000 1 60.000 1 40.000 1 30.000 1 30.000 1 30.000 1 25.000 1 20.000 1 20.000 1 20.000 1 20.000 1 20.000 1 35.000 1

 

Structure your program to show statistics for each line of the file, (similar to the original Python, Perl, and AWK examples below), followed by summary statistics for the file. When showing example output just show a few line statistics and the full end summary.

 

=={{header|Ada}}==

{{libheader|Simple components for Ada}}

with Strings_Edit; use Strings_Edit;

with Strings_Edit.Floats; use Strings_Edit.Floats;

Close (File);

Put_Line ("Syntax error at " & Image (Current.Line) & ':' & Image (Max.Pointer));

<pre>

Average 10.47915 over 129628

Max. 589 false readings start at 1136:20 stamped 1993-2-9

</pre>

=={{header|ALGOL 68}}==

{{trans|python}}

<!--{{does not work with|ALGOL 68|Standard - argc and argv are extensions}} -->

{{works with|ALGOL 68G|Any - tested with release mk15-0.8b.fc9.i386}}

<!--{{does not work with|ELLA ALGOL 68|Any (with appropriate job cards) - argc and argv are extensions}} -->

INT no data max := -1; # Max consecutive flags<0 in lines of file #

FLEX[0]STRING no data max line; # ... and line number(s) where it occurs #

upb list := UPB no data max line;

printf(($l"Maximum run"f(p)" of "g(-0)" consecutive false reading"f(p)" ends at line starting with date"f(p)": "$,

Command:

$ a68g ./Data_Munging.a68 - data

<pre>

Line: 1991-03-30 Reject: 0 Accept: 24 Line tot: 240.000 Line avg: 10.000

Maximum run of 24 consecutive false readings ends at line starting with date: 1991-04-01

</pre>

 

=={{header|AutoHotkey}}==

 

SetFormat, float, 4.2

Totavg := TotSum / TotValid

FileAppend, `n`nDays %Lines%`nMaximal wrong readings: %maxwrong% from %startwrongdate% at %startoccurrence% to %lastwrongdate% at %lastoccurrence%`n`n, %result%

<pre>Day: 1990-01-01 sum: 590.00 avg: 26.82 Readings: 22/24.00

Day: 1990-01-02 sum: 410.00 avg: 17.08 Readings: 24/24.00

 

=={{header|AWK}}==

nodata = 0; # Current run of consecutive flags<0 in lines of file

nodata_max=-1; # Max consecutive flags<0 in lines of file

 

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n", nodata_max, nodata_maxline

<pre>bash$ awk -f readings.awk readings.txt | tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

 

=={{header|Batch File}}==

setlocal ENABLEDELAYEDEXPANSION

set maxrun= 0

echo Line: %date% Accept: %count:~-3% tot: %sum:~-8% avg: %mean:~-8%

 

<pre>

C:\ >batch-fileparsing.bat readings-2.txt

Line: 1990-01-01 Accept: 22 tot: 590.000 avg: 26.818

...

max false: 589 from 1993-02-09 2 until 1993-03-05 14

</pre>

 

=={{header|C}}==

#include <stdlib.h>

#include <string.h>

fclose(outfile);

return 0;

<pre>1990-01-01 Reject: 2 Accept: 22 Average: 26.818

1990-01-02 Reject: 0 Accept: 24 Average: 17.083

 

=={{header|C++}}==

#include <fstream>

#include <string>

cout << "Maximum number of consecutive bad readings is " << badCountMax << endl;

cout << "Ends on date " << badDate << endl;

<pre>1990-01-01 Reject: 2 Accept: 22 Average: 26.818

1990-01-02 Reject: 0 Accept: 24 Average: 17.083

Ends on date 1993-03-05</pre>

 

 

 

 

 

 

 

 

 

 

 

 

 

 

=={{header|D}}==

{{trans|Python}}

 

 

int noData, noDataMax = -1;

int fileValues;

 

double lineTotal = 0.0;

int lineValues;

 

const date = parts[0];

const fields = parts[1 .. $];

 

for (int i; i < fields.length; i += 2) {

 

if (flag < 1) {

}

 

if (noDataMax == noData && noData > 0)

noDataMaxLine ~= date.idup;

}

 

noData = 0;

 

lineTotal += value;

lineValues++;

}

 

fileTotal += lineTotal;

fileValues += lineValues;

}

 

writefln("Total = %10.3f", fileTotal);

writefln("Readings = %6d", fileValues);

 

writefln("\nMaximum run(s) of %d consecutive false " ~

 

 

=={{header|Forth}}==

{{works with|GNU Forth}}

 

\ 1991-03-30[\t10.000\t[-]1]*24

total-sum f@ total-n @ .mean cr ;

 

=={{header|Go}}==

 

import (

)

 

 

func main() {

<pre>

...

 

=={{header|Haskell}}==

import Numeric

import Control.Arrow

(\(t,n) -> printf totalFmt n t (t/fromIntegral n)) $ fst summ

mapM_ ((\(l, d1,d2) -> printf maxFmt l d1 d2)

<pre>Some lines:

 

 

=={{header|Icon}} and {{header|Unicon}}==

 

procedure main()

else

write(fout,"No bad runs of data")

<pre>...

Line: 2004-12-28 Reject: 1 Accept: 23 Line_tot: 77.80000000000001 Line_avg: 3.382608695652174

=={{header|J}}==

'''Solution:'''

parseLine=: 10&({. ,&< (_99&".;._1)@:}.) NB. custom parser

summarize=: # , +/ , +/ % # NB. count,sum,mean

589

]StartDates=: Dates {~ (>:@I.@e.&MaxRun (24 <.@%~ +/)@{. ]) RunLengths

'''Formatting Output'''<br>

Define report formatting verbs:

labels=. , ];.2 'Line: Accept: Line_tot: Line_avg: '

labels , x ,. 7j0 10j3 10j3 ": y

'maxrun dates'=. x

out=. out,LF,'Maximum run(s) of ',(": maxrun),' consecutive false readings ends at line(s) starting with date(s): ',dates

Line: Accept: Line_tot: Line_avg:

2004-12-28 23 77.800 3.383

Average: 10.497

 

 

=={{header|JavaScript}}==

{{works with|JScript}}

var show_lines = 5;

var file_stats = {

function dec3(value) {

return Math.round(value * 1e3) / 1e3;

<pre>Line: 1990-01-01 Reject: 2 Accept: 22 Line_tot: 590 Line_avg: 26.818

Line: 1990-01-02 Reject: 0 Accept: 24 Line_tot: 410 Line_avg: 17.083

 

Maximum run of 589 consecutive false readings ends at 1993-03-05</pre>

=={{header|Lua}}==

io.input( filename )

 

print( string.format( "Readings: %d", file_lines ) )

print( string.format( "Average: %f", file_sum/file_cnt_data ) )

<pre>Output:

File: readings.txt

Maximum 589 consecutive false readings starting at 1993-02-09.</pre>

 

Scan[(a=Position[#[[3;;All;;2]],1];

Print["Line:",#[[1]] ,"\tReject:", 24 - Length[a], "\t Accept:", Length[a], "\tLine_tot:",

Print["\nFile(s) : ",FileName,"\nTotal : ",AccountingForm@GlobalSum,"\nReadings : ",Nb,

"\nAverage : ",GlobalSum/Nb,"\n\nMaximum run(s) of ",MaxRunRecorded,

 

<pre>Line:1990-01-01 Reject:2 Accept:22 Line_tot:590. Line_avg:26.8182

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s):{1993-03-05}</pre>

 

 

=={{header|OCaml}}==

try Some(input_line ic)

with End_of_file -> None

Printf.printf "Maximum run(s) of %d consecutive false readings \

ends at line starting with date(s): %s\n"

 

=={{header|Perl}}==

===An AWK-like solution===

use warnings;

 

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n",

<pre>bash$ perl -f readings.pl readings.txt | tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

 

===An object-oriented solution===

use warnings;

 

$parser->_push_bad_range_if_necessary

}

<pre>$ perl readings.pl < readings.txt | tail

average(2004-12-27) = 57.100 / 23 = 2.483

$</pre>

 

 

 

 

 

<pre>

 

=={{header|PicoLisp}}==

{{trans|AWK}}

Put the following into an executable file "readings":

 

(let (NoData 0 NoDataMax -1 NoDataMaxline "!" TotFile 0 NumFile 0)

" consecutive false readings ends at line starting with date(s): " NoDataMaxline ) ) )

 

Then it can be called as

<pre>$ ./readings readings.txt |tail

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05

$</pre>

 

=={{header|PL/I}}==

 

declare line character (2000) varying;

finish_up:

 

=={{header|PureBasic}}==

Define File$, InLine$, Part$, i, Out$, ErrEnds$, Errcnt, ErrMax

Define lsum.d, tsum.d, rejects, val.d, readings

;

Print("Press ENTER to exit"): Input()

<pre>...

Line: 2004-12-27 Rejects: 1 Accepts: 23 Line_tot: 57.100 Line_avg: 2.483

 

=={{header|Python}}==

import sys

 

 

print "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s" % (

<pre>bash$ /cygdrive/c/Python26/python readings.py readings.txt|tail

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

 

=={{header|R}}==

dfr <- read.delim("readings.txt")

#Calculate daily means

#Calculate time between good measurements

times <- strptime(dfr[1,1], "%Y-%m-%d", tz="GMT") + 3600*seq(1,24*nrow(dfr),1)

 

=={{header|REXX}}==

grandOKs=0

 

sum=0

flg=0

OKs=0

 

parse var Idata data.j flag.j Idata

 

OKs=OKs+1

sum=sum+data.j

end

flg=flg+1

end

end /*j*/

 

grandOKs=grandOKs+OKs

grandSum=grandSum+sum

grandFlg=grandFlg+flg

end /*recs*/

 

else Gavg='[n/a]'

call sy

call sy copies('═',60)

call sy copies('═',60)

∙

∙

 

=={{header|Ruby}}==

total = { "num_readings" => 0, "num_good_readings" => 0, "sum_readings" => 0.0 }

invalid_count = 0

printf "Average = %.3f\n", total['sum_readings']/total['num_good_readings']

puts ""

 

=={{header|Scala}}==

 

A fully functional solution, minus the fact that it uses iterators:

import scala.io.Source

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, invalidCount, startDate))

}

A quick&dirty solution:

def main(args: Array[String]) {

var totalSum = 0.0

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, maxInvalidCount, maxInvalidDate))

}

Last few lines of the sample output (either version):

<pre>

Line: 2004-12-29 Reject: 1 Accept: 23 Line_tot: 56.300 Line_avg: 2.448

Maximum run(s) of 589 consecutive false readings began at 1993-02-09

</pre>

 

 

=={{header|Tcl}}==

set max_invalid_run_end ""

set tot_file 0

puts "Average = [format %.3f [expr {$tot_file / $num_file}]]"

puts ""

 

=={{header|Ursala}}==

and booleans (type <code>%ebXLm</code>) in the parsed data. The same function is used to compute the daily and the

cumulative statistics.

#import nat

#import flo

@nmrSPDSL -&~&,leql$^; ^/length ~&zn&-@hrZPF+ rlc both ~&rZ+-

 

last few lines of output:

<pre>

summary accept: 129403 total: 1358393.4 average: 10.497

maximum of 589 consecutive false readings ending on line 1993-03-05

</pre>

 

=={{header|Vedit macro language}}==

{{trans|AWK}}

Vedit does not have floating point data type, so fixed point calculations are used here.

File_Open("output.txt")

#51 = Buf_Num // Edit buffer for output file

IT("Maximum run(s) of ") Num_Ins(#13, LEFT+NOCR)

IT(" consecutive false readings ends at line starting with date(s): ") Reg_Ins(15)

<pre>

Line: 2004-12-28 Reject: 1 Accept: 23 Line tot: 77.800 Line avg: 3.383

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05

</pre>

 

{{omit from|Openscad}}