Text processing/1: Difference between revisions

Often data is produced by one program, in the wrong format for later use by another program or person. In these situations another program can be written to parse and transform the original data into a format useful to the other. The term "Data Munging" is [http://www.google.co.uk/search?q=%22data+munging%22 often] used in programming circles for this task.

 

<pre>I have to analyse data files that have the following format:

Each row corresponds to 1 day and the field logic is: $1 is the date,

 

The data is [http://www.eea.europa.eu/help/eea-help-centre/faqs/how-do-i-obtain-eea-reports free to download and use] and is of this format:

<pre style="overflow:scroll">

1991-03-30 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1

 

Structure your program to show statistics for each line of the file, (similar to the original Python, Perl, and AWK examples below), followed by summary statistics for the file. When showing example output just show a few line statistics and the full end summary.

 

=={{header|Ada}}==

{{libheader|Simple components for Ada}}

with Strings_Edit; use Strings_Edit;

with Strings_Edit.Floats; use Strings_Edit.Floats;

Close (File);

Put_Line ("Syntax error at " & Image (Current.Line) & ':' & Image (Max.Pointer));

The implementation performs minimal checks. The average is calculated over all valid data. For the maximal chain of consequent invalid data, the source line number, the column number, and the time stamp of the first invalid data is printed.

{{out|Sample output}}

Max. 589 false readings start at 1136:20 stamped 1993-2-9

</pre>

 

=={{header|ALGOL 68}}==

{{works with|ALGOL 68G|Any - tested with release mk15-0.8b.fc9.i386}}

<!--{{does not work with|ELLA ALGOL 68|Any (with appropriate job cards) - argc and argv are extensions}} -->

INT no data max := -1; # Max consecutive flags<0 in lines of file #

FLEX[0]STRING no data max line; # ... and line number(s) where it occurs #

upb list := UPB no data max line;

printf(($l"Maximum run"f(p)" of "g(-0)" consecutive false reading"f(p)" ends at line starting with date"f(p)": "$,

Command:

$ a68g ./Data_Munging.a68 - data

Maximum run of 24 consecutive false readings ends at line starting with date: 1991-04-01

</pre>

 

=={{header|AutoHotkey}}==

 

SetFormat, float, 4.2

Totavg := TotSum / TotValid

FileAppend, `n`nDays %Lines%`nMaximal wrong readings: %maxwrong% from %startwrongdate% at %startoccurrence% to %lastwrongdate% at %lastoccurrence%`n`n, %result%

{{out|Sample output}}

<pre>Day: 1990-01-01 sum: 590.00 avg: 26.82 Readings: 22/24.00

 

=={{header|AWK}}==

nodata = 0; # Current run of consecutive flags<0 in lines of file

nodata_max=-1; # Max consecutive flags<0 in lines of file

 

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n", nodata_max, nodata_maxline

{{out|Sample output}}

<pre>bash$ awk -f readings.awk readings.txt | tail

 

=={{header|Batch File}}==

setlocal ENABLEDELAYEDEXPANSION

set maxrun= 0

echo Line: %date% Accept: %count:~-3% tot: %sum:~-8% avg: %mean:~-8%

 

{{out}}

<pre>

 

=={{header|BBC BASIC}}==

IF file% = 0 THEN PRINT "Could not open test data file" : END

PRINT "Overall mean = " ; Total / Count%

@% = &90A

{{out}}

<pre>

 

=={{header|C}}==

#include <stdlib.h>

#include <string.h>

fclose(outfile);

return 0;

{{out|Sample output}}

<pre>1990-01-01 Reject: 2 Accept: 22 Average: 26.818

 

=={{header|C++}}==

#include <fstream>

#include <string>

cout << "Maximum number of consecutive bad readings is " << badCountMax << endl;

cout << "Ends on date " << badDate << endl;

{{out}}

<pre>1990-01-01 Reject: 2 Accept: 22 Average: 26.818

Maximum number of consecutive bad readings is 589

Ends on date 1993-03-05</pre>

 

=={{header|COBOL}}==

PROGRAM-ID. data-munging.

 

 

GOBACK

 

{{Out|Example output}}

 

=={{header|Common Lisp}}==

(defvar *max-invalid*)

(defvar *max-invalid-date*)

(format t "~%Maximum run(s) of ~a consecutive false readings ends at ~

line starting with date(s): ~a~%"

{{out|Example output}}

<pre>...

=={{header|D}}==

{{trans|Python}}

import std.stdio, std.conv, std.string;

 

"readings ends at line starting with date(s): %-(%s, %)",

noDataMax, noDataMaxLine);

The output matches that of the [[#Python|Python]] version.

 

=={{header|Erlang}}==

The function file_contents/1 is used by [[Text_processing/2]]. Please update the user if you make any interface changes.

 

-module( text_processing ).

 

{_Previous, Value_flags} = lists:foldr( fun file_content_line_value_flag/2, {[], []}, Rest ), % Preserve order

{binary:bin_to_list( Date_binary ), Value_flags}.

{{out}}

<pre>

=={{header|Forth}}==

{{works with|GNU Forth}}

 

\ 1991-03-30[\t10.000\t[-]1]*24

total-sum f@ total-n @ .mean cr ;

 

 

=={{header|Go}}==

 

import (

maxRun, maxDate)

}

{{out}}

<pre>

 

=={{header|Haskell}}==

import Numeric

import Control.Arrow

(\(t,n) -> printf totalFmt n t (t/fromIntegral n)) $ fst summ

mapM_ ((\(l, d1,d2) -> printf maxFmt l d1 d2)

{{out}}

<pre>Some lines:

 

 

=={{header|Icon}} and {{header|Unicon}}==

 

procedure main()

else

write(fout,"No bad runs of data")

{{out|Sample output}}

<pre>...

=={{header|J}}==

'''Solution:'''

parseLine=: 10&({. ,&< (_99&".;._1)@:}.) NB. custom parser

summarize=: # , +/ , +/ % # NB. count,sum,mean

589

]StartDates=: Dates {~ (>:@I.@e.&MaxRun (24 <.@%~ +/)@{. ]) RunLengths

'''Formatting Output'''<br>

Define report formatting verbs:

labels=. , ];.2 'Line: Accept: Line_tot: Line_avg: '

labels , x ,. 7j0 10j3 10j3 ": y

'maxrun dates'=. x

out=. out,LF,'Maximum run(s) of ',(": maxrun),' consecutive false readings ends at line(s) starting with date(s): ',dates

{{out|Show output}}

Line: Accept: Line_tot: Line_avg:

2004-12-28 23 77.800 3.383

Average: 10.497

 

 

=={{header|Java}}==

{{works with|Java|7}}

import java.util.*;

import static java.lang.System.out;

}

}

 

<pre>1990-01-01 out: 2 in: 22 tot: 590.000 avg: 26.818

=={{header|JavaScript}}==

{{works with|JScript}}

var show_lines = 5;

var file_stats = {

function dec3(value) {

return Math.round(value * 1e3) / 1e3;

{{out}}

<pre>Line: 1990-01-01 Reject: 2 Accept: 22 Line_tot: 590 Line_avg: 26.818

 

Maximum run of 589 consecutive false readings ends at 1993-03-05</pre>

 

=={{header|Lua}}==

io.input( filename )

 

print( string.format( "Readings: %d", file_lines ) )

print( string.format( "Average: %f", file_sum/file_cnt_data ) )

<pre>Output:

File: readings.txt

Maximum 589 consecutive false readings starting at 1993-02-09.</pre>

 

Scan[(a=Position[#[[3;;All;;2]],1];

Print["Line:",#[[1]] ,"\tReject:", 24 - Length[a], "\t Accept:", Length[a], "\tLine_tot:",

Print["\nFile(s) : ",FileName,"\nTotal : ",AccountingForm@GlobalSum,"\nReadings : ",Nb,

"\nAverage : ",GlobalSum/Nb,"\n\nMaximum run(s) of ",MaxRunRecorded,

 

<pre>Line:1990-01-01 Reject:2 Accept:22 Line_tot:590. Line_avg:26.8182

=={{header|Nim}}==

{{trans|Python}}

 

var

nodata = 0

nodataMax = -1

 

totFile = 0.0

 

for filename in commandLineParams():

var

totLine = 0.0

numLine = 0

 

 

if flag < 1:

inc nodata

numFile += numLine

 

 

echo ""

 

File(s) = readings.txt

Readings = 129403

 

 

=={{header|OCaml}}==

try Some(input_line ic)

with End_of_file -> None

Printf.printf "Maximum run(s) of %d consecutive false readings \

ends at line starting with date(s): %s\n"

 

=={{header|Perl}}==

===An AWK-like solution===

use warnings;

 

printf "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s\n",

{{out|Sample output}}

<pre>bash$ perl -f readings.pl readings.txt | tail

 

===An object-oriented solution===

use warnings;

 

$parser->_push_bad_range_if_necessary

}

{{out|Sample output}}

<pre>$ perl readings.pl < readings.txt | tail

$</pre>

 

 

 

 

{{out}}

<pre>

 

=={{header|PicoLisp}}==

{{trans|AWK}}

Put the following into an executable file "readings":

 

(let (NoData 0 NoDataMax -1 NoDataMaxline "!" TotFile 0 NumFile 0)

" consecutive false readings ends at line starting with date(s): " NoDataMaxline ) ) )

 

Then it can be called as

<pre>$ ./readings readings.txt |tail

 

=={{header|PL/I}}==

 

declare line character (2000) varying;

finish_up:

 

=={{header|PureBasic}}==

Define File$, InLine$, Part$, i, Out$, ErrEnds$, Errcnt, ErrMax

Define lsum.d, tsum.d, rejects, val.d, readings

;

Print("Press ENTER to exit"): Input()

{{out|Sample output}}

<pre>...

 

=={{header|Python}}==

import sys

 

 

print "\nMaximum run(s) of %i consecutive false readings ends at line starting with date(s): %s" % (

{{out|Sample output}}

<pre>bash$ /cygdrive/c/Python26/python readings.py readings.txt|tail

 

=={{header|R}}==

dfr <- read.delim("readings.txt")

#Calculate daily means

#Calculate time between good measurements

times <- strptime(dfr[1,1], "%Y-%m-%d", tz="GMT") + 3600*seq(1,24*nrow(dfr),1)

 

=={{header|Racket}}==

;; Use SRFI 48 to make %n.nf formats convenient.

(require (prefix-in srfi/48: srfi/48)) ; SRFI 48: Intermediate Format Strings

(unless (zero? N) (srfi/48:format #t "Average = ~10,3F~%" (/ sum N)))

(srfi/48:format #t "~%Maximum run(s) of ~a consecutive false readings ends at line starting with date(s): ~a~%"

{{out|Sample run}}

<pre>$ racket 1.rkt readings/readings.txt | tail

 

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05</pre>

 

=={{header|REXX}}==

grandOKs=0

 

sum=0

flg=0

OKs=0

 

parse var Idata data.j flag.j Idata

 

OKs=OKs+1

sum=sum+data.j

end

flg=flg+1

end

end /*j*/

 

grandOKs=grandOKs+OKs

grandSum=grandSum+sum

grandFlg=grandFlg+flg

end /*recs*/

 

else Gavg='[n/a]'

call sy

call sy copies('═',60)

call sy copies('═',60)

∙

∙

 

=={{header|Ruby}}==

total = { "num_readings" => 0, "num_good_readings" => 0, "sum_readings" => 0.0 }

invalid_count = 0

printf "Average = %.3f\n", total['sum_readings']/total['num_good_readings']

puts ""

 

=={{header|Scala}}==

 

A fully functional solution, minus the fact that it uses iterators:

import scala.io.Source

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, invalidCount, startDate))

}

A quick&dirty solution:

def main(args: Array[String]) {

var totalSum = 0.0

println(report format (files mkString " ", totalSum, totalSize, totalSum / totalSize, maxInvalidCount, maxInvalidDate))

}

Last few lines of the sample output (either version):

<pre>

</pre>

Though it is easier to show when the consecutive false readings ends, if longest run is the last thing in the file, it hasn't really "ended".

 

=={{header|Tcl}}==

set max_invalid_run_end ""

set tot_file 0

puts "Average = [format %.3f [expr {$tot_file / $num_file}]]"

puts ""

 

=={{header|Ursala}}==

and booleans (type <code>%ebXLm</code>) in the parsed data. The same function is used to compute the daily and the

cumulative statistics.

#import nat

#import flo

@nmrSPDSL -&~&,leql$^; ^/length ~&zn&-@hrZPF+ rlc both ~&rZ+-

 

last few lines of output:

<pre>

summary accept: 129403 total: 1358393.4 average: 10.497

maximum of 589 consecutive false readings ending on line 1993-03-05

</pre>

 

{{trans|AWK}}

Vedit does not have floating point data type, so fixed point calculations are used here.

File_Open("output.txt")

#51 = Buf_Num // Edit buffer for output file

IT("Maximum run(s) of ") Num_Ins(#13, LEFT+NOCR)

IT(" consecutive false readings ends at line starting with date(s): ") Reg_Ins(15)

{{out|Sample output}}

<pre>

Maximum run(s) of 589 consecutive false readings ends at line starting with date(s): 1993-03-05

</pre>

 

{{omit from|Openscad}}