Text processing/2: Difference between revisions

{{task|Text processing}}

The following task concerns data that came from a pollution monitoring station with twenty-four instruments monitoring twenty-four aspects of pollution in the air. Periodically a record is added to the file, each record being a line of 49 fields separated by white-space, which can be one or more space or tab characters.

 

i.e. a datestamp followed by twenty-four repetitions of a floating-point instrument value and that instrument's associated integer flag. Flag values are >= 1 if the instrument is working and < 1 if there is some problem with it, in which case that instrument's value should be ignored.

 

1991-03-30 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1

1991-03-31 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 10.000 1 20.000 1 20.000 1 20.000 1 35.000 1 50.000 1 60.000 1 40.000 1 30.000 1 30.000 1 30.000 1 25.000 1 20.000 1 20.000 1 20.000 1 20.000 1 20.000 1 35.000 1

</pre>

 

# Confirm the general field format of the file.

# Identify any DATESTAMPs that are duplicated.

# Report the number of records that have good readings for all instruments.

 

=={{header|Ada}}==

{{libheader|Simple components for Ada}}

with Ada.Text_IO; use Ada.Text_IO;

with Strings_Edit; use Strings_Edit;

Close (File);

Put_Line ("Valid records " & Image (Count) & " of " & Image (Line_No) & " total");

Sample output

<pre>

 

=={{header|Aime}}==

{

integer i;

text s;

 

}

 

error("bad date format");

}

 

i = 1;

while (i < 49) {

i += 1;

i += 1;

}

}

 

main(void)

{

file f;

list l;

 

goods = 0;

 

 

if (!trap(check_format, l)) {

 

}

}

 

 

{{out}} (the "reading.txt" needs to be converted to UNIX end-of-line)

 

=={{header|AutoHotkey}}==

 

data = %A_scriptdir%\readings.txt

 

msgbox, Duplicate Dates:`n%wrongDates%`nRead Lines: %lines%`nValid Lines: %valid%`nwrong lines: %totwrong%`nDuplicates: %TotWrongDates%`nWrong Formatted: %unvalidformat%`n

 

Sample Output:

 

If their are any scientific notation fields then their will be an e in the file:

Quick check on the number of fields:

Full check on the file format using a regular expression:

Full check on the file format as above but using regular expressions allowing intervals (gnu awk):

 

 

 

Accomplished by counting how many times the first field occurs and noting any second occurrences.

1990-03-25

1991-03-31

1993-03-28

1995-03-26

 

 

 

<div style="width:100%;overflow:scroll">

Total records 5471 OK records 5017 or 91.7017 %

</div>

 

=={{header|C}}==

#include <string.h>

#include <stdlib.h>

read_file("readings.txt");

return 0;

 

{{out}}

 

5017 out 5471 lines good

</pre>

 

=={{header|C sharp|C#}}==

using System.Collections.Generic;

using System.Text.RegularExpressions;

}

}

 

<pre>

1993-03-28 is duplicated at Lines : 1183,1184

1995-03-26 is duplicated at Lines : 1910,1911

</pre>

 

=={{header|COBOL}}==

{{works with|OpenCOBOL}}

PROGRAM-ID. text-processing-2.

 

INSPECT input-data (offset:) TALLYING data-len

FOR CHARACTERS BEFORE delim

 

{{out}}

 

=={{header|D}}==

import std.stdio, std.array, std.string, std.regex, std.conv,

std.algorithm;

repeatedDates.byKey.filter!(k => repeatedDates[k] > 1));

writeln("Good reading records: ", goodReadings);

{{out}}

<pre>Duplicated timestamps: 1990-03-25, 1991-03-31, 1992-03-29, 1993-03-28, 1995-03-26

 

=={{header|Eiffel}}==

class

APPLICATION

 

end

{{out}}

<pre>

=={{header|Erlang}}==

Uses function from [[Text_processing/1]]. It does some correctness checks for us.

-module( text_processing2 ).

 

 

value_flag_records() -> 24.

{{out}}

<pre>

 

=={{header|F Sharp|F#}}==

let file = @"readings.txt"

 

ok <- ok + 1

printf "%d records were ok\n" ok

Prints:

Date 1990-03-25 is duplicated

Date 1991-03-31 is duplicated

Date 1995-03-26 is duplicated

5017 records were ok

=={{header|Fortran}}==

The trouble with the dates rather suggests that they should be checked for correctness in themselves, and that the sequence check should be that each new record advances the date by one day. Daynumber calculations were long ago presented by H. F. Fliegel and T.C. van Flandern, in Communications of the ACM, Vol. 11, No. 10 (October, 1968).

Rather than copy today's data to a PDATA holder so that on the next read the new data may be compared to the old, a two-row array is used, with IT flip-flopping 1,2,1,2,1,2,... Comparison of the data as numerical values rather than text strings means that different texts that evoke the same value will not be regarded as different. If the data format were invalid, there would be horrible messages. There aren't, so ... the values should be read and plotted...

 

Crunches a set of hourly data. Starts with a date, then 24 pairs of value,indicator for that day, on one line.

INTEGER Y,M,D !Year, month, and day.

900 CLOSE(IN) !Done.

END !Spaghetti rules.

 

Output:

 

=={{header|Go}}==

 

import (

fmt.Println(uniqueGood,

"unique dates with good readings for all instruments.")

{{out}}

<pre>

 

=={{header|Haskell}}==

import Data.List (nub, (\\))

 

putStr (unlines ("duplicated dates:": duplicatedDates (map date inputs)))

putStrLn ("number of good records: " ++ show (length $ goodRecords inputs))

 

this script outputs:

duplicated timestamps that are on well-formed records.

 

dups := set()

goodRecords := 0

}

 

Sample run:

 

=={{header|J}}==

dat=: TAB readdsv jpath '~temp/readings.txt'

Dates=: getdate"1 >{."1 dat

1992 3 29

1993 3 28

 

=={{header|Java}}==

{{trans|C++}}

{{works with|Java|1.5+}}

import java.util.regex.*;

import java.io.*;

}

}

The program produces the following output:

<pre>

=={{header|JavaScript}}==

{{works with|JScript}}

function analyze_func(filename) {

var dates_seen = {};

 

var analyze = analyze_func('readings.txt');

=={{header|jq}}==

{{works with|jq|with regex support}}

 

For this problem, it is convenient to use jq in a pipeline: the first invocation of jq will convert the text file into a stream of JSON arrays (one array per line):

 

The second part of the pipeline performs the task requirements. The following program is used in the second invocation of jq.

 

'''Generic Utilities'''

def runs:

reduce .[] as $item

def is_integral: test("^[-+]?[0-9]+$");

 

 

'''Validation''':

def validate_line(nr):

def validate_date:

def validate_lines:

. as $in

 

'''Check for duplicate timestamps'''

 

'''Number of valid readings for all instruments''':

# but does check the validity of the record, including the date format:

def number_of_valid_readings:

and all(range(0; 24) | $in[2*. + 2] | (is_integral and tonumber >= 1) );

 

 

'''Generate Report'''

"\nChecking for duplicate timestamps:",

duplicate_timestamps,

{{out}}

'''Part 1: Simple demonstration'''

 

To illustrate that the program does report invalid lines, we first use the six lines at the top but mangle the last line.

field 1 in line 6 has an invalid date: 991-04-03

line 6 has 47 fields

]

 

 

'''Part 2: readings.txt'''

Checking for duplicate timestamps:

[

]

 

=={{header|Lua}}==

io.input( filename )

 

for i = 1, #bad_format do

print( " ", bad_format[i] )

Output:

<pre>Lines read: 5471

 

</pre>

 

Select[Tally@data[[;;,1]], #[[2]]>1&][[;;,1]]//Column

Print["number of good records: ", Count[(Times@@#[[3;;All;;2]])& /@ data, 1],

<pre>duplicated dates:

1990-03-25

1993-03-28

1995-03-26

number of good records: 5017 (out of a total of 5471)</pre>

 

=={{header|MATLAB}} / {{header|Octave}}==

 

% READDAT reads readings.txt file

%

dix = find(diff(d)==0) % check for to consequtive timestamps with zero difference

 

 

<pre>>> [val,count]=readdat;

number of valid records: 5017

</pre>

=={{header|OCaml}}==

open Str

 

let strip_cr str =

 

let map_records =

aux (e::acc) tail

| [_] -> invalid_arg "invalid data"

in

aux [] ;;

aux acc tl

| [] ->

in

aux [] ;;

 

let record_ok (_,record) =

let sum_ok =

List.fold_left (fun sum this ->

if is_ok this then succ sum else sum) 0 record

in

 

let num_good_records =

let li = split (regexp "[ \t]+") line in

let records = map_records (List.tl li)

(date, records)

 

let ic = open_in "readings.txt" in

let rec read_loop acc =

in

let inputs = read_loop [] in

 

Printf.printf "number of good records: %d\n" (num_good_records inputs);

 

this script outputs:

 

=={{header|Perl}}==

use constant FIELDS => 49;

 

map {" $_\n"}

grep {$dates{$_} > 1}

 

Output:

1995-03-26</pre>

 

 

=={{header|PHP}}==

$missformcount = 0;

$totalcount = 0;

foreach ($duplicates as $key => $val){

echo $val . ' at Line : ' . $key . '<br>';

<pre>Valid records 5017 of 5471 total

Duplicates :

1993-03-28 at Line : 1184

1995-03-26 at Line : 1911</pre>

 

=={{header|PL/I}}==

/* To process readings produced by automatic reading stations. */

 

put skip list ('There were ' || k-faulty || ' good readings' );

end check;

 

=={{header|PowerShell}}==

$goodLineCount = 0

get-content c:\temp\readings.txt |

}

[string]$goodLineCount + " good lines"

 

Output:

 

An alternative using regular expression syntax:

$dateHash = @{}

$goodLineCount = 0

}

[string]$goodLineCount + " good lines"

 

Output:

5017 good lines

</pre>

=={{header|PureBasic}}==

Using regular expressions.

#instrumentCount = 24

 

CloseConsole()

EndIf

Sample output:

<pre>Duplicate date: 1990-03-25 occurs on lines 85 and 84.

 

=={{header|Python}}==

import zipfile

import StringIO

#readings = StringIO.StringIO(zfs.read('readings.txt'))

readings = open('readings.txt','r')

The results indicate 5013 good records, which differs from the Awk implementation. The final few lines of the output are as follows

<pre style="height:10ex;overflow:scroll">

* Generate mostly summary information that is easier to compare to other solutions.

 

import zipfile

import StringIO

readings = open('readings.txt','r')

<pre>bash$ /cygdrive/c/Python26/python munge2.py

Duplicate dates:

 

=={{header|R}}==

dfr <- read.delim("d:/readings.txt", colClasses=c("character", rep(c("numeric", "integer"), 24)))

dates <- strptime(dfr[,1], "%Y-%m-%d")

# Number of rows with no bad values

flags <- as.matrix(dfr[,seq(3,49,2)])>0

 

=={{header|Racket}}==

(read-decimal-as-inexact #f)

;; files to read is a sequence, so it could be either a list or vector of files

 

(printf "~a records have good readings for all instruments~%"

Example session:

<pre>$ racket 2.rkt readings/readings.txt

duplicate datestamp: 1995-03-26 at line: 1911 (first seen at: 1910)

5013 records have good readings for all instruments</pre>

 

=={{header|REXX}}==

<br><br>

The program has (negated) code to write the report to a file in addition to the console.

numeric digits 20 /*allow for bigger numbers. */

ifid='READINGS.TXT' /*name of the input file. */

return y//100\==0 | y//400==0 /*apply the 100 and the 400 year rule.*/

/*────────────────────────────────────────────────────────────────────────────*/

'''output''' &nbsp; when using the default input file:

<pre style="height:35ex">

 

=={{header|Ruby}}==

 

def munge2(readings, debug=false)

open('readings.txt','r') do |readings|

munge2(readings)

 

=={{header|Scala}}==

{{works with|Scala|2.8}}

import scala.io.Source

import scala.collection.immutable.{TreeMap => Map}

dateMap.valuesIterable.sum))

}

 

Sample output:

 

=={{header|Sidef}}==

 

ARGF.each { |line|

var i = 0;

}

 

say "#{good_records} good records out of #{$.} total";

say 'Repeated timestamps:';

{{out}}

<pre>

1993-03-28

1995-03-26

</pre>

 

=={{header|Tcl}}==

 

set total [llength $data]

set correct $total

 

puts "$correct records with good readings = [expr $correct * 100.0 / $total]%"

<pre>$ tclsh munge2.tcl

Duplicate datestamp: 1990-03-25

To demonstate a different method to iterate over the file, and different ways to verify data types:

 

array set seen {}

set fh [open readings.txt]

 

puts "total: $total"

Results:

<pre>duplicate date on line 85: 1990-03-25

compiled and run in a single step, with the input file accessed as a list of strings

pre-declared in readings_dot_txt

#import nat

 

#show+

 

output:

<pre>5017 good readings

 

=={{header|VBScript}}==

Set objFile = objFSO.OpenTextFile(objFSO.GetParentFolderName(WScript.ScriptFullName) &_

"\readings.txt",1)

 

objFile.Close

 

{{Out}}

* Reads flag value and checks if it is positive

* Requires 24 value/flag pairs on each line

File_Open("|(PATH_ONLY)\output.txt")

#51 = Buf_Num // Edit buffer for output file

IT("Date format errors: ") Num_Ins(#14)

IT("Invalid data records:") Num_Ins(#15)

Sample output:

1991-03-31: duplicate record at 456

1992-03-29: duplicate record at 820

Date format errors: 0

Invalid data records: 454

 

=={{header|zkl}}==

// I can't use it to select all fields, only verify them

re:=RegExp(0'|^(\d+-\d+-\d+)| + 0'|\s+\d+\.\d+\s+-*\d+| * 24 + ".+$");

reg datep,N, good=0, dd=0;

N=n; // since n is local to this scope

if (not re.search(line)){ println("Line %d: malformed".fmt(n)); continue; }

good+=1;

}

{{out}}

<pre>