User defined pipe and redirection operators
If the language supports operator definition, then:
- create "user defined" the equivalents of the Unix shell "<", "|", ">", "<<", ">>" and $(cmd) operators.
- Provide simple equivalents of: cat, tee, grep, & uniq, but as filters/procedures native to the specific language.
- Replicate the below sample shell script, but in the specific language
- Specifically do not cache the entire stream before the subsequent filter/procedure starts. Pass each record on as soon as available through each of the filters/procedures in the chain.
Alternately: if the language does not support operator definition then replace with:
- define the procedures: input(cmd,stream), pipe(stream,cmd), output(stream, stream), whereis(array), append(stream)
For bonus Kudos: Implement the shell "&" concept as a dyadic operator in the specific language. e.g.:
( head x & tail x & wait ) | grep test
Sample shell script: ¢ draft - pending a better (more interesting) suggestion ¢
aa="$(
(
head -4 < List_of_computer_scientists.lst;
cat List_of_computer_scientists.lst | grep ALGOL | tee ALGOL_pioneers.lst;
tail -4 List_of_computer_scientists.lst
) | sort | uniq | tee the_important_scientists.lst | grep aa
);
echo "Pioneer: $aa"
Input Records:
Name | Areas of interest |
---|---|
Wil van der Aalst | business process management, process mining, Petri nets |
Hal Abelson | intersection of computing and teaching |
Serge Abiteboul | database theory |
Samson Abramsky | game semantics |
Leonard Adleman | RSA, DNA computing |
Manindra Agrawal | polynomial-time primality testing |
Luis von Ahn | human-based computation |
Alfred Aho | compilers book, the 'a' in AWK |
Stephen R. Bourne | Bourne shell, portable ALGOL 68C compiler |
Kees Koster | ALGOL 68 |
Lambert Meertens | ALGOL 68, ABC (programming language) |
Peter Naur | BNF, ALGOL 60 |
Guido van Rossum | Python (programming language) |
Adriaan van Wijngaarden | Dutch pioneer; ARRA, ALGOL |
Dennis E. Wisnosky | Integrated Computer-Aided Manufacturing (ICAM), IDEF |
Stephen Wolfram | Mathematica |
William Wulf | compilers |
Edward Yourdon | Structured Systems Analysis and Design Method |
Lotfi Zadeh | fuzzy logic |
Arif Zaman | Pseudo-random number generator |
Albert Zomaya | Australian pioneer of scheduling in parallel and distributed systems |
Konrad Zuse | German pioneer of hardware and software |
These records can be declared in any format appropriate to the specific language. eg table, array, list, table or text file etc.
Output:
Pioneer: Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
ALGOL 68
See User defined pipe and redirection operators/ALGOL 68
File: Iterator_pipe_operators.a68
MODE
PAGEIN = PAGE,
PAGEAPPEND = REF PAGE,
PAGEOUT = REF PAGE;
MODE
MOID = VOID,
YIELDLINE = PROC(LINE)VOID,
GENLINE = PROC(YIELDLINE)VOID,
FILTER = PROC(GENLINE)GENLINE, # the classic shell filter #
MANYTOONE = PROC([]GENLINE)GENLINE; # eg cat, as in con[cat]enate #
PRIO =: = 5, << = 5, >> = 5;
OP < = (FILTER filter, PAGEIN page)GENLINE: filter(READ page),
< = (MANYTOONE cmd, PAGEIN page)GENLINE: cmd(READ page),
<< = (FILTER filter, PAGEIN page)GENLINE: filter(READ page),
> = (GENLINE gen, PAGEOUT page)VOID: gen(WRITE page),
>> = (GENLINE gen, PAGEAPPEND page)VOID: gen(APPEND page),
=: = (GENLINE gen, FILTER filter)GENLINE: filter(gen),
=: = (GENLINE gen, MANYTOONE cmd)GENLINE: cmd(gen);
File: Iterator_pipe_utilities.a68
# Sample ''utilities'' PROCedure declarations #
PROC cat = ([]GENLINE argv)GENLINE:~;
PROC tee = ([]YIELDLINE args filter)FILTER:~;
PROC grep = (STRING pattern, []GENLINE argv)GENLINE:~;
PROC uniq = (GENLINE arg)GENLINE:~;
PROC sort = (GENLINE arg)GENLINE:~;
PROC head = (INT n, []GENLINE args)GENLINE:~;
PROC tail = (INT n, []GENLINE args)GENLINE:~;
File: Iterator_pipe_page.a68
# Sample ''pipe I/O'' OPerator declarations #
OP READ = (PAGEIN page)GENLINE:~;
OP WRITE = (PAGEOUT page)YIELDLINE: ~;
OP APPEND = (PAGEAPPEND page)YIELDLINE:~;
File: test_Iterator_pipe_page.a68
#!/usr/local/bin/a68g --script #
# First define what kind of record (aka LINE) we are piping and filtering #
FORMAT line fmt = $xg$;
MODE
LINE = STRING,
PAGE = FLEX[0]LINE,
BOOK = FLEX[0]PAGE;
PR READ "Iterator_pipe_page.a68" PR
PR READ "Iterator_pipe_operators.a68" PR
PR READ "Iterator_pipe_utilities.a68" PR
PAGE list of computer scientists = (
"Wil van der Aalst - business process management, process mining, Petri nets",
"Hal Abelson - intersection of computing and teaching",
"Serge Abiteboul - database theory",
"Samson Abramsky - game semantics",
"Leonard Adleman - RSA, DNA computing",
"Manindra Agrawal - polynomial-time primality testing",
"Luis von Ahn - human-based computation",
"Alfred Aho - compilers book, the 'a' in AWK",
"Stephen R. Bourne - Bourne shell, portable ALGOL 68C compiler",
"Kees Koster - ALGOL 68",
"Lambert Meertens - ALGOL 68, ABC (programming language)",
"Peter Naur - BNF, ALGOL 60",
"Guido van Rossum - Python (programming language)",
"Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL",
"Dennis E. Wisnosky - Integrated Computer-Aided Manufacturing (ICAM), IDEF",
"Stephen Wolfram - Mathematica",
"William Wulf - compilers",
"Edward Yourdon - Structured Systems Analysis and Design Method",
"Lotfi Zadeh - fuzzy logic",
"Arif Zaman - Pseudo-random number generator",
"Albert Zomaya - Australian pioneer of scheduling in parallel and distributed systems",
"Konrad Zuse - German pioneer of hardware and software"
);
PAGE algol pioneers list, the scientists list;
PAGE aa;
# Now do a bit of plumbing: #
cat((
head(4, ) < list of computer scientists,
cat(READ list of computer scientists) =: grep("ALGOL", ) =: tee(WRITE algol pioneers list),
tail(4, READ list of computer scientists)
)) =: sort =: uniq =: tee(WRITE the scientists list) =: grep("aa", ) >> aa;
# Finally check the result: #
print((
"Pioneer: ", aa, newline,
"Number of Algol pioneers: ", whole( UPB algol pioneers list, 0 ), newline,
"Number of scientists: ", whole( UPB the scientists list, 0 ), newline
))
Output:
Pioneer: Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL Number of Algol pioneers: 5 Number of scientists: 13
Go
package main
import (
"bufio"
"fmt"
"io"
"io/ioutil"
"regexp"
"sort"
"strings"
)
// fake file system
var fs = make(map[string]string)
type file struct{ name string }
func (f file) Write(p []byte) (int, error) {
fs[f.name] += string(p)
return len(p), nil
}
// role of > operator
func toName(name string, src io.Reader) {
io.Copy(file{name}, src)
}
// role of < operator
func fromName(name string) io.Reader {
return strings.NewReader(fs[name])
}
func tee(in io.Reader, name string) io.Reader {
return io.TeeReader(in, file{name})
}
func grep(in io.Reader, pat string) io.Reader {
pr, pw := io.Pipe()
go func() {
bf := bufio.NewReader(in)
for {
line, readErr := bf.ReadString('\n')
if match, _ := regexp.MatchString(pat, line); match {
_, writeErr := io.WriteString(pw, line)
if writeErr != nil {
return
}
}
if readErr != nil {
pw.CloseWithError(readErr)
return
}
}
pw.Close()
}()
return pr
}
func uniq(in io.Reader) io.Reader {
pr, pw := io.Pipe()
go func() {
bf := bufio.NewReader(in)
var last string
for {
s, readErr := bf.ReadString('\n')
switch readErr {
case nil:
if s != last {
_, writeErr := io.WriteString(pw, s)
if writeErr != nil {
return
}
last = s
}
continue
case io.EOF:
if s > "" && s+"\n" != last {
_, writeErr := io.WriteString(pw, s+"\n")
if writeErr != nil {
return
}
}
pw.Close()
default:
pw.CloseWithError(readErr)
}
return
}
}()
return pr
}
func head(in io.Reader, lines int) io.Reader {
pr, pw := io.Pipe()
go func() {
if lines <= 0 {
lines = 10
}
for bf := bufio.NewReader(in); lines > 0; lines-- {
s, readErr := bf.ReadString('\n')
_, writeErr := io.WriteString(pw, s)
if writeErr != nil {
return
}
if readErr == nil {
continue
}
if readErr == io.EOF {
if s > "" {
io.WriteString(pw, "\n")
}
}
pw.CloseWithError(readErr)
return
}
pw.Close()
}()
return pr
}
func tail(in io.Reader, lines int) io.Reader {
pr, pw := io.Pipe()
go func() {
if lines <= 0 {
lines = 10
}
ring := make([]string, lines)
rn := 0
full := false
bf := bufio.NewReader(in)
var readErr error
var s string
for {
s, readErr = bf.ReadString('\n')
if readErr == nil {
ring[rn] = s
rn++
if rn == lines {
rn = 0
full = true
}
continue
}
if readErr == io.EOF && s > "" {
ring[rn] = s + "\n"
rn++
}
break
}
writeLines := func(start, end int) {
for i := start; i < end; i++ {
if _, err := io.WriteString(pw, ring[i]); err != nil {
return
}
}
}
if full {
writeLines(rn, lines)
}
writeLines(0, rn)
pw.CloseWithError(readErr)
}()
return pr
}
func sortPipe(in io.Reader) io.Reader {
pr, pw := io.Pipe()
go func() {
b, err := ioutil.ReadAll(in)
if len(b) > 0 {
if b[len(b)-1] == '\n' {
b = b[:len(b)-1]
}
list := strings.Split(string(b), "\n")
sort.Strings(list)
for _, s := range list {
if _, err := io.WriteString(pw, s+"\n"); err != nil {
return
}
}
}
pw.CloseWithError(err)
}()
return pr
}
func main() {
fs["List_of_computer_scientists.lst"] =
`Wil van der Aalst business process management, process mining, Petri nets
Hal Abelson intersection of computing and teaching
Serge Abiteboul database theory
Samson Abramsky game semantics
Leonard Adleman RSA, DNA computing
Manindra Agrawal polynomial-time primality testing
Luis von Ahn human-based computation
Alfred Aho compilers book, the 'a' in AWK
Stephen R. Bourne Bourne shell, portable ALGOL 68C compiler
Kees Koster ALGOL 68
Lambert Meertens ALGOL 68, ABC (programming language)
Peter Naur BNF, ALGOL 60
Guido van Rossum Python (programming language)
Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL
Dennis E. Wisnosky Integrated Computer-Aided Manufacturing (ICAM), IDEF
Stephen Wolfram Mathematica
William Wulf compilers
Edward Yourdon Structured Systems Analysis and Design Method
Lotfi Zadeh fuzzy logic
Arif Zaman Pseudo-random number generator
Albert Zomaya Australian pioneer of scheduling in parallel and distributed systems
Konrad Zuse German pioneer of hardware and software
`
toName("aa", grep(tee(uniq(sortPipe(io.MultiReader(
head(fromName("List_of_computer_scientists.lst"), 4),
tee(grep(fromName("List_of_computer_scientists.lst"), "ALGOL"),
"ALGOL_pioneers.lst"),
tail(fromName("List_of_computer_scientists.lst"), 4)))),
"the_important_scientists.lst"), "aa"))
fmt.Print("Pioneer: ", fs["aa"])
showCount("Number of ALGOL pioneers", "ALGOL_pioneers.lst")
showCount("Number of scientists", "the_important_scientists.lst")
}
func showCount(heading, name string) {
if data, ok := fs[name]; ok {
lines := strings.Split(data, "\n")
n := len(lines)
if lines[n-1] == "" {
n--
}
fmt.Printf("%s: %v\n", heading, n)
} else {
fmt.Println(name, "not found")
}
}
Output:
Pioneer: Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL Number of ALGOL pioneers: 5 Number of scientists: 13
J
If we ignore the gratuitous complexity requirements of this task, it boils down to this:
Step 0: get the data. The task does not specify how to get the data, so here I use lynx, which is readily available on most unix-like systems, including cygwin. Note that lynx needs to be in the OS PATH when running j.
require 'task'
data=:<;._2 shell 'lynx -dump -nolist -width=999 http://en.wikipedia.org/wiki/List_of_computer_scientists'
Step 1: define task core algorithms:
grep=: +./@E.S:0 # ]
Step 2: select and display the required data:
;'aa' grep 'ALGOL' grep data
* Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
As for the concept of a pipe that presents data one record at a time to a downstream function, that corresponds to the J operator @
and we could achieve the "left to right" syntax mechanism by explicitly ordering its arguments 2 :'v@u'
but it's not clear how to demonstrate that usefully, in this task. (And, I could write a lot of code, to accomplish what's being accomplished here with the two successive greps, but I find that concept distasteful and tedious.)
However, note also that J's sort (/:~
) and uniq (~.
) operations would work just fine on this kind of data. For example:
;'aa' grep 'ALGOL' grep data,data
* Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
* Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
;'aa' grep ~. 'ALGOL' grep data,data
* Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
That said, this implements most (perhaps all) of the required complexities:
declare=: erase@boxopen
tee=: 4 :0
if._1=nc boxopen x do.(x)=: '' end.
(x)=: (do x),y
y
)
grep=: 4 :'x (+./@E.S:0 # ]) y'
pipe=:2 :'v@(u"0)' NB. small pipe -- spoon feed one record at a time
PIPE=:2 :0 NB. big pipe -- feed everything all together
v u y
:
v (,x)"_ y NB. syntactic sugar, beware of tooth decay
)
head=: {.
tail=: -@[ {. ]
sort=: /:~
uniq=: ~.
cat=: ]
echo=: smoutput@;
declare;:'ALGOL_pioneers the_important_scientists'
aa=: ;do TXT=:0 :0 -.LF
(
(
4 head data
),(
cat pipe
('ALGOL'&grep) pipe
('ALGOL_pioneers'&tee)
data
),(
4 tail data
)) ''PIPE
sort PIPE
uniq PIPE
('the_important_scientists'&tee) PIPE
('aa'&grep)
''
)
echo 'Pioneer:';aa
This produces the result:
Pioneer: * Adriaan van Wijngaarden - Dutch pioneer; ARRA, ALGOL
Julia
Uses Julia's Channels as asynchronous pipes. Note that the $( .. ) Unix command, which gathers stdin output into a single variable, is not used. Instead, a common output channel for various tasks is used to gather output. Sort() rather than sort() is used so as not to overload Base.sort().
const LISTDATA = """
Wil van der Aalst business process management, process mining, Petri nets
Hal Abelson intersection of computing and teaching
Serge Abiteboul database theory
Samson Abramsky game semantics
Leonard Adleman RSA, DNA computing
Manindra Agrawal polynomial-time primality testing
Luis von Ahn human-based computation
Alfred Aho compilers book, the 'a' in AWK
Stephen R. Bourne Bourne shell, portable ALGOL 68C compiler
Kees Koster ALGOL 68
Lambert Meertens ALGOL 68, ABC (programming language)
Peter Naur BNF, ALGOL 60
Guido van Rossum Python (programming language)
Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL
Dennis E. Wisnosky Integrated Computer-Aided Manufacturing (ICAM), IDEF
Stephen Wolfram Mathematica
William Wulf compilers
Edward Yourdon Structured Systems Analysis and Design Method
Lotfi Zadeh fuzzy logic
Arif Zaman Pseudo-random number generator
Albert Zomaya Australian pioneer of scheduling in parallel and distributed systems
Konrad Zuse German pioneer of hardware and software
"""
datafilename = "List_of_computer_scientists.lst"
stat(datafilename).size == 0 && (fd = open(datafilename, "a"); write(fd, LISTDATA); close(fd))
channelstream() = Channel{String}(200)
closewhenempty(c) = @async begin while !isempty(c) sleep(rand()) end; close(c); end
function head(filename, numlines, chan=channelstream())
fd = open(filename)
buffer = String[]
while !eof(fd)
push!(buffer, readline(fd, keep=true))
length(buffer) == numlines && break
end
close(fd)
@async begin
for line in buffer
put!(chan, line)
end
closewhenempty(chan)
end
return chan
end
function cat(filename::AbstractString, chan=channelstream())
@async begin
fd = open(filename)
while !eof(fd)
put!(chan, readline(fd, keep=true))
end
close(fd)
closewhenempty(chan)
end
return chan
end
function grep(inchan::Channel, target, outchan = channelstream())
@async begin
try
while isopen(inchan)
line = take!(inchan)
if occursin(target, line)
put!(outchan, line)
end
end
catch;
end
closewhenempty(outchan)
end
return outchan
end
grep(target) = (chan) -> grep(chan, target)
function tee(inchan::Channel, filename, outchan=channelstream())
fd = open(filename, "w")
@async begin
while isopen(inchan)
try
line = take!(inchan)
write(fd, line)
put!(outchan, line)
catch;
break
end
end
close(fd)
closewhenempty(outchan)
end
return outchan
end
tee(filename, outchan=channelstream()) = (inchan) -> tee(inchan, filename, outchan)
function tail(filename, numlines, chan=channelstream())
fd = open(filename)
buffer = String[]
while !eof(fd)
push!(buffer, readline(fd, keep=true))
length(buffer) > numlines && popfirst!(buffer)
end
@async begin
for line in buffer
put!(chan, line)
end
closewhenempty(chan)
end
return chan
end
function Sort(inchan, outchan = channelstream())
buffer = String[]
try
while isopen(inchan)
push!(buffer, take!(inchan))
end
catch;
end
@async begin
for line in sort!(buffer)
put!(outchan, line)
end
closewhenempty(outchan)
end
return outchan
end
Sort() = (chan) -> Sort(chan)
function uniq(inchan, outchan = channelstream())
alreadyseen = Set{String}()
@async begin
while isopen(inchan)
try
line = take!(inchan)
if !(line in alreadyseen)
push!(alreadyseen, line)
put!(outchan, line)
end
catch;
break
end
end
closewhenempty(outchan)
end
return outchan
end
uniq() = (chan) -> uniq(chan)
function print_lines(chan)
@async begin
while isopen(chan)
line = take!(chan)
print(line)
end
end
end
print_lines() = (chan) -> print_lines(chan)
const commonoutchan = channelstream()
head("List_of_computer_scientists.lst", 4, commonoutchan)
cat("List_of_computer_scientists.lst") |> grep("ALGOL") |> tee("Algol_pioneers.lst", commonoutchan)
tail("List_of_computer_scientists.lst", 4, commonoutchan)
Sort(commonoutchan) |> uniq() |> tee("the_important_scientists.lst") |>
grep("aa") |> print_lines()
- Output:
Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL
Perl
Implementing only stream chaining, cat, grep and tee. Oddly enough, I don't feel the urge to implement all of the more-or-less-the-same features asked for by the task.
use strict;
use 5.10.0;
package IO::File;
sub readline { CORE::readline(shift) } # icing, not essential
package Stream;
use Exporter 'import';
# Only overload one operator. "file | stream" and "stream | stream"
# are not ambiguous like with shell commands.
use overload '|' => \&chain;
sub new {
my $cls = shift;
bless { args => [@_] }, ref $cls || $cls;
}
sub chain {
my ($left, $right, $swap) = @_;
($left, $right) = ($right, $left) if $swap;
if (!ref $left) {
my $h;
open $h, $left and $left = $h or die $left
}
if (!ref $right) {
# output file not implemented: don't know where I'd ever use it
my $h;
open $h, '>', $right and $right = $h or die $right
}
if (ref $left and $left->isa(__PACKAGE__)) {
$left->{output} = $right;
}
if (ref $right and $right->isa(__PACKAGE__)) {
$right->{input} = $left;
}
$right;
}
# Read a line and do something to it. By default it's this dummy
# pass-through function. Overriding it defines a subclass' behavior
sub transform { shift; shift }
sub readline {
my $obj = shift;
my $line;
return $line = <STDIN> unless defined $obj->{input};
while (1) {
$line = $obj->{input}->readline or return;
return $line if $line = $obj->transform($line);
}
}
package Cat;
use parent -norequire, 'Stream';
# Dummy, exactly the same as Stream. Except now we can invoke
# as Cat::ter, instead of Stream::ter, which is not even a word
sub ter { Cat->new(@_) }
package Grep;
use parent -norequire, 'Stream';
sub transform {
my ($obj, $line) = @_;
for (@{$obj->{args}}) {
return $line if ($line =~ $_)
}
return;
}
sub per { Grep->new(@_) }
package Tee;
use parent -norequire, 'Stream';
sub er{
my $obj = Tee->new(@_);
@{$obj->{tees}} =
map { open my $h, '>', $_ or die $_; $h }
@{$obj->{args}};
delete $obj->{args};
$obj
}
sub transform {
my ($obj, $line) = @_;
print $_ $line for @{$obj->{tees}};
$line;
}
package main;
my $chain =
'/etc/services' # head of chain; omit to use STDIN
| Cat::ter # don't really need this line
| Grep::per(qr/tcp/)
| Tee::er('/tmp/t1', '/tmp/t2')
| Grep::per(qr/170/)
| Tee::er('/tmp/t3')
;
print while $_ = $chain->readline;
Phix
You could of course do things more character-by-character or line-by-line, and/or farm things out to separate threads/tasks, but the latter would need some suspend/resume/scheduling, along with explicit eof markers. The distributed version also has a couple of alternatives for pipe_head() and pipe_tail(), along with a class-less version that is compatible with pwa/p2js.
-- demo\rosetta\Fake_Redirection.exw without js -- class (see other version in distro) constant fs = new_dict() -- fake file system class pipe -- fake pipe string data = "" integer idx = 0 function getch() if idx<length(data) then idx += 1 return data[idx] end if return -1 end function function getln() if idx<length(data) then integer start = idx+1 idx = find('\n',data,start) return data[start..idx] end if return -1 end function procedure putch(integer ch) data &= ch end procedure procedure putln(string line) data &= line end procedure function readall() return split(data,'\n',no_empty:=true) end function function rawdata() return data end function end class function joinup(sequence lines) return join(lines,"\n")&"\n" end function procedure toName(string name, pipe src) -- role of > operator setd(name,src.rawdata(),fs) end procedure function fromName(string name) -- role of < operator return new(pipe,{getd(name,fs)}) end function function tee(pipe pin, string name) string data = pin.rawdata() setd(name,data,fs) return new(pipe,{data}) end function function grep(pipe pin, string pat) pipe res = new() while true do object line = pin.getln() if atom(line) then exit end if if match(pat,line) then res.putln(line) end if end while return res end function function multireader(sequence pipes) pipe res = new() for i=1 to length(pipes) do pipe p = pipes[i] res.putln(p.rawdata()) end for return res end function function pipe_head(pipe pin, integer lines) pipe res = new() for i=1 to lines do object line = pin.getln() if atom(line) then exit end if res.putln(line) end for return res -- or, nicer/neater but potentially much wasted effort: -- return new(pipe,{joinup(head(pin.readall(),lines))}) end function function pipe_tail(pipe pin, integer lines) -- sequence ring = repeat("",lines) -- integer rn = 0, full = false -- while true do -- object line = pin.getln() -- if atom(line) then exit end if -- rn += 1 -- if rn>lines then {rn,full} = {1,true} end if -- ring[rn] = line -- end while -- ring = iff(full?ring[rn+1..$]:{}) & ring[1..rn] -- return new(pipe,{joinup(ring)}) return new(pipe,{joinup(tail(pin.readall(),lines))}) end function function sort_unique(pipe pin) return new(pipe,{joinup(unique(pin.readall()))}) end function procedure showCount(string heading, name) if getd_index(name,fs)=NULL then crash("not found") end if integer n = length(split(getd(name,fs), "\n")) printf(1,"%s: %d\n", {heading, n}) end procedure constant lcs_txt = """ Wil van der Aalst business process management, process mining, Petri nets Hal Abelson intersection of computing and teaching Serge Abiteboul database theory Samson Abramsky game semantics Leonard Adleman RSA, DNA computing Manindra Agrawal polynomial-time primality testing Luis von Ahn human-based computation Alfred Aho compilers book, the 'a' in AWK Stephen R. Bourne Bourne shell, portable ALGOL 68C compiler Kees Koster ALGOL 68 Lambert Meertens ALGOL 68, ABC (programming language) Peter Naur BNF, ALGOL 60 Guido van Rossum Python (programming language) Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL Dennis E. Wisnosky Integrated Computer-Aided Manufacturing (ICAM), IDEF Stephen Wolfram Mathematica William Wulf compilers Edward Yourdon Structured Systems Analysis and Design Method Lotfi Zadeh fuzzy logic Arif Zaman Pseudo-random number generator Albert Zomaya Australian pioneer of scheduling in parallel and distributed systems Konrad Zuse German pioneer of hardware and software """, mainlist = "List_of_computer_scientists.lst" setd(mainlist,lcs_txt,fs) toName("aa", grep(tee(sort_unique(multireader({pipe_head(fromName(mainlist), 4), tee(grep(fromName(mainlist), "ALGOL"), "ALGOL_pioneers.lst"), pipe_tail(fromName(mainlist), 4)})), "the_important_scientists.lst"), "aa")) printf(1,"Pioneer: %s", getd("aa",fs)) showCount("Number of ALGOL pioneers", "ALGOL_pioneers.lst") showCount("Number of scientists", "the_important_scientists.lst") ?"done" {} = wait_key() abort(0)
- Output:
Pioneer: Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL Number of ALGOL pioneers: 5 Number of scientists: 13
Racket
#lang racket
(module racksh racket
(require (for-syntax racket/syntax))
(provide (except-out (all-from-out racket) #%app sort)
(rename-out [shell-app #%app]))
(define-syntax (shell-app stx)
(define (=? x y)
(eq? (if (syntax? x) (syntax-e x) x) (if (syntax? y) (syntax-e y) y)))
(define (err msg) (raise-syntax-error 'shell msg stx))
(define (make-call xs)
(cond [(null? xs) (err "empty form")]
[(string? (syntax-e (car xs))) #`(shell #,(car xs) #,@(cdr xs))]
[(eq? #\{ (syntax-property (car xs) 'paren-shape)) (cons #'void xs)]
[else xs]))
(syntax-case stx ()
[(_ x ...) (eq? #\{ (syntax-property stx 'paren-shape))
(let loop ([xs (reverse (syntax->list #'(\; x ...)))]
[form '()] [thunks '()] [I #f] [O #f] [seq '()])
(cond [(null? xs) #`(begin #,@seq)]
[(=? '\; (car xs))
(loop (cdr xs) '() '() #f #f
(if (and (null? form) (null? thunks)) seq
(let* ([form (make-call form)]
[r (if (null? thunks) form
#`(pipe (list (λ() #,form) #,@thunks)))]
[r (if I #`(#,@I (λ() #,r)) r)]
[r (if O #`(#,@O (λ() #,r)) r)])
(cons r seq))))]
[(=? '\| (car xs))
(loop (cdr xs) '() (cons #`(λ() #,(make-call form)) thunks) I O
seq)]
[(or (=? '< (car xs)) (=? '<< (car xs)))
(cond [(null? form) (err "missing expression after < or <<")]
[I (err "duplicate < or << specified")]
[else (loop (cdr xs) (cdr form) thunks
(if (=? '< (car xs))
#`(with-input-from-file #,(car form))
#`(with-input-from-string #,(car form)))
O seq)])]
[(or (=? '> (car xs)) (=? '>> (car xs)))
(cond [(null? form) (err "missing expression after > or >>")]
[O (err "duplicate > or >> specified")]
[else (loop (cdr xs) (cdr form) thunks I
#`(with-output-to-file #,(cadr xs) #:exists
'#,(if (=? '> (car xs)) 'truncate 'append))
seq)])]
[else (loop (cdr xs) (cons (car xs) form) thunks I O seq)]))]
[(_ x ...) #'(x ...)]))
(define (pipe thunks)
(if (null? (cdr thunks)) ((car thunks))
(let-values ([(I O) (make-pipe)])
(parameterize ([current-output-port O])
(thread (λ() (dynamic-wind void (car thunks)
(λ() (close-output-port O))))))
(parameterize ([current-input-port I]) (pipe (cdr thunks))))))
(define (shell cmd . args)
(apply system* (find-executable-path cmd)
(map (λ(x) (if (string? x) x
(with-output-to-string (λ() (display x)))))
args)))
;; implements a common interface of reading a bunch of files; '- means
;; stdin; no files means just stdin
(define (call/files files proc)
(if (null? files) (proc (current-input-port))
(let-values ([(I O) (make-pipe)])
(thread
(λ() (for ([file (in-list files)])
(if (eq? '- file)
(copy-port (current-input-port) O)
(call-with-input-file file (λ(i) (copy-port i O)))))
(close-output-port O)))
(proc I))))
(define-syntax (define-io stx)
(syntax-case stx ()
[(_ (name . xs) E ...)
(with-syntax ([io-name (format-id #'name "io-~a" #'name)])
#'(begin (provide (rename-out [io-name name]))
(define (io-name . xs) E ...)))]))
(define-io (echo . xs)
(for-each display (add-between xs " "))
(newline))
(define-io (cat . files)
(call/files files (λ(I) (copy-port I (current-output-port)))))
(define-io (sort . files)
(display-lines (sort (call/files files port->lines) string<?)))
(define-io (head n . files)
(call/files files
(λ(I) (for ([l (in-lines I)] [i (in-range n)]) (displayln l)))))
(define-io (tail n . files)
(display-lines (take-right (call/files files port->lines) n)))
(define-io (grep rx . files)
(call/files files
(λ(I) (for ([l (in-lines I)] #:when (regexp-match? rx l)) (displayln l)))))
(define-io (uniq . files)
(call/files files
(λ(I) (let loop ([last #f])
(define line (read-line I))
(unless (eof-object? line)
(unless (equal? line last) (displayln line))
(loop line))))))
(define-io (tee file)
(call-with-output-file file #:exists 'truncate
(λ(O) (for ([l (in-lines (current-input-port))])
(displayln l O) (displayln l)))))
(provide $)
(define-syntax-rule ($ E ...)
(with-output-to-string (λ() E ...))))
(module sample (submod ".." racksh)
{\;
define file "List_of_computer_scientists.lst" \;
define aa ($
{{ head 4 < file \;
cat file \| grep "ALGOL" \;
tail 4 < file \;
} \| sort \| uniq \| tee "the_important_scientists.lst" \| grep "aa"
}) \;
echo "Pioneer:" aa}
)
(require 'sample)
Raku
(formerly Perl 6) Implementing cat, redirect(<), head, tail and tee. I share the same general lack of enthusiasm as the Perl example author to implement the other shell ops. Many of the ops already exist in some form in Raku, though they may have slightly different syntax and precedence as the shell ops. I didn't bother implementing pipe (|) as Raku pretty much has chaining semantics built in.
The less than '<' operator is very low level in Raku and would be troublesome to override, so is implemented as 'redirect'. The sort and unique directives don't really have any effect on the final result string, it would be the same without them, but it is following the task example.
This assumes that the data is in a tab separated file "List_of_computer_scientists.lst" in the current directory.
sub cat ($fname) { lazy $fname.IO.lines };
sub head ($count, @positional) { @positional.head($count) }
sub redirect ($fname) { cat($fname) }
sub tail (Int $count, Str $fname) { $fname.IO.lines.tail($count) }
sub infix:<tee> ($stream, $fname) {
# need to reify the lazy list to write to a file
my @values = @$stream[^Inf].grep: *.defined;
# meh. not really going to write the files
# $fname.IO.put @values.join("\n")
# just pass the reified values along
@values.List
}
my $aa = ((
(head 4, redirect 'List_of_computer_scientists.lst'),
cat('List_of_computer_scientists.lst') .grep( /'ALGOL'/ ) tee 'ALGOL_pioneers.lst',
tail 4, 'List_of_computer_scientists.lst'
).flat .sort .unique tee 'the_important_scientists.lst') .grep: /'aa'/;
say "Pioneer: $aa";
- Output:
Pioneer: Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL
Tcl
The syntax of redirections is slightly out, as they're inserted as explicit pipeline elements, and standard Tcl syntax is used to pull in results from sub-pipelines (because it is vastly simpler):
package require Tcl 8.6
# Helpers
proc aspipe {input cmd args} {
tailcall coroutine pipe[incr ::pipes] eval {yield [info coroutine];} \
[list $cmd $input {*}$args] {;break}
}
proc forpipe {input var body} {
upvar 1 $var v
while {[llength [info commands $input]]} {
set v [$input]
uplevel 1 $body
}
}
# Pipeline framework; parses, collects results as newline-separated lines
proc pipeline args {
if {![llength $args]} {error "no pipeline components"}
set p [aspipe {} eval {while {[gets stdin line]>=0} {yield $line}}]
set oi -1
foreach ni [lsearch -all [lappend args "|"] "|"] {
set cmd [lrange $args [expr {$oi+1}] [expr {$ni-1}]]
set p [aspipe $p {*}$cmd]
set oi $ni
}
set accum {}
forpipe $p line {
lappend accum $line
}
return [join $accum \n]
}
# Pipeline implementations - redirections
proc << {in args} {
foreach string $args {
foreach line [split $string "\n"] {
yield $line
}
}
}
proc < {in filename} {
set f [open $filename]
while {[gets $f line] >= 0} {
yield $line
}
close $f
}
proc > {in filename} {
set f [open $filename w]
forpipe $in line {
puts $f $line
}
close $f
}
proc >> {in filename} {
set f [open $filename a]
forpipe $in line {
puts $f $line
}
close $f
}
# Pipeline implementations - "commands"
proc cat {in args} {
foreach filename $args {
if {$filename eq "-"} {
forpipe $in line {
yield $line
}
} else {
set f [open $filename]
while {[gets $f line] >= 0} {
yield $line
}
close $f
}
}
}
proc head {in count} {
forpipe $in line {
if {[incr i] <= $count} {
yield $line
}
}
}
proc tail {in count} {
incr count -1
set accum {}
forpipe $in line {
set accum [lrange [lappend accum $line] end-$count end]
}
foreach item $accum {yield $item}
}
proc grep {in RE} {
forpipe $in line {
if {[regexp $RE $line]} {yield $line}
}
}
proc sort {in} {
set accum {}
forpipe $in line {
lappend accum $line
}
foreach line [lsort $accum] {yield $line}
}
proc uniq {in} {
forpipe $in line {
if {![info exists prev] || $prev ne $line} {
yield $line
}
set prev $line
}
}
proc wc {in {type "words"}} {
set count 0
switch $type {
words { set RE {\S+} }
lines { set RE {.*} }
}
forpipe $in line {
incr count [regexp -all $RE $line]
}
yield $count
}
proc tee {in filename} {
set f [open $filename w]
forpipe $in line {
puts $f $line
yield $line
}
close $f
}
Sample pipeline:
set file "List_of_computer_scientists.lst"
set aa [pipeline \
<< [pipeline < $file | head 4] [pipeline < $file | grep ALGOL | tee "ALGOL_pioneers.txt"] [pipeline < $file | tail 4] \
| sort | uniq | tee "the_important_scientists.lst" | grep aa]
puts "Pioneer: $aa"
Wren
Although Wren supports operator overloading, there are a number of restrictions which would make simulating the Unix shell operators awkward or even imposible. As in the Phix (and Go) examples, I've therefore used named methods instead.
import "./seq" for Lst
var FS = {} // fake file system
var JoinUp = Fn.new { |lines| lines.join("\n") + "\n" }
class Pipe { // fake pipe
static fromName(name) { Pipe.new(FS[name]) } // role of < operator
static multireader(pipes) {
var res = Pipe.new("")
for (i in 0...pipes.count) {
var p = pipes[i]
res.putln(p.rawData)
}
return res
}
construct new(data) {
_data = data
_idx = -1
}
getln() {
if (_idx < _data.count-1) {
var start = _idx + 1
_idx = _data.indexOf("\n", start)
return (_idx >= 0) ? _data[start.._idx] : ""
}
return ""
}
putln(line) { _data = _data + line }
readAll() { _data.split("\n").where { |s| s != "" }.toList }
rawData { _data }
toName(name) { FS[name] = _data } // role of > operator
tee(name) { Pipe.new(FS[name] = _data) }
grep(pat) {
var res = Pipe.new("")
while (true) {
var line = getln()
if (line == "") break
if (line.indexOf(pat) >= 0) res.putln(line)
}
return res
}
head(lines) { Pipe.new(JoinUp.call(readAll().take(lines).toList)) }
tail(lines) {
var t = readAll()
if (t.count >= lines) t = t[-lines..-1]
return Pipe.new(JoinUp.call(t))
}
sortUnique { Pipe.new(JoinUp.call(Lst.distinct(readAll()))) }
}
var showCount = Fn.new { |heading, name|
if (!FS[name]) Fiber.abort("not found")
var n = FS[name].split("\n").count { |s| s != "" }
System.print("%(heading): %(n)")
}
var lcsTxt = """
Wil van der Aalst business process management, process mining, Petri nets
Hal Abelson intersection of computing and teaching
Serge Abiteboul database theory
Samson Abramsky game semantics
Leonard Adleman RSA, DNA computing
Manindra Agrawal polynomial-time primality testing
Luis von Ahn human-based computation
Alfred Aho compilers book, the 'a' in AWK
Stephen R. Bourne Bourne shell, portable ALGOL 68C compiler
Kees Koster ALGOL 68
Lambert Meertens ALGOL 68, ABC (programming language)
Peter Naur BNF, ALGOL 60
Guido van Rossum Python (programming language)
Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL
Dennis E. Wisnosky Integrated Computer-Aided Manufacturing (ICAM), IDEF
Stephen Wolfram Mathematica
William Wulf compilers
Edward Yourdon Structured Systems Analysis and Design Method
Lotfi Zadeh fuzzy logic
Arif Zaman Pseudo-random number generator
Albert Zomaya Australian pioneer of scheduling in parallel and distributed systems
Konrad Zuse German pioneer of hardware and software
"""
var mainList = "List_of_computer_scientists.lst"
FS[mainList] = lcsTxt
var p = Pipe.fromName(mainList)
var pipes = [p.head(4), p.grep("ALGOL").tee("ALGOL_pioneers.lst"), p.tail(4)]
var p2 = Pipe.multireader(pipes).sortUnique.tee("the_important_scientists.lst").grep("aa").toName("aa")
System.write("Pioneer: %(FS["aa"])")
showCount.call("Number of ALGOL pioneers", "ALGOL_pioneers.lst")
showCount.call("Number of scientists", "the_important_scientists.lst")
- Output:
Pioneer: Adriaan van Wijngaarden Dutch pioneer; ARRA, ALGOL Number of ALGOL pioneers: 5 Number of scientists: 13