Word frequency: Difference between revisions

 

=={{header|11l}}==

L(word) re:‘\w+’.find_strings(File(‘135-0.txt’).read().lowercase())

cnt[word]++

 

{{out}}

{{works with|Ada|Ada|2012}}

 

with Ada.Text_IO;

with Ada.Integer_Text_IO;

end loop;

end Word_Frequency;

{{out}}

<pre>

{{works with|ALGOL 68G|Any - tested with release 2.8.3.win32}}

Uses the associative array implementations in [[ALGOL_68/prelude]].

# use the associative array in the Associate array/iteration task #

# but with integer values #

print( ( whole( top counts[ i ], -6 ), ": ", top words[ i ], newline ) )

OD

{{out}}

<pre>

{{works with|GNU APL}}

 

⍝⍝ NOTE: input text is assumed to be encoded in ISO-8859-1

⍝⍝ (The suggested example '135-0.txt' of Les Miserables on

the of and a to

41042 19952 14938 14526 13942

 

=={{header|AppleScript}}==

 

For simplicity here, words are considered to be uninterrupted sequences of letters and/or digits.

The set text is too messy to warrant faffing around with anything more sophisticated.

set filePath to POSIX path of ((path to desktop as text) & "www.rosettacode.org:Word frequency:135-0.txt")

set n to 10

 

{{output}}

The: 41092

Of: 19954

Was: 8622

That: 7924

 

=={{header|Arturo}}==

 

freqs: #[]

r: {/[[:alpha:]]+/}

loop findFrequency "https://www.gutenberg.org/files/135/135-0.txt" 10 'pair [

print pair

 

{{out}}

 

=={{header|AutoHotkey}}==

FileRead, H, % A_temp "\tempfile.txt"

FileDelete, % A_temp "\tempfile.txt"

}

MsgBox % "Freq`tWord`n" result

Outputs:<pre>Freq Word

41036 The

 

=={{header|AWK}}==

# syntax: GAWK -f WORD_FREQUENCY.AWK [-v show=x] LES_MISERABLES.TXT

#

exit(0)

}

{{out}}

<pre>

==={{header|QB64}}===

This is a rather long code. I fulfilled the requirement with QB64. It "cleans" each word so it takes as a word anything that begins and ends with a letter. It works with arrays. Amazing the speed of QB64 to do this job with such a big file as Les Miserables.txt.

OPTION _EXPLICIT

 

 

END SUB

 

{{output}}

==={{header|BaCon}}===

Removing all punctuation, digits, tabs and carriage returns. So "This", "this" and "this." are the same. Full support for UTF8 characters in words. The code itself could be smaller, but for sake of clarity all has been written explicitly.

OPTION COLLAPSE TRUE

 

FOR i = 0 TO 9

PRINT term$[i], " : ", frequency(term$[i])

{{output}}

<pre>

You could cut the length of this down drastically if you didn't need to be able to recall the word at nth position and wished only to display the top 10 words.

 

@echo off

 

goto:eof

 

 

 

 

= MergeSort { Local variable declarations. }

types

& !most-frequent-words { Return the last 10 terms. }

)

'''Output'''

<pre> (6661.it)

{{libheader|GLib}}

Words are defined by the regular expression "\w+".

#include <stdio.h>

#include <glib.h>

return EXIT_FAILURE;

return EXIT_SUCCESS;

 

{{out}}

=={{header|C sharp|C#}}==

{{trans|D}}

using System.Collections.Generic;

using System.IO;

}

}

{{out}}

<pre>Rank Word Frequency

 

=={{header|C++}}==

#include <cstdlib>

#include <fstream>

return 0;

}

 

{{out}}

===Alternative===

{{trans|C#}}

#include <iostream>

#include <fstream>

 

return 0;

{{out}}

<pre>Rank Word Frequency

9 he 6814

10 had 6139</pre>

 

=={{header|Clojure}}==

(->> file

slurp

frequencies

(sort-by val >)

 

{{Out}}

 

=={{header|COBOL}}==

IDENTIFICATION DIVISION.

PROGRAM-ID. WordFrequency.

CLOSE Word-File Output-File.

END-PROGRAM.

 

{{Out}}

 

=={{header|Common Lisp}}==

(defun count-word (n pathname)

(with-open-file (s pathname :direction :input)

(dolist (word words) (incf (gethash word hash 0)))

(maphash #'(lambda (e n) (push `(,e . ,n) ac)) hash) ac)

 

{{Out}}

 

=={{header|Crystal}}==

require "regex"

 

.sort { |a, b| b[1] <=> a[1] }[0..9] # sort and get the first 10 elements

.each_with_index(1) { |(word, n), i| puts "#{i} \t #{word} \t #{n}" } # print the result

 

{{out}}

 

=={{header|D}}==

import std.array : appender, split;

import std.range : take;

writefln("%4s %-10s %9s", rank++, word.k, word.v);

}

 

{{out}}

{{libheader| System.RegularExpressions}}

{{Trans|C#}}

program Word_frequency;

 

readln;

end.

{{out}}

<pre>

</pre>

=={{header|F Sharp}}==

open System.IO

open System.Text.RegularExpressions

let g=Regex("[A-Za-zÀ-ÿ]+").Matches(File.ReadAllText "135-0.txt")

[for n in g do yield n.Value.ToLower()]|>List.countBy(id)|>List.sortBy(fun n->(-(snd n)))|>List.take 10|>List.iter(fun n->printfn "%A" n)

{{out}}

<pre>

=={{header|Factor}}==

This program expects stdin to read from a file via the command line. ( e.g. invoking the program in Windows: <tt>>factor word-count.factor < input.txt</tt> ) The definition of a word here is simply any string surrounded by some combination of spaces, punctuation, or newlines.

USING: ascii io math.statistics prettyprint sequences

splitting ;

lines " " join " .,?!:;()\"-" split harvest [ >lower ] map

sorted-histogram <reversed> 10 head .

{{out}}

<pre>

 

=={{header|FreeBASIC}}==

#Include "file.bi"

type tally

print "time for operation ";timer-tm;" seconds"

sleep

{{out}}

<pre>

There are two sample programs below. First, a simple but powerful method that works in old versions of Frink:

for w = select[wordList[read[normalizeUnicode["https://www.gutenberg.org/files/135/135-0.txt", "UTF-8"]]], %r/[[:alnum:]]/ ]

d.increment[lc[w], 1]

 

 

{{out}}

Next, a "showing off" one-liner that works in recent versions of Frink that uses the <CODE>countToArray</CODE> function which easily creates sorted frequency lists and the <CODE>formatTable</CODE> function that formats into a nice table with columns lined up, and still performs full Unicode-aware normalization, capitalization, and word-breaking:

 

 

{{out}}

=={{header|FutureBasic}}==

Task said: "Feel free to explicitly state the thoughts behind the program decisions." Thus the heavy comments.

include "NSLog.incl"

 

CFDictionaryRef dict

 

if caseSensitive == NO then textStr = fn StringLowercaseString( textStr )

 

CFStringRef tempStr = fn ArrayComponentsJoinedByString( fn StringComponentsSeparatedByCharactersInSet( textStr, fn CharacterSetInvertedSet( fn CharacterSetLetterSet ) ), @" " )

 

CountedSetRef freqencies = fn CountedSetWithArray( tempArr )

 

EnumeratorRef enumRef = fn CountedSetObjectEnumerator( freqencies )

 

CFMutableArrayRef wordArr = fn MutableArrayWithCapacity( 0 )

 

NSInteger totalWords = 0

for wrd in array

totalWords++

next

 

CFStringRef resultStr = fn StringWithFormat( @"%@", mutStr )

end fn = resultStr

 

HandleEvents

<pre>

1 41095 the

22910 1 isabella

 

Elapsed time: 595.407963 milliseconds.

</pre>

=={{header|Go}}==

{{trans|Kotlin}}

 

import (

fmt.Printf("%2d %-4s %5d\n", rank, word, freq)

}

 

{{out}}

=={{header|Groovy}}==

Solution:

def mapCounts = [:]

content.toLowerCase().split(/\W+/).each {

println "Rank Word Frequency\n==== ==== ========="

(0..<n).each { printf ("%4d %-4s %9d\n", it+1, top[it].key, top[it].value) }

 

Test:

 

Output:

===Lazy IO with pure Map, arrows===

{{trans|Clojure}}

 

import Control.Category -- (>>>)

>>> take n

>>> print)

{{Out}}

<pre>

===Lazy IO, map of IORefs===

Using IORefs as values in the map seems to give a ~2x speedup on large files. The below code is based on https://github.com/composewell/streamly-examples/blob/master/examples/WordFrequency.hs , but still using lazy IO to avoid the extra library dependency (in production you should [https://stackoverflow.com/questions/5892653/whats-so-bad-about-lazy-i-o use a streaming library] like streamly/conduit/io-streams):

module Main where

 

in mapM readRef $ M.toList freqtable

print $ take maxw $ sortOn (Down . snd) counts

{{Out}}

<pre>

===Lazy IO, short code, but not streaming===

Or, perhaps a little more simply, though not streaming (will read everything into memory, don't use on big files):

import qualified Data.Text as T

 

 

main :: IO ()

{{Out}}

<pre>(40370,"the")

 

=={{header|Java}}==

{{trans|Kotlin}}

import java.nio.file.Files;

import java.nio.file.Path;

}

}

{{out}}

<pre>Rank Word Frequency

may not begin with hyphen. Thus "the-the" would count as one word, and "-the" would be excluded.

 

< 135-0.txt jq -nR --argjson n 10 '

def bow(stream):

| from_entries

'

====Output====

{

"the": 41087,

"it": 6661

}

 

=={{header|Julia}}==

{{works with|Julia|1.0}}

using FreqTables

 

words = split(replace(txt, r"\P{L}"i => " "))

table = sort(freqtable(words); rev=true)

 

{{out}}

"he" │ 6816

"had" │ 6140</pre>

 

=={{header|KAP}}==

The below program defines the function 'stats' which accepts a filename containing the text.

 

content ← "[\\h,.\"'\n-]+" regex:split unicode:toLower io:readFile file

sorted ← (⍋⊇⊢) content

words ← selection / sorted

{⍵[10↑⍒⍵[;1];]} words ,[0.5] ≢¨ sorted ⊂⍨ +\selection

{{out}}

<pre>┏━━━━━━━━━━━━┓

 

There is no change in the results if the numerals 0-9 are also regarded as letters.

 

import java.io.File

for ((word, freq) in wordGroups)

System.out.printf("%2d %-4s %5d\n", rank++, word, freq)

 

{{out}}

 

=={{header|Liberty BASIC}}==

dim lines$(150000)

open "135-0.txt" for input as #txt

close #txt

end

{{out}}

<pre>Count Word

=={{header|Lua}}==

{{works with|lua|5.3}}

-- This program takes two optional command line arguments. The first (arg[1])

-- specifies the input file, or defaults to standard input. The second

io.write(string.format('%7d %s\n', array[i][1] , array[i][2]))

end

 

{{Out}}

 

=={{header|Mathematica}} / {{header|Wolfram Language}}==

{{out}}

<pre>

 

=={{header|MATLAB}} / {{header|Octave}}==

function [result,count] = word_frequency()

URL='https://www.gutenberg.org/files/135/135-0.txt';

fprintf(1,'%d\t%s\n',count(k),result{k})

end

 

{{out}}

 

=={{header|Nim}}==

 

proc take[T](s: openArray[T], n: int): seq[T] = s[0 ..< min(n, s.len)]

wordFrequencies.sort

for (word, count) in toSeq(wordFrequencies.pairs).take(10):

{{out}}

<pre>40377 the

 

=={{header|Objeck}}==

use Collection;

use RegEx;

};

}

 

Output:

=={{header|OCaml}}==

 

let n =

try int_of_string Sys.argv.(1)

List.iter (fun (word, count) ->

Printf.printf "%d %s\n" count word

 

{{out}}

=={{header|Perl}}==

{{trans|Raku}}

 

 

qr/[a-z]+/, # simple 7-bit ASCII

qr/\w+/, # word characters with underscore

);

 

my %words;

printf "%-7s %6d\n", $w, $words{$w};

last if ++$c >= $top;

}

 

{{out}}

the 41089

of 19949

was 8621

that 7924

 

=={{header|Phix}}==

<span style="color: #008080;">without</span> <span style="color: #008080;">javascript_semantics</span>

<span style="color: #0000FF;">?</span><span style="color: #008000;">"loading..."</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

<span style="color: #7060A8;">traverse_dict</span><span style="color: #0000FF;">(</span><span style="color: #7060A8;">routine_id</span><span style="color: #0000FF;">(</span><span style="color: #008000;">"visitor"</span><span style="color: #0000FF;">),</span><span style="color: #000000;">0</span><span style="color: #0000FF;">,</span><span style="color: #000000;">wf</span><span style="color: #0000FF;">,</span><span style="color: #004600;">true</span><span style="color: #0000FF;">)</span>

{{out}}

<pre>

 

=={{header|Phixmonti}}==

 

"loading..." ?

-1 * get ?

endfor

{{out}}

<pre>loading...

 

=={{header|PHP}}==

<?php

 

}

$i++;

{{out}}

<pre>

=={{header|Picat}}==

To get the book proper, the header and footer are removed. Here are some tests with different sets of characters to split the words (<code>split_char/1</code>).

NTop = 10,

File = "les_miserables.txt",

split_chars(all,"\n\r \t,;!.?()[]”\"-“—-__‘’*").

split_chars(space_punct,"\n\r \t,;!.?").

 

{{out}}

 

=={{header|PicoLisp}}==

(setq *Skip (chop *Delim))

 

(if (idx 'B W T) (inc (car @)) (set W 1)) ) ) )

(for L (head 10 (flip (by val sort (idx 'B))))

{{out}}

<pre>

=={{header|Prolog}}==

{{works with|SWI Prolog}}

read_file_to_string(File, String, [encoding(utf8)]),

re_split("\\w+", String, Words),

 

main:-

 

{{out}}

 

=={{header|PureBasic}}==

 

Structure wordcount

EndIf

 

{{out}}

<pre>

===Collections===

====Python2.7====

import re

import string

 

if __name__ == "__main__":

 

{{Out}}

 

====Python3.6====

from re import findall

 

if __name__ == "__main__":

n = int(input('How many?: '))

 

{{Out}}

===Sorted and groupby===

{{Works with|Python|3.7}}

Word count task from Rosetta Code

http://www.rosettacode.org/wiki/Word_count#Python

if __name__ == '__main__':

main()

{{Out}}

<pre>('the', 40372)

 

===Collections, Sorted and Lambda===

#!/usr/bin/python3

import collections

if i == count - 1:

break

{{Out}}

<pre>[ 1] the : 41039

 

=={{header|R}}==

I chose to remove apostrophes only if they're followed by an s (so "mom" and "mom's" will show up as the same word but "they" and "they're" won't). I also chose not to remove hyphens.

wordcount<-function(file,n){

punctuation=c("`","~","!","@","#","$","%","^","&","*","(",")","_","+","=","{","[","}","]","|","\\",":",";","\"","<",",",">",".","?","/","'s")

return(df[1:n,])

}

{{Out}}

<pre>

9 it 2308

10 i 1845

</pre>

 

=={{header|Racket}}==

 

(define (all-words f (case-fold string-downcase))

 

(module+ main

 

{{out}}

=={{header|Raku}}==

(formerly Perl 6)

Note: much of the following exposition is no longer critical to the task as the requirements have been updated, but is left here for historical and informational reasons.

 

This is slightly trickier than it appears initially. The task specifically states: "A word is a sequence of one or more contiguous letters", so contractions and hyphenated words are broken up. Initially we might reach for a regex matcher like /\w+/ , but \w includes underscore, which is not a letter but a punctuation connector; and this text is '''full''' of underscores since that is how Project Gutenberg texts denote italicized text. The underscores are not actually parts of the words though, they are markup.

 

 

Actually, in this case /A-Za-z/ returns '''very nearly''' the correct answer. Unfortunately, the name "Alèthe" appears once (only once!) in the text, gets incorrectly split into Al & the, and incorrectly reports 41089 occurrences of "the".

 

Here is a sample that shows the result when using various different matchers.

my $file = $filename.IO.slurp.lc.subst(/ (<[\w]-[_]>'-')\n(<[\w]-[_]>) /, {$0 ~ $1}, :g );

rx/ <[a..z]>+ /, # simple 7-bit ASCII

rx/ \w+ /, # word characters with underscore

rx/ <[\w]-[_]>+ /, # word characters without underscore

for @matcher -> $reg {

say "\nTop $top using regex: ", $reg.raku;

}

 

{{out}}

Since REXX doesn't support UTF-8 encodings, code was added to this REXX version to

support the accented letters in the mandated input file.

parse arg fID top . /*obtain optional arguments from the CL*/

if fID=='' | fID=="," then fID= 'les_mes.txt' /*None specified? Then use the default.*/

end /*#*/

say commas(totW) ' words found ('commas(c) "unique) in " commas(#),

{{out|output|text=&nbsp; when using the default inputs:}}

<pre>

Inspired by version 1 and adapted for ooRexx.

It ignores all characters other than a-z and A-Z (which are translated to a-z).

Call time 'R'

abc='abcdefghijklmnopqrstuvwxyz'

tops=tops+words(tl) /*correctly handle the tied rankings. */

end

{{out}}

<pre>We found 22820 different words

 

=={{header|Ring}}==

# project : Word count

 

b = temp

return [a, b]

Output:

<pre>

 

=={{header|Ruby}}==

class String

def wc

 

open('135-0.txt') { |n| n.read.wc[-10,10].each{|n| puts n[0].to_s+"->"+n[1].to_s} }

{{out}}

<pre>

===Tally and max_by===

{{Works with|Ruby|2.7}}

count = open("135-0.txt").read.downcase.scan(RE).tally.max_by(10, &:last)

count.each{|ar| puts ar.join("->") }

{{out}}

<pre>the->41092

</pre>

===Chain of Enumerables===

.downcase

.scan(/\w+/)

w[1]

}

{{out}}

<pre>[ 1] the : 41040

 

=={{header|Rust}}==

use std::collections::HashMap;

use std::fs::File;

fn main() {

word_count(File::open("135-0.txt").expect("File open error"), 10)

 

{{out}}

{{Out}}

Best seen running in your browser [https://scastie.scala-lang.org/EP2Fm6HXQrC1DwtSNvnUzQ Scastie (remote JVM)].

 

object WordCount extends App {

println(s"\nSuccessfully completed without errors. [total ${scala.compat.Platform.currentTime - executionStart} ms]")

 

{{out}}

<pre>Rank Word Frequency

to get words from a fle. The words are [http://seed7.sourceforge.net/libraries/string.htm#lower(in_string) converted to lower case], to assure that "The" and "the" are considered the same.

 

include "gethttp.s7i";

include "strifile.s7i";

end for;

end for;

 

{{out}}

 

=={{header|Sidef}}==

var file = File(ARGV[0] \\ '135-0.txt')

 

top.each { |pair|

say "#{pair.key}\t-> #{pair.value}"

{{out}}

<pre>

 

=={{header|Simula}}==

$ cim -m64 word-count.sim

;

 

END

{{out}}

<pre>

6 garbage collection(s) in 0.2 seconds.

</pre>

 

=={{header|Swift}}==

 

func printTopWords(path: String, count: Int) throws {

} catch {

print(error.localizedDescription)

 

{{out}}

 

=={{header|Tcl}}==

while { [gets stdin line] >= 0 } {

foreach word [regexp -all -inline {[A-Za-z]+} $line] {

foreach {word count} [lrange $sorted 0 [expr {$head * 2 - 1}]] {

puts "$count\t$word"

 

./wordcount-di.tcl 10 < 135-0.txt

=={{header|TMG}}==

McIlroy's Unix TMG:

/* Only lowercase letters can constitute a word in text. */

/* (c) 2020, Andrii Makukha, 2-clause BSD licence. */

/* Character classes */

letter: <<abcdefghijklmnopqrstuvwxyz>>;

 

Unix TMG didn't have <tt>tolower</tt> builtin. Therefore, you would use it together with <tt>tr</tt>:

 

Additionally, because 1972 TMG only understood ASCII characters, you might want to strip down the diacritics (e.g., é → e):

 

=={{header|UNIX Shell}}==

{{works with|zsh}}

This is derived from Doug McIlroy's original 6-line note in the ACM article cited in the task.

 

 

This is Doug McIlroy's original solution but follows other solutions in importing the task's text file from the web and directly specifying the 10 most commonly used words.

 

 

{{Out}}

In order to use it, you have to adapt the PATHFILE Const.

 

Option Explicit

 

If d.Exists(Word) Then _

DisplayFrequencyOf = d(Word)

{{out}}

<pre>Words different in this book : 25884

I've taken the view that 'letter' means either a letter or digit for Unicode codepoints up to 255. I haven't included underscore, hyphen nor apostrophe as these usually separate compound words.

 

 

 

var fileName = "135-0.txt"

var freq = keyVals[rank-1].value

Fmt.print("$2d $-4s $5d", rank, word, freq)

 

{{out}}

=={{header|XQuery}}==

 

$uri := 'https://www.gutenberg.org/files/135/135-0.txt'

return

return <word key="{$key}" count="{$count}"/>

)[position()=(1 to $maxentries)]

{{out}}

<word key="the" count="41092"/>

<word key="of" count="19954"/>

<word key="that" count="7924"/>

<word key="it" count="6661"/>

 

=={{header|zkl}}==

 

// words may have leading or trailing "_", ie "the" and "_the"

RegExp("[a-z]+").pump.fp1(Dictionary().incV)) // line-->(word:count,..)

.toList().copy().sort(fcn(a,b){ b[1]<a[1] })[0,count.toInt()] // hash-->list

{{out}}

<pre>