Tokenize a string: Difference between revisions

 

=={{header|ACL2}}==

(if (or (endp xs) (eql (first xs) delim))

(mv nil (rest xs))

(progn$ (cw (first strs))

(cw (coerce (list delim) 'string))

 

<pre>&gt; (print-with (split-str "Hello,How,Are,You,Today" #\,) #\.)

Hello.How.Are.You.Today.</pre>

 

=={{header|ActionScript}}==

var tokens:Array = hello.split(",");

trace(tokens.join("."));

 

// Or as a one-liner

 

=={{header|Ada}}==

 

begin

end loop;

 

=={{header|ALGOL 68}}==

 

OP +:= = (REF FLEX[]STRING in out, STRING item)VOID:(

printf(($g"."$, string split(beetles, ", "),$l$));

printf(($g"."$, char split(beetles, ", "),$l$))

John Lennon.Paul McCartney.George Harrison.Ringo Starr.

John.Lennon..Paul.McCartney..George.Harrison..Ringo.Starr.

 

=={{header|AutoHotkey}}==

stringsplit, string, string, `,

loop, % string0

{

msgbox % string%A_Index%

 

=={{header|AWK}}==

 

s = "Hello,How,Are,You,Today"

split(s, arr, ",")

}

print

 

A more ''idiomatic'' way for AWK is

 

{

for(i=1; i <= NF; i++) printf $i ".";

print ""

 

which "tokenize" each line of input and this is achieved by using "," as field separator

 

=={{header|BASIC}}==

 

parseMe = "Hello,How,Are,You,Today"

 

PRINT "."; parsed(L0);

NEXT

 

=={{header|Batch File}}==

setlocal enabledelayedexpansion

call :tokenize %1 res

for %%i in (%str%) do set %2=!%2!.%%i

set %2=!%2:~1!

 

''Demo''

Hello.How.Are.You.Today

 

 

=={{header|Bracmat}}==

Solution that employs string pattern matching to spot the commas

& :?ReverseList

& whl

)

& out$!List

Solution that starts by evaluating the input and employs the circumstance that the comma is a list constructing binary operator and that the string does not contain any other characters that are interpreted as operators on evaluation.

& :?ReverseList

& whl

)

& out$!List

 

=={{header|C}}==

This example uses the ''strtok()'' function to separate the tokens. This function is destructive (replacing token separators with '\0'), so we have to make a copy of the string (using ''strdup()'') before tokenizing. ''strdup()'' is not part of [[ANSI C]], but is available on most platforms. It can easily be implemented with a combination of ''strlen()'', ''malloc()'', and ''strcpy()''.

 

#include<stdio.h>

#include<stdlib.h>

 

return 0;

 

Another way to accomplish the task without the built-in string functions is to temporarily modify the separator character. This method does not need any additional memory, but requires the input string to be writeable.

 

 

void doprint(const char *s) {

}

 

char *olds = s;

char olddelim = delim;

tokenize(array, ',', doprint);

return 0;

 

=={{header|C sharp|C#}}==

// or Regex.Split ( "Hello,How,Are,You,Today", "," );

// (Regex is in System.Text.RegularExpressions namespace)

string[] strings = str.Split(',');

 

=={{header|C++}}==

std::getline() is typically used to tokenize strings on a single-character delimiter

 

#include <sstream>

#include <vector>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."));

std::cout << '\n';

 

{{works with|C++98}}

C++ allows the user to redefine what is considered whitespace. If the delimiter is whitespace, tokenization becomes effortless.

 

#include <locale>

#include <sstream>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."));

std::cout << '\n';

 

{{works with|C++98}}

The boost library has multiple options for easy tokenization.

 

#include <vector>

#include <iterator>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."))

std::cout << '\n';

 

=={{header|CoffeeScript}}==

 

arr = "Hello,How,Are,You,Today".split ","

console.log arr.join "."

 

=={{header|Common Lisp}}==

There are libraries out there that handle splitting (e.g., [http://www.cliki.net/SPLIT-SEQUENCE SPLIT-SEQUENCE], and the more-general [http://weitz.de/cl-ppcre/ CL-PPCRE]), but this is a simple one-off, too. When the words are written with write-with-periods, there is no final period after the last word.

 

(loop for start = 0 then (1+ finish)

for finish = (position #\, string :start start)

 

(defun write-with-periods (strings)

 

 

 

 

{{out}}

<pre>Hello.How.Are.You.Today</pre>

 

=={{header|Delphi}}==

 

program TokenizeString;

 

 

end.

 

The result is:

 

Hello

How

You

Today

 

=={{header|E}}==

 

=={{header|Erlang}}==

-export([start/0]).

 

Lst = string:tokens("Hello,How,Are,You,Today",","),

io:fwrite("~s~n", [string:join(Lst,".")]),

 

=={{header|Euphoria}}==

sequence out

integer first, delim

for i = 1 to length(s) do

puts(1, s[i] & ',')

 

=={{header|Factor}}==

 

=={{header|Fantom}}==

A string can be split on a given character, returning a list of the intervening strings.

 

class Main

{

}

}

 

=={{header|Forth}}==

There is no standard string split routine, but it is easily written. The results are saved temporarily to the dictionary.

 

here >r 2swap

begin

1 ?do dup 2@ type ." ." cell+ cell+ loop 2@ type ;

 

 

=={{header|Fortran}}==

{{works with|Fortran|90 and later}}

 

CHARACTER(23) :: str = "Hello,How,Are,You,Today"

END DO

 

=={{header|GAP}}==

# [ "Hello", "How", "Are", "You", "Today" ]

 

JoinStringsWithSeparator(last, ".");

 

=={{header|Go}}==

 

import (

s := "Hello,How,Are,You,Today"

fmt.Println(strings.Join(strings.Split(s, ","), "."))

 

=={{header|Groovy}}==

 

=={{header|Haskell}}==

 

The necessary operations are unfortunately not in the standard library (yet), but simple to write:

 

splitBy _ [] = []

splitBy f list = first : splitBy f (dropWhile f rest) where

-- using regular expression to split:

import Text.Regex

 

Tokenizing can also be realized by using unfoldr and break:

Hello

How

Are

You

* You need to import the modules Data.List and Control.Arrow

 

 

=={{header|HicEst}}==

 

nWords = INDEX(string, ',', 256) + 1

DO i = 1, nWords

WRITE(APPend) TRIM(CHAR(i, maxWordLength, list)), '.'

 

=={{header|Icon}} and {{header|Unicon}}==

A := []

every writes(!A,".")

write()

 

->ss

->

 

=={{header|Io}}==

 

=={{header|J}}==

] t=: <;._1 ',',s

+-----+---+---+---+-----+

 

'.' (I.','=s)}s NB. two steps combined

 

Alternatively using the system library/script <tt>strings</tt>

',' splitstring s

+-----+---+---+---+-----+

 

'.' joinstring ',' splitstring s

 

<tt>splitstring</tt> and <tt>joinstring</tt> also work with longer "delimiters":

 

But, of course, this could be solved with simple string replacement:

 

 

The task asks us to ''Separate the string "Hello,How,Are,You,Today" by commas into an array (or list) so that each element of it stores a different word.'' but for many purposes the original string is an adequate data structure. Note also that given a string, a list of "word start" indices and "word length" integers can be logically equivalent to having an "array of words" -- and, depending on implementation details may be a superior or inferior choice to some other representation. But, in current definition of this task, the concept of "word length" plays no useful role.

Note also that J provides several built-in concepts of parsing: split on leading delimiter, split on trailing delimiter, split J language words. Also, it's sometimes more efficient to append to a string than to prepend to it. So a common practice for parsing on an embedded delimiter is to append a copy of the delimiter to the string and then use the appended result:

 

 

Here '''fn''' is applied to each ',' delimited substring and the results are assembled into an array.

 

Or, factoring out the names:

 

=={{header|Java}}==

 

The first is by splitting the String into an array of Strings. The separator is actually a regular expression so you could do very powerful things with this, but make sure to escape any characters with special meaning in regex.

 

{{works with|Java|1.4+}}

 

String words[] = toTokenize.split(",");//splits on one comma, multiple commas yield multiple splits

for(int i=0; i<words.length; i++) {

System.out.print(words[i] + ".");

 

The other way is to use StringTokenizer. It will skip any empty tokens. So if two commas are given in line, there will be an empty string in the array given by the split function, but no empty string with the StringTokenizer object. This method takes more code to use, but allows you to get tokens incrementally instead of all at once.

 

{{works with|Java|1.0+}}

 

StringTokenizer tokenizer = new StringTokenizer(toTokenize, ",");

while(tokenizer.hasMoreTokens()) {

System.out.print(tokenizer.nextToken() + ".");

 

=={{header|JavaScript}}==

 

 

=={{header|LabVIEW}}==

To tokenize the string, we use the Search/Split String function to split the string by its first comma. Add the beginning (up to, but not including the comma) to the end of the array, remove the first comma from the rest of the string, and pass it back through the shift register to the loop's next iteration. This is repeated until the string is empty. Printing is a simple matter of concatenation.<br/>

{{VI solution|LabVIEW_Tokenize_a_string.png}}

 

 

=={{header|Logo}}==

{{works with|UCB Logo}}

output parse map [ifelse ? = :sep ["| |] [?]] :str

 

This form is more robust, doing the right thing if there are embedded spaces.

if empty? :str [output lput :w :acc]

ifelse equal? first :str :by ~

[output (split butfirst :str :by lput :w :acc)] ~

[output (split butfirst :str :by :acc lput first :str :w)]

 

 

 

 

 

 

 

end

 

 

=={{header|M4}}==

define(`set',`define(`$1[$2]',`$3')')

define(`get',`defn($1[$2])')

define(`show',

`ifelse(eval(j<n),1,`get(a,j).`'define(`j',incr(j))`'show')')

 

Hello.How.Are.You.Today.

 

 

=={{header|MATLAB}} / {{header|Octave}}==

 

 

 

=={{header|MAXScript}}==

for word in (filterString "Hello,How,Are,You,Today" ",") do

(

output += (word + ".")

)

 

=={{header|Mercury}}==

:- module string_tokenize.

:- interface.

Tokens = string.split_at_char((','), "Hello,How,Are,You,Today"),

io.write_list(Tokens, ".", io.write_string, !IO),

 

=={{header|MMIX}}==

EOS IS 0

NL IS 10

LDBU t,tp

PBNZ t,2B % UNTIL EOB(uffer)

~/MIX/MMIX/Progs> mmix tokenizing

Hello

You

Today

 

=={{header|Modula-3}}==

 

IMPORT IO, TextConv;

END;

IO.Put("\n");

=={{header|MUMPS}}==

NEW I,J,INP

SET INP="Hello,how,are,you,today"

NEW J FOR J=1:1:I WRITE INP(J) WRITE:J'=I "."

KILL I,J,INP // Kill is optional. "New" variables automatically are killed on "Quit"

 

In use:

USER>D TOKENS^ROSETTA

Hello.how.are.you.today

 

=={{header|Nemerle}}==

using System.Console;

using Nemerle.Utility.NString;

// a quick in place list comprehension takes care of that

}

 

=={{header|NetRexx}}==

* 20.08.2012 Walter Pachl derived from REXX Version 3

**********************************************************************/

Say ss.word(i)'.'

End

Output as in REXX version

 

=={{header|Objeck}}==

}

 

=={{header|Objective-C}}==

{{works with|Cocoa}}

 

NSArray *tokens = [text componentsSeparatedByString:@","];

NSString *result = [tokens componentsJoinedByString:@"."];

 

=={{header|OCaml}}==

To split on a single-character separator:

 

 

 

 

 

 

 

=={{header|OpenEdge/Progress}}==

i_c AS CHAR

):

MESSAGE

tokenizeString( "Hello,How,Are,You,Today" )

---------------------------

Message

OK

---------------------------

 

=={{header|Oz}}==

{System.printInfo T#"."}

 

=={{header|Pascal}}==

{{works with|Free_Pascal}}

 

{$mode objfpc}{$H+}

 

const

var

I: Integer;

begin

 

The result is:

 

=={{header|Perl}}==

 

 

=={{header|PHP}}==

{{works with|PHP|5.x}}

 

$str = 'Hello,How,Are,You,Today';

echo implode('.', explode(',', $str));

 

=={{header|PicoLisp}}==

 

=={{header|Pike}}==

 

=={{header|PL/I}}==

declare s character (100) initial ('Hello,How,Are,You,Today');

declare n fixed binary (31);

put skip list (string(table));

end;

 

=={{header|Pop11}}==

First show the use of sysparse_string to break up a string and make a list of strings.

 

lvars list;

sysparse_string('the cat sat on the mat') -> list;

;;; print the list of strings

list =>

 

By giving it an extra parameter 'true' we can make it recognize numbers and produce a list of strings and numbers

 

sysparse_string('one 1 two 2 three 3 four 4', true) -> list;

;;; print the list of strings and numbers

** <true>

isinteger(list(2))=>

 

Now show some uses of the built in procedure sys_parse_string, which allows more options:

 

true -> pop_pr_quotes;

;;;

;;; print the list of strings

strings =>

 

If {% ... %} were used instead of [% ... %] the result would be

a vector (i.e. array) of strings rather than a list of strings.

 

;;; print the vector

strings =>

It is also possible to give sys_parse_string a 'conversion' procedure, which is applied to each of the tokens.

E.g. it could be used to produce a vector of numbers, using the conversion procedure 'strnumber', which converts a string to a number:

 

{% sys_parse_string('100 101 102 103 99.9 99.999', strnumber) %} -> numbers;

;;; the result is a vector containing integers and floats,

;;; which can be printed thus:

numbers =>

 

Using lower level pop-11 facilities to tokenise the string:

 

lvars str='Hello,How,Are,You,Today';

;;; Iterate over string

endif;

;;; Reverse the list

 

Since the task requires to use array we convert list to array

 

destlist(ls);

;;; Build a vector from them

printf(ar(i), '%s.');

endfor;

 

We could use list directly for printing:

 

printf(i, '%s.');

 

so the conversion to vector is purely to satisfy task formulation.

 

=={{header|PowerShell}}==

{{works with|PowerShell|1}}

 

{{works with|PowerShell|2}}

 

 

=={{header|Prolog}}==

{{works with|SWI Prolog}}

splitup(Sep,[A|AL],B) --> [A], {\+ [A] = Sep }, splitup(Sep,AL,B).

splitup(Sep,[],[B|BL]) --> Sep, splitup(Sep,B,BL).

phrase(splitup(".",Tokens),Backtogether),

string_to_list(ABack,Backtogether),

?- start.

Hello.How.Are.You.Today

 

 

 

 

=={{header|Python}}==

{{works with|Python|2.5}}{{works with|Python|3.0}}

 

tokens = text.split(',')

 

Or if interpretation of the task description means you don't need to keep an intermediate array:

 

=={{header|R}}==

junk <- strsplit(text, split=",")

 

or the one liner

 

 

=={{header|Raven}}==

 

=={{header|REBOL}}==

tokens: parse original ","

dotted: "" repeat i tokens [append dotted rejoin [i "."]]

 

Original: Hello,How,Are,You,Today

Dotted: Hello.How.Are.You.Today.

 

=={{header|Retro}}==

: char ( -$ ) " " ;

: tokenize ( $-$$ )

[ tokenize action dup 1 <> ] while drop

^buffer'get drop ;

 

This will suffice to split a string into an array of substrings. It is used like this:

 

 

Since the buffer' vocabulary creates a zero-terminated buffer, we can display it using the each@ combinator and a simple quote:

 

 

=={{header|REXX}}==

===version 1===

 

Hello.

How.

Are.

You.

</pre>

 

===version 2===

 

 

 

 

=={{header|Ruby}}==

 

 

=={{header|Scala}}==

 

=={{header|Scheme}}==

{{works with|Guile}}

(define s "Hello,How,Are,You,Today")

(define words (map match:substring (list-matches "[^,]+" s)))

(display (list-ref words n))

(if (< n (- (length words) 1))

 

(with SRFI 13)

(define words (string-tokenize s (char-set-complement (char-set #\,))))

 

=={{header|Seed7}}==

 

 

=={{header|Slate}}==

 

=={{header|Smalltalk}}==

array := 'Hello,How,Are,You,Today' subStrings: $,.

 

Some implementations also have a ''join:'' convenience method that allows the following shorter solution:

 

 

The solution displaying a trailing period would be:

 

array := 'Hello,How,Are,You,Today' subStrings: $,.

 

=={{header|SNOBOL4}}==

For this task, it's convenient to define Perl-style split( ) and join( ) functions.

 

split t = table()

sp1 str pos(0) (break(chs) | rem) $ t<i = i + 1>

* # Test and display

output = join('.',split(',','Hello,How,Are,You,Today'))

 

Hello.How.Are.You.Today

 

=={{header|Standard ML}}==

 

Test:

 

 

=={{header|Tcl}}==

Generating a list form a string by splitting on a comma:

 

Joining the elements of a list by a period:

 

Thus the whole thing would look like this:

 

If you'd like to retain the list in a variable with the name "words", it would only be marginally more complex:

 

(In general, the <tt>regexp</tt> command is also used in Tcl for tokenization of strings, but this example does not need that level of complexity.)

<code>tr</code> knows nothing about arrays, so this solution only changes each comma to a period.

 

 

=={{header|TUSCRIPT}}==

$$ MODE TUSCRIPT

SET string="Hello,How,Are,You,Today"

SET string=SPLIT (string,":,:")

SET string=JOIN (string,".")

 

=={{header|TXR}}==

sequences of non-commas.

 

@(coll)@{token /[^,]+/}@(end)

@(output)

@(rep)@token.@(last)@token@(end)

 

Different approach. Collect tokens, each of

before a comma, or else extends to the end of the line.

 

@(coll)@(maybe)@token,@(or)@token@(end)@(end)

@(output)

@(rep)@token.@(last)@token@(end)

 

Using TXR Lisp:

 

 

=={{header|UNIX Shell}}==

{{works with|Bourne Shell}}

 

(IFS=,

printf '%s.' $string

 

----

{{works with|Bourne Again SHell}}

{{works with|Public Domain Korn SHell|5.2.14}}

stripchar-l ()

#removes the specified character from the left side of the string

join "$( split "$list" "$input_delimiter" )" \

"$contains_a_space" "$output_delimiter";

 

''Example''

 

 

=={{header|UnixPipes}}==

{{works with|Bourne Shell}}

(IFS=, read -r A B; echo "$A".; test -n "$B" && (echo "$B" | token))

}

 

 

=={{header|Ursala}}==

second order function parameterized by the delimiter. Character

literals are preceded by a backquote.

 

token_list = sep`, 'Hello,How,Are,You,Today'

#cast %s

 

'Hello.How.Are.You.Today'

 

=={{header|Vala}}==

 

 

 

=={{header|Vedit macro language}}==

The contents of each text register is then displayed to user, separated by a period.

 

Ins_Text("Hello,How,Are,You,Today")

 

}

 

 

 

 

 

 

 

 

=={{header|XPL0}}==

include c:\cxpl\codes;

int I, J, K, Char;

for K:= 4 downto 0 do [Text(0, addr Array(K,0)); ChOut(0, ^.)];

CrLf(0);

 

The 'addr' operator is used to fetch the 32-bit address of Array rather

Today.You.Are.How.Hello.

</pre>

 

=={{header|Zsh}}==

tokens=(${(s:,:)str})

 

Or, using SH_SPLIT_WORD:

 

 

{{omit from|PARI/GP|No real capacity for string manipulation}}