Tokenize a string: Difference between revisions

{{trans|Python}}

V tokens = text.split(‘,’)

{{out}}

=={{header|360 Assembly}}==

TOKSTR CSECT

USING TOKSTR,R13 base register

PG DC CL80' ' buffer

YREGS

{{out}}

<pre>

=={{header|8080 Assembly}}==

org 100h

jmp demo

period: db '. $'

hello: db 'Hello,How,Are,You,Today$'

{{out}}

<pre>Hello. How. Are. You. Today.</pre>

=={{header|8086 Assembly}}==

org 100h

section .text

hello: db 'Hello,How,Are,You,Today$'

section .bss

{{out}}

<pre>Hello. How. Are. You. Today. </pre>

=={{header|AArch64 Assembly}}==

{{works with|as|Raspberry Pi 3B version Buster 64 bits}}

/* ARM assembly AARCH64 Raspberry PI 3B */

/* program strTokenize64.s */

/* for this file see task include a file in language AArch64 assembly */

.include "../includeARM64.inc"

{{Output}}

<pre>

=={{header|ACL2}}==

(if (or (endp xs) (eql (first xs) delim))

(mv nil (rest xs))

(progn$ (cw (first strs))

(cw (coerce (list delim) 'string))

{{out}}

The user must type in the monitor the following command after compilation and before running the program!<pre>SET EndProg=*</pre>

{{libheader|Action! Tool Kit}}

INCLUDE "D2:ALLOCATE.ACT" ;from the Action! Tool Kit. You must type 'SET EndProg=*' from the monitor after compiling, but before running this program!

Clear(items,@count)

{{out}}

[https://gitlab.com/amarok8bit/action-rosetta-code/-/raw/master/images/Tokenize_a_string.png Screenshot from Atari 8-bit computer]

=={{header|ActionScript}}==

var tokens:Array = hello.split(",");

trace(tokens.join("."));

// Or as a one-liner

=={{header|Ada}}==

use Ada.Text_IO, Ada.Containers, Ada.Strings, Ada.Strings.Fixed, Ada.Strings.Maps;

Put (S & ".");

end loop;

=={{header|ALGOL 68}}==

OP +:= = (REF FLEX[]STRING in out, STRING item)VOID:(

printf(($g"."$, string split(beetles, ", "),$l$));

printf(($g"."$, char split(beetles, ", "),$l$))

{{out}}

<pre>

Note: the "splitnumber" macro cannot separate a number converted to a string by the "XTOSTR" function, because this function "rounds" the number to the decimal position by default.

#include <hopper.h>

back

{{Out}}

<pre>Output:

=={{header|APL}}==

abc. 123. X. ⍝ 3 strings (char vectors), each with a period at the end.

=={{header|AppleScript}}==

intercalate(".", splitOn(",", "Hello,How,Are,You,Today"))

end run

set my text item delimiters to dlm

return strJoined

{{Out}}

<pre>Hello.How.Are.You.Today</pre>

Or,

set tokens to the text items of "Hello,How,Are,You,Today"

set my text item delimiters to "."

{{Out}}

=={{header|ARM Assembly}}==

{{works with|as|Raspberry Pi}}

/* ARM assembly Raspberry PI */

bx lr

=={{header|Arturo}}==

{{out}}

=={{header|Astro}}==

let tokens = text.split(||,||)

=={{header|AutoHotkey}}==

stringsplit, string, string, `,

loop, % string0

{

msgbox % string%A_Index%

=={{header|AWK}}==

s = "Hello,How,Are,You,Today"

split(s, arr, ",")

}

print

A more ''idiomatic'' way for AWK is

{

for(i=1; i <= NF; i++) printf $i ".";

print ""

which "tokenize" each line of input and this is achieved by using "," as field separator

=={{header|BASIC}}==

==={{header|Applesoft BASIC}}===

110 GOSUB 200"TOKENIZE

120 FOR I = 1 TO N

290 A$(N) = A$(N) + C$

300 NEXT TI

==={{header|BaCon}}===

BaCon includes extensive support for ''delimited strings''.

string$ = "Hello,How,Are,You,Today"

' Or simply replace the delimiter

{{out}}

==={{header|BASIC256}}===

tokens$ = explode(instring$,",")

print tokens$[i]; ".";

next i

==={{header|BBC BASIC}}===

{{works with|BBC BASIC for Windows}}

text$ = "Hello,How,Are,You,Today"

PRINT array$(i%) "." ;

NEXT

==={{header|Commodore BASIC}}===

Based on the AppleSoft BASIC version.

10 REM TOKENIZE A STRING ... ROSETTACODE.ORG

20 T$ = "HELLO,HOW,ARE,YOU,TODAY"

270 NEXT L

280 RETURN

==={{header|FreeBASIC}}===

redim tokens(0 to 0) as string

dim as string*1 ch

for i as uinteger = 0 to ubound(tokens)

print tokens(i);".";

==={{header|Liberty BASIC}}===

For i = 0 To 4

array$(i) = Word$("Hello,How,Are,You,Today", (i + 1), ",")

Next i

==={{header|PowerBASIC}}===

PowerBASIC has a few keywords that make parsing strings trivial: <code>PARSE</code>, <code>PARSE$</code>, and <code>PARSECOUNT</code>. (<code>PARSE$</code>, not shown here, is for extracting tokens one at a time, while <code>PARSE</code> extracts all tokens at once into an array. <code>PARSECOUNT</code> returns the number of tokens found.)

DIM parseMe AS STRING

parseMe = "Hello,How,Are,You,Today"

MSGBOX outP

==={{header|PureBasic}}===

'''As described

For i=1 To 5

ForEach MyStrings()

Print(MyStrings()+".")

'''Still, easier would be

==={{header|QBasic}}===

parseMe = "Hello,How,Are,You,Today"

PRINT "."; parsed(L0);

NEXT

==={{header|Run BASIC}}===

FOR i = 1 to 5

textArray$(i) = word$(text$,i,",")

print textArray$(i);" ";

==={{header|VBScript}}===

====One liner====

In fact, the Visual Basic solution (below) could have done the same, as Join() is available.

Unlike PowerBASIC, there is no need to know beforehand how many tokens are in the string -- <code>Split</code> automagically builds the array for you.

Dim parseMe As String, parsed As Variant

parseMe = "Hello,How,Are,You,Today"

MsgBox outP

=={{header|Batch File}}==

setlocal enabledelayedexpansion

call :tokenize %1 res

for %%i in (%str%) do set %2=!%2!.%%i

set %2=!%2:~1!

''Demo''

=={{header|Bracmat}}==

Solution that employs string pattern matching to spot the commas

& :?ReverseList

& whl

)

& out$!List

Solution that starts by evaluating the input and employs the circumstance that the comma is a list constructing binary operator and that the string does not contain any other characters that are interpreted as operators on evaluation.

& :?ReverseList

& whl

)

& out$!List

=={{header|C}}==

This example uses the ''strtok()'' function to separate the tokens. This function is destructive (replacing token separators with '\0'), so we have to make a copy of the string (using ''strdup()'') before tokenizing. ''strdup()'' is not part of [[ANSI C]], but is available on most platforms. It can easily be implemented with a combination of ''strlen()'', ''malloc()'', and ''strcpy()''.

#include<stdio.h>

#include<stdlib.h>

return 0;

Another way to accomplish the task without the built-in string functions is to temporarily modify the separator character. This method does not need any additional memory, but requires the input string to be writeable.

typedef void (*callbackfunc)(const char *);

tokenize(array, ',', doprint);

return 0;

=={{header|C sharp|C#}}==

// or Regex.Split ( "Hello,How,Are,You,Today", "," );

// (Regex is in System.Text.RegularExpressions namespace)

string[] strings = str.Split(',');

Console.WriteLine(String.Join(".", strings));

=={{header|C++}}==

std::getline() is typically used to tokenize strings on a single-character delimiter

#include <sstream>

#include <vector>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."));

std::cout << '\n';

{{works with|C++98}}

C++ allows the user to redefine what is considered whitespace. If the delimiter is whitespace, tokenization becomes effortless.

#include <locale>

#include <sstream>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."));

std::cout << '\n';

{{works with|C++98}}

The boost library has multiple options for easy tokenization.

#include <vector>

#include <iterator>

copy(v.begin(), v.end(), std::ostream_iterator<std::string>(std::cout, "."))

std::cout << '\n';

=={{header|Ceylon}}==

{{works with|Ceylon 1.2}}

value input = "Hello,How,Are,You,Today";

value tokens = input.split(','.equals);

print(".".join(tokens));

=={{header|CFEngine}}==

{

reports:

"${with}" with => join(".", splitstring("Hello,How,Are,You,Today", ",", 99));

}

{{out}}

<pre>cf-agent -KIf ./tokenize-a-string.cf

=={{header|Clojure}}==

Using native Clojure functions and Java Interop:

Using the clojure.string library:

=={{header|CLU}}==

% and returns each substring in order.

tokenize = iter (s: string, c: char) yields (string)

stream$putl(po, part || ".")

end

{{out}}

<pre>Hello.

This can be made to handle more complex cases; UNSTRING allows multiple delimiters, capture of which delimiter was used for each field, a POINTER for starting position (set on ending), along with match TALLYING.

identification division.

program-id. tokenize.

goback.

end program tokenize.

{{out}}

=={{header|CoffeeScript}}==

arr = "Hello,How,Are,You,Today".split ","

console.log arr.join "."

=={{header|ColdFusion}}==

=== Classic tag based CFML ===

<cfoutput>

<cfset wordListTag = "Hello,How,Are,You,Today">

#Replace( wordListTag, ",", ".", "all" )#

</cfoutput>

{{Output}}

<pre>

=== Script Based CFML ===

wordList = "Hello,How,Are,You,Today";

splitList = replace( wordList, ",", ".", "all" );

writeOutput( splitList );

{{Output}}

<pre>

There are libraries out there that handle splitting (e.g., [http://www.cliki.net/SPLIT-SEQUENCE SPLIT-SEQUENCE], and the more-general [http://weitz.de/cl-ppcre/ CL-PPCRE]), but this is a simple one-off, too. When the words are written with write-with-periods, there is no final period after the last word.

(loop for start = 0 then (1+ finish)

for finish = (position #\, string :start start)

(defun write-with-periods (strings)

=={{header|Cowgol}}==

include "strings.coh";

print(".\n");

i := i + 1;

{{out}}

<pre>Hello.

=={{header|Crystal}}==

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=={{header|D}}==

import std.stdio, std.string;

"Hello,How,Are,You,Today".split(',').join('.').writeln;

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=== Using String.split ===

{{libheader| System.SysUtils}}

program Tokenize_a_string;

end.

=== Using TStringList ===

program TokenizeString;

end.

The result is:

Hello

How

You

Today

=={{header|Dyalect}}==

var strings = str.Split(',')

{{out}}

=={{header|Déjà Vu}}==

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=={{header|E}}==

=={{header|Elena}}==

ELENA 4.x:

import extensions;

console.print(s,".")

}

=={{header|Elixir}}==

tokens = String.split("Hello,How,Are,You,Today", ",")

IO.puts Enum.join(tokens, ".")

=={{header|Erlang}}==

-export([start/0]).

Lst = string:tokens("Hello,How,Are,You,Today",","),

io:fwrite("~s~n", [string:join(Lst,".")]),

=={{header|Euphoria}}==

sequence out

integer first, delim

for i = 1 to length(s) do

puts(1, s[i] & ',')

=={{header|F_Sharp|F#}}==

=={{header|Factor}}==

=={{header|Falcon}}==

'''VBA/Python programmer's approach to this solution, not sure if it's the most falconic way'''

/* created by Aykayayciti Earl Lamont Montgomery

April 9th, 2018 */

> b

{{out}}

<pre>

A string can be split on a given character, returning a list of the intervening strings.

class Main

{

}

}

=={{header|Fennel}}==

{{trans|Lua}}

(let [pattern (string.format "([^%s]+)" sep)

fields {}]

(let [str "Hello,How,Are,You,Today"]

=={{header|Forth}}==

There is no standard string split routine, but it is easily written. The results are saved temporarily to the dictionary.

here >r 2swap

begin

1 ?do dup 2@ type ." ." cell+ cell+ loop 2@ type ;

=={{header|Fortran}}==

{{works with|Fortran|90 and later}}

CHARACTER(23) :: str = "Hello,How,Are,You,Today"

END DO

=={{header|Frink}}==

println[join[".", split[",", "Hello,How,Are,You,Today"]]]

=={{header|Gambas}}==

'''[https://gambas-playground.proko.eu/?gist=218e240236cdf1419a405abfed906ed3 Click this link to run this code]'''

Dim sString As String[] = Split("Hello,How,Are,You,Today")

Print sString.Join(".")

Output:

<pre>

=={{header|GAP}}==

# [ "Hello", "How", "Are", "You", "Today" ]

JoinStringsWithSeparator(last, ".");

=={{header|Genie}}==

init

words:array of string[] = str.split(",")

joined:string = string.joinv(".", words)

{{out}}

=={{header|Go}}==

import (

s := "Hello,How,Are,You,Today"

fmt.Println(strings.Join(strings.Split(s, ","), "."))

=={{header|Groovy}}==

=={{header|Haskell}}==

'''Using Data.Text'''

import Data.Text (splitOn,intercalate)

import qualified Data.Text.IO as T (putStrLn)

Output: Hello.How.Are.You.Today

The necessary operations are unfortunately not in the standard library (yet), but simple to write:

splitBy _ [] = []

splitBy f list = first : splitBy f (dropWhile f rest) where

-- using regular expression to split:

import Text.Regex

Tokenizing can also be realized by using unfoldr and break:

Hello

How

Are

You

* You need to import the modules Data.List and Control.Arrow

=={{header|HicEst}}==

nWords = INDEX(string, ',', 256) + 1

DO i = 1, nWords

WRITE(APPend) TRIM(CHAR(i, maxWordLength, list)), '.'

=={{header|Icon}} and {{header|Unicon}}==

A := []

"Hello,How,Are,You,Today" ? {

every writes(!A,".")

write()

{{out}}

A Unicon-specific solution is:

procedure main()

every writes(!A,".")

write()

One wonders what the expected output should be with the input string ",,,,".

=={{header|Io}}==

=={{header|J}}==

] t=: <;._1 ',',s

+-----+---+---+---+-----+

'.' (I.','=s)}s NB. two steps combined

Alternatively using the system library/script <tt>strings</tt>

',' splitstring s

+-----+---+---+---+-----+

'.' joinstring ',' splitstring s

<tt>splitstring</tt> and <tt>joinstring</tt> also work with longer "delimiters":

But, of course, this could be solved with simple string replacement:

The task asks us to ''Separate the string "Hello,How,Are,You,Today" by commas into an array (or list) so that each element of it stores a different word.'' but for many purposes the original string is an adequate data structure. Note also that given a string, a list of "word start" indices and "word length" integers can be logically equivalent to having an "array of words" -- and, depending on implementation details may be a superior or inferior choice to some other representation. But, in current definition of this task, the concept of "word length" plays no useful role.

Note also that J provides several built-in concepts of parsing: split on leading delimiter, split on trailing delimiter, split J language words. Also, it's sometimes more efficient to append to a string than to prepend to it. So a common practice for parsing on an embedded delimiter is to append a copy of the delimiter to the string and then use the appended result:

Here '''fn''' is applied to each ',' delimited substring and the results are assembled into an array.

Or, factoring out the names:

=={{header|Java}}==

{{works with|Java|1.8+}}

{{works with|Java|1.4+}}

String words[] = toTokenize.split(",");//splits on one comma, multiple commas yield multiple splits

for(int i=0; i<words.length; i++) {

System.out.print(words[i] + ".");

The other way is to use StringTokenizer. It will skip any empty tokens. So if two commas are given in line, there will be an empty string in the array given by the split function, but no empty string with the StringTokenizer object. This method takes more code to use, but allows you to get tokens incrementally instead of all at once.

{{works with|Java|1.0+}}

StringTokenizer tokenizer = new StringTokenizer(toTokenize, ",");

while(tokenizer.hasMoreTokens()) {

System.out.print(tokenizer.nextToken() + ".");

=={{header|JavaScript}}==

{{works with|Firefox|2.0}}

=={{header|jq}}==

"Hello,How,Are,You,Today"

=={{header|Jsish}}==

Being in the ECMAScript family, Jsi is blessed with many easy to use character, string and array manipulation routines.

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=={{header|Julia}}==

s = "Hello,How,Are,You,Today"

a = split(s, ",")

println("Splits into ", a)

println("Reconstitutes to \"", t, "\"")

{{out}}

=={{header|K}}==

{{out}}

=={{header|Klingphix}}==

{{out}}

<pre>Hello.How.Are.You.Today.

=={{header|Kotlin}}==

{{works with|Kotlin|1.0b4}}

val input = "Hello,How,Are,You,Today"

println(input.split(',').joinToString("."))

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=={{header|Ksh}}==

#!/bin/ksh

######

{{out}}<pre>Hello.How.Are.You.Today</pre>

=={{header|Lambdatalk}}==

{S.replace , by . in Hello,How,Are,You,Today}.

-> Hello.How.Are.You.Today.

=={{header|Lang5}}==

=={{header|LDPL}}==

DATA:

explode/words is text vector

add 1 and i in i

repeat

=={{header|LFE}}==

> (set split (string:tokens "Hello,How,Are,You,Today" ","))

("Hello" "How" "Are" "You" "Today")

> (string:join split ".")

"Hello.How.Are.You.Today"

=={{header|Lingo}}==

_player.itemDelimiter = ","

output = ""

delete the last char of output

put output

=={{header|Logo}}==

{{works with|UCB Logo}}

output parse map [ifelse ? = :sep ["| |] [?]] :str

This form is more robust, doing the right thing if there are embedded spaces.

if empty? :str [output lput :w :acc]

ifelse equal? first :str :by ~

[output (split butfirst :str :by lput :w :acc)] ~

[output (split butfirst :str :by :acc lput first :str :w)]

=={{header|Logtalk}}==

Using Logtalk built-in support for Definite Clause Grammars (DCGs) and representing the strings as atoms for readbility:

:- object(spliting).

:- end_object.

{{out}}

<pre>

=={{header|Lua}}==

Split function callously stolen from the lua-users wiki

local sep, fields = sep or ":", {}

local pattern = string.format("([^%s]+)", sep)

local str = "Hello,How,Are,You,Today"

{{out}}

<pre>Hello.How.Are.You.Today</pre>

=={{header|M2000 Interpreter}}==

Module CheckIt {

Function Tokenize$(s){

}

Checkit

=={{header|M4}}==

define(`set',`define(`$1[$2]',`$3')')

define(`get',`defn($1[$2])')

define(`show',

`ifelse(eval(j<n),1,`get(a,j).`'define(`j',incr(j))`'show')')

{{out}}

=={{header|Maple}}==

{{Out|Output}}

<pre>"Hello.How.Are.You.Today"</pre>

=={{header|Mathematica}}/{{header|Wolfram Language}}==

=={{header|MATLAB}} / {{header|Octave}}==

s=strsplit('Hello,How,Are,You,Today',',')

fprintf(1,'%s.',s{:})

{{out}}

=={{header|Maxima}}==

=={{header|MAXScript}}==

for word in (filterString "Hello,How,Are,You,Today" ",") do

(

output += (word + ".")

)

=={{header|Mercury}}==

:- module string_tokenize.

:- interface.

Tokens = string.split_at_char((','), "Hello,How,Are,You,Today"),

io.write_list(Tokens, ".", io.write_string, !IO),

=={{header|min}}==

{{works with|min|0.19.3}}

=={{header|MiniScript}}==

=={{header|MMIX}}==

EOS IS 0

NL IS 10

LDBU t,tp

PBNZ t,2B % UNTIL EOB(uffer)

{{out}}

<pre>

=={{header|Modula-3}}==

IMPORT IO, TextConv;

END;

IO.Put("\n");

=={{header|MUMPS}}==

NEW I,J,INP

SET INP="Hello,how,are,you,today"

NEW J FOR J=1:1:I WRITE INP(J) WRITE:J'=I "."