Multisplit

Multisplit
You are encouraged to solve this task according to the task description, using any language you may know.

It is often necessary to split a string into pieces based on several different (potentially multi-character) separator strings, while still retaining the information about which separators were present in the input. This is particularly useful when doing small parsing tasks. The task is to write code to demonstrate this.

The function (or procedure or method, as appropriate) should take an input string and an ordered collection of separators. The order of the separators is significant: The delimiter order represents priority in matching, with the first defined delimiter having the highest priority. In cases where there would be an ambiguity as to which separator to use at a particular point (e.g., because one separator is a prefix of another) the separator with the highest priority should be used. Delimiters can be reused and the output from the function should be an ordered sequence of substrings.

Test your code using the input string “a!===b=!=c” and the separators “==”, “!=” and “=”.

For these inputs the string should be parsed as "a" (!=) "" (==) "b" (=) "" (!=) "c", where matched delimiters are shown in parentheses, and separated strings are quoted, so our resulting output is "a", empty string, "b", empty string, "c". Note that the quotation marks are shown for clarity and do not form part of the output.

Extra Credit: provide information that indicates which separator was matched at each separation point and where in the input string that separator was matched.

Ada

multisplit.adb: <lang Ada>with Ada.Containers.Indefinite_Doubly_Linked_Lists; with Ada.Text_IO;

procedure Multisplit is

  package String_Lists is new Ada.Containers.Indefinite_Doubly_Linked_Lists
    (Element_Type => String);
  use type String_Lists.Cursor;

  function Split
    (Source     : String;
     Separators : String_Lists.List)
     return       String_Lists.List
  is
     Result             : String_Lists.List;
     Next_Position      : Natural := Source'First;
     Prev_Position      : Natural := Source'First;
     Separator_Position : String_Lists.Cursor;
     Separator_Length   : Natural;
     Changed            : Boolean;
  begin
     loop
        Changed            := False;
        Separator_Position := Separators.First;
        while Separator_Position /= String_Lists.No_Element loop
           Separator_Length :=
             String_Lists.Element (Separator_Position)'Length;
           if Next_Position + Separator_Length - 1 <= Source'Last
             and then Source
               (Next_Position .. Next_Position + Separator_Length - 1) =
               String_Lists.Element (Separator_Position)
           then
              if Next_Position > Prev_Position then
                 Result.Append
                   (Source (Prev_Position .. Next_Position - 1));
              end if;
              Result.Append (String_Lists.Element (Separator_Position));
              Next_Position := Next_Position + Separator_Length;
              Prev_Position := Next_Position;
              Changed       := True;
              exit;
           end if;
           Separator_Position := String_Lists.Next (Separator_Position);
        end loop;
        if not Changed then
           Next_Position := Next_Position + 1;
        end if;
        if Next_Position > Source'Last then
           Result.Append (Source (Prev_Position .. Source'Last));
           exit;
        end if;
     end loop;
     return Result;
  end Split;

  Test_Input      : constant String := "a!===b=!=c";
  Test_Separators : String_Lists.List;
  Test_Result     : String_Lists.List;
  Pos             : String_Lists.Cursor;

begin

  Test_Separators.Append ("==");
  Test_Separators.Append ("!=");
  Test_Separators.Append ("=");
  Test_Result := Split (Test_Input, Test_Separators);
  Pos         := Test_Result.First;
  while Pos /= String_Lists.No_Element loop
     Ada.Text_IO.Put (" " & String_Lists.Element (Pos));
     Pos := String_Lists.Next (Pos);
  end loop;
  Ada.Text_IO.New_Line;
  -- other order of separators
  Test_Separators.Clear;
  Test_Separators.Append ("=");
  Test_Separators.Append ("!=");
  Test_Separators.Append ("==");
  Test_Result := Split (Test_Input, Test_Separators);
  Pos         := Test_Result.First;
  while Pos /= String_Lists.No_Element loop
     Ada.Text_IO.Put (" " & String_Lists.Element (Pos));
     Pos := String_Lists.Next (Pos);
  end loop;

end Multisplit;</lang>

output:

 a != == b = != c
 a != = = b = != c

C

What kind of silly parsing is this? <lang C>#include <stdio.h>

1. include <string.h>

void parse_sep(char *str, char **pat, int len) { int i, slen; while (*str != '\0') { for (i = 0; i < len || !putchar(*(str++)); i++) { slen = strlen(pat[i]); if (strncmp(str, pat[i], slen)) continue; printf("{%.*s}", slen, str); str += slen; break; } } }

int main() { char *seps[] = { "==", "!=", "=" }; parse_sep("a!===b=!=c", seps, 3);

return 0; }</lang>output<lang>a{!=}{==}b{=}{!=}c</lang>

C++

using the Boost library tokenizer! <lang cpp>#include <iostream>

1. include <boost/tokenizer.hpp>
2. include <string>

int main( ) {

  std::string str( "a!===b=!=c" ) , output ;
  typedef boost::tokenizer<boost::char_separator<char> > tokenizer ;
  boost::char_separator<char> separator ( "==" , "!=" ) , sep ( "!" )  ;
  tokenizer mytok( str , separator ) ;
  tokenizer::iterator tok_iter = mytok.begin( ) ;
  for ( ; tok_iter != mytok.end( ) ; ++tok_iter )
     output.append( *tok_iter ) ;
  tokenizer nexttok ( output , sep ) ;
  for ( tok_iter = nexttok.begin( ) ; tok_iter != nexttok.end( ) ;

++tok_iter )

     std::cout << *tok_iter << " " ;
  std::cout << '\n' ;
  return 0 ;

}</lang> Output:

a b c

C#

Extra Credit Solution

<lang c sharp>using System; using System.Collections.Generic; using System.Linq; using System.Text;

namespace Multisplit {

   internal static class Program
   {
       private static void Main(string[] args)
       {
           foreach (var s in "a!===b=!=c".Multisplit(true, "==", "!=", "=")) // Split the string and return the separators.
           {
               Console.Write(s); // Write the returned substrings and separators to the console.
           }
           Console.WriteLine();
       }

       private static IEnumerable<string> Multisplit(this string s, bool returnSeparators = false,
                                                     params string[] delimiters)
       {
           var currentString = new StringBuilder(); /* Initiate the StringBuilder. This will hold the current string to return
                                                     * once we find a separator. */

           int index = 0; // Initiate the index counter at 0. This tells us our current position in the string to read.

           while (index < s.Length) // Loop through the string.
           {
               // This will get the highest priority separator found at the current index, or null if there are none.
               string foundDelimiter =
                   (from delimiter in delimiters
                    where s.Length >= index + delimiter.Length &&
                          s.Substring(index, delimiter.Length) == delimiter
                    select delimiter).FirstOrDefault();

               if (foundDelimiter != null)
               {
                   yield return currentString.ToString(); // Return the current string.
                   if (returnSeparators) // Return the separator, if the user specified to do so.
                       yield return
                           string.Format("{{\"{0}\", ({1}, {2})}}",
                                         foundDelimiter,
                                         index, index + foundDelimiter.Length);
                   currentString.Clear(); // Clear the current string.
                   index += foundDelimiter.Length; // Move the index past the current separator.
               }
               else
               {
                   currentString.Append(s[index++]); // Add the character at this index to the current string.
               }
           }

           if (currentString.Length > 0)
               yield return currentString.ToString(); // If we have anything left over, return it.
       }
   }

}</lang>

Sample Output

a{"!=", (1, 3)}{"==", (3, 5)}b{"=", (6, 7)}{"!=", (7, 9)}c

CoffeeScript

<lang coffeescript> multi_split = (text, separators) ->

 # Split text up, using separators to break up text and discarding
 # separators.
 #
 # Returns an array of strings, which can include empty strings when
 # separators are found either adjacent to each other or at the
 # beginning/end of the text.
 #
 # Separators have precedence, according to their order in the array,
 # and each separator should be at least one character long.
 result = []
 i = 0
 s = 
 while i < text.length
   found = false
   for separator in separators
     if text.substring(i, i + separator.length) == separator
       found = true
       i += separator.length
       result.push s
       s = 
       break
   if !found
     s += text[i]
     i += 1
 result.push s
 result
 

console.log multi_split 'a!===b=!=c', ['==', '!=', '='] # [ 'a', , 'b', , 'c' ] console.log multi_split , ['whatever'] # [ ] </lang>

D

<lang d>import std.stdio, std.array, std.algorithm;

string[] multiSplit(in string s, in string[] divisors) /*pure nothrow*/ {

   string[] result;
   auto rest = s.idup;

   while (true) {

bool done = true;

       string delim;
       {
           string best;
           foreach (div; divisors) {
               const maybe = find(rest, div);
               if (maybe.length > best.length) {
                   best = maybe;
                   delim = div;
                   done = false;
               }
           }
       }

result.length++; if (done) { result[$ - 1] = rest.idup; return result; } else { const t = findSplit(rest, delim); result[$ - 1] = t[0].idup; rest = t[2]; }

   }

}

void main() {

   immutable s = "a!===b=!=c";
   immutable divs = ["==", "!=", "="];
   writeln(multiSplit(s, divs).join(" {} "));

}</lang> Output (separator locations indicated by braces):

a {}  {} b {}  {} c

F#

If we ignore the "Extra Credit" requirements and skip 'ordered separators' condition (i.e. solving absolute different task), this is exactly what one of the overloads of .NET's String.Split method does. Using F# Interactive: <lang fsharp>> "a!===b=!=c".Split([|"=="; "!="; "="|], System.StringSplitOptions.None);; val it : string [] = [|"a"; ""; "b"; ""; "c"|]

> "a!===b=!=c".Split([|"="; "!="; "=="|], System.StringSplitOptions.None);; val it : string [] = [|"a"; ""; ""; "b"; ""; "c"|]</lang>

System.StringSplitOptions.None specifies that empty strings should be included in the result.

Go

<lang go>package main

import (

   "fmt"
   "strings"

)

func ms(txt string, sep []string) (ans []string) {

   for txt > "" {
       sepMatch := ""
       posMatch := len(txt)
       for _, s := range sep {
           if p := strings.Index(txt, s); p >= 0 && p < posMatch {
               sepMatch = s
               posMatch = p
           }
       }
       ans = append(ans, txt[:posMatch])
       txt = txt[posMatch+len(sepMatch):]
   }
   return

}

func main() {

   fmt.Printf("%q\n", ms("a!===b=!=c", []string{"==", "!=", "="}))

}</lang> Output:

["a" "" "b" "" "c"]

Icon and Unicon

<lang Icon>procedure main()

  s := "a!===b=!=c"
  # just list the tokens
  every writes(multisplit(s,["==", "!=", "="])," ") | write()
  
  # list tokens and indices
  every ((p := "") ||:= t := multisplit(s,sep := ["==", "!=", "="])) | break write() do 
     if t == !sep then writes(t," (",*p+1-*t,") ") else writes(t," ")
     

end

procedure multisplit(s,L) s ? while not pos(0) do {

  t := =!L | 1( arb(), match(!L)|pos(0) )
  suspend t
  }

end

procedure arb() suspend .&subject[.&pos:&pos <- &pos to *&subject + 1] end</lang>

Sample Output:

a != == b = != c
a != (2) == (4) b = (7) != (8) c

J

<lang j>multisplit=:4 :0

 'sep begin'=.|:t=. y /:~&.:(|."1)@;@(i.@#@[ ,.L:0"0 I.@E.L:0) x
 end=. begin + sep { #@>y
 last=.next=.0
 r=.2 0$0
 while.next<#begin do.
   r=.r,.(last}.x{.~next{begin);next{t
   last=.next{end
   next=.1 i.~(begin>next{begin)*.begin>:last
 end.
 r=.r,.;~last}.x

)</lang>

Explanation:

First find all potentially relevant separator instances, and sort them in increasing order, by starting location and separator index. sep is separator index, and begin is starting location. end is ending location.

Then, loop through the possibilities, skipping over those separators which would overlap with previously used separators.

The result consists of two rows: The first row is the extracted substrings, the second row is the "extra credit" part -- for each extracted substring, the numbers in the second row are the separator index (0 for the first index, 1 for the second, ...), and the location in the original string where the separator appeared. Note that the very last substring does not have a separator following it, so the extra credit part is blank for that substring.

Example use:

<lang j> S=:'a!===b=!=c'

  S multisplit '==';'!=';'='

┌───┬───┬───┬───┬─┐ │a │ │b │ │c│ ├───┼───┼───┼───┼─┤ │1 1│0 3│2 6│1 7│ │ └───┴───┴───┴───┴─┘

  S multisplit '=';'!=';'=='

┌───┬───┬───┬───┬───┬─┐ │a │ │ │b │ │c│ ├───┼───┼───┼───┼───┼─┤ │1 1│0 3│0 4│0 6│1 7│ │ └───┴───┴───┴───┴───┴─┘

  'X123Y' multisplit '1';'12';'123';'23';'3'

┌───┬───┬─┐ │X │ │Y│ ├───┼───┼─┤ │0 1│3 2│ │ └───┴───┴─┘</lang>

JavaScript

Based on Ruby example.

Library: Underscore.js

<lang JavaScript>RegExp.escape = function(text) {

   return text.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, "\\$&");

}

multisplit = function(string, seps) {

   var sep_regex = RegExp(_.map(seps, function(sep) { return RegExp.escape(sep); }).join('|'));
   return string.split(sep_regex);

}</lang>

Perl

<lang Perl>#!/usr/bin/perl -w use strict ; sub multisplit {

  my ( $string , $pattern ) = @_ ;
  split( /($pattern)/, $string, -1 ) ;

}

my $phrase = "a!===b=!=c" ; my $pattern = "==|!=|=" ; print "$_ ," foreach multisplit( $phrase , $pattern ) ; print "\n" ;</lang> Output:

a ,!= , ,== ,b ,= , ,!= ,c ,

Perl 6

<lang perl6>sub multisplit($str, @seps) { $str.split(/ ||@seps /, :all) }

my @chunks = multisplit( 'a!===b=!=c==d', < == != = > );

1. Print the strings.

say @chunks».Str.perl;

1. Print the positions of the separators.

for grep Match, @chunks -> $s {

   say "  $s   from $s.from() to $s.to()";

}</lang> Output:

("a", "!=", "", "==", "b", "=", "", "!=", "c", "==", "d")
  !=	from 1 to 3
  ==	from 3 to 5
  =	from 6 to 7
  !=	from 7 to 9
  ==	from 10 to 12

Using the array @seps in a pattern automatically does alternation. By default this would do longest-term matching (that is, | semantics), but we can force it to do left-to-right matching by embedding the array in a short-circuit alternation (that is, || semantics). As it happens, with the task's specified list of separators, it doesn't make any difference.

Perl 6 automatically returns Match objects that will stringify to the matched pattern, but can also be interrogated for their match positions, as illustrated above by post-processing the results two different ways.

PicoLisp

<lang PicoLisp>(de multisplit (Str Sep)

  (setq Sep (mapcar chop Sep))
  (make
     (for (S (chop Str) S)
        (let L
           (make
              (loop
                 (T (find head Sep (circ S))
                    (link
                       (list
                          (- (length Str) (length S))
                          (pack (cut (length @) 'S)) ) ) )
                 (link (pop 'S))
                 (NIL S (link NIL)) ) )
           (link (pack (cdr (rot L))))
           (and (car L) (link @)) ) ) ) )

(println (multisplit "a!===b=!=c" '("==" "!=" "="))) (println (multisplit "a!===b=!=c" '("=" "!=" "==")))</lang> Output:

("a" (1 "!=") NIL (3 "==") "b" (6 "=") NIL (7 "!=") "c")
("a" (1 "!=") NIL (3 "=") NIL (4 "=") "b" (6 "=") NIL (7 "!=") "c")

Pike

<lang Pike>string input = "a!===b=!=c"; array sep = ({"==", "!=", "=" });

array result = replace(input, sep, `+("\0", sep[*], "\0"))/"\0"; result; Result: ({ "a", "!=", "", "==", "b", "=", "", "!=", "c" })

int pos = 0; foreach(result; int index; string data) {

   if ((<"==", "!=", "=">)[data])
       result[index] = ({ data, pos });
   pos+=sizeof(data);

}

result; Result: ({"a", ({"!=", 1}), "", ({"==", 3}), "b", ({"=", 6}), "", ({"!=", 7}), "c"})</lang>

Prolog

Works with SWI-Prolog. <lang Prolog>multisplit(_LSep, ) --> {!}, [].

multisplit(LSep, T) --> {next_sep(LSep, T, [], Token, Sep, T1)}, ( {Token \= },[Token], {!}; []), ( {Sep \= },[Sep], {!}; []), multisplit(LSep, T1).

next_sep([], T, Lst, Token, Sep, T1) :- % if we can't find any separator, the game is over ( Lst = [] -> Token = T, Sep = , T1 = ;

% we sort the list to get nearest longest separator predsort(my_sort, Lst, [(_,_, Sep)|_]), atomic_list_concat([Token|_], Sep, T), atom_concat(Token, Sep, Tmp), atom_concat(Tmp, T1, T)).

next_sep([HSep|TSep], T, Lst, Token, Sep, T1) :- sub_atom(T, Before, Len, _, HSep), next_sep(TSep, T, [(Before, Len,HSep) | Lst], Token, Sep, T1).

next_sep([_HSep|TSep], T, Lst, Token, Sep, T1) :- next_sep(TSep, T, Lst, Token, Sep, T1).

my_sort(<, (N1, _, _), (N2, _, _)) :- N1 < N2.

my_sort(>, (N1, _, _), (N2, _, _)) :- N1 > N2.

my_sort(>, (N, N1, _), (N, N2, _)) :- N1 < N2.

my_sort(<, (N, N1, _), (N, N2, _)) :- N1 > N2. </lang> Output :

?- multisplit(['==', '!=', '='], 'ax!===b=!=c', Lst, []).
Lst = [ax,'!=',==,b,=,'!=',c] .

Python

Using Regular expressions

<lang python>>>> import re >>> def ms2(txt="a!===b=!=c", sep=["==", "!=", "="]): if not txt or not sep: return [] ans = m = [] for m in re.finditer('(.*?)(?:' + '|'.join('('+re.escape(s)+')' for s in sep) + ')', txt): ans += [m.group(1), (m.lastindex-2, m.start(m.lastindex))] if m and txt[m.end(m.lastindex):]: ans += [txt[m.end(m.lastindex):]] return ans

>>> ms2() ['a', (1, 1), , (0, 3), 'b', (2, 6), , (1, 7), 'c'] >>> ms2(txt="a!===b=!=c", sep=["=", "!=", "=="]) ['a', (1, 1), , (0, 3), , (0, 4), 'b', (0, 6), , (1, 7), 'c']</lang>

Not using RE's

<lang python>>>> def ms(txt="a!===b=!=c", sep=["==", "!=", "="]): if not txt or not sep: return [] size = [len(s) for s in sep] ans, pos0 = [], 0 def getfinds(): return [(-txt.find(s, pos0), -sepnum, size[sepnum]) for sepnum, s in enumerate(sep) if s in txt[pos0:]]

finds = getfinds() while finds: pos, snum, sz = max(finds) pos, snum = -pos, -snum ans += [ txt[pos0:pos], [snum, pos] ] pos0 = pos+sz finds = getfinds() if txt[pos0:]: ans += [ txt[pos0:] ] return ans

>>> ms() ['a', [1, 1], , [0, 3], 'b', [2, 6], , [1, 7], 'c'] >>> ms(txt="a!===b=!=c", sep=["=", "!=", "=="]) ['a', [1, 1], , [0, 3], , [0, 4], 'b', [0, 6], , [1, 7], 'c']</lang>

Alternative version <lang python>def min_pos(List): return List.index(min(List))

def find_all(S, Sub, Start = 0, End = -1, IsOverlapped = 0): Res = [] if End == -1: End = len(S) if IsOverlapped: DeltaPos = 1 else: DeltaPos = len(Sub) Pos = Start while True: Pos = S.find(Sub, Pos, End) if Pos == -1: break Res.append(Pos) Pos += DeltaPos return Res

def multisplit(S, SepList): SepPosListList = [] SLen = len(S) SepNumList = [] ListCount = 0 for i, Sep in enumerate(SepList): SepPosList = find_all(S, Sep, 0, SLen, IsOverlapped = 1) if SepPosList != []: SepNumList.append(i) SepPosListList.append(SepPosList) ListCount += 1 if ListCount == 0: return [S] MinPosList = [] for i in range(ListCount): MinPosList.append(SepPosListList[i][0]) SepEnd = 0 MinPosPos = min_pos(MinPosList) Res = [] while True: Res.append( S[SepEnd : MinPosList[MinPosPos]] ) Res.append([SepNumList[MinPosPos], MinPosList[MinPosPos]]) SepEnd = MinPosList[MinPosPos] + len(SepList[SepNumList[MinPosPos]]) while True: MinPosPos = min_pos(MinPosList) if MinPosList[MinPosPos] < SepEnd: del SepPosListList[MinPosPos][0] if len(SepPosListList[MinPosPos]) == 0: del SepPosListList[MinPosPos] del MinPosList[MinPosPos] del SepNumList[MinPosPos] ListCount -= 1 if ListCount == 0: break else: MinPosList[MinPosPos] = SepPosListList[MinPosPos][0] else: break if ListCount == 0: break Res.append(S[SepEnd:]) return Res

S = "a!===b=!=c" multisplit(S, ["==", "!=", "="]) # output: ['a', [1, 1], , [0, 3], 'b', [2, 6], , [1, 7], 'c'] multisplit(S, ["=", "!=", "=="]) # output: ['a', [1, 1], , [0, 3], , [0, 4], 'b', [0, 6], , [1, 7], 'c']</lang>

Ruby

The simple method, using a regular expression to split the text.

<lang ruby>text = 'a!===b=!=c' separators = ['==', '!=', '=']

def multisplit_simple(text, separators)

 sep_regex = Regexp.new(separators.collect {|sep| Regexp.escape(sep)}.join('|'))
 text.split(sep_regex)

end

p multisplit_simple(text, separators) ["a", "", "b", "", "c"] => nil p multisplit_simple(text, ['=', '!=', '==']) ["a", "", "", "b", "", "c"] => nil</lang>

The version that also returns the information about the separations.

<lang ruby>def multisplit(text, separators)

 sep_regex = Regexp.new(separators.collect {|sep| Regexp.escape(sep)}.join('|'))
 separator_info = []
 pieces = []
 i = prev = 0
 while i = text.index(sep_regex, i)
   separator = Regexp.last_match(0)
   pieces << text[prev .. i-1]
   separator_info << [separator, i]
   i = i + separator.length
   prev = i
 end
 pieces << text[prev .. -1]
 [pieces, separator_info]

end

p multisplit(text, separators)

1. => [["a", "", "b", "", "c"], [["!=", 1], ["==", 3], ["=", 6], ["!=", 7]]]</lang>

Also demonstrating a method to rejoin the string given the separator information.

<lang ruby>def multisplit_rejoin(info)

 str = info[0].zip(info[1])[0..-2].inject("") {|str, (piece, (sep, idx))| str << piece << sep} 
 str << info[0].last

end

p multisplit_rejoin(multisplit(text, separators)) == text

1. => true</lang>

Run BASIC

<lang runbasic>str$ = "a!===b=!=c" sep$ = "=== != =! b =!="

while word$(sep$,i+1," ") <> ""

i = i + 1
theSep$ = word$(sep$,i," ")
split$  = word$(str$,1,theSep$)
print i;" ";split$;" Sep By: ";theSep$

wend</lang>

1 a!     Sep By: ===
2 a      Sep By: !=
3 a!===b Sep By: =!
4 a!===  Sep By: b
5 a!===b Sep By: =!=

Tcl

This simple version does not retain information about what the separators were: <lang tcl>proc simplemultisplit {text sep} {

   set map {}; foreach s $sep {lappend map $s "\uffff"}
   return [split [string map $map $text] "\uffff"]

} puts [simplemultisplit "a!===b=!=c" {"==" "!=" "="}]</lang>

Output:

a {} b {} c

However, to keep the match information a more sophisticated technique is best. Note that the most natural model of result here is to return the split substrings as a separate list to the match information (because the two collections of information are of different lengths). <lang tcl>proc multisplit {text sep} {

   foreach s $sep {lappend sr [regsub -all {\W} $s {\\&}]}
   set sepRE [join $sr "|"]
   set pieces {}
   set match {}
   set start 0
   while {[regexp -indices -start $start -- $sepRE $text found]} {

lassign $found x y lappend pieces [string range $text $start [expr {$x-1}]] lappend match [lsearch -exact $sep [string range $text {*}$found]] $x set start [expr {$y + 1}]

   }
   return [list [lappend pieces [string range $text $start end]] $match]

}</lang> Demonstration code: <lang tcl>set input "a!===b=!=c" set matchers {"==" "!=" "="} lassign [multisplit $input $matchers] substrings matchinfo puts $substrings puts $matchinfo</lang> Output:

a {} b {} c
1 1 0 3 2 6 1 7

TXR

Here, the separators are embedded into the syntax rather than appearing as a datum. Nevertheless, this illustrates how to do that small tokenizing task with various separators.

The clauses of choose are applied in parallel, and all potentially match at the current position in the text. However :shortest tok means that only that clause survives (gets to propagate its bindings and position advancement) which minimizes the length of the string which is bound to the tok variable. The :gap 0 makes the horizontal collect repetitions strictly adjacent. This means that coll will quit when faced with a nonmatching suffix portion of the data rather than scan forward (no gap allowed!). This creates an opportunity for the tail variable to grab the suffix which remains, which may be an empty string.

<lang txr>@(next :args) @(coll :gap 0)@(choose :shortest tok)@\

               @tok@{sep /==/}@\
             @(or)@\
               @tok@{sep /!=/}@\
             @(or)@\
               @tok@{sep /=/}@\
             @(end)@(end)@tail

@(output) @(rep)"@tok" {@sep} @(end)"@tail" @(end)</lang>

Runs:

$ ./txr multisplit.txr 'a!===b=!=c'
"a" {!=} "" {==} "b" {=} "" {!=} "c"
$ ./txr  multisplit.txr 'a!===!==!=!==b'
"a" {!=} "" {==} "" {!=} "" {=} "" {!=} "" {!=} "" {=} "b"
$ ./txr  multisplit.txr ''
""
$ ./txr  multisplit.txr 'a'
"a"
$ ./txr  multisplit.txr 'a='
"a" {=} ""
$ ./txr  multisplit.txr '='
"" {=} ""
$ ./txr  multisplit.txr '=='
"" {==} ""
$ ./txr  multisplit.txr '==='
"" {==} "" {=} ""