Category talk:Wren-upc

User-perceived characters

In Unicode a user-perceived character (or grapheme cluster) can comprise one or more codepoints and the process of splitting a string into such grapheme clusters is described in Unicode Standard Annex #29.

Given the complexity of this process, Wren doesn't have built-in support for it and this module aims to remedy that situation. It is based on Oliver Kuederle's Unicode Text Segmentation for Go library which is subject to the MIT License and is currently based on Unicode version 12.0.

Although the source code file is large by Wren library standards (over 1900 lines), approximately 1600 lines of this are needed to describe the property table which provides the raw material for text segmentation. In the interests of brevity, I have omitted the comments which accompanied the original table which should be referred to if any explanation is needed.

I tried repeatedly to post the source code for this module to RC but was unable to do so - kept getting a 502 'bad gateway' error. I suspect this is something to do with the size of the code (circa 75K bytes) though, when I tried to post it in chunks, that didn't work either.

Anyway, I've now posted the code to this GitHub gist and may do the same in future with my larger submissions.