Unicode strings: Difference between revisions

As the world gets smaller each day, internationalization becomes more and more important.   For handling multiple languages, [[Unicode]] is your best friend.

 

 

How well prepared is your programming language for Unicode?

*   [[Terminal control/Display an extended character]]

<br><br>

 

=={{header|80386 Assembly}}==

{{works with|ALGOL 68G|Any - tested with release [http://sourceforge.net/projects/algol68/files/algol68g/algol68g-1.18.0/algol68g-1.18.0-9h.tiny.el5.centos.fc11.i386.rpm/download 1.18.0-9h.tiny].}}

{{wont work with|ELLA ALGOL 68|Any (with appropriate job cards) - tested with release [http://sourceforge.net/projects/algol68/files/algol68toc/algol68toc-1.8.8d/algol68toc-1.8-8d.fc9.i386.rpm/download 1.8-8d] - due to extensive use of '''format'''[ted] ''transput''.}}

# -*- coding: utf-8 -*- #

 

))

 

{{out}}

<pre>

=={{header|Arturo}}==

 

 

print ["text:" text]

print ["contains string '好'?:" contains? text "好"]

print ["contains character '平'?:" contains? text `平`]

 

{{out}}

How broad/deep does the language support Unicode? What encodings (e.g. UTF-8, UTF-16, etc) can be used? - There is no inbuilt support for Unicode, but all encodings can be represented through hexadecimal strings.

 

{{works with|BBC BASIC for Windows}}

* How easy is it to present Unicode strings in source code?

'''Code example:'''

(whether this listing displays correctly will depend on your browser)

*FONT Times New Roman, 20

B$ += CHR$?A%

NEXT

[[Image:unicode_bbc.gif]]

 

 

=={{header|C}}==

#include <stdlib.h>

#include <locale.h>

#endif

return 0;

 

=={{header|C sharp|C#}}==

Default unicode strings for most implementations. Unicode chars can be used on variable and function names.

Tested in SBCL 1.2.7 and ECL 13.5.1

(defvar ♥♦♣♠ "♥♦♣♠")

(defun ✈ () "a plane unicode function")

 

=={{header|D}}==

import std.uni; // standard package for normalization, composition/decomposition, etc..

import std.utf; // standard package for decoding/encoding, etc...

 

// escape sequences like what is defined in C are also allowed inside of strings and characters.

 

=={{header|DWScript}}==

ELENA supports both UTF8 and UTF16 strings, Unicode identifiers are also supported:

 

{

var 四十二 := "♥♦♣♠"; // UTF8 string

var строка := "Привет"w; // UTF16 string

{{out}}

<pre>

 

=={{header|Elixir}}==

 

=={{header|Erlang}}==

The simplified explanation is that Erlang allows Unicode in comments/data/file names/etc, but not in function or variable names.

 

=={{header|Go}}==

The <code>string</code> data type represents a read-only sequence of bytes, conventionally but not necessarily representing UTF-8-encoded text.

A number of built-in features interpret <code>string</code>s as UTF-8. For example,

var u rune

for i, u = range "voilà" {

fmt.Println(i, u)

{{out}}

<pre>

 

In contrast,

for i := 0; i < len(w); i++ {

fmt.Println(i, w[i])

}

{{out}}

<pre>

Unicode characters can be represented directly in J strings:

 

 

By default, they are represented as utf-8:

 

 

The above string requires 12 literal elements to represent the four characters using utf-8.

However, they can be represented as utf-16 instead:

 

♥♦♣♠

#7 u:'♥♦♣♠'

 

The above string requires 4 literal elements to represent the four characters using utf-16. (7 u: string produces a utf-16 result.)

These forms are not treated as equivalent:

 

 

The utf-8 string of literals is a different string of literals from the utf-16 string.

unless the character literals themselves are equivalent:

 

 

Here, we were dealing with ascii characters, so the four literals needed to represent the characters using utf-8 matched the four literals needed to represent the characters using utf-16.

When this is likely to be an issue, you should enforce a single representation. For example:

 

1

'♥♦♣♠' -:&(8&u:) 7 u:'♥♦♣♠'

 

 

Output uses characters in whatever format they happen to be in.

=={{header|Julia}}==

Non-ASCII strings in Julia are UTF8-encoded by default, and Unicode identifiers are also supported:

julia> println(四十二)

And you can also specify unicode characters by ordinal:

 

=={{header|Kotlin}}==

 

Here's a simple example of using both unicode identifiers and unicode strings in Kotlin:

 

fun main(args: Array<String>) {

val åäö = "as⃝df̅ ♥♦♣♠ 頰"

println(åäö)

 

{{out}}

 

=={{header|langur}}==

 

Identifiers are ASCII only. Comments and string literals may use Unicode.

 

 

 

The len() function returns the number of code points in a string.

 

Using a for of loop over a string gives the code point indices, and using a for in loop over a string gives the code point numbers.

 

See langurlang.org for more details.

Variable names can not contain anything but ASCII.

 

#unicode -> append('\u9830')

#unicode

#unicode -> get (2)

'<br />'

{{out}}

<pre>♥♦♣♠頰

 

Here is example UFT-8 encoding:

> (set encoded (binary ("åäö ð" utf8)))

#B(195 165 195 164 195 182 32 195 176)

 

Display it in native Erlang format:

 

> (io:format "~tp~n" (list encoded))

<<"åäö ð"/utf8>>

 

Example UFT-8 decoding:

> (unicode:characters_to_list encoded 'utf8)

"åäö ð"

 

=={{header|Lingo}}==

In recent versions (since v11.5) of Lingo's only implementation "Director" UTF-8 is the default encoding for both scripts and strings. Therefor Unicode string literals can be specified directly in the code, and also variable names support Unicode. To represent/deal with string data in other encodings, you have to use the ByteArray data type. Various ByteArray as well as FileIO methods support an optional 'charSet' parameter that allows to transcode data to/from UTF-8 on the fly. The supported 'charSet' strings can be displayed like this:

-- ["big5", "cp1026", "cp866", "ebcdic-cp-us", "gb2312", "ibm437", "ibm737",

"ibm775", "ibm850", "ibm852", "ibm857", "ibm861", "ibm869", "iso-8859-1",

"windows-1256", "windows-1257", "windows-1258", "windows-874",

"x-ebcdic-greekmodern", "x-mac-ce", "x-mac-cyrillic", "x-mac-greek",

 

=={{header|Locomotive Basic}}==

It should be added however that the character set can be easily redefined from BASIC with the SYMBOL and SYMBOL AFTER commands, so the CPC character set can be turned into e.g. Latin-1. As two-byte UTF-8 characters can be converted to Latin-1, at least a subset of Unicode can be printed in this way:

 

20 ' define German umlauts as in Latin-1

30 SYMBOL AFTER 196

200 ' zero-terminated UTF-8 string

210 DATA &48,&C3,&A4,&6C,&6C,&C3,&B6,&20,&4C,&C3,&BC,&64,&77,&69,&67,&2E,&20,&C3,&84,&C3,&96,&C3,&9C

Produces this (slightly nonsensical) output:

 

[[File:Unicode print locomotive basic.png]]

 

=={{header|M2000 Interpreter}}==

 

 

Font "Arial"

Mode 32

القديم=10

Print القديم+1=11 ' true

 

=={{header|Mathematica}}/{{header|Wolfram Language}}==

 

=={{header|Perl}}==

 

print "$四十二"; # voilà

$x = "\N{sigma} \U\N{sigma}";

$y = "\x{2708}";

 

 

The default of IO behavior can also be set in <code>PERL_UNICODE</code>.

 

=={{header|PicoLisp}}==

PicoLisp can directly handle _only_ Unicode (UTF-8) strings. So the problem is rather how to handle non-Unicode strings: They must be pre- or post-processed by external tools, typically with pipes during I/O. For example, to read a line from a file in 8859 encoding:

 

=={{header|Pike}}==

writing it out.

 

#charset utf8

void main()

write( string_to_utf8(nånsense) );

}

{{Out}}

<pre>

λ ä

</pre>

 

=={{header|Python}}==

Python supports writing Unicode literals in any encoding, but you have to declare the encoding being used. This is done by including a special comment as either the first or second line of the source file:

# -*- coding: latin-1 -*-

 

u = 'abcdé'

In Python 3, the default encoding is UTF-8. Before that it was ASCII.

 

=={{header|Racket}}==

 

#lang racket

 

;; and in fact the standard language makes use of some of these

(λ(x) x) ; -> an identity function

 

Further points:

(formerly Perl 6)

 

 

 

Non-Unicode strings are represented as Buf types rather than Str types, and Unicode operations may not be applied to Buf types without some kind of explicit conversion. Only ASCIIish operations are allowed on buffers.

 

 

* Supports the normalized forms NFC, NFD, NFKC, and NFKD, and character equivalence as specified in [http://unicode.org/reports/tr15/ Unicode technical report #15].

* Works seamlessly with upper plane and private use plane character codepoints.

* Provides tools to deal with strings that contain invalid Unicode characters.

 

In general, it tries to make dealing with Unicode "just work".

 

Raku intends to support Unicode even better than Perl 5, which already does a great job in recent versions of accessing large swaths of Unicode spec. functionality. Raku improves on Perl 5 primarily by offering explicitly typed strings that always know which operations are sensical and which are not.

 

=={{header|REXX}}==

 

=={{header|Ring}}==

see "Hello, World!"

 

ok

ring_see("Converted To (Hindi): " + cText + nl)

{{out}}

<pre>

Unicode strings are no problem:

 

 

Unicode code is no problem either:

 

array.inject(:+)

end

 

puts Σ([4,5,6]) #=>15

Ruby 2.2 introduced a method to normalize unicode strings:

p bad = "¿como\u0301 esta\u0301s?" # => "¿comó estás?"

p bad.unicode_normalized? # => false

p bad.unicode_normalize! # => "¿comó estás?"

p bad.unicode_normalized? # => true

 

Since Ruby 2.4 Ruby strings have full Unicode case mapping.

=={{header|Scala}}==

{{libheader|Scala}}

 

def charToInt(s: String) = {

val a = "$abcde¢£¤¥©ÇßçIJijŁłʒλπ•₠₡₢₣₤₥₦₧₨₩₪₫€₭₮₯₰₱₲₳₴₵₵←→⇒∙⌘☺☻ア字文𪚥".

map(c => "%s\t\\u%04X".format(c, c.toInt)).foreach(println)

{{out}}

<pre style="height:20ex;overflow:scroll">true true

Swift has an [https://swiftdoc.org/v5.1/type/string/ advanced string type] that defaults to i18n operations and exposes encoding through views:

 

print(flag.characters.count)

// Prints "1"

print(nfc == nfd) //NFx: true

print(nfc == nfkx) //NFKx: false

 

Swift [https://forums.swift.org/t/string-s-abi-and-utf-8/17676 apparently uses a null terminiated char array] for storage to provide compatibility with C, but does a lot of work under-the-covers to make things more ergonomic:

=={{header|Sidef}}==

Sidef use UTF-8 encoding for pretty much everything, such as source files, chars, strings, stdout, stderr and stdin.

class 国際( なまえ, Straße ) {

 

民族.each { |garçon|

garçon.言え;

{{out}}

<pre>

 

Japanese test case:

@(freeform)

@(coll)@{STANZA /[^\n\x3000 ]+/}@(end)@/.*/

 

Test data: Japanese traditional song:

 

Vala strings are UTF-8 encoded by default. In order to print them correctly on the screen, use stdout.printf instead of print.

 

=={{header|Visual Basic .NET}}==

See the C# for some general information about the .NET runtime.

Below is an example of certain parts based of the information in the D entry.

 

Sub Main()

End Sub

 

{{out}}

<pre>some text

WDTE supports Unicode in both identifiers and strings. WDTE is very loose about identifier rules. If it doesn't conflict with a syntactic structure, such as a keyword, literal, or operator, than it's allowed as an identifier.

 

 

 

=={{header|Wren}}==

 

The standard library does not support normalization but the above module does allow one to split a string into ''user perceived characters'' (or ''graphemes'').

for (c in w) {

System.write("%(c) ") // prints the 5 Unicode 'characters'.

System.print(" %(zwe.bytes.count) bytes: %(zwe.bytes.toList.join(" "))")

System.print(" %(zwe.codePoints.count) code-points: %(zwe.codePoints.toList.join(" "))")

 

{{out}}

 

* How broad/deep does the language support Unicode? What encodings (e.g. UTF-8, UTF-16, etc) can be used? There is no inbuilt support for Unicode, but all encodings can be represented through hexadecimal strings. A decoder and output routine would need to be written, but this is easy to do on the Spectrum.

 

{{omit from|GUISS}}