Unicode strings: Difference between revisions

{{works with|ALGOL 68G|Any - tested with release [http://sourceforge.net/projects/algol68/files/algol68g/algol68g-1.18.0/algol68g-1.18.0-9h.tiny.el5.centos.fc11.i386.rpm/download 1.18.0-9h.tiny].}}

{{wont work with|ELLA ALGOL 68|Any (with appropriate job cards) - tested with release [http://sourceforge.net/projects/algol68/files/algol68toc/algol68toc-1.8.8d/algol68toc-1.8-8d.fc9.i386.rpm/download 1.8-8d] - due to extensive use of '''format'''[ted] ''transput''.}}

# -*- coding: utf-8 -*- #

))

{{out}}

<pre>

=={{header|Arturo}}==

print ["text:" text]

print ["contains string '好'?:" contains? text "好"]

print ["contains character '平'?:" contains? text `平`]

{{out}}

'''Code example:'''

(whether this listing displays correctly will depend on your browser)

*FONT Times New Roman, 20

B$ += CHR$?A%

NEXT

[[Image:unicode_bbc.gif]]

=={{header|C}}==

#include <stdlib.h>

#include <locale.h>

#endif

return 0;

=={{header|C sharp|C#}}==

Default unicode strings for most implementations. Unicode chars can be used on variable and function names.

Tested in SBCL 1.2.7 and ECL 13.5.1

(defvar ♥♦♣♠ "♥♦♣♠")

(defun ✈ () "a plane unicode function")

=={{header|D}}==

import std.uni; // standard package for normalization, composition/decomposition, etc..

import std.utf; // standard package for decoding/encoding, etc...

// escape sequences like what is defined in C are also allowed inside of strings and characters.

=={{header|DWScript}}==

ELENA 4.x:

{

var 四十二 := "♥♦♣♠"; // UTF8 string

console.writeLine:строка;

console.writeLine:四十二;

{{out}}

<pre>

The <code>string</code> data type represents a read-only sequence of bytes, conventionally but not necessarily representing UTF-8-encoded text.

A number of built-in features interpret <code>string</code>s as UTF-8. For example,

var u rune

for i, u = range "voilà" {

fmt.Println(i, u)

{{out}}

<pre>

In contrast,

for i := 0; i < len(w); i++ {

fmt.Println(i, w[i])

}

{{out}}

<pre>

Unicode characters can be represented directly in J strings:

By default, they are represented as utf-8:

The above string requires 12 literal elements to represent the four characters using utf-8.

However, they can be represented as utf-16 instead:

♥♦♣♠

#7 u:'♥♦♣♠'

The above string requires 4 literal elements to represent the four characters using utf-16. (7 u: string produces a utf-16 result.)

These forms are not treated as equivalent:

The utf-8 string of literals is a different string of literals from the utf-16 string.

unless the character literals themselves are equivalent:

Here, we were dealing with ascii characters, so the four literals needed to represent the characters using utf-8 matched the four literals needed to represent the characters using utf-16.

When this is likely to be an issue, you should enforce a single representation. For example:

1

'♥♦♣♠' -:&(8&u:) 7 u:'♥♦♣♠'

Here, we see that even when comparing non-ascii characters, we can coerce both arguments to be utf-8 or utf-16 and in either case the resulting literal strings match. (8 u: string produces a utf-8 result.)

=={{header|Julia}}==

Non-ASCII strings in Julia are UTF8-encoded by default, and Unicode identifiers are also supported:

julia> println(四十二)

And you can also specify unicode characters by ordinal:

=={{header|Kotlin}}==

Here's a simple example of using both unicode identifiers and unicode strings in Kotlin:

fun main(args: Array<String>) {

val åäö = "as⃝df̅ ♥♦♣♠ 頰"

println(åäö)

{{out}}

The following is an example of using the "any" modifier on a string literal.

Indexing on a string indexes by code point. The index may be a single number, a range, or an array of such things.

Variable names can not contain anything but ASCII.

#unicode -> append('\u9830')

#unicode

#unicode -> get (2)

'<br />'

{{out}}

<pre>♥♦♣♠頰

Here is example UFT-8 encoding:

> (set encoded (binary ("åäö ð" utf8)))

#B(195 165 195 164 195 182 32 195 176)

Display it in native Erlang format:

> (io:format "~tp~n" (list encoded))

<<"åäö ð"/utf8>>

Example UFT-8 decoding:

> (unicode:characters_to_list encoded 'utf8)

"åäö ð"

=={{header|Lingo}}==

In recent versions (since v11.5) of Lingo's only implementation "Director" UTF-8 is the default encoding for both scripts and strings. Therefor Unicode string literals can be specified directly in the code, and also variable names support Unicode. To represent/deal with string data in other encodings, you have to use the ByteArray data type. Various ByteArray as well as FileIO methods support an optional 'charSet' parameter that allows to transcode data to/from UTF-8 on the fly. The supported 'charSet' strings can be displayed like this:

-- ["big5", "cp1026", "cp866", "ebcdic-cp-us", "gb2312", "ibm437", "ibm737",

"ibm775", "ibm850", "ibm852", "ibm857", "ibm861", "ibm869", "iso-8859-1",

"windows-1256", "windows-1257", "windows-1258", "windows-874",

"x-ebcdic-greekmodern", "x-mac-ce", "x-mac-cyrillic", "x-mac-greek",

=={{header|Locomotive Basic}}==

It should be added however that the character set can be easily redefined from BASIC with the SYMBOL and SYMBOL AFTER commands, so the CPC character set can be turned into e.g. Latin-1. As two-byte UTF-8 characters can be converted to Latin-1, at least a subset of Unicode can be printed in this way:

20 ' define German umlauts as in Latin-1

30 SYMBOL AFTER 196

200 ' zero-terminated UTF-8 string

210 DATA &48,&C3,&A4,&6C,&6C,&C3,&B6,&20,&4C,&C3,&BC,&64,&77,&69,&67,&2E,&20,&C3,&84,&C3,&96,&C3,&9C

Produces this (slightly nonsensical) output:

Font "Arial"

Mode 32

القديم=10

Print القديم+1=11 ' true

=={{header|Mathematica}}/{{header|Wolfram Language}}==

=={{header|Perl}}==

print "$四十二"; # voilà

$x = "\N{sigma} \U\N{sigma}";

$y = "\x{2708}";

The default of IO behavior can also be set in <code>PERL_UNICODE</code>.

=={{header|PicoLisp}}==

PicoLisp can directly handle _only_ Unicode (UTF-8) strings. So the problem is rather how to handle non-Unicode strings: They must be pre- or post-processed by external tools, typically with pipes during I/O. For example, to read a line from a file in 8859 encoding:

=={{header|Pike}}==

writing it out.

#charset utf8

void main()

write( string_to_utf8(nånsense) );

}

{{Out}}

<pre>

=={{header|Python}}==

Python supports writing Unicode literals in any encoding, but you have to declare the encoding being used. This is done by including a special comment as either the first or second line of the source file:

# -*- coding: latin-1 -*-

u = 'abcdé'

In Python 3, the default encoding is UTF-8. Before that it was ASCII.

=={{header|Racket}}==

#lang racket

;; and in fact the standard language makes use of some of these

(λ(x) x) ; -> an identity function

Further points:

Raku programs and strings are all in Unicode and operate at a grapheme abstraction level, which is agnostic to underlying encodings or normalizations. (These are generally handled at program boundaries.) Opened files default to UTF-8 encoding. All Unicode character properties are in play, so any appropriate characters may be used as parts of identifiers, whitespace, or user-defined operators. For instance:

Non-Unicode strings are represented as Buf types rather than Str types, and Unicode operations may not be applied to Buf types without some kind of explicit conversion. Only ASCIIish operations are allowed on buffers.

=={{header|Ring}}==

see "Hello, World!"

ok

ring_see("Converted To (Hindi): " + cText + nl)

{{out}}

<pre>

Unicode strings are no problem:

Unicode code is no problem either:

array.inject(:+)

end

puts Σ([4,5,6]) #=>15

Ruby 2.2 introduced a method to normalize unicode strings:

p bad = "¿como\u0301 esta\u0301s?" # => "¿comó estás?"

p bad.unicode_normalized? # => false

p bad.unicode_normalize! # => "¿comó estás?"

p bad.unicode_normalized? # => true

Since Ruby 2.4 Ruby strings have full Unicode case mapping.

=={{header|Scala}}==

{{libheader|Scala}}

def charToInt(s: String) = {

val a = "$abcde¢£¤¥©ÇßçIJijŁłʒλπ•₠₡₢₣₤₥₦₧₨₩₪₫€₭₮₯₰₱₲₳₴₵₵←→⇒∙⌘☺☻ア字文𪚥".

map(c => "%s\t\\u%04X".format(c, c.toInt)).foreach(println)

{{out}}

<pre style="height:20ex;overflow:scroll">true true

Swift has an [https://swiftdoc.org/v5.1/type/string/ advanced string type] that defaults to i18n operations and exposes encoding through views:

print(flag.characters.count)

// Prints "1"

print(nfc == nfd) //NFx: true

print(nfc == nfkx) //NFKx: false

Swift [https://forums.swift.org/t/string-s-abi-and-utf-8/17676 apparently uses a null terminiated char array] for storage to provide compatibility with C, but does a lot of work under-the-covers to make things more ergonomic:

=={{header|Sidef}}==

Sidef use UTF-8 encoding for pretty much everything, such as source files, chars, strings, stdout, stderr and stdin.

class 国際( なまえ, Straße ) {

民族.each { |garçon|

garçon.言え;

{{out}}

<pre>

Japanese test case:

@(freeform)

@(coll)@{STANZA /[^\n\x3000 ]+/}@(end)@/.*/

Test data: Japanese traditional song:

Vala strings are UTF-8 encoded by default. In order to print them correctly on the screen, use stdout.printf instead of print.

=={{header|Visual Basic .NET}}==

See the C# for some general information about the .NET runtime.

Below is an example of certain parts based of the information in the D entry.

Sub Main()

End Sub

{{out}}

<pre>some text

WDTE supports Unicode in both identifiers and strings. WDTE is very loose about identifier rules. If it doesn't conflict with a syntactic structure, such as a keyword, literal, or operator, than it's allowed as an identifier.

=={{header|Wren}}==

The standard library does not support normalization but the above module does allow one to split a string into ''user perceived characters'' (or ''graphemes'').

for (c in w) {

System.write("%(c) ") // prints the 5 Unicode 'characters'.

System.print(" %(zwe.bytes.count) bytes: %(zwe.bytes.toList.join(" "))")

System.print(" %(zwe.codePoints.count) code-points: %(zwe.codePoints.toList.join(" "))")

{{out}}