Unicode strings: Difference between revisions
m
syntax highlighting fixup automation
Alextretyak (talk | contribs) (Added 11l) |
Thundergnat (talk | contribs) m (syntax highlighting fixup automation) |
||
Line 88:
{{works with|ALGOL 68G|Any - tested with release [http://sourceforge.net/projects/algol68/files/algol68g/algol68g-1.18.0/algol68g-1.18.0-9h.tiny.el5.centos.fc11.i386.rpm/download 1.18.0-9h.tiny].}}
{{wont work with|ELLA ALGOL 68|Any (with appropriate job cards) - tested with release [http://sourceforge.net/projects/algol68/files/algol68toc/algol68toc-1.8.8d/algol68toc-1.8-8d.fc9.i386.rpm/download 1.8-8d] - due to extensive use of '''format'''[ted] ''transput''.}}
<
# -*- coding: utf-8 -*- #
Line 388:
))
)</
{{out}}
<pre>
Line 401:
=={{header|Arturo}}==
<
print ["text:" text]
Line 407:
print ["contains string '好'?:" contains? text "好"]
print ["contains character '平'?:" contains? text `平`]
print ["text as ascii:" as.ascii text]</
{{out}}
Line 462:
'''Code example:'''
(whether this listing displays correctly will depend on your browser)
<
*FONT Times New Roman, 20
Line 519:
B$ += CHR$?A%
NEXT
= LEFT$(B$)</
[[Image:unicode_bbc.gif]]
Line 535:
=={{header|C}}==
C is not the most unicode friendly language, to put it mildly. Generally using unicode in C requires dealing with locales, manage data types carefully, and checking various aspects of your compiler. Directly embedding unicode strings in your C source might be a bad idea, too; it's safer to use their hex values. Here's a short example of doing the simplest string handling: print it.<
#include <stdlib.h>
#include <locale.h>
Line 564:
#endif
return 0;
}</
=={{header|C sharp|C#}}==
Line 580:
Default unicode strings for most implementations. Unicode chars can be used on variable and function names.
Tested in SBCL 1.2.7 and ECL 13.5.1
<
(defvar ♥♦♣♠ "♥♦♣♠")
(defun ✈ () "a plane unicode function")
</syntaxhighlight>
=={{header|D}}==
<
import std.uni; // standard package for normalization, composition/decomposition, etc..
import std.utf; // standard package for decoding/encoding, etc...
Line 607:
// escape sequences like what is defined in C are also allowed inside of strings and characters.
}</
=={{header|DWScript}}==
Line 620:
ELENA 4.x:
<
{
var 四十二 := "♥♦♣♠"; // UTF8 string
Line 627:
console.writeLine:строка;
console.writeLine:四十二;
}</
{{out}}
<pre>
Line 646:
The <code>string</code> data type represents a read-only sequence of bytes, conventionally but not necessarily representing UTF-8-encoded text.
A number of built-in features interpret <code>string</code>s as UTF-8. For example,
<
var u rune
for i, u = range "voilà" {
fmt.Println(i, u)
}</
{{out}}
<pre>
Line 663:
In contrast,
<
for i := 0; i < len(w); i++ {
fmt.Println(i, w[i])
}
</syntaxhighlight>
{{out}}
<pre>
Line 699:
Unicode characters can be represented directly in J strings:
<
♥♦♣♠</
By default, they are represented as utf-8:
<
12</
The above string requires 12 literal elements to represent the four characters using utf-8.
Line 711:
However, they can be represented as utf-16 instead:
<
♥♦♣♠
#7 u:'♥♦♣♠'
4</
The above string requires 4 literal elements to represent the four characters using utf-16. (7 u: string produces a utf-16 result.)
Line 720:
These forms are not treated as equivalent:
<
0</
The utf-8 string of literals is a different string of literals from the utf-16 string.
Line 727:
unless the character literals themselves are equivalent:
<
1</
Here, we were dealing with ascii characters, so the four literals needed to represent the characters using utf-8 matched the four literals needed to represent the characters using utf-16.
Line 734:
When this is likely to be an issue, you should enforce a single representation. For example:
<
1
'♥♦♣♠' -:&(8&u:) 7 u:'♥♦♣♠'
1</
Here, we see that even when comparing non-ascii characters, we can coerce both arguments to be utf-8 or utf-16 and in either case the resulting literal strings match. (8 u: string produces a utf-8 result.)
Line 790:
=={{header|Julia}}==
Non-ASCII strings in Julia are UTF8-encoded by default, and Unicode identifiers are also supported:
<
julia> println(四十二)
voilà</
And you can also specify unicode characters by ordinal:
<
✈</
=={{header|Kotlin}}==
Line 803:
Here's a simple example of using both unicode identifiers and unicode strings in Kotlin:
<
fun main(args: Array<String>) {
val åäö = "as⃝df̅ ♥♦♣♠ 頰"
println(åäö)
}</
{{out}}
Line 824:
The following is an example of using the "any" modifier on a string literal.
<
Indexing on a string indexes by code point. The index may be a single number, a range, or an array of such things.
Line 847:
Variable names can not contain anything but ASCII.
<
#unicode -> append('\u9830')
#unicode
Line 853:
#unicode -> get (2)
'<br />'
#unicode -> get (4) -> integer</
{{out}}
<pre>♥♦♣♠頰
Line 864:
Here is example UFT-8 encoding:
<
> (set encoded (binary ("åäö ð" utf8)))
#B(195 165 195 164 195 182 32 195 176)
</
Display it in native Erlang format:
<
> (io:format "~tp~n" (list encoded))
<<"åäö ð"/utf8>>
</syntaxhighlight>
Example UFT-8 decoding:
<
> (unicode:characters_to_list encoded 'utf8)
"åäö ð"
</syntaxhighlight>
=={{header|Lingo}}==
In recent versions (since v11.5) of Lingo's only implementation "Director" UTF-8 is the default encoding for both scripts and strings. Therefor Unicode string literals can be specified directly in the code, and also variable names support Unicode. To represent/deal with string data in other encodings, you have to use the ByteArray data type. Various ByteArray as well as FileIO methods support an optional 'charSet' parameter that allows to transcode data to/from UTF-8 on the fly. The supported 'charSet' strings can be displayed like this:
<
-- ["big5", "cp1026", "cp866", "ebcdic-cp-us", "gb2312", "ibm437", "ibm737",
"ibm775", "ibm850", "ibm852", "ibm857", "ibm861", "ibm869", "iso-8859-1",
Line 893:
"windows-1256", "windows-1257", "windows-1258", "windows-874",
"x-ebcdic-greekmodern", "x-mac-ce", "x-mac-cyrillic", "x-mac-greek",
"x-mac-icelandic", "x-mac-turkish"]</
=={{header|Locomotive Basic}}==
Line 917:
It should be added however that the character set can be easily redefined from BASIC with the SYMBOL and SYMBOL AFTER commands, so the CPC character set can be turned into e.g. Latin-1. As two-byte UTF-8 characters can be converted to Latin-1, at least a subset of Unicode can be printed in this way:
<
20 ' define German umlauts as in Latin-1
30 SYMBOL AFTER 196
Line 938:
200 ' zero-terminated UTF-8 string
210 DATA &48,&C3,&A4,&6C,&6C,&C3,&B6,&20,&4C,&C3,&BC,&64,&77,&69,&67,&2E,&20,&C3,&84,&C3,&96,&C3,&9C
220 DATA &20,&C3,&A4,&C3,&B6,&C3,&BC,&20,&56,&69,&65,&6C,&65,&20,&47,&72,&C3,&BC,&C3,&9F,&65,&21,&00</
Produces this (slightly nonsensical) output:
Line 978:
<syntaxhighlight lang="m2000 interpreter">
Font "Arial"
Mode 32
Line 1,001:
القديم=10
Print القديم+1=11 ' true
</syntaxhighlight>
=={{header|Mathematica}}/{{header|Wolfram Language}}==
Line 1,112:
=={{header|Perl}}==
In Perl, "Unicode" means "UTF-8". If you want to include utf8 characters in your source file, unless you have set <code>PERL_UNICODE</code> environment correctly, you should do<syntaxhighlight lang
Inside the script, utf8 characters can be used both as identifiers and literal strings, and built-in string functions will respect it:<
print "$四十二"; # voilà
print uc($四十二); # VOILÀ</
or you can specify unicode characters by name or ordinal:<
$x = "\N{sigma} \U\N{sigma}";
$y = "\x{2708}";
print scalar reverse("$x $y"); # ✈ Σ σ</
Regular expressions also have support for unicode based on properties, for example, finding characters that's normally written from right to left:<
When it comes to IO, one should specify whether a file is to be opened in utf8 or raw byte mode:<
open OUT, ">:raw", "file_byte";</
The default of IO behavior can also be set in <code>PERL_UNICODE</code>.
Line 1,142:
=={{header|PicoLisp}}==
PicoLisp can directly handle _only_ Unicode (UTF-8) strings. So the problem is rather how to handle non-Unicode strings: They must be pre- or post-processed by external tools, typically with pipes during I/O. For example, to read a line from a file in 8859 encoding:
<
=={{header|Pike}}==
Line 1,159:
writing it out.
<syntaxhighlight lang="pike">
#charset utf8
void main()
Line 1,170:
write( string_to_utf8(nånsense) );
}
</syntaxhighlight>
{{Out}}
<pre>
Line 1,179:
=={{header|Python}}==
Python supports writing Unicode literals in any encoding, but you have to declare the encoding being used. This is done by including a special comment as either the first or second line of the source file:
<
# -*- coding: latin-1 -*-
u = 'abcdé'
print(ord(u[-1]))</
In Python 3, the default encoding is UTF-8. Before that it was ASCII.
Line 1,190:
=={{header|Racket}}==
<syntaxhighlight lang="racket">
#lang racket
Line 1,208:
;; and in fact the standard language makes use of some of these
(λ(x) x) ; -> an identity function
</syntaxhighlight>
Further points:
Line 1,223:
Raku programs and strings are all in Unicode and operate at a grapheme abstraction level, which is agnostic to underlying encodings or normalizations. (These are generally handled at program boundaries.) Opened files default to UTF-8 encoding. All Unicode character properties are in play, so any appropriate characters may be used as parts of identifiers, whitespace, or user-defined operators. For instance:
<syntaxhighlight lang="raku"
say ∛27; # prints 3</
Non-Unicode strings are represented as Buf types rather than Str types, and Unicode operations may not be applied to Buf types without some kind of explicit conversion. Only ASCIIish operations are allowed on buffers.
Line 1,269:
=={{header|Ring}}==
<
see "Hello, World!"
Line 1,281:
ok
ring_see("Converted To (Hindi): " + cText + nl)
</syntaxhighlight>
{{out}}
<pre>
Line 1,297:
Unicode strings are no problem:
<
str.include?("好") # => true</
Unicode code is no problem either:
<
array.inject(:+)
end
puts Σ([4,5,6]) #=>15
</syntaxhighlight>
Ruby 2.2 introduced a method to normalize unicode strings:
<
p bad = "¿como\u0301 esta\u0301s?" # => "¿comó estás?"
p bad.unicode_normalized? # => false
p bad.unicode_normalize! # => "¿comó estás?"
p bad.unicode_normalized? # => true
</syntaxhighlight>
Since Ruby 2.4 Ruby strings have full Unicode case mapping.
Line 1,320:
=={{header|Scala}}==
{{libheader|Scala}}
<
def charToInt(s: String) = {
Line 1,347:
val a = "$abcde¢£¤¥©ÇßçIJijŁłʒλπ•₠₡₢₣₤₥₦₧₨₩₪₫€₭₮₯₰₱₲₳₴₵₵←→⇒∙⌘☺☻ア字文𪚥".
map(c => "%s\t\\u%04X".format(c, c.toInt)).foreach(println)
}</
{{out}}
<pre style="height:20ex;overflow:scroll">true true
Line 1,417:
Swift has an [https://swiftdoc.org/v5.1/type/string/ advanced string type] that defaults to i18n operations and exposes encoding through views:
<
print(flag.characters.count)
// Prints "1"
Line 1,432:
print(nfc == nfd) //NFx: true
print(nfc == nfkx) //NFKx: false
</syntaxhighlight>
Swift [https://forums.swift.org/t/string-s-abi-and-utf-8/17676 apparently uses a null terminiated char array] for storage to provide compatibility with C, but does a lot of work under-the-covers to make things more ergonomic:
Line 1,484:
=={{header|Sidef}}==
Sidef use UTF-8 encoding for pretty much everything, such as source files, chars, strings, stdout, stderr and stdin.
<
class 国際( なまえ, Straße ) {
Line 1,502:
民族.each { |garçon|
garçon.言え;
}</
{{out}}
<pre>
Line 1,557:
Japanese test case:
<
@(freeform)
@(coll)@{STANZA /[^\n\x3000 ]+/}@(end)@/.*/
</syntaxhighlight>
Test data: Japanese traditional song:
Line 1,627:
Vala strings are UTF-8 encoded by default. In order to print them correctly on the screen, use stdout.printf instead of print.
<
=={{header|Visual Basic .NET}}==
See the C# for some general information about the .NET runtime.
Below is an example of certain parts based of the information in the D entry.
<syntaxhighlight lang="text">Module Module1
Sub Main()
Line 1,653:
End Sub
End Module</
{{out}}
<pre>some text
Line 1,665:
WDTE supports Unicode in both identifiers and strings. WDTE is very loose about identifier rules. If it doesn't conflict with a syntactic structure, such as a keyword, literal, or operator, than it's allowed as an identifier.
<
プリント 'これは実験です。';</
=={{header|Wren}}==
Line 1,682:
The standard library does not support normalization but the above module does allow one to split a string into ''user perceived characters'' (or ''graphemes'').
<
for (c in w) {
System.write("%(c) ") // prints the 5 Unicode 'characters'.
Line 1,703:
System.print(" %(zwe.bytes.count) bytes: %(zwe.bytes.toList.join(" "))")
System.print(" %(zwe.codePoints.count) code-points: %(zwe.codePoints.toList.join(" "))")
System.print(" %(Graphemes.clusterCount(zwe)) grapheme")</
{{out}}
|