UTF-8 encode and decode: Difference between revisions

{{trans|Python}}

 

R ‘U+’hex(ch.code).zfill(4)

 

V chars = [‘A’, ‘ö’, ‘Ж’, ‘€’]

L(char) chars

 

{{out}}

 

=={{header|8th}}==

hex \ so bytes print nicely

 

 

bye

Output:<pre>

A 41

 

=={{header|Action!}}==

BYTE ARRAY hex=['0 '1 '2 '3 '4 '5 '6 '7 '8 '9 'A 'B 'C 'D 'E 'F]

 

StrUnicode(res) PutE()

OD

{{out}}

[https://gitlab.com/amarok8bit/action-rosetta-code/-/raw/master/images/UTF-8_encode_and_decode.png Screenshot from Atari 8-bit computer]

{{works with|Ada|Ada|2012}}

 

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;

with Ada.Integer_Text_IO;

end;

end loop;

{{out}}

<pre>Character Unicode UTF-8 encoding (hex)

Despite this complexity, what actually gets generated is highly optimizable C code. Note that the following demonstration requires no ATS-specific library support whatsoever.

 

 

UTF-8 encoding and decoding in ATS2.

 

println! ("SUCCESS")

 

{{out}}

 

=={{header|AutoHotkey}}==

Bytes := hex>=0x10000 ? 4 : hex>=0x0800 ? 3 : hex>=0x0080 ? 2 : hex>=0x0001 ? 1 : 0

Prefix := [0, 0xC0, 0xE0, 0xF0]

DllCall("msvcrt.dll\" v, "Int64", value, "Str", s, "UInt", OutputBase, "CDECL")

return s

(comment

0x0041

}

MsgBox % output

{{out}}

<pre>

=={{header|BaCon}}==

BaCon supports UTF8 natively.

 

CONST letter$ = "A ö Ж € 𝄞"

FOR x IN letter$

PRINT x, TAB$(1), "U+", HEX$(UCS(x)), TAB$(2), COIL$(LEN(x), HEX$(x[_-1] & 255))

{{out}}

<pre>Char Unicode UTF-8 (hex)

 

=={{header|C}}==

#include <stdio.h>

#include <stdlib.h>

return 0;

}

Output

Character Unicode UTF-8 encoding (hex)

----------------------------------------

𝄞 U+1d11e f0 9d 84 9e

 

 

=={{header|C sharp|C#}}==

 

using System.Text;

 

20AC € E2-82-AC

1D11E 𝄞 F0-9D-84-9E */

 

 

Helper functions

 

(defun ascii-byte-p (octet)

"Return t if octet is a single-byte 7-bit ASCII char.

when (= (nth i templates) (logand (nth i bitmasks) octet))

return i)))

 

Encoder

 

(defun unicode-to-utf-8 (int)

"Take a unicode code point, return a list of one to four UTF-8 encoded bytes (octets)."

;; return the list of UTF-8 encoded bytes.

byte-list))

 

Decoder

 

(defun utf-8-to-unicode (byte-list)

"Take a list of one to four utf-8 encoded bytes (octets), return a code point."

(error "calculated number of bytes doesnt match the length of the byte list")))

(error "first byte in the list isnt a lead byte"))))))

 

The test

 

(defun test-utf-8 ()

"Return t if the chosen unicode points are encoded and decoded correctly."

;; return t if all are t

(every #'= unicodes-orig unicodes-test)))

 

Test output

 

CL-USER> (test-utf-8)

character A, code point: 41, utf-8: 41

character 𝄞, code point: 1D11E, utf-8: F0 9D 84 9E

T

 

=={{header|D}}==

import std.stdio;

 

writefln("%s %7X [%(%X, %)]", c, unicode, bytes);

}

 

{{out}}

=={{header|Elena}}==

ELENA 4.x :

import extensions;

"𝄞".printAsString().printAsUTF8Array().printAsUTF32();

console.printLine();

{{out}}

<pre>

 

=={{header|F_Sharp|F#}}==

// Unicode character point to UTF8. Nigel Galloway: March 19th., 2018

let fN g = match List.findIndex (fun n->n>g) [0x80;0x800;0x10000;0x110000] with

|2->[0xe0+(g&&&0xf000>>>12);0x80+(g&&&0xfc0>>>6);0x80+(g&&&0x3f)]

|_->[0xf0+(g&&&0x1c0000>>>18);0x80+(g&&&0x3f000>>>12);0x80+(g&&&0xfc0>>>6);0x80+(g&&&0x3f)]

{{out}}

<pre>

{{works with|gforth|0.7.9_20191121}}

{{works with|lxf|1.6-982-823}}

over + swap ?do

i c@ 3 .r loop ;

\ can also be written as

\ 'A' test 'ö' test 'Ж' test '€' test '𝄞' test

{{out}}

<pre>

If you also want to see the implementation of <code>xc!+</code> and <code>xc@+</code>, here it is (<code>u8!+</code> is the UTF-8 implementation of <code>xc!+</code>, and likewise for <code>u8@+</code>):

 

 

$80 Constant max-single-byte

REPEAT $7F xor 2* or r>

BEGIN over $80 u>= WHILE tuck c! 1+ REPEAT nip ;

 

=={{header|Go}}==

===Implementation===

This implementation is missing all checks for invalid data and so is not production-ready, but illustrates the basic UTF-8 encoding scheme.

 

import (

rune(b[3]&mbMask)

}

{{out}}

<pre>

 

===Library/language===

 

import (

fmt.Printf("%-7c U+%04X\t%-12X\t%c\n", codepoint, codepoint, encoded, decoded)

}

{{out}}

<pre>

 

Alternately:

 

import (

fmt.Printf("%-7c U+%04X\t%-12X\t%c\n", codepoint, codepoint, encoded, decoded)

}

{{out}}

<pre>

=={{header|Groovy}}==

{{trans|Java}}

 

class UTF8EncodeDecode {

}

}

{{out}}

<pre>Char Name Unicode UTF-8 encoded Decoded

Example makes use of [http://hackage.haskell.org/package/bytestring <tt>bytestring</tt>] and [http://hackage.haskell.org/package/text <tt>text</tt>] packages:

 

import qualified Data.ByteString as ByteString (pack, unpack)

(printf "U+%04X" codepoint :: String)

(intercalate " " (map (printf "%02X") values))

{{out}}

<pre>

=={{header|J}}==

'''Solution:'''

ucp=: 9&u: NB. converts to unicode from UTF-8 or unicode codepoint integer

 

'''Examples:'''

AöЖ€𝄞

ucp 65 246 1046 8364 119070

1d11e

utf8@dfh ucp_hex utf8 65 246 1046 8364 119070

 

=={{header|Java}}==

{{works with|Java|7+}}

import java.util.Formatter;

 

}

}

{{out}}

<pre>

=={{header|JavaScript}}==

An implementation in ECMAScript 2015 (ES6):

/***************************************************************************\

|* Pure UTF-8 handling without detailed error reporting functionality. *|

?( m&0x07)<<18|( n&0x3f)<<12|( o&0x3f)<<6|( p&0x3f)<<0

:(()=>{throw'Invalid UTF-8 encoding!'})()

The testing inputs:

const

str=

:[ [ a,b,c]]

,inputs=zip3(str,cps,cus);

The testing code:

console.log(`\

${'Character'.padEnd(16)}\

${`[${[...utf8encode(cp)].map(n=>n.toString(0x10))}]`.padEnd(16)}\

${utf8decode(cu).toString(0x10).padStart(8,'U+000000')}`)

and finally, the output from the test:

<pre>

 

'''Preliminaries'''

# output: the corresponding binary array, most significant bit first

def binary_digits:

.result += .power * $b

| .power *= 2)

'''Encode to UTF-8'''

# output: the corresponding decimal number of that codepoint.

def utf8_encode:

end

| map(binary_to_decimal)

'''Decode an array of UTF-8 bytes'''

# output: the corresponding decimal number of that codepoint.

def utf8_decode:

else $d[0][-3:] + $d[1][$mb:] + $d[2][$mb:] + $d[3][$mb:]

end

'''Task'''

[ "A", "ö", "Ж", "€", "𝄞" ][]

| . as $glyph

| "Glyph \($glyph) => \($encoded) => \($decoded) => \([$decoded]|implode)" ;

 

{{out}}

<pre>

Julia supports by default UTF-8 encoding.

 

enc = Vector{UInt8}(t)

dec = String(enc)

println(dec, " → ", enc)

 

{{out}}

 

=={{header|Kotlin}}==

 

fun utf8Encode(codePoint: Int) = String(intArrayOf(codePoint), 0, 1).toByteArray(Charsets.UTF_8)

System.out.printf("%-${n}s %c\n", s, decoded)

}

 

{{out}}

=={{header|langur}}==

{{works with|langur|0.8.4}}

 

for .cp in "AöЖ€𝄞" {

val .utf8rep = join " ", map f $"\.b:X02;", .utf8

writeln $"\.cpstr:-11; U+\.cp:X04:-8; \.utf8rep;"

 

{{out}}

- byteArray.toHexString (intStart, intLen): returns hex string representation of byte array (e.g. for printing)<br />

- byteArray.readRawString (intLen, [strCharSet="UTF-8"]): reads a fixed number of bytes as a string

put "Character Unicode (int) UTF-8 (hex) Decoded"

repeat with c in chars

ba = bytearray(c)

put col(c, 12) & col(charToNum(c), 16) & col(ba.toHexString(1, ba.length), 14) & ba.readRawString(ba.length)

Helper function for table formatting

str = string(val)

repeat with i = str.length+1 to len

end repeat

return str

{{out}}

<pre>

=={{header|Lua}}==

{{works with|Lua|5.3}}

-- Accept an integer representing a codepoint.

-- Return the values of the individual octets.

end

end

{{out}}

<pre>

 

=={{header|M2000 Interpreter}}==

Module EncodeDecodeUTF8 {

a$=string$("Hello" as UTF8enc)

}

EncodeDecodeUTF8

{{out}}

<pre>

 

=={{header|Mathematica}}/{{header|Wolfram Language}}==

{{out}}

<pre>{65, 195, 182, 208, 150, 226, 130, 172}

For this purpose, using sequences or bytes is not natural. Here is a way to proceed using the module “unicode”.

 

 

const UChars = ["\u0041", "\u00F6", "\u0416", "\u20AC", "\u{1D11E}"]

r = s.toRune

# Display.

 

{{out}}

In this section, we provide two procedures to convert a Unicode code point to a UTF-8 sequence of bytes and conversely, without using the module “unicode”. We provide also a procedure to convert a sequence of bytes to a string in order to print it. The algorithm is the one used by the Go solution.

 

 

const

# Display.

echo &"""{s.toString:>5} U+{c.int.toHex(5)} {s.map(toHex).join(" ")}"""

 

{{out}}

 

=={{header|Perl}}==

use strict;

use warnings;

} split //, $utf8;

print "\n";

 

{{out}}

As requested in the task description:

 

<span style="color: #008080;">constant</span> <span style="color: #000000;">tests</span> <span style="color: #0000FF;">=</span> <span style="color: #0000FF;">{</span><span style="color: #000000;">#0041</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">#00F6</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">#0416</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">#20AC</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">#1D11E</span><span style="color: #0000FF;">}</span>

<span style="color: #7060A8;">printf</span><span style="color: #0000FF;">(</span><span style="color: #000000;">1</span><span style="color: #0000FF;">,</span><span style="color: #008000;">"#%04x -> {%s} -> {%s}\n"</span><span style="color: #0000FF;">,{</span><span style="color: #000000;">codepoint</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">hex</span><span style="color: #0000FF;">(</span><span style="color: #000000;">s</span><span style="color: #0000FF;">,</span><span style="color: #008000;">"#%02x"</span><span style="color: #0000FF;">),</span><span style="color: #000000;">hex</span><span style="color: #0000FF;">(</span><span style="color: #000000;">r</span><span style="color: #0000FF;">,</span><span style="color: #008000;">"#%04x"</span><span style="color: #0000FF;">)})</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">for</span>

 

{{out}}

 

=={{header|Processing}}==

 

Integer[] code_points = {0x0041, 0x00F6, 0x0416, 0x20AC, 0x1D11E};

tel_1 += 30; tel_2 = 50;

}

 

 

The encoding and decoding procedure are kept simple and designed to work with an array of 5 elements for input/output of the UTF-8 encoding for a single code point at a time. It was decided not to use a more elaborate example that would have been able to operate on a buffer to encode/decode more than one code point at a time.

 

 

Procedure UTF8_encode(x, Array encoded_codepoint.a(1)) ;x is codepoint to encode, the array will contain output

Print(#CRLF$ + #CRLF$ + "Press ENTER to exit"): Input()

CloseConsole()

Sample output:

<pre> Unicode UTF-8 Decoded

 

=={{header|Python}}==

#!/usr/bin/env python3

from unicodedata import name

chars = ['A', 'ö', 'Ж', '€', '𝄞']

for char in chars:

{{out}}

<pre>Character Name Unicode UTF-8 encoding (hex)

 

=={{header|Racket}}==

 

(define char-map

(map (curryr number->string 16) bites)

(bytes->string/utf-8 (list->bytes bites))

{{out}}

<pre>#\A A (41) A LATIN-CAPITAL-LETTER-A

{{works with|Rakudo|2017.02}}

Pretty much all built in to the language.

 

for < A ö Ж € 𝄞 😜 👨‍👩‍👧‍👦> -> $char {

printf " %-5s | %-43s | %6s | %-7s | %12s |%4s\n", $char, $char.uninames.join(','), $char.ords.join(' '),

('U+' X~ $char.ords».base(16)).join(' '), $char.encode('UTF8').list».base(16).Str, $char.encode('UTF8').decode;

{{out}}

<pre>Character| Name | Ordinal| Unicode | UTF-8 encoded | decoded

=={{header|Ruby}}==

 

character_arr = ["A","ö","Ж","€","𝄞"]

for c in character_arr do

puts ""

end

{{out}}

<pre>

 

=={{header|Rust}}==

let chars = vec!('A', 'ö', 'Ж', '€', '𝄞');

chars.iter().for_each(|c| {

});

}

{{out}}

<pre>

=={{header|Scala}}==

=== Imperative solution===

 

val codePoints = Seq(0x0041, 0x00F6, 0x0416, 0x20AC, 0x1D11E)

printf(s"%-${w}c %-36s %-7s %-${16 - w}s%c%n",

codePoint, Character.getName(codePoint), leftAlignedHex, s, utf8Decode(bytes))

=== Functional solution===

 

object UTF8EncodeAndDecode extends App {

 

println(s"\nSuccessfully completed without errors. [total ${scala.compat.Platform.currentTime - executionStart} ms]")

=== Composable and testable solution===

 

object UTF8EncodeAndDecode extends TheMeat with App {

 

}

 

=={{header|Seed7}}==

include "unicode.s7i";

include "console.s7i";

hex(utf8) rpad 22 <& utf8ToStri(utf8));

end for;

 

{{out}}

 

=={{header|Sidef}}==

code.chr.encode('UTF-8').bytes.map{.chr}

}

assert_eq(n, decoded.ord)

say "#{decoded} -> #{encoded}"

{{out}}

<pre>

=={{header|Swift}}==

In Swift there's a difference between UnicodeScalar, which is a single unicode code point, and Character which may consist out of multiple UnicodeScalars, usually because of combining characters.

 

func encode(_ scalar: UnicodeScalar) -> Data {

print("character: \(decoded), code point: U+\(String(scalar.value, radix: 16)), \tutf-8: \(formattedBytes)")

}

{{out}}

<pre>

=={{header|Tcl}}==

Note: Tcl can handle Unicodes only up to U+FFFD, i.e. the Basic Multilingual Plane (BMP, 16 bits wide). Therefore, the fifth test fails as expected.

set u [format %c $int]

set bytes {}

lappend res [encoder $test] -> [decoder [encoder $test]]

puts $res

<pre>

0x0041 41 -> A

While perhaps not as readable as the above, this version handles beyond-BMP codepoints by manually composing the utf-8 byte sequences and emitting raw bytes to the console. <tt>encoding convertto utf-8</tt> command still does the heavy lifting where it can.

 

scan $codepoint %llx cp

if {$cp < 0x10000} {

set utf8 [utf8 $codepoint]

puts "[format U+%04s $codepoint]\t$utf8\t[hexchars $utf8]"

 

{{out}}<pre>U+0041 A 41

 

=={{header|VBA}}==

Dim y() As Byte

Dim r As Long

Debug.Print String$(8 - Len(s), " "); s

Next cpi

A 41 41 41

ö F6 C3 B6 F6

 

=={{header|Vlang}}==

fn decode(s string) ?[]u8 {

return hex.decode(s)

println("${codepoint:-7} U+${codepoint:04X}\t${encoded:-12}\t${decoded.bytestr()}")

}

{{out}}

<pre>Char Unicode UTF-8 encoded Decoded

=={{header|Wren}}==

The utf8_decode function was translated from the Go entry.

 

var utf8_encode = Fn.new { |cp| String.fromCodePoint(cp).bytes.toList }

var uni = String.fromCodePoint(cp2)

System.print("%(Fmt.s(-11, uni)) %(Fmt.s(-37, test[0])) U+%(Fmt.s(-8, Fmt.Xz(4, cp2))) %(utf8)")

 

{{out}}

 

=={{header|zkl}}==

foreach utf,unicode_int in (T( T("\U41;",0x41), T("\Uf6;",0xf6),

T("\U416;",0x416), T("\U20AC;",0x20ac), T("\U1D11E;",0x1d11e))){

 

println("%s %s %9s %x".fmt(char,char2,"U+%x".fmt(unicode_int),utf_int));

Int.len() --> number of bytes in int. This could be hard coded because UTF-8

has a max of 6 bytes and (0x41).toBigEndian(6) --> 0x41,0,0,0,0,0 which is