Posts

Showing posts with the label utf-8

How does UTF-8 encoding know a character is one byte or two bytes

  That’s not how UTF-8 works. Characters U+0000 through U+007F (aka ASCII) are stored as single bytes. They are the only characters whose codepoints numerically match their UTF-8 presentation. For example, U+0041 becomes  0x41  which is  01000001  in binary. All other characters are represented with multiple bytes. U+0080 through U+07FF use two bytes each, U+0800 through U+FFFF use three bytes each, and U+10000 through U+10FFFF use four bytes each. Computers know where one character ends and the next one starts because UTF-8 was designed so that the single-byte values used for ASCII do not overlap with those used in multi-byte sequences. The bytes  0x00  through  0x7F  are only used for ASCII and nothing else; the bytes above  0x7F  are only used for multi-byte sequences and nothing else. Furthermore, the bytes that are used at the beginning of the multi-byte sequences also cannot occur in any other position in those sequences. Beca...