Bytes Encoding
We always hear that data is stored in binary, zeros and ones. Still, it isn't clear how this binary is interpreted and shown as strings or other readable data.
Encoding and Decoding
The data stored in files is already byte array. The renderer has the metadata to understand which encoding is used. It uses this information to decode the bytes into characters and show them as strings.
When you save a file, the renderer encodes the string into bytes using the encoding specified in the metadata.
When we define variables, the files are encoded and stored in the programming language's default encoding. Same applies to files generated by compiler as well.
The java compiler already stores encoded strings in the class files. If we open the file using javap then we will see the actual string. This is just the rendering logic of the tool.
Byte Data Representation
Bytes are just 8-bit binary data. When you run programs, the debugger doesn't show the raw binary. It shows the decimal values.
How applications send bytes?
An app sends an HTTP request or response. The headers are joined into one string, split by new lines. New line is also a Unicode character. Then each char is just encoded to UTF-8 and sent out.
Same process will be applied for request/response body.
Any data coming into the app is always binary. It's decoded by whatever encoding was used to send it.
Byte Order
When bytes move between systems, the receiver must know the order to process the bytes of a character.
Byte order is about the order of bytes in a multibyte value. If a Unicode character is many bytes, byte order says which byte is first.
Byte order has two forms.
- Big Endian - Means the bits are processed from left to right. Like the usual way of reading.
- Little Endian - Here the bits are processed from right to left. The opposite way of reading.
the Big-Endian / Little-Endian naming comes from Gulliver's Travels, where the Lilliputians argue over whether to break eggs on the little-end or big-end.
Character Boundary
When a full string is streamed as UTF-8 data, the receiver must know which bytes go with which character. This matters because UTF-8 uses multiple bytes to support all languages.
The first few bits of a byte tell the decoder how many bytes, with the current one, form a character. No extra byte marks the start of a new character.
0xxxxxxx: A single-byte character. 110xxxxx: A two-byte character. 1110xxxx: A three-byte character. 11110xxx: A four-byte character.