30 votes
Accepted

Why does UTF-8 waste several bits in its encoding

This is done so that you can detect when you are in the middle of a multi-byte sequence. When looking at UTF-8 data, you know that if you see 10xxxxxx, that you are in the middle of a multibyte ...
user avatar
14 votes
Accepted

What is the encoding used in Git's binary patches?

Aha, it's RFC1924's version of the base85 encoding, which uses 5 ASCII characters to represent 4 bytes (80% efficiency): static const char en85[] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '...
user avatar
10 votes
Accepted

UTF-8 questions

Answer to Question 1: why is it that the first 2 bits of every continuation byte are set to “10”? It lets you land at a random place in the sequence and unambiguously work back to the beginning of ...
user avatar
  • 1,035
7 votes
Accepted

Why does ISO 8859-1 contain letter-free diacritics?

Note: when some important missing characters (such as the Euro symbol €) were added to the character set to create ISO8859-15, some mostly unused characters had to go, and this included the letter-...
user avatar
6 votes

Why does UTF-8 waste several bits in its encoding

The official way lets the decoder know when it's in the middle of the tuple and it knows to skip bytes (or go backwards) until the byte starts with 0 or 11; this prevents garbage values when a single ...
user avatar
6 votes
Accepted

When is it beneficial to not use utf-8?

For an external encoding (i.e., an encoding of things not inside your program) it is very hard to beat UTF-8; it supports every character your users might ever reasonably need and there's lots of ...
user avatar
4 votes

Why does ISO 8859-1 contain letter-free diacritics?

ISO based Latin-1 on ECMA-094, which based it on the DEC Multinational Character Set so Europeans could use the DEC VT220. The first 128 code points of every 8-bit character set had to be the same as ...
user avatar
  • 1,513
3 votes

Compressing EBCDIC file vs UTF8

First thoughts EBCDIC is a format with an 8 bit encoding. The basic EBCDIC character set has plenty of unused space, so not all the potential of the 8 bits is used. But it has codepages for handling ...
user avatar
  • 66.8k
3 votes
Accepted

Name of data structure where a string determines its value

Type-Length-Value or Tag-Length-Value. Both protocols describe a mechanism for encoding optional information in multiple tuplets of variable length into text-based messages, using three fields: type, ...
user avatar
3 votes

Why does UTF-8 waste several bits in its encoding

Short answer, your proposal does not differentiate between the first byte and continuation bytes. The bit pattern at the high end of the first byte tells you with how many bytes the actual character ...
user avatar
  • 159
2 votes

How is encoding handled correctly during copy-paste between programs?

The simplest solution is to use a standard encoding. For example, in Windows, one standard encoding is "unicode", which refers to UTF-16, the encoding recommended for Windows applications. The ...
user avatar
2 votes
Accepted

Fast and simple hex compression

The current protocol encodes the data into hex What protocol? TCP certainly doesn't do that. I believe you're actually referring to binary to text encoding. This is a method of converting binary ...
user avatar
2 votes

Why does UTF-8 waste several bits in its encoding

What hasn't been mentioned is that if you have a correct sequence of code points, and a pointer that is guaranteed to point to the first byte of a code point, with UTF-8 you can very easily find the ...
user avatar
  • 37.4k
2 votes

Reduce number of digits by converting to alphanumeric data

Encode the number as base 64 using a fixed size representation and restrict one of the characters to be non-numeric to prevent collisions with existing identifiers. Base 64 uses 'A'-'Z' to represent 0-...
user avatar
  • 151
2 votes

Reduce number of digits by converting to alphanumeric data

This is precisely why storing business defined identifiers as database identifiers becomes problematic. The business changes their rules, and this forces downstream changes in databases. If you were ...
user avatar
1 vote
Accepted

How to identify encoding of a text string?

@MartinGrey, I think your question is perfectly sensible, but it's like Bob says in a comment, your misconception is in the idea that handling arbitrary data in arbitrary encodings, and decoding it ...
user avatar
  • 3,948
1 vote

Reduce number of digits by converting to alphanumeric data

Unclear elements It would be useful to clarify the following points: Is the - currently stored in the string ? Is the length control on the client side based on the visible string or on the ...
user avatar
  • 66.8k
1 vote
Accepted

Reduce number of digits by converting to alphanumeric data

When this really solves the problem, why not? Sometimes, a pragmatic solution with some "duct tape" is required to get things done, even if it does not look beautiful from a design perspective. ...
user avatar
  • 184k
1 vote
Accepted

Dictionary Coding for OLAP

Let us start with your "products" as a representative example: OLAP databases which involve products are typically used to create queries like which are the top 10 selling products over the last 3 ...
user avatar
  • 184k
1 vote

Windows compatibility with Unix/Linux newline "\n"

This question is really about a software application's "customer base". To answer your question, you have to know whether your customers might be inconvenienced if your application generates output ...
user avatar
  • 16.4k
1 vote

Windows compatibility with Unix/Linux newline "\n"

As far as Windows and C# is involved you can always use the Environment.Newline to determine the default new line character of the system the program is ran on. also, you can use text.Replace("\n",...
user avatar
1 vote

Windows compatibility with Unix/Linux newline "\n"

Windows Notepad (notepad.exe) doesn't interpret a standalone \n as a new line. It's not necessarily "modern" but pretty much "mainstream". If you're writing text files, every day user should be able ...
user avatar
  • 1,326
1 vote

How is encoding handled correctly during copy-paste between programs?

A program transferring text can do one of three things: Transfer without telling which encoding is used, transfer using a standard encoding, or transfer while specifying the encoding. The receiving ...
user avatar
  • 37.4k
1 vote

cross-application text formatting

Some applications, among which LibreOffice at Linux, use multiple clipboards if I'm not mistaken. The clipboard is the place where things are temporarily stored when pressing CTL+C, and of ...
user avatar
1 vote

When is it beneficial to not use utf-8?

The answer is that UTF-8 is by far the best general-purpose data interchange encoding, and is almost mandatory if you are using any of the other protocols that build on it (mail, XML, HTML, etc). ...
user avatar
  • 8,075

Only top scored, non community-wiki answers of a minimum length are eligible