30
votes
Accepted
Why does UTF-8 waste several bits in its encoding
This is done so that you can detect when you are in the middle of a multi-byte sequence. When looking at UTF-8 data, you know that if you see 10xxxxxx, that you are in the middle of a multibyte ...
- 14.6k
14
votes
Accepted
What is the encoding used in Git's binary patches?
Aha, it's RFC1924's version of the base85 encoding, which uses 5 ASCII characters to represent 4 bytes (80% efficiency):
static const char en85[] = {
'0', '1', '2', '3', '4', '5', '6', '7', '8', '...
- 337
10
votes
Accepted
UTF-8 questions
Answer to Question 1: why is it that the first 2 bits of every continuation byte are set to “10”?
It lets you land at a random place in the sequence and unambiguously work back to the beginning of ...
- 1,035
7
votes
Accepted
Why does ISO 8859-1 contain letter-free diacritics?
Note: when some important missing characters (such as the Euro symbol €) were added to the character set to create ISO8859-15, some mostly unused characters had to go, and this included the letter-...
- 97.6k
6
votes
Why does UTF-8 waste several bits in its encoding
The official way lets the decoder know when it's in the middle of the tuple and it knows to skip bytes (or go backwards) until the byte starts with 0 or 11; this prevents garbage values when a single ...
- 25.5k
6
votes
Accepted
When is it beneficial to not use utf-8?
For an external encoding (i.e., an encoding of things not inside your program) it is very hard to beat UTF-8; it supports every character your users might ever reasonably need and there's lots of ...
- 6,317
4
votes
Why does ISO 8859-1 contain letter-free diacritics?
ISO based Latin-1 on ECMA-094, which based it on the DEC Multinational Character Set so Europeans could use the DEC VT220. The first 128 code points of every 8-bit character set had to be the same as ...
- 1,513
3
votes
Compressing EBCDIC file vs UTF8
First thoughts
EBCDIC is a format with an 8 bit encoding. The basic EBCDIC character set has plenty of unused space, so not all the potential of the 8 bits is used. But it has codepages for handling ...
- 66.8k
3
votes
Accepted
Name of data structure where a string determines its value
Type-Length-Value or Tag-Length-Value.
Both protocols describe a mechanism for encoding optional information in multiple tuplets of variable length into text-based messages, using three fields: type, ...
- 195k
3
votes
Why does UTF-8 waste several bits in its encoding
Short answer, your proposal does not differentiate between the first byte and continuation bytes.
The bit pattern at the high end of the first byte tells you with how many bytes the actual character ...
- 159
2
votes
How is encoding handled correctly during copy-paste between programs?
The simplest solution is to use a standard encoding. For example, in Windows, one standard encoding is "unicode", which refers to UTF-16, the encoding recommended for Windows applications. The ...
- 3,864
2
votes
Accepted
Fast and simple hex compression
The current protocol encodes the data into hex
What protocol? TCP certainly doesn't do that.
I believe you're actually referring to binary to text encoding. This is a method of converting binary ...
- 88.3k
2
votes
Why does UTF-8 waste several bits in its encoding
What hasn't been mentioned is that if you have a correct sequence of code points, and a pointer that is guaranteed to point to the first byte of a code point, with UTF-8 you can very easily find the ...
- 37.4k
2
votes
Reduce number of digits by converting to alphanumeric data
Encode the number as base 64 using a fixed size representation and restrict one of the characters to be non-numeric to prevent collisions with existing identifiers. Base 64 uses 'A'-'Z' to represent 0-...
- 151
2
votes
Reduce number of digits by converting to alphanumeric data
This is precisely why storing business defined identifiers as database identifiers becomes problematic. The business changes their rules, and this forces downstream changes in databases.
If you were ...
- 29k
1
vote
Accepted
How to identify encoding of a text string?
@MartinGrey, I think your question is perfectly sensible, but it's like Bob says in a comment, your misconception is in the idea that handling arbitrary data in arbitrary encodings, and decoding it ...
- 3,948
1
vote
Reduce number of digits by converting to alphanumeric data
Unclear elements
It would be useful to clarify the following points:
Is the - currently stored in the string ?
Is the length control on the client side based on the visible string or on the ...
- 66.8k
1
vote
Accepted
Reduce number of digits by converting to alphanumeric data
When this really solves the problem, why not? Sometimes, a pragmatic solution with some "duct tape" is required to get things done, even if it does not look beautiful from a design perspective.
...
- 184k
1
vote
Accepted
Dictionary Coding for OLAP
Let us start with your "products" as a representative example: OLAP databases which involve products are typically used to create queries like
which are the top 10 selling products over the last 3 ...
- 184k
1
vote
Windows compatibility with Unix/Linux newline "\n"
This question is really about a software application's "customer base".
To answer your question, you have to know whether your customers might be inconvenienced if your application generates output ...
- 16.4k
1
vote
Windows compatibility with Unix/Linux newline "\n"
As far as Windows and C# is involved you can always use the Environment.Newline
to determine the default new line character of the system the program is ran on.
also, you can use text.Replace("\n",...
- 139
1
vote
Windows compatibility with Unix/Linux newline "\n"
Windows Notepad (notepad.exe) doesn't interpret a standalone \n as a new line. It's not necessarily "modern" but pretty much "mainstream".
If you're writing text files, every day user should be able ...
- 1,326
1
vote
How is encoding handled correctly during copy-paste between programs?
A program transferring text can do one of three things: Transfer without telling which encoding is used, transfer using a standard encoding, or transfer while specifying the encoding.
The receiving ...
- 37.4k
1
vote
cross-application text formatting
Some applications, among which LibreOffice at Linux, use multiple clipboards if I'm not mistaken.
The clipboard is the place where things are temporarily stored when pressing CTL+C, and of ...
- 191
1
vote
When is it beneficial to not use utf-8?
The answer is that UTF-8 is by far the best general-purpose data interchange encoding, and is almost mandatory if you are using any of the other protocols that build on it (mail, XML, HTML, etc).
...
- 8,075
Only top scored, non community-wiki answers of a minimum length are eligible