28
The final '=' or '==' in Base64 is there only to make the number of characters a multiple of 4. You can remove it, since you can always put it back later on. Note that Base64 is so called because it uses 64 distinct characters. Uppercase letters, lowercase letters, and digits, that's 62. So Base64 also uses '/' and '+', which may or may not fit your bill.
...
25
This is done so that you can detect when you are in the middle of a multi-byte sequence. When looking at UTF-8 data, you know that if you see 10xxxxxx, that you are in the middle of a multibyte character, and should back up in the stream until you see either 0xxxxxx or 11xxxxxx. Using your scheme, bytes 2 or 3 could easily end up with patters like either ...
10
The only relevant feature of UTF-7 (over UTF-8 for example) is that it's a 7-bit encoding, just like good old ASCII is. That means that it works over a system that is not 8-bit clean.
The only large-scale system where this even matters today is mail servers (don't ask me why they didn't fix this problem 10-20 years ago, most servers did, but some ostensibly ...
7
Thanks to Charles Salvia's comment, I found a method in the IMAP module that helped:
require "net/imap"
Net::IMAP.decode_utf7(mail_body)
7
Note: when some important missing characters (such as the Euro symbol €) were added to the character set to create ISO8859-15, some mostly unused characters had to go, and this included the letter-free diacritics. So, the designers of ISO8859-1 may have been very smart people and may have had good reasons, but apparently nobody understood them!
However, ...
6
Aha, it's RFC1924's version of the base85 encoding, which uses 5 ASCII characters to represent 4 bytes (80% efficiency):
static const char en85[] = {
'0', '1', '2', '3', '4', '5', '6', '7', '8', '9',
'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J',
'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T',
'U', 'V', 'W', 'X', 'Y', 'Z',
'a', 'b',...
6
The official way lets the decoder know when it's in the middle of the tuple and it knows to skip bytes (or go backwards) until the byte starts with 0 or 11; this prevents garbage values when a single byte gets corrupted.
6
For an external encoding (i.e., an encoding of things not inside your program) it is very hard to beat UTF-8; it supports every character your users might ever reasonably need and there's lots of support in many OSes and tools. (The one place that counts as an exception to this is in file names, where you must use the platform's conventions if you want any ...
4
XML is defined to be UTF-8 or UTF-16 with BOM in the absence of an explicit encoding in the prologue: http://www.w3.org/TR/REC-xml/#charencoding
If you're reading/writing an XML file using a standard parser/serializer, you should always use FileInputStream / FileOutputStream and rely on the library doing the right thing.
If you're getting input from an ...
4
VDPAU only accelerates the decoding/playing of video. I don't know if the time taken to copy the raw video to the GPU encode, and then read back the compressed data would really mean there was much benefit from a GPU.
With playback you are forced to copy the data to the GPU anyway and it's obvisuly better to copy the smaller compressed source stream.
There ...
4
Why are you using GET in the first place? Just POST the sensitive data over HTTPS, and add suitable CSRF protection.
This is assuming that the information is sensitive in the sense that the authorized user and the server may see it, but others (other users and attackers) must not.
If the user isn't allowed to see the information either, then just don't ...
4
Re-encoding characters is generally done when the receiving system can't process them. For example, BASE64 is representing data using 6 bits (26, hence 64) of characters to represent longer data sequences (the sometimes-appearing "==" at the end is padding for alignment). This is because your picture file in email may have 0xFE in it and your mail server ...
4
ISO based Latin-1 on ECMA-094, which based it on the DEC Multinational Character Set so Europeans could use the DEC VT220. The first 128 code points of every 8-bit character set had to be the same as ASCII for backward-compatibility. Indeed, back in the bad old days, misconfigured network hardware often interpreted the high bit as an error-correction code ...
3
Short answer, your proposal does not differentiate between the first byte and continuation bytes.
The bit pattern at the high end of the first byte tells you with how many bytes the actual character is built. These patterns provide also some error recognition while parsing a string. If you are reading the (seemingly) first byte of a character and you get ...
3
Type-Length-Value or Tag-Length-Value.
Both protocols describe a mechanism for encoding optional information in multiple tuplets of variable length into text-based messages, using three fields: type, length and value. The type and length are fixed in size (typically 1-4 bytes), and the value field is of variable size. These fields are used as follows:
...
2
You can compress the data with e.g. gzip, bzip2 or lzma and then run through base64 to limit the used character set. This is beneficial only on larger strings of hundreds of bytes or more.
2
The rationale behind UTF-32 is simple: It's the most straightforward representation of Unicode code points. So why isn't everything in UTF-32? Two main reasons:
One is size. UTF-32 requires 4 bytes for every character. For text that uses only characters in the Basic Multilingual Place, this is twice as much space as UTF-16. For English text, it's 4 ...
2
A string isn't a bag of bytes and a specification of the encoding : in most languages a string is a bag of bytes, and in best cases an implicit encoding defined by the language. It's your responsability as a coder to know what to do depending on the encoding (edit : as tdhammer pointed, most modern language explicitly forbids manipulating strings in other ...
2
If you know that the encoding is always going to be UTF-8, you could use the InputStreamReader class which allow you to specify an encoding, and pass those instances to StreamSource. Otherwise, I'd expect the system to use the encoding specified in the XML file, and if it's not specified then either use the system default encoding or make a best guess (that ...
answered May 31 '13 at 14:07
Michael Borgwardt
44.5k11 gold badges106 silver badges159 bronze badges
2
What hasn't been mentioned is that if you have a correct sequence of code points, and a pointer that is guaranteed to point to the first byte of a code point, with UTF-8 you can very easily find the pointer to the first byte of the previous code point (skip all bytes that start with 01xx xxxx). With your encoding, it's impossible without potentially ...
2
The simplest solution is to use a standard encoding. For example, in Windows, one standard encoding is "unicode", which refers to UTF-16, the encoding recommended for Windows applications. The programs which accept clipboard input have to be able to interpret the encoding. This is all documented on MSDN.
Unicode (Windows)
Standard Clipboard Formats
2
The current protocol encodes the data into hex
What protocol? TCP certainly doesn't do that.
I believe you're actually referring to binary to text encoding. This is a method of converting binary data to textual data so that it can be sent over systems that only allow text.
Hex itself is just a way to present a number for viewing, not really meant for ...
2
Text compression algorithms
Word based compression
You already explained yourself the basic principles of a dictionary compression that works by word. However interesting article that you referred to is about optimization of this process.
With your example of 62 characters original text broken down into 25 word example (counting spaces), you found that ...
2
Let us start with your "products" as a representative example: OLAP databases which involve products are typically used to create queries like
which are the top 10 selling products over the last 3 months?
which 10 products brought us the most revenue over the last year?
in which locations did we have tho most shortage in delivery of certain products?
All ...
1
As far as Windows and C# is involved you can always use the Environment.Newline
to determine the default new line character of the system the program is ran on.
also, you can use text.Replace("\n","\r\n") to switch to windows return.
There are still compatibility issues when managing files and especially office related ones, some arcane COM apis are ...
1
Windows Notepad (notepad.exe) doesn't interpret a standalone \n as a new line. It's not necessarily "modern" but pretty much "mainstream".
If you're writing text files, every day user should be able to edit, don't focus on \n only and instead write your program in a way to accept all three styles (since some older programs might even use \r only).
1
Characters are one of the most confusing things in computer science. This and the reason that computer science was mostly starting in English speaking countries which adopted the ASCII (American Standard Code for Information Interchange) are responsible for the miserable existence characters have to live in a computer. Chinese and emojis were not in question ...
1
Some applications, among which LibreOffice at Linux, use multiple clipboards if I'm not mistaken.
The clipboard is the place where things are temporarily stored when pressing CTL+C, and of LibreOffice I know it saves it in different clipboards, in different formats. These formats vary from plain-text to rich and might even include images.
There's also a ...
1
The answer is that UTF-8 is by far the best general-purpose data interchange encoding, and is almost mandatory if you are using any of the other protocols that build on it (mail, XML, HTML, etc).
However, UTF-8 is a multi-byte encoding and relatively new, so there are lots of situations where it is a poor choice. Here are a few.
Internal encoding in ...
Only top voted, non community-wiki answers of a minimum length are eligible
