Questions tagged [text-encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-3
votes
1answer
103 views

How to identify encoding of a text string? [closed]

I guess most of you already met them. You get them from your data sources, see them in your logs, or in the output from your legacy systems. Some strings you can't really read. To derive any useful ...
2
votes
2answers
80 views

Compressing EBCDIC file vs UTF8

Today I went across a weird case for which I have no explanation, so here I am. I have two files with identical content, but one is encoded in UTF-8 and the other one is in IBM EBCDIC. Both of them ...
4
votes
1answer
215 views

UTF-8 questions

When you encode a code point to code units based on UTF-8, then if the code point fits on 7 bits, the most significant bit is set to zero so that it tells you it is a character which is stored on 1 ...
2
votes
4answers
224 views

Reduce number of digits by converting to alphanumeric data

We have an app that receives a web service request, processes it and sends it back to our client by another web service call. There is a unique field in the request, a tracking Id, which currently ...
0
votes
1answer
78 views

Name of data structure where a string determines its value

A few times in my career I've found myself writing decoders for responses from IoT products or weird apis that insted of using JSON or XML as a response, they reply with something like ...
1
vote
1answer
59 views

Dictionary Coding for OLAP

Let's say we have a generic table like below: id, name, price, quantity 20 product_x 5,00 100 20 product_y 5,00 100 20 ...
1
vote
3answers
280 views

Algorithm for optimizing text compression

I am looking for text compression algorithms (natural language compression, rather than compression of arbitrary binary data). I have seen for example An Efficient Compression Code for Text ...
3
votes
4answers
2k views

Windows compatibility with Unix/Linux newline “\n”

A follow-up to Difference between '\n' and '\r\n'. It's been few decades since the schism was introduced. Nowadays, when documents are being exchanged over the internet, typically ...
3
votes
1answer
949 views

What is the encoding used in Git's binary patches?

Git can generate patches/diffs for binary files as well as for text files. I'm trying to figure out what encoding it uses for its binary patches. Here is an example: diff --git a/www/images/...
-1
votes
2answers
120 views

How to find charset name by byte codes?

I have a printer and SDK to work with it in Java. Printer working well with english letters and digits but doesn't print correctly special symbols like 'ä' or 'ê'. I suppose that I need to convert ...
-1
votes
2answers
3k views

Fast and simple hex compression

I'm working on a project that requires a TCP connection between a client and server. The current protocol encodes the data into hex and then sends it. However, hex increases the length of the payload ...
0
votes
1answer
643 views

What type of encoding does wikipedia use?

I have the wikipedia data dump and trying to decode special characters in the page titles, except a lot of characters don't match up the "standard" ascii encoding (referencing from here.) As an ...
4
votes
2answers
209 views

Why does ISO 8859-1 contain letter-free diacritics?

ISO 8859-1 contains a few letter-free diacritics: The diaeresis (¨), the acute accent (´), the cedilla (¸) and the macron (¯).¹ Why were they included? As far as I know (please correct me if I am ...
6
votes
2answers
4k views

How is encoding handled correctly during copy-paste between programs?

Suppose a program A opens a text file A using encoding A to decode the file, and a program B opens a text file B using encoding B. When we copy some text from file B in program B to file A in ...
1
vote
2answers
55 views

cross-application text formatting

I am not sure whether this question is a good fit for this site, but if it is not, please let me know and I will take it down. If it is off-topic, some general info on where I can look for these ...
17
votes
4answers
2k views

Why does UTF-8 waste several bits in its encoding

According to the Wikipedia article, UTF-8 has this format: First code Last code Bytes Byte 1 Byte 2 Byte 3 Byte 4 point point Used U+0000 U+007F 1 0xxxxxxx U+0080 U+...
0
votes
1answer
4k views

VB.NET - Serial Key Generating (Encryption, Hashing, Encoding?) Algorithm

I am trying to create a basic licensing system where I take a unique ID from the client computer, and I get this Hexadecimal string (hyphens removed e.g. "84-18-CE-...."): "...
1
vote
2answers
2k views

When is it beneficial to not use utf-8? [duplicate]

When is it beneficial to use encodings other than UTF-8? Aside from dealing with pre-unicode documents, that is. And more importantly, why isn't UTF-8 the default in most languages? That is, why do I ...
4
votes
4answers
15k views

What encoding is used by javax.xml.transform.Transformer? [closed]

Please can you answer a couple of questions based on the code below (excludes the try/catch blocks), which transforms input XML and XSL files into an output XSL-FO file: File xslFile = new File("...
7
votes
2answers
16k views

Does FFMpeg support gpu acceleration of media encoding/decoding?

I was wondering if ffmpeg supported gpu acceleration. I was reading on their websites and came across contradicting information. http://www.ffmpeg.org/general.html#Video-Codecs -H.264 / AVC / MPEG-4 ...
3
votes
2answers
1k views

Unable to debug an encodded javascript?

I’m having some problems debugging an encoded javacscript. This script I’m referring to given in this link over here. The encoding here is simple and it works by shifting the unicodes values to ...
8
votes
2answers
2k views

How relevant is UTF-7 when it comes to parsing emails?

I recently implemented incoming emails for an application and boy, did I open the gates of hell? Since then every other day an email arrives that makes the app fail in a different way. One of those ...
2
votes
2answers
759 views

What is really happening when we change encoding in a string?

http://php.net/manual/en/function.mb-convert-encoding.php Say I do: $encoded = mb_convert_encoding ($original); That looks like simple enough. WHat I am imagining is the following $original has a ...
4
votes
3answers
3k views

URL Encryption vs. Encoding

At the moment non/semi sensitive information is sent from one page to another via GET on our web application. Such as user ID or page number requested etc. Sometimes slightly more sensitive ...
13
votes
4answers
22k views

What type of encoding can I use to make a string shorter?

I am interested in encoding a string I have and I am curious if there is a type of encoding that can be used that will only include alpha and numeric characters and would preferably shorten the number ...
41
votes
8answers
3k views

Why are there multiple Unicode encodings?

I thought Unicode was designed to get around the whole issue of having lots of different encoding due to a small address space (8 bits) in most of the prior attempts (ASCII, etc.). Why then are there ...