Questions tagged [character-encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
573
votes
1answer
71k views

Is the use of “utf8=✓” preferable to “utf8=true”?

I have recently seen a few URIs containing the query parameter "utf8=✓". My first impression (after thinking "mmm, looks cool") was that this could be used to detect a broken character encoding. So, ...
133
votes
2answers
575k views

How to detect the encoding of a file?

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters). When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of "...
98
votes
5answers
149k views

What is the advantage of choosing ASCII encoding over UTF-8?

All characters in ASCII can be encoded using UTF-8 without an increase in storage (both requires a byte of storage). UTF-8 has the added benefit of character support beyond "ASCII-characters". If ...
65
votes
2answers
39k views

Why do so many hashed and encrypted strings end in an equals sign?

I work in C# and MSSQL and as you'd expect I store my passwords salted and hashed. When I look at the hash stored in an nvarchar column (for example the out the box aspnet membership provider). I've ...
35
votes
3answers
140k views

Why do we need to put N before strings in Microsoft SQL Server?

I'm learning T-SQL. From the examples I've seen, to insert text in a varchar() cell, I can write just the string to insert, but for nvarchar() cells, every example prefix the strings with the letter N....
31
votes
8answers
7k views

Should character encodings besides UTF-8 (and maybe UTF-16/UTF-32) be deprecated?

A pet peeve of mine is looking at so many software projects that have mountains of code for character set support. Don't get me wrong, I'm all for compatibility, and I'm happy that text editors let ...
27
votes
7answers
2k views

Is the carriage-return char considered obsolete

I wrote an open source library that parses structured data but intentionally left out carriage-return detection because I don't see the point. It adds additional complexity and overhead for little/no ...
24
votes
5answers
7k views

What issues lead people to use Japanese-specific encodings rather than Unicode?

At work I come across a lot of Japanese text files in Shift-JIS and other encodings. It causes many mojibake (unreadable character) problems for all computer users. Unicode was intended to solve this ...
17
votes
4answers
2k views

Why does UTF-8 waste several bits in its encoding

According to the Wikipedia article, UTF-8 has this format: First code Last code Bytes Byte 1 Byte 2 Byte 3 Byte 4 point point Used U+0000 U+007F 1 0xxxxxxx U+0080 U+...
16
votes
2answers
7k views

Is UTF-16 fixed-width or variable-width? Why doesn't UTF-8 have byte-order problem?

Is UTF-16 fixed-width or variable-width? I got different results from different sources: From http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF: UTF-16 stores Unicode characters in sixteen-bit ...
11
votes
3answers
5k views

Should my source code be in UTF-8?

I feel that often you don't really choose what format your code is in. I mean most of my tools in the past have decided for me. Or I haven't really even thought about it. I was using TextPad on ...
8
votes
2answers
2k views

How relevant is UTF-7 when it comes to parsing emails?

I recently implemented incoming emails for an application and boy, did I open the gates of hell? Since then every other day an email arrives that makes the app fail in a different way. One of those ...
7
votes
2answers
2k views

Should I HTML encode all output from my API?

I am creating a RESTful JSON API to access data from our website where the content is in German. A handful of the fields will return formatted HTML while most are single lines of text although they ...
6
votes
2answers
3k views

Why was ASCII needed?

With an encoding such as EBCDIC being in existence already (and being 8 bit to boot), what was the need to invent yet another encoding and a 7 bit one at that? Why was ASCII invented and what ...
6
votes
2answers
5k views

How is encoding handled correctly during copy-paste between programs?

Suppose a program A opens a text file A using encoding A to decode the file, and a program B opens a text file B using encoding B. When we copy some text from file B in program B to file A in ...
5
votes
3answers
524 views

What limitation will we face if each user-perceived character is assigned to one codepoint?

What limitations will we have if Unicode standards had decided to assign one and only one codepoint to every user-perceived character? Currently, Unicode has code-points that correspond to combining ...
4
votes
1answer
221 views

UTF-8 questions

When you encode a code point to code units based on UTF-8, then if the code point fits on 7 bits, the most significant bit is set to zero so that it tells you it is a character which is stored on 1 ...
4
votes
3answers
2k views

When should I *not* use Unicode? [duplicate]

Unicode seems that its becoming more and more ubiquitous these days if it's not already, but I have to wonder if there are any domains were Unicode isn't the best implementation choice. Are there any ...
4
votes
2answers
207 views

Is a newline convention not part of an encoding?

There are different conventions of representing the new line character in different types of OSes. Does the newline convention have nothing to do with what encoding is used? Is the newline ...
4
votes
2answers
997 views

What steps can I take to avoid character encoding issues in a web application?

In previous web applications I've built, I've had issues with users entering exotic characters into forms which get stored strangely in the database, and sometimes appear different or double-encoded ...
4
votes
4answers
16k views

What encoding is used by javax.xml.transform.Transformer? [closed]

Please can you answer a couple of questions based on the code below (excludes the try/catch blocks), which transforms input XML and XSL files into an output XSL-FO file: File xslFile = new File("...
4
votes
1answer
1k views

Is the BOM optional for UTF-16 and UTF-32?

I used to think that the BOM is optional for UTF-8, but mandatory for UTF-16 and UTF-32. But then I have read the following (in this article): Let's look just at the ones that Notepad supports. ...
3
votes
3answers
19k views

How does it matter if a character is 8 bit or 16 bit or 32 bit

Well, I am reading Programing Windows with MFC, and I came across Unicode and ASCII code characters. I understood the point of using Unicode over ASCII, but what I do not get is how and why is it ...
3
votes
2answers
16k views

I can type ⅓, ⅔ and ½ but can I type 3/3 and 2/2 using unicode? [closed]

I can type ⅓, ⅔ and ½ but can I type 3/3 and 2/2 using unicode? I know that from a mathematical point of view the fractions 2/2 = 3/3 = 1 but I am typing a list where I want to indicate that you have ...
3
votes
2answers
23k views

How many bytes can I fit into a QR code, with low error correction?

I need to fit at least 300 bytes (or much much more) into a QR code and think I can do this by mapping each byte into the associated ISO/IEC 8859-1 character located here. Since each byte (1-255) ...
3
votes
3answers
617 views

Was API hooking done as needed for Stuxnet to work? I don't think so

Caveat: I am a political science student and I have tried my level best to understand the technicalities; if I still sound naive please overlook that. In the Symantec report on Stuxnet, the authors ...
3
votes
1answer
312 views

Encoding to ASCII where original encoding of string is not known

Given that we do not know the encoding of a string what is the best way to make sure that it is transformed to say ASCII? Also in such situations we are willing to accept potential loss of data.
2
votes
2answers
2k views

Why html entity names for characters like ¥ € ¢ © ®

It make sense to use entity names for describing <a> as per shown below code. <!doctype html> <html> <head> <title> My First Webpage</title> &...
2
votes
3answers
2k views

Should I convert the whole project to UTF-8?

I am working on a highly customized shop software, based on a open-source one, written in PHP and usual web techniques (CSS, HTML, JS). I did a lot of customization in the past months/years and ...
2
votes
2answers
108 views

What's the difference between the range of characters you can use in a script and a script's encoding?

The two concepts seem equal to me, but I'm not really sure I understand encoding well enough to confirm that this is the case.
2
votes
2answers
1k views

How to detect client character encoding?

I programmed a telnet server using C as programming language but I have a problem to send characters with emphases (é, è, à ...). The character encoding is different between the telnet clients (...
2
votes
1answer
162 views

What Unicode Transformation Format is being represented when just Unicode is written?

Many programs will supply one or more of the following as file encoding formats: UTF-8, UTF-16, UTF-32 and simply Unicode. How do I know what Unicode Transformation Format Unicode is referring to? I'm ...
2
votes
2answers
7k views

What does the python Codecs module do?

I just read through the documentation on the Codecs module, but I guess my knowledge/experience of comp sci doesn't run deep enough yet for me to comprehend it. It's for dealing with encoding/...
2
votes
2answers
102 views

Compressing EBCDIC file vs UTF8

Today I went across a weird case for which I have no explanation, so here I am. I have two files with identical content, but one is encoded in UTF-8 and the other one is in IBM EBCDIC. Both of them ...
2
votes
2answers
301 views

What are the commonly confused encodings that may result in identical test data?

I'm fixing code that is using ASCIIEncoding in some places and UTF-8 encoding in other functions. Since we aren't using the UTF-8 features, all of our unit tests passed, but I want to create a ...
2
votes
2answers
769 views

What is really happening when we change encoding in a string?

http://php.net/manual/en/function.mb-convert-encoding.php Say I do: $encoded = mb_convert_encoding ($original); That looks like simple enough. WHat I am imagining is the following $original has a ...
1
vote
3answers
182 views

Is it ok to use localized character encodings for code?

Is it ok to use a localized encoding (i.e. ISO-8859-15) instead of ANSI/UTF-8/some standard for code? What when you have names that do not translate very well into english code? Or when your company ...
1
vote
1answer
762 views

What encoding are the HTTP status and header lines?

If I was going to write a parser for HTTP, would I be able to assume the encoding of the HTTP headers and status line? Until I read the charset or encoding header, how could I tell what the encoding ...
1
vote
3answers
208 views

Does printing out numbers involve converting the numbers into characters?

In programming languages, when numbers (either integer, or real) are printed out, are they firstly converted to the codes of the readable characters that are meant to represent the numbers, and then ...
1
vote
3answers
6k views

Is path in Set-Cookie URL encoded?

I'm writing some code that sets cookies and I'm wondering about the exact semantics of the Set-Cookie header. Imagine the following HTTP header line: Set-Cookie: name=value; Path=/%20 For with path ...
1
vote
2answers
2k views

When is it beneficial to not use utf-8? [duplicate]

When is it beneficial to use encodings other than UTF-8? Aside from dealing with pre-unicode documents, that is. And more importantly, why isn't UTF-8 the default in most languages? That is, why do I ...
1
vote
1answer
249 views

How to cross-reference many character encodings with ASCII OR UTFx?

I'm working with a binary structure, the goal of which is to index the significance of specific bits for any character encoding so that we may trigger events while doing specific checks against the ...
1
vote
1answer
66 views

Does the SHA256 hashing algorithm change based on the content encoding?

I am starting to look into how to implement SHA256 in JavaScript, and found this for example. It requires UTF-8 encoding it sounds like. Another one I saw required/supported only ASCII encoding and ...
1
vote
2answers
75 views

In python, what or who is character encoding information for?

If you go to www.htmlbasictutor.ca/character-encoding.htm you will find the following description of character encoding. Character encoding tells the browser and validator what set of characters to ...
1
vote
1answer
141 views

Differentiating Between ASCII and Unicode in File Spec

I am developing against a file spec that lists the data type for certain fields as CHAR(<length>) The spec is for a fixed width flat file. In most cases, possible values to populate the fields ...
1
vote
1answer
44 views

Layout Behavior of Characters (question about unicode standard)

I've been reading Unicode's core specification (see https://www.unicode.org/versions/latest/). I mostly understood what the text was explaining in section 2.1 Architectural Context until it started ...
1
vote
1answer
97 views

Create and implement new encoding

I'm working on a project with huge files that contain only the set {[0-9],.}. Encoding in UTF-8 or ASCII make huge files. I wonder if I could find a way to encode in only 4 bits (make those file 16 ...
1
vote
1answer
97 views

String encoding in string instances from different languages

I recently got a requirement to develop a chat-like application, or rather, a foundation of classes and methods that would allow certain applications to have chat-like features. The framework must be ...
0
votes
2answers
756 views

Why Unicode Encoding/Decoding is Necessary in JavaScript

I am wondering why unicode encoding is necessary in JavaScript. I am looking at utf8.js as an example. I am also looking at the utf8 spec, but am not really following the different pieces of data. ...
0
votes
1answer
456 views

Java takes 2 bytes to represent character?

In general a character is represented in 1 byte i.e. 8 bits . This is I believe true for all text editors even for databases like oracle. 1 byte can represent 2^8 = 256 Characters. My question is when ...