Questions tagged [unicode]
Unicode is intended to be a universal character set for describing all the characters required for written text incorporating all writing systems, technical symbols and punctuation.
2
votes
1answer
83 views
Differentiating Between ASCII and Unicode in File Spec
I am developing against a file spec that lists the data type for certain fields as
CHAR(<length>)
The spec is for a fixed width flat file. In most cases, possible values to populate the fields ...
2
votes
3answers
427 views
How to align on both word size and cache lines in x86
From what it sounds like, a 64 bit processor means aligning to 64 bits, which means if you have unicode utf-8 stored in there, each 8-bit chunk would take up 64 bits of space. That doesn't really make ...
-1
votes
2answers
220 views
How does MS word renders different fonts?
My main goal is described here.
How can Microsoft Word or Wordpad or other word editing software render fonts when these fonts seems to not follow the same rules?
How do they render characters ...
0
votes
2answers
333 views
Why Unicode Encoding/Decoding is Necessary in JavaScript
I am wondering why unicode encoding is necessary in JavaScript. I am looking at utf8.js as an example. I am also looking at the utf8 spec, but am not really following the different pieces of data. ...
0
votes
1answer
196 views
Java takes 2 bytes to represent character?
In general a character is represented in 1 byte i.e. 8 bits . This is I believe true for all text editors even for databases like oracle. 1 byte
can represent 2^8 = 256 Characters.
My question is when ...
37
votes
2answers
6k views
Should UTF-8 CSV files contain a BOM (byte order mark)?
Our line-of-business software allows the user to save certain data as CSV. Since there are a lot of different formats (all called "CSV") in use in the wild, we are tying to decide what the "default ...
4
votes
1answer
345 views
Is the BOM optional for UTF-16 and UTF-32?
I used to think that the BOM is optional for UTF-8, but mandatory for UTF-16 and UTF-32.
But then I have read the following (in this article):
Let's look just at the ones that Notepad supports.
...
3
votes
3answers
501 views
Why does Unicode have separate codepoints for characters with identical glyphs?
(Not entirely sure whether this should go in the information-security StackExchange instead; feel free to move it there if that's where it belongs.)
Unicode has many, many instances of pairs or ...
1
vote
1answer
239 views
Unicode Telugu language characters
I am developing a mobile app in android in which I use Telugu (Indian language) texts. On my mobile Telugu language alphabets are available. Therefore, I am not facing any problem for testing my app. ...
8
votes
1answer
322 views
Do C++'s iterator categories forbid writing a UTF-8 iterator adapter?
I've been working on a UTF-8 iterator adapter. By which, I mean an adapter that turns an iterator to a char or unsigned char sequence into an iterator to a char32_t sequence. My work here was inspired ...
0
votes
2answers
4k views
I can type ⅓, ⅔ and ½ but can I type 3/3 and 2/2 using unicode? [closed]
I can type ⅓, ⅔ and ½ but can I type 3/3 and 2/2 using unicode? I know that from a mathematical point of view the fractions 2/2 = 3/3 = 1 but I am typing a list where I want to indicate that you have ...
2
votes
1answer
130 views
What Unicode Transformation Format is being represented when just Unicode is written?
Many programs will supply one or more of the following as file encoding formats: UTF-8, UTF-16, UTF-32 and simply Unicode. How do I know what Unicode Transformation Format Unicode is referring to? I'm ...
7
votes
3answers
2k views
Compiling for string and wstring
I'm creating a library. I want to use it in multiple projects which may use multi-byte or unicode (std::string or std::wstring). I've adopted the old MS method of conditional compiling:
namespace ...
1
vote
1answer
377 views
Cross-platform unicode support for Python command line tool
I am working on a large command line tool, written for Python 2.6+ and supported for Windows, OS X and Linux. The target users are developers but it is also being auto-invoked by CI-systems etc. In ...
8
votes
1answer
554 views
How do you mix left-to-right and right-to-left scripts without your files looking crazy?
Say your native language is Hebrew, and you're working in a programming language like Python 3, which lets you put Hebrew in source code. Good for you! You've got a dict:
d = {'a': 1}
and you want ...
86
votes
5answers
9k views
Would UTF-8 be able to support the inclusion of a vast alien language with millions of new characters?
In the event an alien invasion occurred and we were forced to support their languages in all of our existing computer systems, is UTF-8 designed in a way to allow for their possibly vast amount of ...
4
votes
2answers
196 views
Why does ISO 8859-1 contain letter-free diacritics?
ISO 8859-1 contains a few letter-free diacritics: The diaeresis (¨), the acute accent (´), the cedilla (¸) and the macron (¯).¹
Why were they included? As far as I know (please correct me if I am ...
1
vote
1answer
4k views
What is the difference between “Wide character” set and “Unicode character set”? [closed]
Today I was reading my favourite C++ Programming book (C++ Primer Plus) and the section which was about variables and character sets in C++,
however I got really confused about Unicode and Wide ...
0
votes
0answers
75 views
How can I resolve Unicode Hex Value Mismatches between WordML and XSL:FO?
We have an important legal document that our app generates in WordML, with foreign characters represented via Unicode. These foreign characters vary widely, and include languages with special ...
1
vote
2answers
249 views
Is there accepted decimal-based Unicode notation for technical audiences?
When writing for technical audiences, there are various ways to type Unicode representations, but they all seem to be Hexadecimal:
\uFFFF - From C# / Java Strings
\U0000FFFF - From C# / Java Strings
...
1
vote
2answers
1k views
Prime symbol in Python variable name
So I'm a terrible person and I want to name a variable in my mathy-python3 code s′ (that's a prime symbol).
I was under the impression Unicode literals work as identifiers in Python 3, which is why ...
8
votes
5answers
3k views
Why does “charset” really mean “encoding” in common usage?
Something that has long confused me is that so much software uses the terms "charset" and "encoding" as synonyms.
When people refer to a unicode "encoding", they always mean a ruleset for ...
0
votes
1answer
317 views
Simple unicode application?
I want to create simple language learning applications to help friends in learning languages. A simple Java console application would do the trick, but the Windows console does not seem to handle ...
4
votes
4answers
226 views
Technical reasons to prefer coding business logic to support Unicode (when not required)
I have a legacy application in which the UI and business logic are already reasonably well-separated. There is a proposal to separate them even further, turning the core application into a "service" (...
6
votes
1answer
2k views
Does it make sense to choose UTF-32, based on concern that some basic rule will be broken for UTF-8?
I'm working on an cross platform C++ project, which doesn't consider unicode, and need change to support unicode.
There is following two choices, and I need to decide which one to choose.
Using UTF-...
4
votes
3answers
2k views
When should I *not* use Unicode? [duplicate]
Unicode seems that its becoming more and more ubiquitous these days if it's not already, but I have to wonder if there are any domains were Unicode isn't the best implementation choice. Are there any ...
6
votes
5answers
2k views
Using π, φ, λ etc. as variable names while programming? [duplicate]
This is a function in the d3.v3.js file (the graph library D3.js):
function d3_geo_areaRingStart() {
var λ00, φ00, λ0, cosφ0, sinφ0;
d3_geo_area.point = function(λ, φ) {
...
1
vote
1answer
177 views
How can I learn about typography, fonts, glyphs, etc.? [closed]
I know so little about this that I'm having trouble formulating the question.
Apparently due to technical limitations, nastaleeq style of writing Urdu is very difficult, perhaps impossible, given ...
5
votes
3answers
522 views
Consequences of “naïve” vs “naive”?
While using IE autocorrect "naive" got transformed to "naïve"!
My regional settings are Au English, from a Unicode search point of view the two are nothing alike. I am not even sure whether there are ...
6
votes
2answers
945 views
Languages supporting unicode logic operators
Are there any programming languages that support the use of unicode logic operators? For example, many programming languages use "!=" as the "does not equal"
operator, but in mathematics the symbol ...
15
votes
3answers
1k views
A Unicode sentinel value I can use?
I am desiging a file format and I want to do it right. Since it is a binary format, the very first byte (or bytes) of the file should not form valid textual characters (just like in the PNG file ...
4
votes
2answers
166 views
Strategy for website with international strings
What things need to be considered for a Website that contains International strings, for instance Simplified Chinese and English mixed.
UTF8 seems to me a natural choice, including a meta tag. Still,...
6
votes
3answers
830 views
Is O(1) random access into variable length encoding strings useful?
I remember reading that there are no existing data structures which allow for random-access into a variable length encoding, like UTF-8, without requiring additional lookup tables.
The main question ...
29
votes
2answers
13k views
Why does Java use UTF-16 for internal string representation?
I would imagine the reason was fast, array like access to the character at index, but some characters won't fit into 16 bits, so it wouldn't work...
So if you have to handle special cases anyways, ...
35
votes
2answers
2k views
Unicode license
The Unicode Terms of Use state that any software that uses their data files (or a modification of them) should carry the Unicode license references. It seems to me that most Unicode libraries have ...
3
votes
1answer
2k views
understanding the encoding scheme in python 3
I got this error in my program which grab data from different website and write them to a file:
'charmap' codec can't encode characters in position 151618-151624: character maps to <undefined>
...
3
votes
3answers
18k views
How does it matter if a character is 8 bit or 16 bit or 32 bit
Well, I am reading Programing Windows with MFC, and I came across Unicode and ASCII code characters. I understood the point of using Unicode over ASCII, but what I do not get is how and why is it ...
33
votes
3answers
120k views
Why do we need to put N before strings in Microsoft SQL Server?
I'm learning T-SQL. From the examples I've seen, to insert text in a varchar() cell, I can write just the string to insert, but for nvarchar() cells, every example prefix the strings with the letter N....
11
votes
2answers
4k views
Efficient Trie implementation for unicode strings
I have been looking for an efficient String trie implementation. Mostly I have found code like this:
Referential implementation in Java (per wikipedia)
I dislike these implementations for mostly two ...
7
votes
1answer
1k views
How in the earth CHRW produce unicode codes given that it only accept 65k possible input?
http://babelstone.blogspot.com/2005/11/how-many-unicode-characters-are-there.html says there are 1 million unicode characters and around 240k of which are already assigned.
1 million > 240k > 65k
...
4
votes
2answers
191 views
Prerequisites for developing an application with Unicode support [closed]
What could be the necessary prerequisites to be taken when developing an application with Unicode support in the context of
Web applications
Desktop applications
Embedded applications
Prerequisites ...
5
votes
3answers
488 views
What limitation will we face if each user-perceived character is assigned to one codepoint?
What limitations will we have if Unicode standards had decided to assign one and only one codepoint to every user-perceived character?
Currently, Unicode has code-points that correspond to combining ...
13
votes
8answers
2k views
What's the point of adding Unicode identifier support to various language implementations?
I personally find reading code full of Unicode identifiers confusing. In my opinion, it also prevents the code from being easily maintained. Not to mention all the effort required for authors of ...
0
votes
2answers
209 views
Unicode clarification
Why is the length of the some characters e.g. the following 'ᨒ' 3 when it should be 2
ᨒ
U+1a12
1a12 means 6674
2^16 is 65536 so 6674 should take only 2 bytes and not three
1
vote
3answers
243 views
Unicode Explanation Required
Can someone explain what this means?
Unicode defines a codespace of 1,114,112 code points in the range 0hex to 10FFFFhex.
http://en.wikipedia.org/wiki/Unicode
16
votes
3answers
1k views
Is it possible to write a generalized string reverse function that works for all localisations and string types?
I was just watching the Jon Skeet (with Tony the Pony) presentation from Dev-Days.
Although "write a string reverse function" is coding interview 101 - I'm not sure that it's actually possible to ...
15
votes
2answers
7k views
Is UTF-16 fixed-width or variable-width? Why doesn't UTF-8 have byte-order problem?
Is UTF-16 fixed-width or variable-width? I got different results
from different sources:
From http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF:
UTF-16 stores Unicode characters in sixteen-bit ...
24
votes
5answers
5k views
What issues lead people to use Japanese-specific encodings rather than Unicode?
At work I come across a lot of Japanese text files in Shift-JIS and other encodings. It causes many mojibake (unreadable character) problems for all computer users. Unicode was intended to solve this ...
40
votes
8answers
3k views
Why are there multiple Unicode encodings?
I thought Unicode was designed to get around the whole issue of having lots of different encoding due to a small address space (8 bits) in most of the prior attempts (ASCII, etc.).
Why then are there ...
4
votes
3answers
521 views
What should I consider when converting from ASCII to Unicode?
For instance if I had only ASCII characters and then switched to Unicode now I have to consider special characters, and bigger strings or chars. What else along these lines should I be considering?
I'...
