148

On my filesystem (Windows 7) I have some text files (These are SQL script files, if that matters).

When opened with Notepad++, in the "Encoding" menu some of them are reported to have an encoding of "UCS-2 Little Endian" and some of "UTF-8 without BOM".

What is the difference here? They all seem to be perfectly valid scripts. How could I tell what encodings the file have without Notepad++?

Improve this question
6
  • 9
    There is a pretty simple way using Firefox. Open your file using Firefox, then View > Character Encoding. Detailed here.
    – Catherine Gasnier
    Apr 15, 2014 at 16:26
  • use heuristics. checkout enca and chardet for POSIX systems.
    – Janus Troelsen
    Nov 3, 2014 at 21:00
  • 4
    I think an alternative answer is TRIAL and ERROR. iconv in particular is useful for this purpose. Essentially you iterate the corrupted characters strings/text through different encodings to see which one works. You win, when the characters are no longer corrupted. I'd love to answer here, with a programmatic example. But it's unfortunately a protected question.
    – Brandon Bertelsen
    Dec 7, 2015 at 19:03
  • FF is using Mozilla Charset Detectors. Another simple way is opening the file with MS word, it'll guess the files quite correctly even for various ancient Chinese and Japanese codepages
    – phuclv
    May 14, 2018 at 3:39
  • If chardet or chardetect is not available on your system, then you can install the package via your package manager (e.g. apt search chardet — on ubuntu/debian the package is usually called python-chardet or python3-chardet) or via pip with pip install chardet (or pip install cchardet for the faster c-optimized version).
    – ccpizza
    Mar 27, 2019 at 17:12

2 Answers 2

Reset to default
129

Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using.

For example, a file with the first three bytes 0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the characters . Or it might be a different file type entirely.

Notepad++ does its best to guess what encoding a file is using, and most of the time it gets it right. Sometimes it does get it wrong though - that's why that 'Encoding' menu is there, so you can override its best guess.

For the two encodings you mention:

  • The "UCS-2 Little Endian" files are UTF-16 files (based on what I understand from the info here) so probably start with 0xFF,0xFE as the first 2 bytes. From what I can tell, Notepad++ describes them as "UCS-2" since it doesn't support certain facets of UTF-16.
  • The "UTF-8 without BOM" files don't have any header bytes. That's what the "without BOM" bit means.
Improve this answer
11
  • 3
    BOMs: msdn.microsoft.com/en-us/library/windows/desktop/…
    – Jan Doggen
    Feb 15, 2013 at 10:34
  • 2
    Why would a file that starts with a BOM be auto-detected as "UTF-8 without BOM"?
    – Michael Borgwardt
    Feb 15, 2013 at 10:36
  • 3
    And if a file started with 0xFF,0xFE it should be auto-detected as UTF-16, not UCS-2. UCS-2 is probably guessed because it contains mainly ASCII characters and thus every other byte is null.
    – Michael Borgwardt
    Feb 15, 2013 at 10:39
  • 2
    With experience, alas, metadata (“headers”) can also be wrong. The database holding the information could be corrupted, or the original uploader could have got this wrong. (This has been a significant problem for us in the past few months; some data was uploaded as “UTF-8” except it was “really ISO8859-1, since they're the same really?!” Bah! Scientists should be kept away from origination of metadata; they just get it wrong…)
    – Donal Fellows
    Dec 8, 2013 at 19:39
  • 4
    Actually I think it's "funny" that the encoding problem still stays in 2014 since no file in the world will begin with "" and I'm very surprised when I see a HTML page which has been loaded with the wrong encoding.. It's a matter of probability. It's unthinkable to choose the wrong encoding if another encoding would avoid strange chars.. Looking for the encoding which avoids strange chars would work in 99,9999% of cases I guess. But still there are errors.. Also it's a very confusing message to use ascii instead of UTF8 to save space.. it's confusing junior developers this idea of perform..
    – Revious
    Oct 18, 2014 at 18:47
26

You cannot. If you could do that, there would not be so many web sites or text files with “random gibberish” out there. That's why the encoding is usually sent along with the payload as meta data.

In case it's not, all you can do is a “smart guess” but the result is often ambiguous since the same byte sequence might be valid in several encodings.

Improve this answer
5
  • 2
    OK, then, does the Windows OS store that information (meta data) actually somewhere? In the registry probably?
    – Marcel
    Feb 15, 2013 at 10:18
  • You're wrong. That is codepages- not quite the same. There are algorithms to guess at the Unicode encoding.
    – DeadMG
    Feb 15, 2013 at 10:24
  • 6
    @Marcel: No. That's why "text files" are so problematic for anything except pure ASCII.
    – Michael Borgwardt
    Feb 15, 2013 at 10:37
  • 1
    well notepad++ can do this, it can tell you if text file is utf-8 encoded or not
    – user25
    Jan 1, 2018 at 14:31
  • 1
    From the tools I tried, this one was the only that gave precise results, tried Cyrillic and non-standard Japanese. It uses chardet under the hood. Wish I could post it as an answer ;c
    – Klesun
    Jun 10, 2021 at 23:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.