Files generally indicate their encoding with a file header. There are many examples here. However, even reading the header you can never be sure what encoding a file is really using.
For example, a file with the first three bytes
0xEF,0xBB,0xBF is probably a UTF-8 encoded file. However, it might be an ISO-8859-1 file which happens to start with the characters
ï»¿. Or it might be a different file type entirely.
Notepad++ does its best to guess what encoding a file is using, and most of the time it gets it right. Sometimes it does get it wrong though - that's why that 'Encoding' menu is there, so you can override its best guess.
For the two encodings you mention:
- The "UCS-2 Little Endian" files are UTF-16 files (based on what I understand from the info here) so probably start with
0xFF,0xFE as the first 2 bytes. From what I can tell, Notepad++ describes them as "UCS-2" since it doesn't support certain facets of UTF-16.
- The "UTF-8 without BOM" files don't have any header bytes. That's what the "without BOM" bit means.
chardetfor POSIX systems.
iconvin particular is useful for this purpose. Essentially you iterate the corrupted characters strings/text through different encodings to see which one works. You win, when the characters are no longer corrupted. I'd love to answer here, with a programmatic example. But it's unfortunately a protected question.
chardetectis not available on your system, then you can install the package via your package manager (e.g.
apt search chardet— on ubuntu/debian the package is usually called
python3-chardet) or via pip with
pip install chardet(or
pip install cchardetfor the faster c-optimized version).