The character encoding can generally not be determined completely. However, there are many hints:
- ASCII contains only bytes with values below 0x7F, originally it is a 7 bit encoding, but the byte values are simply zero-padded so the first bit is always zero;
- UTF-8 contains ASCII + additional bytes for which the highest bit is set, e.g. the three most significant bits may be set to
110 to indicate that two bytes are used instead of one to encode a character. UTF-8 may contain a Byte Order Mark (BOM), but usually it doesn't. The encoding is identical to ASCII if only ASCII characters are present.
- UTF-16 is usually prefixed with a BOM as it is a 16 bit encoding that may either use big- or little endian w.r.t. the order of the bytes (not the bits inside the bytes). As it is a 16 bit encoding where the only the lowest 7 bits encode ASCII, it is usually easy to recognize humans, and easily distinguished using statistical heuristics as well.
- There are many, many 8 bit encoding schemes, such as Windows-1252 also known as CP-1252. This encoding extends ISO-8859-1 which encodes the Latin-1 character set. This by itself is a form of extended ASCII.
UTF-16 is generally easy to recognize due to the common BOM and many bytes set to zero - at least for Western languages that use Latin-1. UTF-8 usually doesn't have a BOM, but the encoding scheme for additional characters is relatively easy to recognize.
A text editor that only sees ASCII will usually represent them using UTF-8 (now more and more the default) or Windows-1252. Sometimes applications and languages will simply keep to the system default. However, nowadays many text files do not do this and simply default to UTF-8 for all text. It has been the common default on Linux and Android for a long time now.
For older systems usually a system-specific code page was used. One of the more recognizable ones - at least for Westeners - is the IBM code page 437 as it was used for text-based windowing systems and a lot of ANSI art (sometimes incorrectly called ASCII art), going back to the time of DOS. However, quite often these code pages are not easily recognizable, which is why ASCII art often doesn't look good when a text file is opened. It quite often defaults to the system default such as Windows-1252.
It is extremely uncommon, but sometimes other character encodings are used. Some of those are "dialects" of ASCII such as IA5 that are just slightly different. More commonly though they would be text files using a code page for another country, where the first 128 codes are ASCII compatible.
If you come across such an encoding then you could convert to UTF-8 which is generally recognized easily, and it contains all the possible characters from the various code pages.