We have a contact import process we wrote a long time ago that takes a csv file and pulls the data into a database, based on various options and column possibilities. Customers can supply files to us directly for us to import, or once mappings are setup they can upload themselves for automatic processing. Most of the content is English but on occasion we run into issues with foreign language characters. Encoding was never specifically handled and I believe is defaulting to expecting ANSI, leading to characters being corrupted along the way.
We are changing the system to ensure everything to UTF-8 before processing. It is my understanding that to get everything properly to UTF-8 you have to know what the current encoding is. A way to handle that is to ask the user the encoding of the file (not realistic for us) or use a library to guess the encoding.
My question is if there is a best practice here when accepting text files for processing. Should we for example instruct our customers to provide UTF-8 and then run the file through a process that rejects the file if found not to be UTF-8, or actually go through a guessing tool to try our best to convert that could lead to occasional bad guesses? We have files currently being automatically uploaded to us so rejecting could be painful. Any other options here that I am not thinking of?