Skip to main content
Share Your Experience: Take the 2024 Developer Survey
62 votes

Should UTF-8 CSV files contain a BOM (byte order mark)?

Not for UTF-8, but see the various caveats in the comments. It's unnecessary (UTF-8 has no byte order) unlike UTF-16/32 and not recommended in the Unicode standard. It's also quite rare to see UTF-8 ...
Kayaman's user avatar
  • 1,960
11 votes

Should UTF-8 CSV files contain a BOM (byte order mark)?

There still is no widespread convention AFAIK, though certainly UTF-8 is now generally accepted. The BOM is an awful artifact: It is invisible (zero-width space). Some software might break on the ...
Joop Eggen's user avatar
  • 2,059
7 votes

Import large csv files

A Stream does not need to read the entire file into memory, so... Don't bother with Excel at all if you can accumulate needed information while reading records: Sum as records are read Build a ...
radarbob's user avatar
  • 5,823
7 votes
Accepted

Meaningful response to the user after his uploaded CSV was processed?

This depends on the way the CSV is created, maintained or fixed in case of errors, and on how the imported data will be processed after. Let's start with the question of "all or nothing" vs. "reject ...
Doc Brown's user avatar
  • 209k
6 votes
Accepted

Multi-level validation in C#

Here is my solution. It is not perfect, but it is a start. First of all, I have created an enum called ErrorLevel: enum ErrorLevel { Error, Warning } Next, for every validation rule, I have ...
Michael Haddad's user avatar
5 votes

Can the csv format be defined by a regex?

This regexp can tokenize normal CSV, as described in the RFC: /("(?:[^"]|"")*"|[^,"\n\r]*)(,|\r?\n|\r)/ Explanation: ("(?:[^"]|"")*"|[^,"\n\...
Sam Watkins's user avatar
5 votes

Back-end solution for pulling from CSV files

To allow a variety of queries on such data, which transfer only the requested amount of bits and bytes over the network, one needs to preprocess and optimize the data for this purpose, there is no way ...
Doc Brown's user avatar
  • 209k
4 votes
Accepted

Storing survey data from csv to xml

Which is the better encoding really depends on what you're trying to store. If you're storing the data gathered by the survey, it'll be much different than if you're storing a record of the user input ...
Morgen's user avatar
  • 1,091
4 votes
Accepted

Import large csv files

I do not know which, from the other answers, will work best for you. AFAICT, I'd remark, mostly from an implementation standpoint, that they all make sense, but from a design perspective, which one ...
YSharp's user avatar
  • 888
4 votes

Import large csv files

Ummm...Use Excel. Excel can import CSV files. All the functionality you describe here can be done in Excel. If these are recurring tasks, you could write some macros to automate the loading of the ...
Jon Raynor's user avatar
  • 11.6k
3 votes

Import large csv files

Of course you can use bulk import. If you can't access the corporate database for some reasons, you install one locally and use it. Since you seem to be working with Microsoft's stack, SQL Server ...
Arseni Mourzenko's user avatar
3 votes
Accepted

Generating CSV export

There is a #codereview site, which might be better suited to your question format. However! architecturally it has a few problems First and foremost you have not separated out the code that writes ...
Ewan's user avatar
  • 76.4k
2 votes

Should UTF-8 CSV files contain a BOM (byte order mark)?

The only public definition of CSV format is RFC 4180 and such CSV file should use UTF-8 encoding and no byte-order-mark (BOM) because UTF-8 files do not depend on CPU byte order and therefore do not ...
Mikko Rantalainen's user avatar
2 votes

A program design question: Good idea using HDFS in c for reading large data?

For only reading 10 MB of data out of a 600 GB file, you don't need a distributed filesystem, you need a way of reading only what you're interested in. Databases with indexes come to mind or even ...
D. Jurcau's user avatar
  • 557
2 votes

Store line numbers of CSV in the database

There is no general design rule for storing or not storing line or record numbers of a CSV file. You store them when you have a requirement for storing them (like you want to to store the file line-by-...
Doc Brown's user avatar
  • 209k
2 votes

What to do when test data takes up most of repo size?

Is this a common pattern in larger code bases that deal with data processing? Yes. When starting out developing a processing pipeline, checking some "real" inputs against some known good ...
l0b0's user avatar
  • 11.5k
2 votes

System Design: Efficient way to import hundreds of different CSV formats

There is nothing wrong in letting a human define the column mapping for each new data provider if the number of different providers is something around "some hundreds". Give a human ~five ...
Doc Brown's user avatar
  • 209k
2 votes

System Design: Efficient way to import hundreds of different CSV formats

You practically have two options: have a tool that takes in a records definition of a file and a CSV file with data. Basically, for each data provider you define how their records match with your ...
Bogdan's user avatar
  • 3,620
1 vote

Back-end solution for pulling from CSV files

Just keep it in memory Even the full county level CSV data is relatively small (16 MByte) so it would be possible to fully cache it in backend memory without a database and keep a simplistic index on ...
Hans-Martin Mosner's user avatar
1 vote

Passing csv files between regions

Okay, so after switching my PC's "Format" setting to 'French (France)' via control panel, I opened a csv in Excel containing the above line of text. Excel placed the entire line into a single cell. ...
Andrew Stephens's user avatar
1 vote

Multi-level validation in C#

There are several ways you could go. The business objects could implement IValidate which you could define as: public interface IValidate { List<IWarning> GetWarnings(); List<IError&...
Bernhard Hiller's user avatar
1 vote

Need help creating a database of map polygons

There's a solution with a much nicer workflow, although the initial setup is more complex: use a geospatial database like PostGIS, Oracle Spatial, or even the geospatial features of MySQL (although it ...
blm768's user avatar
  • 181
1 vote

Mapping different XML and CSV feeds

If you mean that they should be able to import their existing files, then you will have to create a mapping to translate each of the existing Amazon, Newegg, ... formats into your database. For XML ...
kevin cline's user avatar
  • 33.7k
1 vote

Why do CSV file formats normally use quoting instead of escaping?

The CSV originates from the early seventies (Defined in IBM Fortran 77), it was introduced to give a better data transfer with less errors in punch cards as the previously used fixed length format was ...
plykkegaard's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible