5

I recently joined a software project and discovered that the current architecture relies on line numbers of input data files. For example, the data on line 248 of an input file corresponds to file ID 247.

My gut tells me that this is dangerous and bad practice, but I'm too green to be sure. The people working on the project say that they did it for better performance, and that it shouldn't be a problem as long as we're careful to maintain the same format. I'm worried that future modifications could inadvertently lead to disaster, and doubt that the performance gains are that great.

Would this be considered bad practice? Or is this a perfectly normal thing to do?

Edit for clarification: The data is always generated by machine, not a person, so it's at least safe in that regard. It's a recurrent import, and there are multiple types of files with the same format (ID = line number - 1) but different data. Thus, a mistake in one input file would make it inconsistent with all the other data.

| improve this question | |
  • 8
    Is the input generated by machine or person? – whatsisname Dec 14 '16 at 4:14
  • It is not dangerous if you think of lines as records and line numbers as record IDs. – mouviciel Dec 14 '16 at 9:34
  • 1
    I've once worked on a project for a bank. The ouput of the algoritms was a file that later was associated in a datagram similar to the described in the question and processed in the mainframe. It really depends on your project. – linuxunil Dec 14 '16 at 12:17
  • Is it a single import or recurrent import, if so are the lines and associated files required to retain the same ID across imports ? if generated by person, are comments permissible ? is it possible for a single record to span multiple lines (I would think not but still) ? Can a line termination character be present in the line content itself (think CSV with a multi-line cell text should be fine if appropriately quoted but might skew the line count) ? – Newtopian Dec 14 '16 at 15:26
  • @whatsisname It's always generated by machine, I'll edit my question to reflect that, thanks. – Agargara Dec 18 '16 at 10:12
6

Is it dangerous for a program to rely on line numbers in input data?

No. It should not be dangerous.

Would this be considered bad practice? Or is this a perfectly normal thing to do?

It may be bad practice but it really depends on how the client & service manage the data/api contract.

If the service validates the input format before processing it, it should be fine. If the clients are using a library provided by the service and can call only via this library, then should be fine since the library would enforce this quirky pattern.

If the service just assumes the format is correct, the data is valid and goes about its business; you have a bad practices being implemented. This bad practice may be fine if the cost of corruption is low or non-existent.

PS: I'm not so sure about the performance improvements, but if you are new, it'll make sense to learn about those benefits and how they came about deciding on this particular solution.

| improve this answer | |
  • 1
    There is no validation done on the input file, the code currently just assumes the format is correct. I will suggest doing some kind of validation (such as checking the total line count matches the total number of actual files, etc.) to my team members. Thank you for the detailed answer. – Agargara Dec 18 '16 at 10:21
2

In theory as long as the data format is well defined then it doesn't matter if its 'optimised' in this way.

However, I think you are right to be concerned.

  • no way to atomically process a single line
  • no way to check position in file is correct
  • data which included a line feed character would effect other rows

Adding a single integer Id field seems like a small price to pay for peace of mind

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.