I'm newish to programming and would like to know what people have found to be the most supportable option or rather what people find more understandable. I'm the only one writing code at my firm, but gradually will hand this off to someone else in the future.
The project I'm working on is parsing a huge text file that comes in once a day. The data file uses previous day's data file as a reference so each data file builds upon the previous data file. Using documentation I'm able to parse the text file and populate a MySQL database. Once the text file has been read, the data in the database is manipulated and then dumped into many Excel workbooks. The text file has a huge amount of data and I only need a fraction of it for the Excel workbooks.
My question is what is the proper structure for this program? I see that I have two options.
- Is it better to construct a database that fits the data. So create a database with more tables and fields then is actually needed and possibly a little cryptic in it's layout.
- Is it better to manipulate the data while it's being parsed and put it into more readable, human database?
What I've Done So Far
When I wrote the first version of this program, I went for option 2. Since I didn't know much about relational databases at the time and wanted it very easy for me to understand, visualize, etc. However, now that I'm looking to expand the data that I parse from the text file and since each data file builds on the previous day's data file I'm leaning towards option 1 so later down the line I won't have to spend a day reconstructing the database.
Does this make sense? Please comment if it doesn't. Any help would be greatly appreciated.
Let me start by saying the answers so far have been very helpful. Thank you. After reading some comments though I find that I have been a bit vague in my question. Let me try to be a bit clearer. I'm dealing with financial data of clients. Each account has its own Excel workbook that is used by the financial advisors to make projections and investment decisions. Additional worksheets in the the workbook are used to display the account's holdings and projected future to the client. The data that I get in each data file is separated by a record. Each record has a type...there are something like 13 different record types. Each record type is marked by a particular letter or number. Then after that letter or number there are a specific number of characters that represent that particular record. The record layout differs per record type and there is redundant information between record types which makes it additionally aggravating. Another point of contention is that if the financial advisor buys a new security, one that s/he hasn't bought before then the next business day, only in that day's data file, will the security's information show up (such as the name, security type, etc) and not again unless there's a change in that security. So if I ever need to rebuild the database I would have to start from day one.
So I'm taking this data file, parsing it and writing it to a MySQL database separated into about four tables. I use EF6 to work with the database, and so that was a reason I started with a simple, readable fields and tables. Hope this expanded explanation helps.