2

I have some automated task to be done using certain types of logs generated on the server end of each day. Log files contains specific machine identification "MAC" that is to be re-written in to another text file converted into machine's name.

SAMEPE LOG DATA FILES EACH CONTAINING ~2 MILLIONS LINES IN BINARY FORMAT


-----------|--------|-----------------|----|----------|---------
DATE       |TIME    |MAC_ADDRESS      |PORT|IP_ADDRESS|BLA BLAA
-----------|--------|-----------------|----|----------|----------
19-01-2014 |18:16:42|A3:43:31:11:02:1A|22  |
19-01-2014 |18:16:43|A3:43:31:11:12:1A|22  |
19-01-2014 |18:16:43|A3:43:31:11:12:1A|21  |
19-01-2014 |18:16:44|A3:43:31:11:12:22|80  |
19-01-2014 |18:16:44|A3:43:31:11:12:00|443 |
19-01-2014 |18:16:45|A3:43:31:AA:12:AA|5800|
19-01-2014 |18:16:46|A3:43:31:AA:30:1A|21  |

The above records I need them to be converted to text format, the MAC address replaced with machine name from my machine name database. My current program, read each line from the above file and find the machine name from machine name db matching the MAC_ADDRESS of read data and write each of them to text file.


This is my question.

My program is written in C language. I have two option to manage the machine names DB, first is to have a sqlite db or else second option is to have a flat binary file. I consider the performance. I am afraid, using db for this would be less performance compare to file as it takes times for the connect/disconnect on processing of each line of log. even though, I am not clear on my point still.

Please consider the number of records and suggest me whether is it better to use flat file for sqlite db for storing machine names records.

  • Have you actually tried to measure the performance of the two options? As sqlite uses a local, binary file, I suspect that the difference between an sqlite lookup and your own search won't be significant. – Bart van Ingen Schenau Apr 27 '14 at 12:59
  • really I did not try with sqlite. Current program run on flat file. I just considered migrate sqlite and ended up in question. – Muneer Apr 27 '14 at 13:05
  • How important is performance? Is the current performance good enough? Increasing program complexity to improve performance is often a terrible move if the performance was more than adequate to start with. – Leon Apr 27 '14 at 13:11
  • @leon, In my case performance is the first choice. If a flat file can server more faster or somewhat equals to sqlite db with the type and amount of data available in my case then I will continue with flat file or else If any one have proven result on sqlite db, I can migrate to it. – Muneer Apr 27 '14 at 13:14
  • 1
    How many records are in your machine name DB? Tens, hundreds, thousands? If the number isn't too high, your fastest approach is to prefetch your machine DB into memory and then simply lookup the mac-addresses in memory as you process the log file(s). – Stepan Riha Apr 27 '14 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.