5

I have been tasked with developing a web based (i.e runs in browser) viewer for a proprietary log file.

I have no control over the format of the logs, I just consume them. The log file contains binary data appended by a text message on each line, so part of each line must be de-serialized on read.

What I am posing, is what is the preferred approach here to quickly read the file and make it available for searching and page of text retrieval.

My approaches:

I have a web service read the file and bulk insert its contents

  1. Into a SQL server installation
  2. Into a serverless no-SQL like LiteDb
  3. Into a serverless Sqlite instance with one sqlite file per log file.

The data is wiped after that log file data has not been viewed for a week.

The problem is these approaches works fine on files that are less than 100mb but quickly breaks down with log files that sometime grow to be 2gb.

Even on a 2 gb file, parsing is relatively fast and seemingly not the bottleneck. On the serveless options, writing the parsed data is also relatively quick

Query of full text is not great on any options, though once an index as has time to build on the Sql Server, it's decent.

I am wondering if anyone has any advice / experience with this sort of project - basically, quickly de-serialize a large file and bulk insert it into some sort of data store (maybe an in-memory data-store would be better) and make it available for interaction via a web service.

CLARIFICATIONS:

@Basile - Sorry, I didn't mean to imply that, I just wanted to make sure I was looking down the best avenue and had the best suited solution

@Basile - The log files are dead simple, just a 45 byte binary blob that I read as bytes and copy to a struct (in .NET), then I read the rest of the text until a new line char. Repeat until EOF.

@DocBrown Sorry, I read this, thought it was appropriate see here

@Christophe I don't have 3 DBs, those are just the three approaches I've tested so far.

  • What make you think that Sqlite cannot work on gigabyte-sized files? It could (at least if you create appropriate indexes). See this. And you might parse the log files incrementally. – Basile Starynkevitch Dec 19 '16 at 8:29
  • 1
    You should explain a bit more what these proprietary log files are, what is their format, how are they written (or grown)? Of course, the approach depends a lot on the operating system & file system involved. – Basile Starynkevitch Dec 19 '16 at 8:32
  • Can you clarify why you need to feed 3 different databases ? Can you at which rythm the logs are parsed (every day ? every week ? every message ? – Christophe Dec 19 '16 at 20:23
  • 1
    If you're doing full-text search, consider using a full-text indexer like Solr or Lucene, which underpins it. – Blrfl Dec 19 '16 at 20:46
  • 1
    Please post an anonymized sample of the log file format and of a typical query? – Apalala Feb 5 '17 at 12:59
2

Try Elasticsearch's Kibana, it's made for querying/browsing large logs. https://www.elastic.co/products/kibana

It uses Elasticsearch under the hood which you could interface with directly if you want to.

Otherwise. If your parser is really fast try see if you could 'emit' entries while streaming through your log. You could perhaps get away with doing a full pass (text) scan without first throwing it all into memory or a database.

2

I don't think it is a great idea to preload 2GB of log data. That would make for an unbearable user experience, and if you run this thing on a production server you will set off a bunch of alarms in the NOC.

I would focus on keeping a small memory footprint and reading as little of the file as possible. There are ways to search the file without actually loading it all. Some common use cases:

User wants to see log data from a certain portion of the file, as indicated by dragging a scrollbar.

  1. Open a file stream on the file and compute the target of a Seek operation by multiplying [total file length] * [scrollbar percentage].
  2. Seek the target location
  3. Read until the next newline character; discard
  4. Start reading and displaying records until the screen is full

User wants to see log data for a certain time period

  1. Open the file stream on the file
  2. Divide the file in half; seek the halfway point
  3. Skip to the next newline character, then read the next record
  4. Parse the date/time stamp.
  5. If the date/time is just right, stop
  6. If the date/time is too low, repeat the above operations on the second half of the file (so in step 2 you're actually finding the 3/4th mark)
  7. If the date/time is too high, repeat the above operations on the first half (so in step 2 you're actually finding the 1/4th mark)
  8. Keep recursing until you find the time range that is desired. This is known as a binary search.
2

One concept to keep in mind here is "computing power and IO are very, very dear at the proverbial point of attack but very, very cheap in the background."

So, presuming we aren't talking very real time operations here, I would take a different approach. I would have the web service run as an agent constantly loading the log files and I'd let client query it on demand. That should alleviate the load-time issues as the bulk of the data is already loaded. For disk space concerns the database or app can easily handle truncation duties based on retention policies.

1

Deploy one process that consumes the file and transforms into a data structure that you insert into a database. If the files are potentially GB's in size, then read the file in line by line and process each. This will lessen the memory footprint.

If your database is MSSQL and is version 2008R2 or higher, use the full text index feature. I mention that because it is one I have first hand experience with. Most other DB platforms will have a full text index feature. If not, then turn to lucene or solr (both from Apache) to handle the indexing for search.

You will need to work out a method of looping this ingest to pick up new/incremental items and avoid double ingesting duplicate records. How you do this depends on the log file data structure and how the app producing it.

Your web UI should be another process that acts as a presentation layer for interacting with the pre-ingested and indexed data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.