I have a binary file that I want to parse. The file is broken up into records that are 1024 bytes each. The high level steps needed are:
- Read 1024 bytes at a time from the file.
- Parse each 1024-byte "record" (chunk) and place the parsed data into a map or struct.
- Return the parsed data to the user and any error(s).
Due to I/O constraints, I don't think it makes sense to attempt concurrent reads from the file. However, I see no reason why the 1024-byte records can't be parsed using goroutines so that multiple 1024-byte records are being parsed concurrently. I'm new to Go, so I wanted to see if this makes sense or if there is a better (faster) way:
- A main function opens the file and reads 1024 bytes at a time into byte arrays (records).
- The records are passed to a function that parses the data into a map or struct. The parser function would be called as a goroutine on each record.
- The parsed maps/structs are appended to a slice via a channel. I would preallocate the underlying array managed by the slice as the file size (in bytes) divided by 1024 as this should be the exact number of elements (assuming no errors).
This appears to be a producer with multiple consumers (at least the way I'm thinking about it). I am aware of an example of this pattern in Go, but I'm not sure if this changes when reading contiguously from a file (it seems concurrent reads would slow things down, so only one producer, but many consumers parsing could speed things up—but I need to make sure I don't run out of memory also).