0

I want to collect a lot of files (file data + metadata) from local servers to a central server. Files are important, need to ensure that no files are lost

  • Local servers: implement a collector to collect files, 10k files/day/local server
  • Center servers:
    • minio storage: save files
    • ingestion service: provide an API to receive and insert file info into DB

I am considering the following 2 solutions.

1. API Call

collector

  • Push file into minio, get object name
  • Call API to add file info (object name, metadata) into ingestion service
  • Handling retries when a task fails

2. Only minio

collector:

  • Push file and metadata into minio

central server:

  • Setup minio to publish ObjectCreated events to kafka
  • Consume minio events, call API of ingestion service to add file into into DB.

3. Using data ingest tools

I don't research about these tools, because

  • My data source is a not supported
  • My data processing system is simple

I lean towards solution 2, but I'm a bit worried about minio event persistence. Could you please review above solutions and give me some feedback?

1 Answer 1

1

need to ensure that no files are lost

On each local server, have a midnight cron job produce a manifest.txt of files. It should contain full path names, sizes, and maybe timestamps. We could e.g. use /usr/bin/find to produce the manifest. Maintain manifests in sorted order, so it's easy to diff them. For extra credit, include a SHA-3 hash for each file listed.

Since all files get copied to central server, the manifests will, also.

Daily the central server runs two checks.

  1. Scan the set of known hosts and verify we have a fresh manifest from each host.
  2. Scan each manifest, and verify we have properly received each listed file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.