4

Problem

I am currently investigating a solution to allow fast upload (and retrieval) of data that can be implemented via NoSQL/SQL or a file system on a Centos server with 64 core cpu with 529GB of ram.

The case is as follows:

The users have a need to keep a track of data uploaded onto a server, then at some point retrieve all the data stored on server ready for processing on a different system.

The operations on the data involve upload/import and download/export.

each record uploaded will range between 4mb-5mb

Options

To handle the large amount of data I believe the server we have is more than efficient enough to handle the load. However, we’re looking for a solution to make sure that scalability and backup isn’t going to be an issue in the future. Some options I have considered are:

  • mongoDB(GridFS to chunkate the data).
  • Oracle database (the LOB compression and datafield).

What are your thoughts on the best approach to take for this project? Has anyone had a similar dilemma and what was done to solve it?

Thanks, in advance for your help.

EDIT More Information

The estimated records we are expecting are 2400 as a worst case scenario 9600-12000mb in a day and will arrive in a burst. The users will need a sub select of data and will be used on corporate gigabit LAN network with standard Ethernet cables

Improve this question
4
  • 1
    In order to get a good answer, you need to provide additional information: How many records per unit of time (second, hour, day) are you expecting, and do they will arrive in bursts or at a steady rate? Do users need to select subsets of the data, or will they retrieve all available data at once? What is your network configuration, are all users accessing via LAN, or is will some use the Internet?
    – kdgregory
    Oct 25, 2016 at 12:01
  • 2
    In general, however, "scalability and backup" is best managed via redundancy, rather than having one large server.
    – kdgregory
    Oct 25, 2016 at 12:01
  • Please edit your question and add some more details. Right now, what you've described could be a simple photo sharing web site, easily implemented on small hardware. What makes your problem different?
    – Dan Pichelman
    Oct 25, 2016 at 13:20
  • Added some more information if you need any more ill be happy to help
    – angrymuffins
    Oct 25, 2016 at 13:33

1 Answer 1

Reset to default
1

A scalable architecture depends on thinking through every component and how they interact, not just whether a particular piece of hardware or software can meet the need. And as I said in a comment, redundancy is the standard answer to both scalability and durability.

In your case, the data volumes aren't really that large. I'm sure that your hardware can handle it, using either Mongo or Oracle. But what happens if that server goes down? Or loses a disk? Or your volumes grow.

Here are the three areas that I'd want to fully understand:

Ingest

  • How many clients will be connecting concurrently, and what are their tolerances for waiting? Any single server could have trouble with 2400 concurrent uploads, simply due to context switching. Standard practice is to have a group of front-end servers with a load balancer in front of them.
  • Are you using an HTTP front end or are other technologies allowed? For example, could you write directly to a queue?

Storage

  • What are your requirements for durability and accessibility, and what options do you have to meet those requirements? A RAID-5 volume, for example, will handle a single disk failure but not failure of the database host. Hot backups can mitigate host failures, but require identical configurations (ie, two or more machines with same spec).
  • A sharded database can improve scalability, but again requires additional hardware (including backups).
  • How long does the data have to live, and what are the long-term access patterns? 12 GB/day is really small if it only has to live a week, but it turns into 4 TB/year, which isn't that small.
  • Do you need to maintain the data within your physical plant, or can you use cloud storage?

Retrieval

  • Will the users be retrieving subsets of data based on criteria, or can they accept all data (perhaps with a time range) and filter it locally? The former implies a database with some form of query language, while the latter might be implemented using something like Kafka.
  • Do you need advanced analysis in your queries (for example, medians). That would drive you toward a traditional SQL DBMS versus NoSQL (which would require you to perform such analysis on the client).

Until you've given some thought to those points, it's difficult if not impossible to give a valid answer in this forum. But one thing that I would suggest looking at is a distributed database such as Cassandra.

Improve this answer
1
  • thanks for this, it is a lot to think about and ideally was looking for some suggestions on where to start with this but you have given the insight on what i need to look into and to what i can find based on the points given. thanks marking this as correct for now ;)
    – angrymuffins
    Oct 26, 2016 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.