Job Scheduling, Stored Procedures

Question

I am working on a polyglot web app project that has functionalities that can be separated, but that rely on each other. I am unfamiliar with the following (what key words to even search for, what concepts are in play):

I have a few different sources of disparate data that are separately created/ingested into a data store. the data that is needed to back the actual web app requires clean-up and curation of the different types of data.

As a concrete example, perhaps you have a large book/article datastore and a user datastore. You want to curate and push data from both to a graph store for recommender purposes, and a paired down version of the book store to actually serve the web app and search functionality.

How do you usually go about this? It's not fast changing data, so I wouldn't think a full event-based reactive system is quite necessary. Is the concept to somehow schedule jobs at a specified period?

Answer 1

A key concept/search term may be ETL (Extract, transform, load). That's a formalization of the process you describe: grab data from all over the place, clean it up, and then deliver it. You're in luck, there are both a number of existing tools you may be able to use and a lot written on the topic.

I'm most familiar with Pentaho (Kettle, actually). Don't take this as a recommendation, but more as an example of what can be done...

With Kettle, I pull in loads of data from many sources: files, databases, and even web services. If your data source isn't supported out of the box, you can likely find a plugin that works. Then, I transform the data into an appropriate shape, clean it up, and drop it somewhere else. Again, if your destination isn't supported out of the box, you can likely find a plugin to help. (Actually, writing your own plugin isn't insurmountable.) Best of all, I can schedule all of these processes in any idiosyncratic fashion I see fit.

Be sure to evaluate a bunch of tools to decide what works best for your team.

With regards to best practices, there's much written, but most is technology-specific. The big vendor databases have their own approach. Open source tools have their approach. You don't mention the tech you're using, but you'll likely find more detail by searching for ETL as related to that tech. Generically, best practice recommendations tend to be sort of obvious: know your requirements, logging, that sort of thing. Maybe post secondary questions as you discover details.

Answer 2

To answer my own question on what I ended up going with, and to my judgement the best approach: Streams! Although a much less talked about use-case (you usually only hear about consuming fast data), steam frameworks are a very good way to reason about and a simple way to build ETL pipelines. I build in the Scala/Java ecosystem. Akka Streams (for a low level take) and Apache Flink are great tools for this.