0

Background:

I am in the process of writing a suite of applications and have been tasked with tracking "metadata" across the app. This task requires me to process, export, then import, a large amount of data.

Example

The (M)ain application, will be running and serve as the main collection point for data. Each client will be running their own copy of the same portal style application in their environment, which will be pushing metadata up to "M".

For now the solution I am going with is to write the data to a file in the client applications, and then simply upload them through the UI dashboard for application M. The database and everything else is already taken care of but the issue is what option to go with to write potentially large files that will only be ingested in another application, both of which I have design control of.

With future upgrades in mind to this process (JMS Queuing, potentially web services, etc.), what is the most effective way to write this data out to a file for quick reading/processing in the main application? My initial hunch is to go with JAXB and just have a schema common between everything so it is easy to export/import the data sets as XML files. Would it be better for future upgrades to write the files out as JSON? Is there another option all together I am missing?

TLDR: I have control of both sides of an import/export process with potentially large amounts of data. What is the most efficient way to store/format this data for now with future upgrades to the process in mind?

4
  • have you thought of database dumps/exports over FTP? Aug 9, 2016 at 14:45
  • @TulainsCórdova I have, but part of the metadata collection we want to scrub a lot of PII style information so we can't relate it back to any particular clients using these child applications. So it wouldn't be as simple as a DB dump/export since there is some amount of minor transform that happens. ETL on the DB seems "possible" but a lot more time intensive and trickier.
    – Walls
    Aug 9, 2016 at 15:03
  • I think the reason this question is not getting the deserved attention is because it's not clear enough. As it is now, it's a wall of text and the TLDR section doesn't seem to help much. Aug 9, 2016 at 20:18
  • Since its Java to Java - you may just want to Serialize the objects. docs.oracle.com/javase/tutorial/jndi/objects/serial.html
    – Johnny V
    Aug 15, 2016 at 21:09

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.