I am in the process of writing a suite of applications and have been tasked with tracking "metadata" across the app. This task requires me to process, export, then import, a large amount of data.
The (M)ain application, will be running and serve as the main collection point for data. Each client will be running their own copy of the same portal style application in their environment, which will be pushing metadata up to "M".
For now the solution I am going with is to write the data to a file in the client applications, and then simply upload them through the UI dashboard for application M. The database and everything else is already taken care of but the issue is what option to go with to write potentially large files that will only be ingested in another application, both of which I have design control of.
With future upgrades in mind to this process (JMS Queuing, potentially web services, etc.), what is the most effective way to write this data out to a file for quick reading/processing in the main application? My initial hunch is to go with JAXB and just have a schema common between everything so it is easy to export/import the data sets as XML files. Would it be better for future upgrades to write the files out as JSON? Is there another option all together I am missing?
TLDR: I have control of both sides of an import/export process with potentially large amounts of data. What is the most efficient way to store/format this data for now with future upgrades to the process in mind?