2

I am working on a machine learning project at the moment which requires me to transfer the data from an old Java app(which is also the custodian of the data in current paradigm) to a python service which will do all the machine learning related stuff. So, I have data to the tune of a few GBs that needs to flow through the network.

What would be the most efficient way of transferring that data?

This information may be useful-

  1. The JAVA application is deployed as a 3-tier AWS instance and uses elastic search, postgres and neo4j.
  2. The python application will be deployed on a separate AWS instance.
  3. The data exists in Neo4J, is currently not encoded, but can be written to CSVs, or transformed into objects.

Help is appreciated! Thanks in advance!

  • One time, or on an ongoing basis? – Robert Harvey Aug 30 at 3:46
  • Hey @RobertHarvey! This can be triggered by the user. I can say this will probably happen once a week at most... – Uttakarsh Tikku Aug 30 at 4:30
  • 1
    "most efficient". That requires you to define a metric for efficiency. – Thorbjørn Ravn Andersen Aug 30 at 7:55
  • Hi @ThorbjørnRavnAndersen! The metric that I am most concerned about is network usage. Currently, network traffic is already a big problem. So I guess I am trying to weigh the tradeoff between network usage and speed of transfer. – Uttakarsh Tikku Sep 1 at 9:58
  • What shape is your data? Tabular data you probably won't do better than CSV, for more generic documents you probably want JSON. If your data isn't tabular but is structured more or less the same every time you can probably come up with a better custom format, though doing that can be prone to errors in any edge cases you forget to account for. – Turksarama Sep 1 at 10:42
0

I see this as a perfect example of using stream processing pipelines such as Apache Kafka (or Apache Flink).

The rationale to use them is that you can add as much producers (Java app) or consumers (Python app) as possible. You also do not have to worry about if they work in different speeds as Kafka will buffer it.

Before passing the data to Kafka you might have to serialize it (if it is not ASCII). For that matter you might want to use JSON (Alternatively you could use something like Apache Avro which let you easily partition the data).

  • Thanks for the reply, @Vicente Adolfo Bolea Sánchez! I have decided to go with the approach you suggested. I am currently running proof of concepts using both rabbit mq and kafka to compare their performance. – Uttakarsh Tikku Sep 5 at 12:44
1

Why not just let both apps read from the same database? Or if you cannot do that, you could write the data to S3 with one app and read it from S3 with the other app. The target app can listen to events in S3 for every file which is written and then just load it.

Maybe I am oversimplifying it but it seems easy...(?)

There is also Snowflake SQL www.snowflake.com which you can connect to AWS using their technology called Snowpipe which basically lets you write to a resource in AWS and that will load it into the Snowflake DB (but this is probably overdoing it in this case).

  • Hi! Thanks for your reply. I believe we could share the same database between the two apps, but in such a case, I am also concerned about the interface definitions and versioning. The team working on java related stuff could change data structures and break the python app. So, I was wondering if there is a preferred way of managing the interface definitions. – Uttakarsh Tikku Sep 1 at 10:06
  • @UttakarshTikku Yes if you validate the data being written or read similar to the jsonschema it will let you decide and know if and when the data structure changed to something which would break. If it does, just don't overwrite it and do something else. If it is csv you could find a csv validation. – Niklas Rosencrantz Sep 1 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.