The general question is what kind of mechanism can I use to transfer data to and from publishers and subscribers where publishers or subscribers can be permanently offline? Can message queues be used for this?
I am thinking a message queue style approach where the online service (publisher) publishes messages to a queue online, copy the queue offline, and have the offline service (subscriber) then process all incoming messages. I need to design a solution where the same applies with the roles reversed, where the offline service is the publisher and the online service is the subscriber
The data is physically transported between the networks by way of storage medium. Once the storage medium is plugged into a network a data transfer service reads the storage medium and moves all of the data from the storage medium to the queues.
The way I currently do it is a simple table copy where
- The database has triggers that listen for inserts, updates, or deletes, makes note of the event in a database event table (db_publish_events)
- Later a data transfer service reads all published events from db_publish_events and then copies the entire row to a JSON file with a tag of INSERT, UPDATE, or DELETE
- Then the JSON file is manually transported to the subscribing system and then processed by a data transfer service. Each record transferred in is marked in a database event receiving table (db_received_events).
- A JSON file is downloaded from the subscriber of acknowledgment receipts of all events received by the subscriber.
- The JSON file with receipts is sent back to the original publishing database and changes the state of the db_publish_event to mark it as received so the publisher will stop sending it.
Simple table copy across a network
No data integrity across tables or business event boundaries since one record is transferred at a time.
I am thinking of a solution where the entire event is transferred as one message, so before complex business rule processing splits an event into multiple tables.
Is there a way for commercial or open source messaging software (RabbitMQ) to do a simple USB copy of messages for publishing in an offline queue?