Syncing data from a monolith to a micro-service

Question

we have a monolith that basically reaches close to a 100k IoT devices/routers and tries to gather some information about them.

As you can imagine, this can take close up to 40 hours for the process to complete(due to badly designed monolith + slow network that reaches these devices)

Now I am trying to break this IoT device syncing part out from the monolith, and make it it's own service(with multiple handlers to manage the load to make things faster).

But all the data/credentials required to connect to these devices are still in the Old Monolith. This data changes rarely, probably once a month.

What's the best way to sync data from the old monolith? Should it be a REST API in the old monolith that returns a giant JSON with all the data which the new micro-service can call like say every week?

I know about things like event based architecture which works great for a large team, but I am a lone developer(so limited resources) so I am wondering some practical engineering solution that works and which can avoid making the data syncing part it's own project.

Answer 1

You need to wrap the existing data in an api of some kind. So that whenever it is updated you have a point where you can intercept the update and replicate it to your new system.

Then you can simply manually populate the new system and expect it to stay in sync.

Now you can get to the bit you want to improve, it can call this new API for the data it needs and not interact with the old system.

It may even be possible to totally separate out this data source from the monolith as part of this process. Getting it to call the new API rather than its own data source.