For company abc, teams are separated in terms of mission-critical applications and their extended applications. For example, customer xyz uses mission-critical data, but there are business constraints/flows that cannot be achieved with the mission-critical system alone. To maintain data integrity (mission-critical system is the system of record), web service(s) have been made to interact with the mission-critical data. These web services send one entity (business object) in respect to the mission-critical system.
From the mission-critical side, there is a dependency on some third party tools. One of these tools queues the web service calls into threads, and it has the ability to reject currently queued or new web service calls. Rejection meaning that the data did not make it into the destination system. Although I did not get clarification on all criteria, the typical concerns are used, such as large data set(s), network traffic, or database locks.
As stated above, the extended application is supposed to provide functionality that is more in line with their expected work flow(s). A request has been made to modify extended application data in bulk, such as vendor x was bought out by vendor w so that all of the current occurring transactions must be mapped/set to the new vendor. In this case, there can be 1 - n records that need to be changed, and each record may not exist together as one entity in the mission-critical system.
The last step to this process would be to upload the changed data to the mission-critical system, because external systems (accounting, vendors, etc.) have a requirement to only use data from that system.
There is the possibility of getting a custom web service, but there would still be an issue if the third party tool decides to reject the call after a successful web service call was determined.
Processes do exist where bulk data is exported to files (CSV or spreadsheet), but the use has been focused on data going outside of the company. If a web service could accept a file, then there is sill the rejection issue. If some batch/ETL could be used, then there is a risk with data integrity, since that would create a different branch to manipulate data that is not within the context of the mission-critical system.
An idea had been tossed around to queue the data entries in our own staging/transaction area, so that we could still upload through the web service and keep record, if it was not fully processed. The concern with this idea is the network traffic. (EX: 1000 entries were updated and this data spans 50 entities. The web service would have 50 calls to make with its current format.)
For the design:
Where does the bulk data manipulation fit in respect to the customer request?
Should the mission-critical system define a way to accept bulk data?
Should this only be the responsibility of the team associated with the extended application? (EX: Define a "reasonable" process to the customer in respect to how data integrity is maintained; Or defining staging/transaction areas with some kind of job that only passes the data via web service calls)
With the current company structure (many teams to handle the subsets), what could be added to an individual team's process to help with future functionality requests at a similar level as above? (requests that cross into mission-critical data integrity)