1

I need to call third party APIs where the payload size has a maximum size limit, say, 4 MB. That means I cannot call the API with a payload more than 4MB in size.

How can I solve this in an algorithmic way? Because a simple solution could be like every time I generate a payload, I check the size, and if it exceeds, I can regenerate it. But this is not an effective solution and not good from a performance point of view.

For more detail, let's take this example. There is one API where I can file invoices in one call only. Now I can have thousands of invoices and each invoice could have different data as per business.

In this, how can I determine a benchmark to send a limited number of invoices to meet the size limit requirement?

6
  • Serialize the invoices individually, add the per-item array overhead and loop until it exceeds the limit and then combine them into a single json array? May 15, 2017 at 7:49
  • Or just send a conservatively low number of invoices, there is probably little gain in getting close to the limit. May 15, 2017 at 7:50
  • Thanks @CodesInChaos for your input. serializing individual invoice could be overhead. But It could be possible with batch of invoices. Say 5000 invoices at once and check the limit. Thanks! May 15, 2017 at 8:22
  • 1000 is probably a better number. Maybe even 100. Hard to know without performing some tests. May 15, 2017 at 20:17
  • yes it make sense @RobertHarvey to performing some test. I've done benchmarking and it took 10000 invoices to have 5 MB of size. But here our invoices size and pattern could be different from subscriber to subscriber. So not sure if I can have one full proof test. So as you suggested 1000 or 5000 batch could serialize and tested for the size. May 16, 2017 at 5:43

1 Answer 1

2

If this is about limits to the size of a reply from your server, then answer this: What does the API do if you exceed the 4MB limit? Does it do something sensible? There are two sensible ways: a. The API returns an error indicating that you exceed the limit, in which case you try again, asking for half as many items. Or the API returns a status indicating that out of the n items you wanted, m ≤ n items have actually been processed, so you know m items have been done, and you ask for the remaining ones.

The last case is the best IMO: If you want to get a million items, you request a million and the API tells you that you only got 2,500. So then you ask for 997,500 and so on.

If this is about limits sending data to the server: You generate data for one item after the other, building the data you are going to the server. For each item, you check if adding that item would make your data too big, and in that case you send all the data you had so far except the new item, and then the new item becomes part of the next message to the server.

3
  • API just refuse the connection without processing any records. We've performed test and it takes 10000 invoices to make payload size 5 MB so adding one item at a time and checking the payload size could be overhead. May 16, 2017 at 5:46
  • @FenilRathod left or right you're going to add overhead. The best way is to measure. How big is the overhead? Is it acceptable, how are you going to address failures etc. For instance one strategy could be to just always send 10000 invoices and when the api refuses them, have the program backtrack and feed the same information in smaller portions.
    – Pieter B
    May 16, 2017 at 8:45
  • If you know at which size the problem occurs, then building the payload bit by bit should be minimal overhead. You build it bit by bit anyway; if you have 1000 invoices, you create a payload starting with nothing, adding the first invoice, then the second one. The only difference would be adding up the sizes. If you can't predict where the error happens, that's bad.
    – gnasher729
    May 16, 2017 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.