Tiled images

Tiled images are large images that have been split in smaller square tiles. There are several tiled image formats, with different ways of organizing the tile files. A tiled image on the web can be downloaded only by finding all the individual tile URLs, downloading them, and stitching them back together. This is what a tiled image downloader does.

zoomable image

Building a tiled image downloader

I maintain several tiled image downloaders (mainly dezoomify, and dezoomify-rs). Their implementation is quite simple, the only software architecture challenges are to:

  1. Make as much of the software as possible testable without having to actually make network requests
  2. Support many different tiled image formats without duplicating code.

I call dezoomers the modules of the software that are specific to each tiled image format. The goal is to be able to have dezoomers be as small as possible, and to have the core of the software do most of the work.

Implementing template-only tiled image download

Currently, in dezoomify-rs, the dezoomers provide a list of tile URLs and associated positions, and then the core downloads and stitches the tiles in parallel. That works for most image formats, but not for template-only downloads.

Template-only download is a feature that would allow the software to take as a single input an URL template of the form http://test.com/{x},{y}.jpg, and be able to replace {x} and {y} by coordinates to create the image. The challenge here is that the bounds for x and y are not known in advance, and can be computed only by requesting tiles, and looking at the response of the server: as soon as the server returns a 404, we know that we have reached the maximum value for x, and process to the next line, until the server returns a 404 even for x=0, at which point we know we have reached the maximal value for y.

Here is a simple schema I made of the state machine that could be implemented for template-only downloads:

state machine

The question is: how to architecture the software in a way that:

  1. allows both template-only downloads and traditional dezoomers when all the tile URLs are known in advance.
  2. is efficient (always downloads as many tiles in parallel as possible)
  3. is testable
  4. avoids code duplication
  • 3
    This is a good architectural question for Software Engineering. While it appears to be Too Broad at first glance, the bottom of the post provides specific guidance for directing answers. – Robert Harvey Aug 21 '19 at 17:08

Here is a sketch of an algorithm which should do the trick - functional tools to the rescue:

  • Implement your core downloader not in terms of a list of URLs, but in terms of a stream (or generator) of URLs (I did not do this in Javascript or Rust by myself, only in some other languages, but found several information in the web that both languages support these concepts).

  • The downloader fills the parallel download queue with up to N elements, where N is the maximum number of concurrent downloads allowed. Note such an implementation should be able to work on a "finite" stream as well as on an "infinite" stream (based on a lazy generator function). Getting a 404 then should work as an additional stopping criterion.

  • The downloaded images need to be returned through some asynchronous event (or callback function) to pass them to the "stitcher". This mechanics could also be used to inform the caller about failed downloads.

Now you can this reuse for your traditional downloader as well as for your template-only downloads. The first one should be obvious. The second one works exactly in the two-step way you have scetched above, with the difference that this describes the order of URLs generated for the stream.

  • First, you start with an "infinite" stream of tiles (1,1), (2,1), (3,1), ... to find the limit for x and download the first row

  • Then you provide the URLS for (1,2), (2,2), ..., (MaxX, 2), (1,3), (2,3), ..., also as an "infinite stream", until the download stops.

It should be obvious that this fulfills your requirements 1, 2 and 4. Requirement #3 is quite orthogonal to any algorithm here, one gets the kind of testability you asked for (working without network requests) by making the download functionality replaceable by some "mock" download function.

Hope this was clear enough, if not, don't hesitate to ask.

| improve this answer | |
  • 1
    one gets "testability" by making the download functionality replaceable by some "mock" download function. -- One gets "testability" by writing your tests first. Writing your tests first identifies the ideal API for your functions, not only identifying mockable parameters, but also minimizing the use of mocks. – Robert Harvey Aug 21 '19 at 19:27
  • Thank you for the answer! This idea seems to be the way to go, but there are design questions that still need to be solved: what should be the interface between the dezoomer and the core? The dezoomer has to be able to provide a stream of tile URLs, but it has to be somehow notified of the point at which the stream was stopped in order to find MaxX... – lovasoa Aug 21 '19 at 19:27
  • @lovasoa: Don't expect to obtain an entire design from whole cloth here. That's your job. Our job is to point you in the direction that best suits your specific requirements. – Robert Harvey Aug 21 '19 at 19:30
  • 1
    I think these kind of details are best worked out in code - I gave you a starting point, now its up to you to make this work. – Doc Brown Aug 21 '19 at 19:55
  • 2
    And concerning Rust: all you need to make this work are (lazy) iterators and higher order functions - Rust seems to have both. – Doc Brown Aug 21 '19 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.