I'm writing a pet project to scrape some websites, and write out results to HTML.

Currently my data model and interfaces are:

class Post:
    def __init__(self, title, description, images)
        self.title = title
        self.description = description
        self.images = images

class Image:
    # Images do not store their data; they instead contain a guid, which can be used to retrieve the
    # associated data from a kv-store like interface (which may be just a file system)
    def __init__(self, guid, description):
        self.guid = guid
        self.description = description

class DataStore:
    def load(self):
        # returns all Posts. Note that Image objects do _not_ have a reference to their data.

def outputHTML(posts, fp):
    # write out HTML. Include links to the Images within each Post

Note that an Image does not have a reference to the data, but instead stores a key/guid which can be used to retrieve data from a kv store. This was intentional given that clients (eg: outputHTML) do not need to access the data - they just need to write out a link to the data. Clients can use a provided kv store to retrieve the data if desired. Additionally this decision was made to ensure that a load of the data did not unnecessarily read file data.

However I run into some conceptual issues when considering the return type of a Scraper.

class Scraper:
    def scrape(self):
        # returns a Post
        # needs to also return the _data_ associated with an Image

A Scraper needs to source the image data from the web page. It also constructs and returns the Post data. However the Images the Post reference can not store the binary data, so I'm not sure how this should be returned to the caller.

  • The Image requires a guid, so I could pass in a kv store to the scrape method and have the Scraper insert the binary data directly into the kv store. However I'm resisting this: the scraper should return the data in memory, plain and simple. It's a client responsibility as to what to do with the data.
  • I could have the Scraper return both a Post and a dictionary which maps guids of Image objects contained by the Post to binary data. This doesn't feel elegant as I'm now returning two objects.
  • I could have the scrape method return a distinct class isomorphic to Post, which would contain an equivalent Image class which actually holds the binary data. However these duplication feels unnecessary.
  • I toyed with the idea of having two subclasses of Image: one which contains the data, the other which contains a link to the data. But there is no common interface between the two (eg: what should getGuid return for the subtype which stores the data directly?)

Criticisms/suggestions welcome! I feel that this conflict is arising from insisting the return type of scrape being coupled to the concept of a kv store via the guid parameter in Image.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.