I'm writing a pet project to scrape some websites, and write out results to HTML.
Currently my data model and interfaces are:
class Post: def __init__(self, title, description, images) self.title = title self.description = description self.images = images class Image: # Images do not store their data; they instead contain a guid, which can be used to retrieve the # associated data from a kv-store like interface (which may be just a file system) def __init__(self, guid, description): self.guid = guid self.description = description class DataStore: def load(self): # returns all Posts. Note that Image objects do _not_ have a reference to their data. pass def outputHTML(posts, fp): # write out HTML. Include links to the Images within each Post
Note that an
Image does not have a reference to the data, but instead stores a key/guid which can be used to retrieve data from a kv store. This was intentional given that clients (eg:
outputHTML) do not need to access the data - they just need to write out a link to the data. Clients can use a provided kv store to retrieve the data if desired. Additionally this decision was made to ensure that a load of the data did not unnecessarily read file data.
However I run into some conceptual issues when considering the return type of a
class Scraper: def scrape(self): # returns a Post # needs to also return the _data_ associated with an Image
Scraper needs to source the image data from the web page. It also constructs and returns the
Post data. However the
Post reference can not store the binary data, so I'm not sure how this should be returned to the caller.
Imagerequires a guid, so I could pass in a
kv storeto the
scrapemethod and have the
Scraperinsert the binary data directly into the
kv store. However I'm resisting this: the scraper should return the data in memory, plain and simple. It's a client responsibility as to what to do with the data.
- I could have the
Scraperreturn both a
Postand a dictionary which maps guids of
Imageobjects contained by the
Postto binary data. This doesn't feel elegant as I'm now returning two objects.
- I could have the
scrapemethod return a distinct class isomorphic to
Post, which would contain an equivalent
Imageclass which actually holds the binary data. However these duplication feels unnecessary.
- I toyed with the idea of having two subclasses of
Image: one which contains the data, the other which contains a link to the data. But there is no common interface between the two (eg: what should
getGuidreturn for the subtype which stores the data directly?)
Criticisms/suggestions welcome! I feel that this conflict is arising from insisting the return type of
scrape being coupled to the concept of a kv store via the
guid parameter in