1

Introduction

Let's say that you have such design in your application:

- application  (laravel)
  - web scraper (used by application)
    - query builder (used by web scraper)

The application is what actually uses the Web Scraper to fetch the data from the web, however since we use the Web Scraper as a search engine as well, we build the query string used from the Web Scraper based on user input.

The Web Scraper is the library that actually deals with extracting and parsing HTML files taken from the sites we work in.

The Query Builder makes use of some remote APIs (ie. google maps for coordinates based on user input) to be used in the Web Scraper.

Code Example

class WebsiteScraper 
{
    public function city($city)
    {
        $this->parameters['city'] = CityIdentifierFactory::retrieve($city);
    }
}
class CityIdentifier
{
    const API_URL = "http://maps.google.com/maps/api/geocode/json";

    public function __construct(\GuzzleHttp\Client $client)
    {
        $this->client = $client;
    }

    public function retrieve($city)
    {
        try
        {
            $coordinates = $this->getCoordinates($city);

            if (count($coordinates) <= 0) {
                return [];
            }

            return $coordinates;
        } catch (Exception $e) {
            throw new QueryBuilderCityFetchingException(
                "Whoops! Something wrong happened."
            );
        }
    }

    private function getCoordinates($city)
    {
        $response = $this->client->get(self::API_URL, [
            'query' => [
                'address' => $city,
                'region'  => 'IT'
            ]
        ]);

        $body = json_decode($response->getBody()->getContents());

        return [[
            'city'     => $city,
            'lat'      => $body->results[0]->geometry->location->lat,
            'lng'      => $body->results[0]->geometry->location->lng,
            'location' => $city
        ]];
    }
}
class CityIdentifierFactory
{
    public static function retrieveFromDisk($name)
    {
        $identifier = new FilesystemCityIdentifier(__DIR__ . "/../../assets/provinces.json");
        return $identifier->retrieve($name);
    }

    public static function retrieve($name)
    {
        $identifier = new CityIdentifier(new Client);
        return $identifier->retrieve($name);
    }
}

Problem

The problem is that the Web Scraper is ran thousands of times a day, and Google Maps API have limits that we should not reach (even though we reached them already! :( ). I would like to implement a caching layer in the CityIdentifier component of the Query Builder, so that I avoid to make thousands of queries related to the same city.

Question

Where should I implement this caching layer, knowing that all of these things are going to be dependency-injected inside Laravel (4.2)?

Should it be in the Web Scraper, in the query builder, or in the actual CityIdentifier component?

Where should cache be saved? In an internal folder of the query builder, or in some application folder that is configured in the library?

  • To be honest no one will care where you save your cache, however, you might want him to expire. I see three ways : either you cache the raw content of your query as index of your cache (easy), 2- you can compute if queries are the same even if parameters are not in the same order (harder), 3- you know every data that can be asked by your application, when a specific city is asked, you download all information to your cache and perform the research only in the cache (easy to very hard depending on your needs) – Walfrat Apr 6 '17 at 12:31
3

Preface: I know you are asking about PHP. I have no PHP experience but here is what I would do as a Java developer. I do not know how relevant this makes my answer.

You wrote:

I would like to implement a caching layer in the QueryBuilder

This sounds like a good case for a Decorator. Here you would wrap a "regular" QueryBuilder and add caching behavior. The code might look something like this:

class CachingQueryBuilder implements QueryBuilder {

    QueryBuilder basicQueryBuilder;
    Cache cache;

    @Override Query buildQuery(String input) {

        if(cache.contains(input)) {
            //returned cached contents when possible
            return cache.get(input);
        } else {
            //build query manually, cache it, and return it
            Query q = basicQueryBuilder.buildQuery(input);
            cache.put(q);
            return q;
        } 
    }
}

Google's Guava API (a Java resource) has some great caches at: Guava Caches.

  • Why should I use a decorator? There isn't a situation where I need to use the QueryBuilder without the caching layer. – GiamPy Apr 6 '17 at 13:10
  • @GiamPy -- I would use a decorator to cleanly separate "relying on the cache to provide full (or partial answers)" from "building a Query when the cache is empty". Any system with a cache MUST handle the case where the cache misses (say due to an empty cache or a rare query). So, I disagree when you say "there isn't a situation where I need to use a QB without a cache". A system without a cache is the same as a system with a cache that happens to be empty. A Decorator will simplify your BasicQB because it won't be littered with if-else blocks that depend on "cache.Contains(input)" – Ivan Apr 6 '17 at 13:21
  • Thing is I don't need the caching layer for the entire query builder. I phrased it wrongly, sorry. I need it only for the CityIdentifier class, which goes on the net to fetch some data, and I want to avoid that for repeated identical calls. – GiamPy Apr 6 '17 at 13:43
  • That doesn't change the point of this answer. Your CityIdentifier will be able to fetch informations about a city, while a CachedCityIdentifier will use the cached informations if it exists, or use CityIdentifier otherwise. How much data is cached does not change the need for another class since it's another behavior. – Steve Chamaillard Dec 19 '17 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.