Introduction
Let's say that you have such design in your application:
- application (laravel)
- web scraper (used by application)
- query builder (used by web scraper)
The application is what actually uses the Web Scraper to fetch the data from the web, however since we use the Web Scraper as a search engine as well, we build the query string used from the Web Scraper based on user input.
The Web Scraper is the library that actually deals with extracting and parsing HTML files taken from the sites we work in.
The Query Builder makes use of some remote APIs (ie. google maps for coordinates based on user input) to be used in the Web Scraper.
Code Example
class WebsiteScraper
{
public function city($city)
{
$this->parameters['city'] = CityIdentifierFactory::retrieve($city);
}
}
class CityIdentifier
{
const API_URL = "http://maps.google.com/maps/api/geocode/json";
public function __construct(\GuzzleHttp\Client $client)
{
$this->client = $client;
}
public function retrieve($city)
{
try
{
$coordinates = $this->getCoordinates($city);
if (count($coordinates) <= 0) {
return [];
}
return $coordinates;
} catch (Exception $e) {
throw new QueryBuilderCityFetchingException(
"Whoops! Something wrong happened."
);
}
}
private function getCoordinates($city)
{
$response = $this->client->get(self::API_URL, [
'query' => [
'address' => $city,
'region' => 'IT'
]
]);
$body = json_decode($response->getBody()->getContents());
return [[
'city' => $city,
'lat' => $body->results[0]->geometry->location->lat,
'lng' => $body->results[0]->geometry->location->lng,
'location' => $city
]];
}
}
class CityIdentifierFactory
{
public static function retrieveFromDisk($name)
{
$identifier = new FilesystemCityIdentifier(__DIR__ . "/../../assets/provinces.json");
return $identifier->retrieve($name);
}
public static function retrieve($name)
{
$identifier = new CityIdentifier(new Client);
return $identifier->retrieve($name);
}
}
Problem
The problem is that the Web Scraper is ran thousands of times a day, and Google Maps API have limits that we should not reach (even though we reached them already! :( ). I would like to implement a caching layer in the CityIdentifier component of the Query Builder, so that I avoid to make thousands of queries related to the same city.
Question
Where should I implement this caching layer, knowing that all of these things are going to be dependency-injected inside Laravel (4.2)?
Should it be in the Web Scraper, in the query builder, or in the actual CityIdentifier component?
Where should cache be saved? In an internal folder of the query builder, or in some application folder that is configured in the library?