3

I'm working on a semi-complex web API, the high level design so far looks like this:

|--------|  |------------|  |------------|
|        |--|            |  |            |
|  core  |  |            |  |            |
|        |  |            |  |            |
|--------|  |            |  |            |
    |       | controller |--|     API    |
    |       |            |  |            |
|--------|  |            |  |            |
|        |--|            |  |            |
| engine |  |            |  |            |
|        |  |            |  |            |
|--------|  |------------|  |------------|

From bottom to top, the core is connected with the 'outside' (DB and third-party APIs), in the engine is where all the processing of the data happens (I have some ML models), the data to process comes mainly from the core, so there is a constant flow of data from core to engine (raw data) and from engine back to core (processed data). In the controller I have all the functions used in the API (I have the cache and the prefetch system running here). And lastly, in the API module, I handle the requests and call the functions in the controller.

I have a couple of questions:

  • Is this a good architecture? I have no experience in software engineering, and I basically create this four blocks just to have the all the code organized, so I'm not sure if the architecture is well designed for scale, any tips here?
  • Also, I'm not sure where should I be handling errors: This may sound like a silly question, but I'm not sure in which part of the API should I handle things like invalid parameters or unexpected errors in the code. For example, lets say I have an endpoint /weather with a country parameter to get the weather in this country, where should I handle an invalid country parameter (or an unexpected error in the code), I initially thought that I should catch this as soon as possible and return an error message, but then the low level components wouldn't be able to handle any errors.

This is an example, (using Django for the API):

*api.py*
def get_weather(request, country):
    weather = package.controller.get_weather(country)
    return JsonResponse(weather)


*controller.py*
@cache # This is where the endpoints are cached
def get_weather(country):
    calculated_weather = core.calculate_weather(country)
    return calculated_weather

*engine/weather.py*
def calculate_weather(country):
    raw_data = engine.fetch_raw_weather_data(country)
    calculate_weather


*core/weather.py*
def fetch_raw_weather_data(country):
    # Fetch the data using third party APIs
    return data

The actual API is way more complex but I think it shows the situation. In theory an invalid country name would raise an exception in the engine or the core, depending on the case, if I catch this kind of errors in the controller, then I can return the error message to the API. I'm not sure where to catch unexpected errors, like third party APIs exceptions, or issues on processing the data, should I generate the error message there (in core and engine) and try to return it to the API, or should I manually raise exceptions and handle the message in the api.py module?.

Any suggestions or tips to improve this?

5
  • 2
    I'll don't have time to write an answer, but I wanted to leave a small comment about terminology (just for future reference). The term API refers to just the interface & associated protocols (how you call stuff, which parameters you pass, what they mean, what kind of response is returned). The thing you're building is not an API, it's a web service; an API (application programming interface) is just what the service exposes externally. All kinds of other things have an API (components, libraries, frameworks), it's not a term closely tied to just the web and RESTful services. Nov 11 at 20:23
  • @FilipMilovanović I should have been more precise with the terminology, thanks for pointing that out! I have a question then, although what I'm building is a web service would it be correct to say that the layer API is actually the API of this service, even if it contains logic?
    – Stack
    Nov 11 at 20:38
  • 1
    @Stack (cool user name btw), regarding terminology, the api in your design is usually called controller and contains the logic to process requests and produce responses. What you call controller is called service, e.g. in Java/Spring Nov 11 at 22:00
  • 2
    @Stack, regarding your question, the overall design looks good, I would raise exceptions in the inner layers and let the api layer turn that into meaningful HTTP response messages or appropriate HTTP error codes Nov 11 at 22:03
  • 1
    "would it be correct to say that the layer API is actually the API of this service" - I think that's OK. I guess on some level you could say that the layer is a specific implementation, or a "materialization", of the API (the API itself being a conceptual thing that can, in principle, have several different implementations, all of which facilitate the same interface). But informally, sure, you could say something like the "API layer" or label it just as "API" on a diagram - I think pretty much everyone would be OK with that. Nov 12 at 14:31
2

Let me suggest:

  • The REST API needs to handle any validation errors and transform to any Domain rich objects. For example, check that country has a valid ISO code. Aso nullability, types, minimum length or date consistency, etc. Any of this errors should be handled with the correspondent bad request with a useful message. Unexpected exceptions should be converted to 500 error with no trace information and infrastructure errors can report a generic information like "failed to retrieve weather from X". This may be a good place for translating errors if multi-language is an issue.
  • If the core is connecting to the database and other web-services you may want to separate layers in an Hexagonal architecture. But this depends on the amount of logic that you have. In this case, your infrastructure errors could be transformed. Rest or DDBB error getting the Weather becomes a generic WeatherRepositoryError infrastructure agnostic.
  • In your core there is no problem in throwing exceptions up for Panics (Nullpointer, IndexOutOfBound). Just make sure that they are catch and logged in the API. You will work hard so that those errors and logs tend to zero. How? converting them to infrastructure errors (Service X failed) or domain errors (Weather of Poland still not calculated)
  • Infrastructure error are the hardest: the most important thing is what do you want to do with them. Resilience patterns? Show better messages?
  • If you are very functional you can even try Railway Oriented Programming but make sure to read the WARNING section before.

Anyway. Error handling is hard and Django is an opinionated framework that I have only used as a hobby so things may be a bit special there

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.