API Design for Error Handling

Question

I'm working on a semi-complex web API, the high level design so far looks like this:

|--------|  |------------|  |------------|
|        |--|            |  |            |
|  core  |  |            |  |            |
|        |  |            |  |            |
|--------|  |            |  |            |
    |       | controller |--|     API    |
    |       |            |  |            |
|--------|  |            |  |            |
|        |--|            |  |            |
| engine |  |            |  |            |
|        |  |            |  |            |
|--------|  |------------|  |------------|

From bottom to top, the core is connected with the 'outside' (DB and third-party APIs), in the engine is where all the processing of the data happens (I have some ML models), the data to process comes mainly from the core, so there is a constant flow of data from core to engine (raw data) and from engine back to core (processed data). In the controller I have all the functions used in the API (I have the cache and the prefetch system running here). And lastly, in the API module, I handle the requests and call the functions in the controller.

I have a couple of questions:

• Is this a good architecture? I have no experience in software engineering, and I basically create this four blocks just to have the all the code organized, so I'm not sure if the architecture is well designed for scale, any tips here?
• Also, I'm not sure where should I be handling errors: This may sound like a silly question, but I'm not sure in which part of the API should I handle things like invalid parameters or unexpected errors in the code. For example, lets say I have an endpoint /weather with a country parameter to get the weather in this country, where should I handle an invalid country parameter (or an unexpected error in the code), I initially thought that I should catch this as soon as possible and return an error message, but then the low level components wouldn't be able to handle any errors.

This is an example, (using Django for the API):

*api.py*
def get_weather(request, country):
    weather = package.controller.get_weather(country)
    return JsonResponse(weather)


*controller.py*
@cache # This is where the endpoints are cached
def get_weather(country):
    calculated_weather = core.calculate_weather(country)
    return calculated_weather

*engine/weather.py*
def calculate_weather(country):
    raw_data = engine.fetch_raw_weather_data(country)
    calculate_weather


*core/weather.py*
def fetch_raw_weather_data(country):
    # Fetch the data using third party APIs
    return data

The actual API is way more complex but I think it shows the situation. In theory an invalid country name would raise an exception in the engine or the core, depending on the case, if I catch this kind of errors in the controller, then I can return the error message to the API. I'm not sure where to catch unexpected errors, like third party APIs exceptions, or issues on processing the data, should I generate the error message there (in core and engine) and try to return it to the API, or should I manually raise exceptions and handle the message in the api.py module?.

Any suggestions or tips to improve this?

Answer 1

Let me suggest:

• The REST API needs to handle any validation errors and transform to any Domain rich objects. For example, check that country has a valid ISO code. Aso nullability, types, minimum length or date consistency, etc. Any of this errors should be handled with the correspondent bad request with a useful message. Unexpected exceptions should be converted to 500 error with no trace information and infrastructure errors can report a generic information like "failed to retrieve weather from X". This may be a good place for translating errors if multi-language is an issue.
• If the core is connecting to the database and other web-services you may want to separate layers in an Hexagonal architecture. But this depends on the amount of logic that you have. In this case, your infrastructure errors could be transformed. Rest or DDBB error getting the Weather becomes a generic WeatherRepositoryError infrastructure agnostic.
• In your core there is no problem in throwing exceptions up for Panics (Nullpointer, IndexOutOfBound). Just make sure that they are catch and logged in the API. You will work hard so that those errors and logs tend to zero. How? converting them to infrastructure errors (Service X failed) or domain errors (Weather of Poland still not calculated)
• Infrastructure error are the hardest: the most important thing is what do you want to do with them. Resilience patterns? Show better messages?
• If you are very functional you can even try Railway Oriented Programming but make sure to read the WARNING section before.

Anyway. Error handling is hard and Django is an opinionated framework that I have only used as a hobby so things may be a bit special there