There's a lot loaded in this question. But let's address the first cry for help.
Some one help me with my design paralysis!
- Step 1: stop worrying about "Best practices" and "perfect"
- Step 2: identify 1-3 things that works
- Step 3: figure out which of them is going to fit your problem best
Ok. With that out of the way let's unpack this a bit. You have a service that depends on multiple other services. And you are thinking about the fail state, that's good.
When communicating failure states back to a service's client:
- Identify the appropriate status code
- 424, Failed dependency seems most appropriate for your example
- 503, Service unavailable, also appropriate
- Identify what the message body should include
- Do you include downstream error state? (may or may not be appropriate)
- How about a JSON response with any additional information?
- Or is just the code OK?
- Use logging and log shipping for the rest
Not industry standard, but a personal gripe is that the 500 internal server error is the default error code when your application fails to catch an exception. As a result you may leak information about your code outside of the service that can be used by bad people to exploit it later. When I see a 500 error, the first thing I think is that someone can't handle errors appropriately. Sometimes it is legitimately the most correct response, but most of the time there is something better.
Asynchronous vs. Synchronous programming has their trade-offs. The biggest trade-off is complexity vs. handling more requests with fewer resources.
My personal recommendation is to get something working first. After that, you can see precisely where the design flaws really are. Sometimes it's just not what you assumed it was up front.
I think it's more common to focus on orchestrating multiple instances of microservices before looking at how to rearchitect them. Orchestration is the process of spinning up new instances of services or tearing down instances you no longer need in an automated fashion. Once you have the orchestration problem solved, it becomes easier to look at how to re-architect things.
Any time you have one service depend on another service, you need to look at how you intend to take advantage of multiple instances of that service. Tools that help in that arena are:
- Service Discovery (like Consul and Eureka)
- The circuit breaker pattern (like Hystrix)
- Automatic retry in a load balanced way
Since you have a complicated set up, with services that depend on other services, you also should look into request tracing. In the Spring world the project would be Zipkin, but there are many implementations of the open tracing pattern. There are also multiple server side pieces that can consume the output of the tracing pattern. For example the Elastic Stack has tooling for that built in.
Narrow your focus on the specific problem you are trying to solve, and then solve it. As you expand your view, you will see other needs to support and maintain your software. Add those needs to the backlog and work them off as you can.
When it comes to microservices, I think there are a lot of architectural patterns to pull from but no settled "right way" to do things. There is tooling that helps address a common set of problems. You can start small and simply and grow in to the complete solution over time.