Your formulation lends itself to actors. Just replace "agent" with "actor" and your arrows with "messages" and you're there.
You could take a look at actor libraries like Akka or Actor-based languages like Erlang. The general idea of actor models is also well-described on wikipedia.
Basically, an actor model addresses your requirements nicely:
- Delegation of behavior: Just send the message to another actor
- Hierarchy: Actors are typically already organized in a hierarchy. Usually, this is for fault tolerance, as the higher actors are supervisors, but it may suit your need of letting "the next higher behavior handle" the messages.
- Forward every other observation: You can add an actor specifically with this forwarding behavior in place, independent of the other behaviors/actors.
- Choose an action: Basics of actors, as they always have to choose what to do when an incoming message arrives.
- Timing constraints: message sending for actors directly supports concepts like: wait for an answer, before processing further messages.
Additionally, libraries like Akka also support explicit behavior of actors and changing it. Think of it as replacing the method that handles incoming messages by a complete different method doing different things for the same messages. It's up to you to find out, whether your use cases profit more from having different actors per behavior, or whether modelling actors with changing behaviors is more appropriate.
Edit - to address your concerns:
Yes, actors are inherently parallel, however, the actual communication sequence can be sequentialized. If you let your actors block and wait for an answer, you essentially get a synchronous process (albeit I agree it may no longer be a natural match).
The order could be determined by the environment actor. You send the observation, wait for the resulting action and send the reward. Or behavior-changing actors could be used to design this sequence (i.e. after sending the observation, the actor changes its behavior to a new one in which it can only accept actions).
In terms of having to respond in a certain way, the concept of channels may be interesting to you. In Akka (not sure about Erlang there) you can define typed channels, which basically gives you a type-safe communication where only messages of certain types can be exchanged via the channel. Hence, you can ensure things like the response having to be this or that.