How to design the interactions between these agent behaviors?

Question

Problem

I want to design the class architecture for an agent that interacts with an environment by repeating three steps until the end of an episode: First, the environment provides an observation to the agent. Second, the agent chooses an action in response. Third, the environment rewards the action by a reward. The agent is composed of partial behaviors, for example:

• Act randomly with a given probability and delegate to the next behavior otherwise.
• Simplify the observations and always have the next higher behavior handle them.
• Normalize rewards into a certain range but leave everything else to the next behavior.
• Forward every other observation to the next behavior, and ignore the others. Repeat the next behavior's actions during the ignored observations.
• Choose an action based on the current observation and a strategy learned from the past.

Behaviors

Generally, behaviors receive observations and either return an action, or forward a modified observation to the next behavior. They will eventually get back an action that they can modify and pass back to the previous behavior. However, they can also create and send a new observation to the next behavior first. Each behavior must receive its reward at some point after returning an action and before receiving the next observation.

Example

In summary, observations, actions, and rewards can flow between the behaviors before producing the action for the environment. Here is an example interaction of two behaviors with an environment (observations are blue, actions are red, rewards are yellow):

Question

The behaviors should not know of each other, so that I can add additional behaviors in between existing ones or stop the simulation at any time. How can I design this? Are there any patterns to simplify the dynamic interaction between my behaviors?

Answer 1

Your formulation lends itself to actors. Just replace "agent" with "actor" and your arrows with "messages" and you're there.

You could take a look at actor libraries like Akka or Actor-based languages like Erlang. The general idea of actor models is also well-described on wikipedia.

Basically, an actor model addresses your requirements nicely:

• Delegation of behavior: Just send the message to another actor
• Hierarchy: Actors are typically already organized in a hierarchy. Usually, this is for fault tolerance, as the higher actors are supervisors, but it may suit your need of letting "the next higher behavior handle" the messages.
• Forward every other observation: You can add an actor specifically with this forwarding behavior in place, independent of the other behaviors/actors.
• Choose an action: Basics of actors, as they always have to choose what to do when an incoming message arrives.
• Timing constraints: message sending for actors directly supports concepts like: wait for an answer, before processing further messages.

Additionally, libraries like Akka also support explicit behavior of actors and changing it. Think of it as replacing the method that handles incoming messages by a complete different method doing different things for the same messages. It's up to you to find out, whether your use cases profit more from having different actors per behavior, or whether modelling actors with changing behaviors is more appropriate.

Edit - to address your concerns:

Yes, actors are inherently parallel, however, the actual communication sequence can be sequentialized. If you let your actors block and wait for an answer, you essentially get a synchronous process (albeit I agree it may no longer be a natural match).

The order could be determined by the environment actor. You send the observation, wait for the resulting action and send the reward. Or behavior-changing actors could be used to design this sequence (i.e. after sending the observation, the actor changes its behavior to a new one in which it can only accept actions).

In terms of having to respond in a certain way, the concept of channels may be interesting to you. In Akka (not sure about Erlang there) you can define typed channels, which basically gives you a type-safe communication where only messages of certain types can be exchanged via the channel. Hence, you can ensure things like the response having to be this or that.