I want to design the class architecture for an agent that interacts with an environment by repeating three steps until the end of an episode: First, the environment provides an observation to the agent. Second, the agent chooses an action in response. Third, the environment rewards the action by a reward. The agent is composed of partial behaviors, for example:
- Act randomly with a given probability and delegate to the next behavior otherwise.
- Simplify the observations and always have the next higher behavior handle them.
- Normalize rewards into a certain range but leave everything else to the next behavior.
- Forward every other observation to the next behavior, and ignore the others. Repeat the next behavior's actions during the ignored observations.
- Choose an action based on the current observation and a strategy learned from the past.
Generally, behaviors receive observations and either return an action, or forward a modified observation to the next behavior. They will eventually get back an action that they can modify and pass back to the previous behavior. However, they can also create and send a new observation to the next behavior first. Each behavior must receive its reward at some point after returning an action and before receiving the next observation.
In summary, observations, actions, and rewards can flow between the behaviors before producing the action for the environment. Here is an example interaction of two behaviors with an environment (observations are blue, actions are red, rewards are yellow):
The behaviors should not know of each other, so that I can add additional behaviors in between existing ones or stop the simulation at any time. How can I design this? Are there any patterns to simplify the dynamic interaction between my behaviors?