2

Problem

I want to design the class architecture for an agent that interacts with an environment by repeating three steps until the end of an episode: First, the environment provides an observation to the agent. Second, the agent chooses an action in response. Third, the environment rewards the action by a reward. The agent is composed of partial behaviors, for example:

  • Act randomly with a given probability and delegate to the next behavior otherwise.
  • Simplify the observations and always have the next higher behavior handle them.
  • Normalize rewards into a certain range but leave everything else to the next behavior.
  • Forward every other observation to the next behavior, and ignore the others. Repeat the next behavior's actions during the ignored observations.
  • Choose an action based on the current observation and a strategy learned from the past.

Behaviors

Generally, behaviors receive observations and either return an action, or forward a modified observation to the next behavior. They will eventually get back an action that they can modify and pass back to the previous behavior. However, they can also create and send a new observation to the next behavior first. Each behavior must receive its reward at some point after returning an action and before receiving the next observation.

Example

In summary, observations, actions, and rewards can flow between the behaviors before producing the action for the environment. Here is an example interaction of two behaviors with an environment (observations are blue, actions are red, rewards are yellow):

example interactions of two behaviors and the environment

Question

The behaviors should not know of each other, so that I can add additional behaviors in between existing ones or stop the simulation at any time. How can I design this? Are there any patterns to simplify the dynamic interaction between my behaviors?

  • Do you have any Interface definitions that we can look at? – Robert Harvey Aug 12 '16 at 0:12
  • @RobertHarvey An interface for Behavior would be a observe(x): a or x method to receive observations from the previous behavior and a perform(a): a or x method to receive actions from the next behavior. Both return actions for the previous or observations for the next behavior. Here is an agent implementation for this design. However, it does not take rewards into account. – danijar Aug 12 '16 at 0:31
1

Your formulation lends itself to actors. Just replace "agent" with "actor" and your arrows with "messages" and you're there.

You could take a look at actor libraries like Akka or Actor-based languages like Erlang. The general idea of actor models is also well-described on wikipedia.

Basically, an actor model addresses your requirements nicely:

  • Delegation of behavior: Just send the message to another actor
  • Hierarchy: Actors are typically already organized in a hierarchy. Usually, this is for fault tolerance, as the higher actors are supervisors, but it may suit your need of letting "the next higher behavior handle" the messages.
  • Forward every other observation: You can add an actor specifically with this forwarding behavior in place, independent of the other behaviors/actors.
  • Choose an action: Basics of actors, as they always have to choose what to do when an incoming message arrives.
  • Timing constraints: message sending for actors directly supports concepts like: wait for an answer, before processing further messages.

Additionally, libraries like Akka also support explicit behavior of actors and changing it. Think of it as replacing the method that handles incoming messages by a complete different method doing different things for the same messages. It's up to you to find out, whether your use cases profit more from having different actors per behavior, or whether modelling actors with changing behaviors is more appropriate.

Edit - to address your concerns:

Yes, actors are inherently parallel, however, the actual communication sequence can be sequentialized. If you let your actors block and wait for an answer, you essentially get a synchronous process (albeit I agree it may no longer be a natural match).

The order could be determined by the environment actor. You send the observation, wait for the resulting action and send the reward. Or behavior-changing actors could be used to design this sequence (i.e. after sending the observation, the actor changes its behavior to a new one in which it can only accept actions).

In terms of having to respond in a certain way, the concept of channels may be interesting to you. In Akka (not sure about Erlang there) you can define typed channels, which basically gives you a type-safe communication where only messages of certain types can be exchanged via the channel. Hence, you can ensure things like the response having to be this or that.

  • Thanks, I see some problems, though: My process is synchronous but actors run in parallel. Moreover, I want to enforce certain rules for the Behaviors instances, such as (1) the order of steps in each local observation, action, and reward cycle, or (2) that a behavior must respond with one of two options. Both seem to be hard to implement using actors. What do you think? – danijar Aug 12 '16 at 6:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.