Categorical Agent

I recently completed the first round of testing on the first version of the categorical agent. I messed with adding actions that when given the correct parameters directly affected the reward by a very small amount either probabilistically, in a set amount or when completed in a particular sequence. In each case after a few minutes the agent converged on a policy where it would ONLY select the actions which affected its reward. All cases worked as desired except the case where the correct pattern of actions resulted in the reward. In this case the agent appeared to choose almost randomly between the two actions rather than always completing them in order. The agent can't yet represent its state space in terms of actions previously taken, so this is the expected behavior at the moment. A future revision should fix this.

There's still a lot of work to do - detailed below:

1) Do something about cycles in world representation. The agent occasionally creates cyclic mental structures which result in crashes. Either disallow cyclic structures (simple) or do some more complicated things to prevent getting stuck in cycles (better if we want our agent to be able to reason about cyclic mental structures). If I can't think of a reason to allow a cyclic mental structure I'm just going to disallow them.

2) The agent needs a mechanism for restricting the input of an action to create a more specific action. Moving in a random direction probably won't do anything to increase reward, but moving in a specific direction will.

3) The agent needs to log previous actions taken with inputs, outputs, and the resultant rewards in its mental conceptual structure. This allows the agent to reason about past history, allowing it to always pick the correct pattern of actions to result in the optimal reward.

4) The agent needs to be able to restrict it's state space. Currently the agent's state space contains all actions and all categories. This works fine for the agent in a simple enough environment, but will not scale well for more complicated environments (the agent will become painfully slow). The agent needs to be able to reason about it's mental state space representations' categories and actions and add or remove actions and categories when mutating the state space.

5) Allow categorization of all concepts by their parent concepts if they are in a composite structure. "Distance OF a red ball" vs. "Distance".

6) Perhaps I should make all concepts composites and allow them to have child members for simpler recursion traversal (not sure about this - the agent would have more structural options, but would they be useful?).

The links below give some detailed design information for how the categorical agent works.

Categorical Agent Design

Knowledge Representation