Information Channels

• Information usually comes from multiple sources: observed entities, different sensors, communication channels.
• Agents can differentiate between information channels, because they occupy always the same vector positions.

Entities Features

• Often, a subset of identical features describes different observed entities.
• This is a graph structure!
• Self-Attention can be used to learn the edges of the latent graph Li et al., 2021

$$ \begin{eqnarray} \mathbf{O}_t^a = \begin{bmatrix} ent_1 \\ \vdots \\ ent_k \end{bmatrix}_t^a = \begin{bmatrix} f_{1,1} & \cdots & f_{1,z} \\ \vdots & \ddots & \vdots \\ f_{k,1} & \cdots & f_{k,z} \end{bmatrix} _t^a \end{eqnarray} $$

$$ \begin{eqnarray} \mathbf{S}_t = \begin{bmatrix} ent_1 \\ \vdots \\ ent_k \end{bmatrix}_t = \begin{bmatrix} f_{1,1} & \cdots & f_{1,z} \\ \vdots & \ddots & \vdots \\ f_{k,1} & \cdots & f_{k,z} \end{bmatrix}_t \end{eqnarray} $$

• Some information get lost when dropping concatenation:
  • Which of the vertices represent "me"?
  • Which of the vertices are "collaborative agents"?
• Easier solutions are flags into the vertex features.

$$ \begin{eqnarray} f_{i,\texttt{IS_SELF}}^a = \begin{cases} 1, & \text{if } i = a\\ 0, & \text{otherwise.} \end{cases} \end{eqnarray} $$

$$ \begin{eqnarray} f_{i, \texttt{IS_AGENT}}^a = \begin{cases} 1, & \text{if } i \in A\\ 0, & \text{otherwise.} \end{cases} \end{eqnarray} $$

Transformer Mixer

Transformer hypernetwork:

• Agents hidden states (\(W_1\)): q-values embedding.
• Recurrent mechanism: (\(b_1, W_2, b_2\)): projection over \(Q_{tot}\).
• State embedder: environment reasoning.

Transformer Agent

Similar architecture to UPDET, but:

• Q-Values are derived from the agent hidden state, reinforcing recurrent passing of gradient.
• Separate output layer used for Policy Decoupling.