How are we using information in MARL?

• State-of-the-art approaches in MARL (MADDPG, VDN, QTran, QMix, QPlex, MAPPO, DICG) focus on the learning model.

• Observations and State vectors are gathered from environments and input to function approximators.

• How is this information structured? Are we using it effectively?

Information Channels

• Information usually comes from multiple sources: observed entities, different sensors, communication channels.
• Agents can differentiate between information channels, because they occupy always the same vector positions.

Entities features

• Often, a subset of identical features describe the observed entities.
• This is a graph structure!
  • Vertices -> entities features
  • Edges -> ?
• Self-Attention can be used to learn the edges of the latent graph (Li et al., 2021).

Observations and State Matrices

• We reformulate the entire MARL problem in terms of graphs.
• The goal of the NNs is to learn the latent coordination graph.

Main features

• Extends QMix with the use graph matrices and transformers.
  • Independent q-learners.
  • Centralized mixer which respects the monotonicity constraint.
• Graph reasoning in both the agents and mixer.
  • UPDET (Hu et al. 2022): self-attention in agents.
  • QPlex (Wang et al. 2021), DICG (Li et al. 2021): self-attention in mixer.

Transformer Agent

Similar architecture to UPDET, but:

• Q-Values are derived from the agent hidden state, reinforcing recurrent passing of gradient.
• Separate output layer used for Policy Decoupling.

Transformer Mixer

Transformer hypernetwork:

• Agents hidden states:
  • : q-values embedding
• Recurrent mechanism:
  • : projection over
• State embedder:
  • environment reasoning

Spread

Setup:

• Observation entity features: ,
• State entity features: position+velocity.
• Extended to 6 agents (normally only 3)
• Reported evaluation metric: percentage of landmarks occupied at the conclusion of an episode (POL)

Spread Results

Spread 3v3

Spread 4v4

Spread 5v5

Spread 6v6

StarCraft 2

Setup:

• Original state and observations already implicitly sed a graph-based observation and state
• Reported results for the hardest maps.

SC2 Results

5m_vs_6m

8m_vs_9m

27m_vs_30m

6h_vs_8z

5m_vs_6m

3s5z_vs_3s6z

MM2

corridor

Model Size

Zero-Shot Transfer

Spread: the learned policy is transferable between different team of agents.

Transfer Learning

SC2: learning can be speeded up by transferring a learned policy.

8m_vs_9m to 5m_vs_6m

5s10z to 3s5z_vs_3s6z