In reinforcement learning, an “agent” is an entity which can observe the world and take actions. Multi-agent setups have multiple agents observing and taking actions. The interplay between them is very interesting, but this also introduces some complexity.

Before talking about multiagent setups it’s worth a note about two other RL algorithms.

Proximal Policy Optimization (PPO) is an algorithm containing an Actor and a Critic, trained together. The Actor is a neural network which determines which action to take. The Critic is a neural network which

Self-play

There are a few ways to set up multi-agent training, in two general buckets: “simple” methods and population-based methods.

Simple MARL methods

Independent PPO (IPPO) is a two agent setup, where Agents A and B play matches and train based on the outcomes of those games.

Two-Timescale PPO is similar to IPPO but trains the agents with different learning rates, creating a “leader” agent and a “follower” agent.

Population-based methods

Fictitious Self-Play (FSP) trains agents against all past policies, sampled uniformly.

Prioritized Fictitious Self-Play (PFSP) samples opponents based on how difficult they are. The agent will practice against difficult opponents more often.

League training maintains a “league” of agents. The idea being that a lot of past policies are just bad, so we should only add an agent into the league if it meets some performance threshold. Agents train from games played against other agents in the league.

Policy Space Response Oracle (PSRO) attempts to fill in exploitability gaps in the population of agents by finding a policy which performs best against the group.