MettaGrid is a multi-agent gridworld environment for studying the emergence of cooperation and social behaviors in reinforcement learning agents. The environment features a variety of objects and actions that agents can interact with to manage resources, engage in combat, share with others, and optimize their rewards.
In MettaGrid, agents navigate a gridworld and interact with various objects to manage their energy, harvest resources, engage in combat, and cooperate with other agents. The key dynamics include:
- Energy Management: Agents must efficiently manage their energy, which is required for all actions. They can harvest resources and convert them to energy at charger stations.
- Resource Gathering: Agents can gather resources from generator objects scattered throughout the environment.
- Cooperation and Sharing: Agents have the ability to share resources with other agents and use energy to power the heart altar, which provides rewards.
- Combat: Agents can attack other agents to temporarily freeze them and steal their resources. They can also use shields to defend against attacks.
The environment is highly configurable, allowing for experimentation with different world layouts, object placements, and agent capabilities.
The Agent object represents an individual agent in the environment. Agents can move, rotate, attack, and interact with other objects. Each agent has energy, resources, and shield properties that govern its abilities and interactions.
The Altar object allows agents to spend energy to gain rewards. Agents can power the altar by using the use action when near it. The altar has a cooldown period between uses.
- Using the heart altar costs
altar.use_cost energy. So, no matter how much energy you have, you are always dumping the same amount of energy in it and getting the same amount of reward. - After the heart altar is used, it is unable to be used for altar.cooldown timesteps.
- A single use of the heart altar gives you a single unit of reward:
if
target._type_id == ObjectType.AltarT: self.env._rewards[actor_id] += 1
The Converter object allows agents to convert their harvested resources into energy. Agents can use converters by moving to them and taking the use action. Each use of a converter provides a specified amount of energy and has a cooldown period.
- Using the converter does not cost any energy.
- Using the converter outputs
converter.energy_output.r1energy- see
this.output_energy = cfg[b"energy_output.r1"]in the Converter cppclass
- see
- Using the converter increments resource 2 by one and decrements resource 1 by 1
- There is currently no use for
converter.energy_output.r2andconverter.energy_output.r3 - After the converter is used, it is unable to be used for
converter.cooldowntimesteps
The Generator object produces resources that agents can harvest. Agents can gather resources from generators by moving to them and taking the use action. Generators have a specified capacity and replenish resources over time.
- Using the generator costs
generator.use_cost(currently 0) energy - Using the generator once gives one resource 1
- After the generator is used, it is unable to be used for
generator.cooldowntimesteps
The Wall object acts as an impassable barrier in the environment, restricting agent movement.
The cooldown property determines how long before objects can be used again.
The move action allows agents to move to an adjacent cell in the gridworld. The action has two modes: moving forward and moving backward relative to the agent's current orientation.
The rotate action enables agents to change their orientation within the gridworld. Agents can rotate to face in four directions: down, left, right, and up.
The attack action allows agents to attack other agents or objects within their attack range. Successful attacks freeze the target for freeze_duration timesteps and allow the attacker to steal resources. Further, the attacked agent's energy is set to 0. Attacks have a cost and inflict a damage value. The agent selects from one of nine coordinates within its attack range.
The shield action turns on a shield. When the shield is active, the agent is protected from attacks by other agents. The shield consumes energy defined by upkeep.shield while active. Attack damage is subtracted from the agent's energy, rather than freezing the agent.
The transfer action enables agents to share resources with other agents. Agents can choose to transfer specific resources to another agent in an adjacent cell. It is currently not implemented.
The use action allows agents to interact with objects such as altars, converters, and generators. The specific effects of the use action depend on the target object and can include converting resources to energy, powering the altar for rewards, or harvesting resources from generators.
The gift action allows agents to share resources with other agents. It is currently not implemented.
The swap action allows agents to swap positions with other agents. It is currently not implemented.
The track_last_action property adds the agent's last action to the observation space. It is currently not implemented.
The MettaGrid environment is highly configurable through the use of YAML configuration files. These files specify the layout of the gridworld, the placement of objects, and various properties of the objects and agents.