baba_is_eval logo

baba_is_eval

lennart-finke/baba_is_eval

Evaluates language models' meta-level reasoning through the puzzle game Baba Is You

GraphCanon updated Oct 18, 2026 · GitHub synced Oct 18, 2026

62 stars8 forksLast push Jun 30, 2025 Python

Decision brief

Unverified catalog guidance: baba_is_eval leverages Baba Is You to evaluate language models' meta-level reasoning in an interactive text-based setting through MCP server integration.

Good fit when

  • Unverified catalog guidance: - When you need to test the advanced problem-solving and strategic thinking capabilities of a language model, especially for understanding or creating game rules dynamically
  • Unverified catalog guidance: - For developing or testing models that require interaction with complex, rule-manipulative games to gauge their ability to adapt to changing conditions

Avoid when

  • Unverified catalog guidance: - If your test scenario requires stability or reliability, as baba_is_eval is noted for being unstable due to its alpha status
  • Unverified catalog guidance: - When you need a tool that supports automated testing without the necessity of human supervision or manual setup of game assets and MCP server interaction
Pricing:
freemium - Free to use, but requires purchasing Baba Is You from a platform like Steam for testing.

Observed Jul 16, 2026 · Source: enrich:decision_facts

Verify the decision

Maintenance and security

Full trust report
Maintenance
Dormant (474d since push)
As of Oct 18, 2026
Provenance
Not a fork · Personal account
As of Oct 18, 2026
Security (OSV)
No lockfile
As of Aug 2, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

Package installation is not established from the available documentation. Check the source repository for setup instructions.

git clone https://github.com/lennart-finke/baba_is_eval
Source repository

Similar tools

Same-category neighbours. No typed graph edges are catalogued for this tool yet.

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

Uses an MCP server to interact with the game in text format for language model evaluation.

Capability facts

MCP server
Unknown; MCP server not established by this scan

Source: repo_scan · Oct 18, 2026

Languages
python

Source: github.language · Oct 18, 2026

Categories

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

Documented Python runtime (not runtime-tested)Python

Source: README excerpt (readme_declaration_v2, Oct 18, 2026)

pip install mcp fastmcp pyautogui configparser
Source link

Tags

README

Baba Is Eval https://github.com/user attachments/assets/cb6d0d49 b583 46a2 b262 1cf38279a531 We evaluate language models' meta level reasoning in an interactive setting, like the upcoming ARC AGI 3, except we use the brilliant puzzle game "Baba Is You". In short, the player manip...

For agents

This page has a .md twin and JSON over the API.

Was this helpful?

Anonymous feedback helps us improve pages and translations.