{"data":{"slug":"lennart-finke-baba-is-eval","name":"baba_is_eval","tagline":"Evaluates language models' meta-level reasoning through the puzzle game Baba Is You","github_url":"https://github.com/lennart-finke/baba_is_eval","owner":"lennart-finke","repo":"baba_is_eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/42694673?v=4","primary_language":"Python","stars":62,"forks":8,"topics":["llm-evaluation","llm-tool-call","mcp","mcp-server"],"archived":false,"github_pushed_at":"2025-06-30T20:54:43+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/lennart-finke-baba-is-eval","markdown_url":"https://www.graphcanon.com/tools/lennart-finke-baba-is-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lennart-finke-baba-is-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lennart-finke-baba-is-eval","description":"Claude  et al. play the brilliant puzzle title \"Baba is You\"","homepage_url":null,"license":null,"open_issues":2,"watchers":1,"ai_summary":"Uses an MCP server to interact with the game in text format for language model evaluation.","readme_excerpt":"# Baba Is Eval\n\nhttps://github.com/user-attachments/assets/cb6d0d49-b583-46a2-b262-1cf38279a531\n\nWe evaluate language models' meta-level reasoning in an interactive setting, like the upcoming ARC-AGI-3, except we use the brilliant puzzle game \"Baba Is You\". In short, the player manipulates the game rules by pushing word blocks around to form or destroy rules. We provide an MCP server to interact with the game in text format. The project is currently an alpha and not stable. Contributions are welcome, and brave devs with model credits to spare are invited to give it a try.\n\n## Setup 🔩\n\n- Buy \"Baba Is You\" somewhere, like on Steam: https://store.steampowered.com/app/736260/Baba_Is_You\n- Clone this repo in the `Data` folder\n   - macOS + Steam: `/Users/[username]/Library/Application Support/Steam/steamapps/common/Baba Is You/Baba Is You.app/Contents/Resources/Data`\n   - Windows + Steam (?): `C:\\Program Files (x86)\\Steam\\steamapps\\common\\Baba Is You\\Data`\n- Install prerequisites (or use `uv`)\n```bash\npip install mcp fastmcp pyautogui configparser\n```\n- Run the setup script (after inspecting its contents) (also required before restarting the game, and for changes to `io.lua` which in turn require restarting the game)\n```bash\ncd baba_is_eval\nchmod +x setup.sh\n./setup.sh\n```\n- Open the game, preferably in a command line\n   - macOS + Steam: `/Users/[username]/Library/Application Support/Steam/steamapps/common/Baba Is You/Baba Is You.app/Contents/MacOS/Chowdren`\n   - (Currently we use clicking as a workaround for focusing, you might need to move the window to (800, 500) pixels on monitor one)\n- Start the MCP Server\n```bash\nmcp dev baba_is_eval/game_mcp.py\n```\n\nOn top of this MCP client agnostic setup, you can use a client like Claude Desktop to have a model interact with the server and play the game.\n\n### Available MCP Tools\n\nThe server provides these tools for interacting with the game:\n\n- **`enter_level(level: str)`** - Enter a specific level (e.g., \"1\", \"2\", \"3\")\n- **`get_game_state()`** - Get the current game state as a matrix\n- **`execute_commands(commands: str)`** - Execute movement commands (e.g., \"right,up,down\")\n- **`undo_multiple(n: int)`** - Undo the last n moves\n- **`restart_level()`** - Restart the current level\n- **`leave_level()`** - Exit the current level\n- **`game_rules(topic: str)`** - Get help on game rules\n\nThe game state is returned as a matrix like this:\n```\ny/x |  1  |  2  |  3  |  4  |  5  \n----+-----+-----+-----+-----+-----\n 1  |     |     |     |     |     \n 2  |     | baba|     |     |     \n 3  |     |     | flag|     |     \n 4  |     |     |     |     |     \n 5  |     |     |     |     |     \n```\n\n## Roadmap 🛣️\n\nContributions welcome! Goals listed in order of pressingness.\n\n- [x] Reading and writing using modded Lua\n- [x] MCP integration\n- [ ] Acceptable game I/O reliability\n- [ ] Complete MCP control over game functionality\n- [ ] Automated evaluation logs\n- [ ] More efficient scaffolding\n- [ ] Menu navigation beyond the first 8 levels, to arbitrary levels\n- [ ] Good latency\n- [ ] Parallel execution\n- [ ] Support for community levels\n\n\n## Inner Workings ⚰️\n\nThis works in the dumbest way possible; we reverse engineer the exposed Lua functions, use the mod functionality to read the game state, and write to one of the game state config files using mod hooks, which is then read in by the MCP server. For move and undo inputs, we write Lua files to `commands/` from the MCP server to be read in, if detected, in the `always` mod hook. Perhaps this dooms the project to be brittle and slow forever, but perhaps there is some better way. \n\n\n## Credits\nBaba is You was created by Hempuli. It is a lovely game, go play it yourself!","github_created_at":"2025-06-30T18:14:16+00:00","created_at":"2026-07-15T10:40:19.184662+00:00","updated_at":"2026-09-20T04:25:09.366207+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"llm-tool-call","name":"llm-tool-call"},{"slug":"mcp","name":"mcp"},{"slug":"mcp-server","name":"mcp-server"}],"trust":{"provenance":{"is_fork":false,"github_id":1011379817,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-09T06:00:52.896Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":435,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-08-02T04:00:31.106Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-09T06:00:53.343Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-09T06:00:53.343Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium","summary":"Free to use, but requires purchasing Baba Is You from a platform like Steam for testing."},"requirements":{"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"when_to_use":["- When you need to test the advanced problem-solving and strategic thinking capabilities of a language model, especially for understanding or creating game rules dynamically","- For developing or testing models that require interaction with complex, rule-manipulative games to gauge their ability to adapt to changing conditions","- To focus on the evaluation of models in scenarios where they must interact with an environment through text-based commands and interpret textual feedback"],"when_not_to_use":["- If your test scenario requires stability or reliability, as baba_is_eval is noted for being unstable due to its alpha status","- When you need a tool that supports automated testing without the necessity of human supervision or manual setup of game assets and MCP server interaction"],"source":"enrich:decision_facts","observed_at":"2026-07-16T20:43:46.460Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium - Free to use, but requires purchasing Baba Is You from a platform like Steam for testing."},{"label":"Adopt for","value":"baba_is_eval leverages Baba Is You to evaluate language models' meta-level reasoning in an interactive text-based setting through MCP server integration."},{"label":"License detail","value":"(unknown)"}]}}