{"data":{"slug":"lennart-finke-baba-is-eval","name":"baba_is_eval","tagline":"Claude et al. play the brilliant puzzle title \"Baba is You\"","github_url":"https://github.com/lennart-finke/baba_is_eval","owner":"lennart-finke","repo":"baba_is_eval","owner_avatar_url":"https://avatars.githubusercontent.com/u/42694673?v=4","primary_language":"Python","stars":62,"forks":8,"topics":["llm-evaluation","llm-tool-call","mcp","mcp-server"],"archived":false,"github_pushed_at":"2025-06-30T20:54:43+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/lennart-finke-baba-is-eval","markdown_url":"https://www.graphcanon.com/tools/lennart-finke-baba-is-eval.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lennart-finke-baba-is-eval","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lennart-finke-baba-is-eval","description":"Claude  et al. play the brilliant puzzle title \"Baba is You\"","homepage_url":null,"license":null,"open_issues":2,"watchers":1,"ai_summary":null,"readme_excerpt":"# Baba Is Eval\n\nhttps://github.com/user-attachments/assets/cb6d0d49-b583-46a2-b262-1cf38279a531\n\nWe evaluate language models' meta-level reasoning in an interactive setting, like the upcoming ARC-AGI-3, except we use the brilliant puzzle game \"Baba Is You\". In short, the player manipulates the game rules by pushing word blocks around to form or destroy rules. We provide an MCP server to interact with the game in text format. The project is currently an alpha and not stable. Contributions are welcome, and brave devs with model credits to spare are invited to give it a try.\n\n## Setup 🔩\n\n- Buy \"Baba Is You\" somewhere, like on Steam: https://store.steampowered.com/app/736260/Baba_Is_You\n- Clone this repo in the `Data` folder\n   - macOS + Steam: `/Users/[username]/Library/Application Support/Steam/steamapps/common/Baba Is You/Baba Is You.app/Contents/Resources/Data`\n   - Windows + Steam (?): `C:\\Program Files (x86)\\Steam\\steamapps\\common\\Baba Is You\\Data`\n- Install prerequisites (or use `uv`)\n```bash\npip install mcp fastmcp pyautogui configparser\n```\n- Run the setup script (after inspecting its contents) (also required before restarting the game, and for changes to `io.lua` which in turn require restarting the game)\n```bash\ncd baba_is_eval\nchmod +x setup.sh\n./setup.sh\n```\n- Open the game, preferably in a command line\n   - macOS + Steam: `/Users/[username]/Library/Application Support/Steam/steamapps/common/Baba Is You/Baba Is You.app/Contents/MacOS/Chowdren`\n   - (Currently we use clicking as a workaround for focusing, you might need to move the window to (800, 500) pixels on monitor one)\n- Start the MCP Server\n```bash\nmcp dev baba_is_eval/game_mcp.py\n```\n\nOn top of this MCP client agnostic setup, you can use a client like Claude Desktop to have a model interact with the server and play the game.\n\n### Available MCP Tools\n\nThe server provides these tools for interacting with the game:\n\n- **`enter_level(level: str)`** - Enter a specific level (e.g., \"1\", \"2\", \"3\")\n- **`get_game_state()`** - Get the current game state as a matrix\n- **`execute_commands(commands: str)`** - Execute movement commands (e.g., \"right,up,down\")\n- **`undo_multiple(n: int)`** - Undo the last n moves\n- **`restart_level()`** - Restart the current level\n- **`leave_level()`** - Exit the current level\n- **`game_rules(topic: str)`** - Get help on game rules\n\nThe game state is returned as a matrix like this:\n```\ny/x |  1  |  2  |  3  |  4  |  5  \n----+-----+-----+-----+-----+-----\n 1  |     |     |     |     |     \n 2  |     | baba|     |     |     \n 3  |     |     | flag|     |     \n 4  |     |     |     |     |     \n 5  |     |     |     |     |     \n```\n\n## Roadmap 🛣️\n\nContributions welcome! Goals listed in order of pressingness.\n\n- [x] Reading and writing using modded Lua\n- [x] MCP integration\n- [ ] Acceptable game I/O reliability\n- [ ] Complete MCP control over game functionality\n- [ ] Automated evaluation logs\n- [ ] More efficient scaffolding\n- [ ] Menu navigation beyond the first 8 levels, to arbitrary levels\n- [ ] Good latency\n- [ ] Parallel execution\n- [ ] Support for community levels\n\n\n## Inner Workings ⚰️\n\nThis works in the dumbest way possible; we reverse engineer the exposed Lua functions, use the mod functionality to read the game state, and write to one of the game state config files using mod hooks, which is then read in by the MCP server. For move and undo inputs, we write Lua files to `commands/` from the MCP server to be read in, if detected, in the `always` mod hook. Perhaps this dooms the project to be brittle and slow forever, but perhaps there is some better way. \n\n\n## Credits\nBaba is You was created by Hempuli. It is a lovely game, go play it yourself!","github_created_at":"2025-06-30T18:14:16+00:00","created_at":"2026-07-15T10:40:19.184662+00:00","updated_at":"2026-07-15T10:40:21.946117+00:00","categories":[{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"},{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"}],"tags":[{"slug":"llm-evaluation","name":"llm-evaluation"},{"slug":"llm-tool-call","name":"llm-tool-call"},{"slug":"mcp","name":"mcp"},{"slug":"mcp-server","name":"mcp-server"},{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":1011379817,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-15T10:40:20.135Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":379,"last_release_at":null},"security_summary":{"status":"no_manifest","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-15T10:40:20.584Z","medium_count":0,"scan_profile":"mcp_manifest","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-15T10:40:19.920Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-15T10:40:19.920Z"}}}}