{"data":{"slug":"alibaba-multimodal-industrial-ai-industrybench","name":"IndustryBench","tagline":"A multi-lingual benchmark for evaluating industrial domain knowledge of LLMs.","github_url":"https://github.com/alibaba-multimodal-industrial-ai/IndustryBench","owner":"alibaba-multimodal-industrial-ai","repo":"IndustryBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/283948917?v=4","primary_language":"Python","stars":155,"forks":10,"topics":["industry-benchmark","llm-evaluation"],"archived":false,"github_pushed_at":"2026-06-15T08:33:50+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/alibaba-multimodal-industrial-ai-industrybench","markdown_url":"https://www.graphcanon.com/tools/alibaba-multimodal-industrial-ai-industrybench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/alibaba-multimodal-industrial-ai-industrybench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=alibaba-multimodal-industrial-ai-industrybench","description":"A multi-lingual benchmark for evaluating industrial domain knowledge of LLMs.","homepage_url":null,"license":"MIT","open_issues":1,"watchers":8,"ai_summary":null,"readme_excerpt":"# IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs\n\n[📝Paper](https://arxiv.org/abs/2605.10267) | [🤗HuggingFace Data](https://huggingface.co/datasets/alibaba-multimodal-industrial-ai/IndustryBench)  \n\n**Source-grounded industrial procurement QA** for LLMs: each item is tied to a **Chinese national standard (GB/T)** or a **structured industrial product record**, with **human-reviewed** English, Russian, and Vietnamese renderings **aligned to the same item ids** as the Chinese source.\n\n| | |\n|:---|:---|\n| **Items** | 2,049 |\n| **Languages** | Chinese (source) + EN / RU / VI (aligned) |\n| **Labels** | 7 capability dimensions · 10 industry categories · panel-derived difficulty (easy / medium / hard) |\n| **Sources** | GB/T excerpts + industrial product records (see paper §3) |\n| **Paper** | [arXiv:2605.10267](https://arxiv.org/abs/2605.10267) |\n| **Dataset** | [`alibaba-multimodal-industrial-ai/IndustryBench`](https://huggingface.co/datasets/alibaba-multimodal-industrial-ai/IndustryBench) on Hugging Face |\n\n**Evaluation idea (paper §4):** models answer **closed-book** from the question only; a **calibrated LLM judge** scores **raw** correctness on **0–3**; a **separate safety-violation (SV)** check uses the **source excerpt** (`knowledge_text`). SV hits can zero the effective score—see paper for the full protocol and human calibration (**κ_w ≈ 0.798** on the judge sample).\n\n---\n\n## Who this repo is for\n\n| You want… | Do this |\n|:---|:---|\n| **Only the data** | Use Hugging Face below—no clone required. |\n| **The same scoring pipeline as the paper** | Clone this repo, export a CSV, run `evaluate.py` (below). |\n\n---\n\n## 1. Load the dataset (most users)\n\n```bash\npip install datasets\n```\n\n```python\nfrom datasets import load_dataset\n\nds = load_dataset(\"alibaba-multimodal-industrial-ai/IndustryBench\", split=\"train\")\n# e.g. inspect\nprint(ds[0].keys())\n```\n\nThe Hugging Face UI may show a small **metadata** table (language, license, task tags, etc.) if the dataset `README.md` on the Hub starts with a **`---` YAML block**. That block is optional; a YAML-free template lives in this repo at [`huggingface/README.md`](huggingface/README.md) for you to paste on the Hub if you want that table gone.\n\nTypical columns include `question` / `answer` (Chinese), `question_en` / `answer_en`, `question_ru` / `answer_ru`, `question_vi` / `answer_vi`, `knowledge_text`, `capability`, `difficulty`, `domain`, `industry_primary`, etc. Full schema is documented in the **paper appendix** and on the **HF dataset card** body (markdown below any YAML).\n\n---\n\n## 2. Reproduce the released evaluation script\n\n**Prereqs:** Python 3.10+, `pip install -r requirements.txt`, and an **OpenAI-compatible** HTTP API (any host that exposes `POST …/v1/chat/completions`).\n\n**Steps**\n\n1. Export the HF split to CSV (path can be anything; used as `--data-path`):\n\n   ```python\n   from datasets import load_dataset\n   load_dataset(\"alibaba-multimodal-industrial-ai/IndustryBench\", split=\"train\").to_csv(\"industrybench.csv\")\n   ```\n\n2. Set an API key (`--api-key` **or** env `OPENAI_API_KEY` **or** `DASHSCOPE_API_KEY`).\n\n3. Run (example: DashScope-compatible base + Qwen):\n\n   ```bash\n   python evaluate.py \\\n     --data-path industrybench.csv \\\n     --language zh \\\n     --api-base https://dashscope.aliyuncs.com/compatible-mode/v1 \\\n     --model qwen3-max\n   ```\n\n   - **`--api-base`** — Root URL that ends with **`/v1`**. The script appends **`/chat/completions`** itself. It is **not** the model name.\n   - **`--model`** — Model that **answers** the questions.\n   - **`--judge-model`** — Optional; defaults to `--model`. Set to your judge (e.g. `qwen3-max`) if the answer model differs.\n\n4. Results and checkpoints go under `results/` by default. See `python evaluate.py --help`.\n\n---\n\n## 3. What’s in this repository\n\n| Path | Role |\n|:---|:---|\n| `evaluate.py` | End-to-end multilingual runner: generation → LLM judge (0–3) → optional safety review → CSV. |\n| `requirements.txt` | Mini","github_created_at":"2026-05-12T11:21:40+00:00","created_at":"2026-07-11T12:01:55.218166+00:00","updated_at":"2026-07-11T12:02:21.150262+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"}],"tags":[{"slug":"python","name":"python"},{"slug":"industry-benchmark","name":"industry-benchmark"},{"slug":"llm-evaluation","name":"llm-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":1236570104,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T12:01:56.100Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":26,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":3,"high_count":0,"last_scan_at":"2026-07-11T12:02:01.608Z","medium_count":4,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T12:02:01.094Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T12:02:01.094Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T12:02:01.094Z"}}}}