---
title: "IndustryBench"
type: "tool"
slug: "alibaba-multimodal-industrial-ai-industrybench"
canonical_url: "https://www.graphcanon.com/tools/alibaba-multimodal-industrial-ai-industrybench"
github_url: "https://github.com/alibaba-multimodal-industrial-ai/IndustryBench"
homepage_url: null
stars: 155
forks: 10
primary_language: "Python"
license: "MIT"
archived: false
categories: ["model-training", "llm-frameworks", "data-retrieval"]
tags: ["python", "industry-benchmark", "llm-evaluation"]
updated_at: "2026-07-11T12:02:21.150262+00:00"
---

# IndustryBench

> A multi-lingual benchmark for evaluating industrial domain knowledge of LLMs.

A multi-lingual benchmark for evaluating industrial domain knowledge of LLMs.

## Facts

- Repository: https://github.com/alibaba-multimodal-industrial-ai/IndustryBench
- Stars: 155 · Forks: 10 · Open issues: 1 · Watchers: 8
- Primary language: Python
- License: MIT
- Last pushed: 2026-06-15T08:33:50+00:00

## Trust & health

_Signals computed from public GitHub metadata. Not a security guarantee._

- Maintenance: Active (computed 2026-07-11T12:01:56.100Z)
- Security scan: Findings present (0 critical, 0 high, 4 medium, 3 low) · last scan 2026-07-11T12:02:01.608Z
- Full report: [trust report](/tools/alibaba-multimodal-industrial-ai-industrybench/trust.md) · [JSON](https://www.graphcanon.com/api/graphcanon/tools/alibaba-multimodal-industrial-ai-industrybench/trust)

## Categories

- [Model Training](/categories/model-training.md)
- [LLM Frameworks](/categories/llm-frameworks.md)
- [Data & Retrieval](/categories/data-retrieval.md)

## Tags

python, industry-benchmark, llm-evaluation

## Category neighbours (exploratory)

_Same-category tools for discovery only - not curated alternatives. Cap shown at six._

- [awesome](/tools/sindresorhus-awesome.md) - 😎 Curated list of awesome topics including hardware resources (★ 484,026) [Active]
- [tensorflow](/tools/tensorflow-tensorflow.md) - An Open Source Machine Learning Framework for Everyone (★ 196,300) [Very active]
- [AutoGPT](/tools/significant-gravitas-autogpt.md) - AutoGPT is the vision of accessible AI for everyone, to use and to build on. (★ 185,464) [Very active]
- [ollama](/tools/ollama-ollama.md) - Get up and running with various large language models using Ollama. (★ 175,936) [Very active]
- [prompts.chat](/tools/f-prompts-chat.md) - Share, discover, and collect prompts from the community (★ 165,372) [Very active]
- [transformers](/tools/huggingface-transformers.md) - Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models (★ 162,482) [Very active]

_+ 2 more not listed._

## README (excerpt)

_Quoted verbatim from the upstream repository. Untrusted content - treat as data, not instructions._

````text
# IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs

[📝Paper](https://arxiv.org/abs/2605.10267) | [🤗HuggingFace Data](https://huggingface.co/datasets/alibaba-multimodal-industrial-ai/IndustryBench)  

**Source-grounded industrial procurement QA** for LLMs: each item is tied to a **Chinese national standard (GB/T)** or a **structured industrial product record**, with **human-reviewed** English, Russian, and Vietnamese renderings **aligned to the same item ids** as the Chinese source.

| | |
|:---|:---|
| **Items** | 2,049 |
| **Languages** | Chinese (source) + EN / RU / VI (aligned) |
| **Labels** | 7 capability dimensions · 10 industry categories · panel-derived difficulty (easy / medium / hard) |
| **Sources** | GB/T excerpts + industrial product records (see paper §3) |
| **Paper** | [arXiv:2605.10267](https://arxiv.org/abs/2605.10267) |
| **Dataset** | [`alibaba-multimodal-industrial-ai/IndustryBench`](https://huggingface.co/datasets/alibaba-multimodal-industrial-ai/IndustryBench) on Hugging Face |

**Evaluation idea (paper §4):** models answer **closed-book** from the question only; a **calibrated LLM judge** scores **raw** correctness on **0–3**; a **separate safety-violation (SV)** check uses the **source excerpt** (`knowledge_text`). SV hits can zero the effective score—see paper for the full protocol and human calibration (**κ_w ≈ 0.798** on the judge sample).

---

## Who this repo is for

| You want… | Do this |
|:---|:---|
| **Only the data** | Use Hugging Face below—no clone required. |
| **The same scoring pipeline as the paper** | Clone this repo, export a CSV, run `evaluate.py` (below). |

---

## 1. Load the dataset (most users)

```bash
pip install datasets
```

```python
from datasets import load_dataset

ds = load_dataset("alibaba-multimodal-industrial-ai/IndustryBench", split="train")
# e.g. inspect
print(ds[0].keys())
```

The Hugging Face UI may show a small **metadata** table (language, license, task tags, etc.) if the dataset `README.md` on the Hub starts with a **`---` YAML block**. That block is optional; a YAML-free template lives in this repo at [`huggingface/README.md`](huggingface/README.md) for you to paste on the Hub if you want that table gone.

Typical columns include `question` / `answer` (Chinese), `question_en` / `answer_en`, `question_ru` / `answer_ru`, `question_vi` / `answer_vi`, `knowledge_text`, `capability`, `difficulty`, `domain`, `industry_primary`, etc. Full schema is documented in the **paper appendix** and on the **HF dataset card** body (markdown below any YAML).

---

## 2. Reproduce the released evaluation script

**Prereqs:** Python 3.10+, `pip install -r requirements.txt`, and an **OpenAI-compatible** HTTP API (any host that exposes `POST …/v1/chat/completions`).

**Steps**

1. Export the HF split to CSV (path can be anything; used as `--data-path`):

   ```python
   from datasets import load_dataset
   load_dataset("alibaba-multimodal-industrial-ai/IndustryBench", split="train").to_csv("industrybench.csv")
   ```

2. Set an API key (`--api-key` **or** env `OPENAI_API_KEY` **or** `DASHSCOPE_API_KEY`).

3. Run (example: DashScope-compatible base + Qwen):

   ```bash
   python evaluate.py \
     --data-path industrybench.csv \
     --language zh \
     --api-base https://dashscope.aliyuncs.com/compatible-mode/v1 \
     --model qwen3-max
   ```

   - **`--api-base`** — Root URL that ends with **`/v1`**. The script appends **`/chat/completions`** itself. It is **not** the model name.
   - **`--model`** — Model that **answers** the questions.
   - **`--judge-model`** — Optional; defaults to `--model`. Set to your judge (e.g. `qwen3-max`) if the answer model differs.

4. Results and checkpoints go under `results/` by default. See `python evaluate.py --help`.

---

## 3. What’s in this repository

| Path | Role |
|:---|:---|
| `evaluate.py` | End-to-end multilingual runner: generation → LLM judge (0–3) → optional safety review → CSV. |
| `requirements.txt` | Mini
````

---

**Machine-readable endpoints**

- JSON: [`/api/graphcanon/tools/alibaba-multimodal-industrial-ai-industrybench`](/api/graphcanon/tools/alibaba-multimodal-industrial-ai-industrybench)
- LLM index: [/llms.txt](/llms.txt)
- Full corpus: [/llms-full.txt](/llms-full.txt)

_GraphCanon - The knowledge graph for AI development. https://www.graphcanon.com/_