{"data":{"slug":"hithink-research-bizfinbench","name":"BizFinBench","tagline":"A Business-Driven Real-World Financial Benchmark for Evaluating LLMs","github_url":"https://github.com/HiThink-Research/BizFinBench","owner":"HiThink-Research","repo":"BizFinBench","owner_avatar_url":"https://avatars.githubusercontent.com/u/186997494?v=4","primary_language":"Python","stars":169,"forks":12,"topics":["benchmark","finance","llm","llm-benchmarking","llm-evaluation"],"archived":false,"github_pushed_at":"2026-05-01T01:36:34+00:00","maintenance_label":"Slowing","stars_delta_30d":1,"url":"https://www.graphcanon.com/tools/hithink-research-bizfinbench","markdown_url":"https://www.graphcanon.com/tools/hithink-research-bizfinbench.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/hithink-research-bizfinbench","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=hithink-research-bizfinbench","description":"A Business-Driven Real-World Financial Benchmark for Evaluating LLMs","homepage_url":"https://hithink-research.github.io/BizFinBench/","license":null,"open_issues":0,"watchers":2,"ai_summary":"BizFinBench is designed as a finance-oriented benchmark to evaluate the performance of large language models in real-world business scenarios.","readme_excerpt":"### Install requirements\n```sh\npip install -r requirements.txt\n```\n\n---\n\n### Quick Start – Evaluate a Local Model\n\n```sh\nexport MODEL_PATH=model/Qwen2.5-0.5B   # Path to the model to be evaluated\nexport REMOTE_MODEL_PORT=16668\nexport REMOTE_MODEL_URL=http://127.0.0.1:${REMOTE_MODEL_PORT}/model\nexport MODEL_NAME=Qwen2.5-0.5B\nexport PROMPT_TYPE=chat_template   # Hithink llama3 llama2 none qwen chat_template; chat_template is recommended\n\n---\n\n### Quick Start – Evaluate a Local Model and Score with a Judge Model\n\n```sh\nexport MODEL_PATH=model/Qwen2.5-0.5B   # Path to the model to be evaluated\nexport REMOTE_MODEL_PORT=16668\nexport REMOTE_MODEL_URL=http://127.0.0.1:${REMOTE_MODEL_PORT}/model\nexport MODEL_NAME=Qwen2.5-0.5B\nexport PROMPT_TYPE=chat_template   # llama3 llama2 none qwen chat_template; chat_template is recommended\n\n---\n\n### Quick Start – Evaluate external apis (e.g., chatgpt)\n\n```sh\nexport API_NAME=chatgpt # The api name, currently support chatgpt\nexport API_KEY=xxx # Your api key\nexport MODEL_NAME=gpt-4.1\n\n---\n\n## 📄 License\n  **Usage and License Notices**: The data and code are intended and licensed for research use only.\nLicense: Attribution-NonCommercial 4.0 International It should abide by the policy of OpenAI: https://openai.com/policies/terms-of-use","github_created_at":"2025-05-07T07:28:26+00:00","created_at":"2026-07-11T12:01:35.952814+00:00","updated_at":"2026-09-20T07:00:05.404729+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"finance","name":"finance"},{"slug":"llm","name":"llm"},{"slug":"llm-benchmarking","name":"llm-benchmarking"},{"slug":"llm-evaluation","name":"llm-evaluation"}],"trust":{"provenance":{"is_fork":false,"github_id":979225416,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-20T07:00:02.538Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":142,"last_release_at":null,"stars_delta_30d":1,"open_issues_delta_30d":0},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":90,"high_count":0,"last_scan_at":"2026-07-11T12:01:43.843Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-20T07:00:03.610Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-20T07:00:03.610Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["需要安装所需的Python库以运行评估：pip install -r requirements.txt","环境变量设置包括模型路径、远程模型URL、模型名称以及其他相关参数，如使用API进行测试时的API key等。","必须确保遵守相关的使用和许可政策，这可能涉及研究使用的限制及其他第三方协议条款。"]},"constraints":null,"when_to_use":["For teams专注于金融行业，需要评估其模型在实际业务场景中的表现时。","如果需要使用具体的财务基准来衡量大型语言模型的性能，并且希望该基准能反映真实世界的商业情景，则适合采用BizFinBench。"],"when_not_to_use":["不要在不需要针对财务管理特定挑战进行测试的情况下使用BizFinBench，例如，在评估通用自然语言处理任务时。","如果项目主要关注的是非金融行业的应用，并且所使用的场景与实际的商业交易无直接关联，则不宜采用BizFinBench进行测评。"],"source":"enrich:decision_facts","observed_at":"2026-07-15T09:56:37.590Z"},"constraint_facets":null,"decision_summary":[{"label":"Requirements","value":"需要安装所需的Python库以运行评估：pip install -r requirements.txt; 环境变量设置包括模型路径、远程模型URL、模型名称以及其他相关参数，如使用API进行测试时的API key等。; 必须确保遵守相关的使用和许可政策，这可能涉及研究使用的限制及其他第三方协议条款。"},{"label":"Adopt for","value":"BizFinBench is a finance-specific benchmark for evaluating large language models in real-world business settings."}]}}