{"data":{"slug":"cvs-health-langfair","name":"langfair","tagline":"LangFair is a Python library for conducting use-case level LLM bias and fairness assessments","github_url":"https://github.com/cvs-health/langfair","owner":"cvs-health","repo":"langfair","owner_avatar_url":"https://avatars.githubusercontent.com/u/89528242?v=4","primary_language":"Python","stars":260,"forks":46,"topics":["ai","ai-safety","artificial-intelligence","bias","bias-detection","ethical-ai","fairness","fairness-ai","fairness-ml","fairness-testing","large-language-models","llm","llm-evaluation","llm-evaluation-framework","llm-evaluation-metrics","python","responsible-ai"],"archived":false,"github_pushed_at":"2026-06-29T20:15:23+00:00","maintenance_label":"Active","url":"https://www.graphcanon.com/tools/cvs-health-langfair","markdown_url":"https://www.graphcanon.com/tools/cvs-health-langfair.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/cvs-health-langfair","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=cvs-health-langfair","description":"LangFair is a Python library for conducting use-case level LLM bias and fairness assessments","homepage_url":"https://cvs-health.github.io/langfair/","license":"Other","open_issues":23,"watchers":4,"ai_summary":null,"readme_excerpt":"

\n \n

\n\n# LangFair: Use-Case Level LLM Bias and Fairness Assessments\n\n\n\n\n\n\n\n\nLangFair is a comprehensive Python library designed for conducting bias and fairness assessments of large language model (LLM) use cases. This repository includes various supporting resources, including\n\n- [Documentation site](https://cvs-health.github.io/langfair/) with complete API reference\n- [Comprehensive framework](https://github.com/cvs-health/langfair/tree/main#-choosing-bias-and-fairness-metrics-for-an-llm-use-case) for choosing bias and fairness metrics\n- [Demo notebooks](https://github.com/cvs-health/langfair/tree/main#-example-notebooks) providing illustrative examples\n- [LangFair tutorial](https://medium.com/cvs-health-tech-blog/how-to-assess-your-llm-use-case-for-bias-and-fairness-with-langfair-7be89c0c4fab) on Medium\n- [Software paper](https://arxiv.org/abs/2501.03112v1) on how LangFair compares to other toolkits\n- [Research paper](https://arxiv.org/abs/2407.10853) on our evaluation approach\n\n## 🚀 Why Choose LangFair?\nStatic benchmark assessments, which are typically assumed to be sufficiently representative, often fall short in capturing the risks associated with all possible use cases of LLMs. These models are increasingly used in various applications, including recommendation systems, classification, text generation, and summarization. However, evaluating these models without considering use-case-specific prompts can lead to misleading assessments of their performance, especially regarding bias and fairness risks.\n \nLangFair addresses this gap by adopting a Bring Your Own Prompts (BYOP) approach, allowing users to tailor bias and fairness evaluations to their specific use cases. This ensures that the metrics computed reflect the true performance of the LLMs in real-world scenarios, where prompt-specific risks are critical. Additionally, LangFair's focus is on output-based metrics that are practical for governance audits and real-world testing, without needing access to internal model states.\n\n

\n \n

\n\n**Note:** This diagram illustrates the workflow for assessing bias and fairness in text generation and summarization use cases.\n\n## ⚡ Quickstart Guide\n### (Optional) Create a virtual environment for using LangFair\nWe recommend creating a new virtual environment using venv before installing LangFair. To do so, please follow instructions [here](https://docs.python.org/3/library/venv.html).\n\n### Installing LangFair\nThe latest version can be installed from PyPI:\n\n```bash\npip install langfair\n```\n\n### Usage Examples\nBelow are code samples illustrating how to use LangFair to assess bias and fairness risks in text generation and summarization use cases. The below examples assume the user has already defined a list of prompts from their use case, `prompts`. \n\n##### Generate LLM responses\nTo generate responses, we can use LangFair's `ResponseGenerator` class. First, we must create a `langchain` LLM object. Below we use `ChatVertexAI`, but **any of [LangChain’s LLM classes](https://js.langchain.com/docs/integrations/chat/) may be used instead**. Note that `InMemoryRateLimiter` is to used to avoid rate limit errors.\n```python\nfrom langchain_google_vertexai import ChatVertexAI\nfrom langchain_core.rate_limiters import InMemoryRateLimiter\nrate_limiter = InMemoryRateLimiter(\n requests_per_second=4.5, check_every_n_seconds=0.5, max_bucket_size=280, \n)\nllm = ChatVertexAI(\n model_name=\"gemini-pro\", temperature=0.3, rate_limiter=rate_limiter\n)\n```\nWe can use `ResponseGenerator.generate_responses` to generate 25 responses for each prompt, as is convention for toxicity evaluation.\n```python\nfrom langfair.generator import ResponseGenerator\nrg = ResponseGenerator(langchain_llm=llm)\ngenerat","github_created_at":"2024-09-20T19:11:29+00:00","created_at":"2026-07-11T10:45:24.67394+00:00","updated_at":"2026-07-11T10:45:34.459849+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"computer-vision","name":"Computer Vision","url":"https://www.graphcanon.com/categories/computer-vision","markdown_url":"https://www.graphcanon.com/categories/computer-vision.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/computer-vision"},{"slug":"developer-tools","name":"Developer Tools","url":"https://www.graphcanon.com/categories/developer-tools","markdown_url":"https://www.graphcanon.com/categories/developer-tools.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/developer-tools"}],"tags":[{"slug":"ethical-ai","name":"ethical-ai"},{"slug":"ai-safety","name":"ai-safety"},{"slug":"bias","name":"bias"},{"slug":"ai","name":"ai"},{"slug":"artificial-intelligence","name":"artificial-intelligence"},{"slug":"fairness-ai","name":"fairness-ai"},{"slug":"bias-detection","name":"bias-detection"},{"slug":"fairness","name":"fairness"}],"trust":{"provenance":{"is_fork":false,"github_id":860612327,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:45:25.457Z","maintenance":{"label":"Active","score":82,"methodology":"github_public_v1","releases_90d":0,"days_since_push":11,"last_release_at":"2026-01-09T14:22:35Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T10:45:26.264Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:45:25.865Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-07-11T10:45:25.865Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-07-11T10:45:25.865Z"}}}}