{"data":{"slug":"fiddler-labs-fiddler-auditor","name":"fiddler-auditor","tagline":"Tool to evaluate language models","github_url":"https://github.com/fiddler-labs/fiddler-auditor","owner":"fiddler-labs","repo":"fiddler-auditor","owner_avatar_url":"https://avatars.githubusercontent.com/u/48138171?v=4","primary_language":"Python","stars":194,"forks":24,"topics":["ai-observability","evaluation","generative-ai","langchain","llms","nlp","robustness"],"archived":false,"github_pushed_at":"2024-03-11T02:49:45+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/fiddler-labs-fiddler-auditor","markdown_url":"https://www.graphcanon.com/tools/fiddler-labs-fiddler-auditor.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/fiddler-labs-fiddler-auditor","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=fiddler-labs-fiddler-auditor","description":"Fiddler Auditor is a tool to evaluate language models.","homepage_url":null,"license":"Other","open_issues":15,"watchers":6,"ai_summary":"Fiddler Auditor is an evaluation tool designed for auditing the robustness and reliability of large language models (LLMs) and natural language processing (NLP) models before deployment in production.","readme_excerpt":"# \"Fiddler\n\nAuditing Large Language Models made easy!\n\n\n\n\n\n## What is Fiddler Auditor?\n\n
\n \"Fiddler\n
\n\nLanguage models enable companies to build and launch innovative applications to improve productivity and increase customer satisfaction. \nHowever, it’s been known that LLMs can hallucinate, generate adversarial responses that can harm users, and even expose private information that they were trained on when prompted or unprompted. It's more critical than ever for ML and software application teams to minimize these risks and weaknesses before launching LLMs and NLP models. As a result, it’s important for you to include a process to audit language models thoroughly before production.\nThe Fiddler Auditor enables you to test LLMs and NLP models, identify weaknesses in the models, and mitigate potential adversarial outcomes before deploying them to production.\n\n## Features and Capabilities\n\n

\n

\n \"Fiddler\n
\n

\n\nFiddler Auditor supports\n\n- Red-teaming LLMs for your use-case with prompt perturbation\n- Integration with LangChain\n- Custom evaluation metrics\n- Generative and Discriminative NLP models\n- Comparison of LLMs\n\n

\n

\n \"Example\n An example report generated by the Fiddler Auditor for text-davinci-003. \n
\n

\n\n\n## Installation\n\n### From PyPI\nAuditor is available on PyPI and we test on Python 3.8 and above. We recommend creating a virtual python environment and installing using the following command\n\n```bash\npip install fiddler-auditor\n```\n\n### From source\nYou can install from source after cloning this repo using the following command\n\n```bash\npip install .\n```\n\n## Quick-start guides\n- [Fiddler Auditor Quickstart](https://github.com/fiddler-labs/fiddler-auditor/blob/main/examples/LLM_Evaluation.ipynb) \n- [Evaluate LLMs with custom metrics](https://github.com/fiddler-labs/fiddler-auditor/blob/main/examples/Custom_Evaluation.ipynb) \n- [Prompt injection attack with custom transformation](https://github.com/fiddler-labs/fiddler-auditor/blob/main/examples/Custom_Transformation.ipynb) \n\n\n## Contribution\nWe are continuously updating this library to support language models as they evolve. \n\n- Contributions in the form of suggestions and PRs to Fiddler Auditor are welcome!\n- If you encounter a bug, please feel free to raise issues in this repository.\n\nFor step-by-step instructions follow the [Contribution Guide](CONTRIBUTION.md).\n\n## Community\n- For questions and support, join the [Fiddler Community](https://www.fiddler.ai/slackinvite)\n- Discover the latest guides, videos, and research with the [Fiddler Resources Library](https://www.fiddler.ai/resources)\n- Stay informed by following us on [Twitter](https://twitter.com/fiddlerlabs)\n- Subscribe to our [monthly newsletter](https://www.fiddler.ai/blog#subscribe)\n- [Request a demo](https://www.fiddler.ai/demo)","github_created_at":"2023-05-18T22:58:28+00:00","created_at":"2026-07-11T23:14:41.821024+00:00","updated_at":"2026-08-02T06:00:43.58442+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"}],"tags":[{"slug":"ai-observability","name":"ai-observability"},{"slug":"evaluation","name":"evaluation"},{"slug":"generative-ai","name":"generative-ai"},{"slug":"langchain","name":"langchain"},{"slug":"llms","name":"llms"},{"slug":"nlp","name":"nlp"},{"slug":"robustness","name":"robustness"}],"trust":{"provenance":{"is_fork":false,"github_id":642585279,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-02T06:00:42.838Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":874,"last_release_at":"2023-11-08T23:23:11Z"},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:14:51.036Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-02T06:00:43.304Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-02T06:00:43.304Z"},"license_spdx":{"value":"Other","source":"github.license","observed_at":"2026-08-02T06:00:43.304Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"unknown","summary":"The pricing information for Fiddler Auditor is not specified in the repository data provided."},"requirements":{"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"pricing_model":"unknown","requires_docker":false},"when_to_use":["When you need to perform red-teaming exercises on your LLM using prompt perturbation specific to your use-case","For custom evaluation metrics that can be aligned with the specifics of your business requirements or project goals","If integration with LangChain is required for your NLP pipelines","To compare multiple language models and make informed decisions based on their robustness and reliability"],"when_not_to_use":["When standard evaluation methods suffice and you do not need advanced red-team testing tailored to your specific use-case","If the project does not require or benefit from custom evaluation metrics that address niche concerns beyond general model performance","In scenarios where models are already evaluated using other comprehensive frameworks, making additional evaluations redundant"],"source":"enrich:decision_facts","observed_at":"2026-07-16T19:54:33.769Z"},"constraint_facets":{"min_ram_gb":null,"pricing_model":"unknown","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"unknown - The pricing information for Fiddler Auditor is not specified in the repository data provided."},{"label":"Adopt for","value":"Fiddler Auditor is an evaluation tool for assessing the robustness and reliability of language models prior to their deployment in production."}]}}