GraphCanon
Home/LLM Frameworks/fiddler-auditor
fiddler-auditor logo

fiddler-auditor

Enrichment pending
fiddler-labs/fiddler-auditor

Fiddler Auditor is a tool to evaluate language models.

GitHub

GraphCanon updated today · GitHub synced today

193
Stars
24
Forks
15
Open issues
6
Watchers
2y
Last push
Python OtherCreated May 18, 2023

Trust & integrity

Full report
Maintenance
Dormant (852d since push)
As of today · Source: github_public_v1
Provenance
Not a fork · Organization account
As of today · Source: github_public_v1
Security (OSV)
No lockfile
As of today · Source: none

Public GitHub metadata and optional OSV dependency scans. Signals, not a guarantee. Trust methodology.

Overview

Fiddler Auditor is a tool to evaluate language models.

Capability facts

Languages
python

Source: github.language+pyproject.toml · Jul 11, 2026

Categories

LLM FrameworksModel TrainingInference & Serving

Compatibility

Sourced claims from the README excerpt - not unsourced marketing copy.

LangChain integrationLangChain

Source: README excerpt (regex_v1, Jul 11, 2026)

- Integration with LangChain
Source link
Python runtimePython

Source: README excerpt (regex_v1, Jul 11, 2026)

Auditor is available on PyPI and we test on Python 3.8 and above. We recommend creating a virtual python environment and installing us
Source link

Tags

llmsevaluationnlppythonrobustnessgenerative-aiai-observabilitylangchain

README

Fiddler Auditor

Auditing Large Language Models made easy!

What is Fiddler Auditor?

Fiddler Auditor Capabilities

Language models enable companies to build and launch innovative applications to improve productivity and increase customer satisfaction. However, it’s been known that LLMs can hallucinate, generate adversarial responses that can harm users, and even expose private information that they were trained on when prompted or unprompted. It's more critical than ever for ML and software application teams to minimize these risks and weaknesses before launching LLMs and NLP models. As a result, it’s important for you to include a process to audit language models thoroughly before production. The Fiddler Auditor enables you to test LLMs and NLP models, identify weaknesses in the models, and mitigate potential adversarial outcomes before deploying them to production.

Features and Capabilities

Fiddler Auditor Flow

Fiddler Auditor supports

  • Red-teaming LLMs for your use-case with prompt perturbation
  • Integration with LangChain
  • Custom evaluation metrics
  • Generative and Discriminative NLP models
  • Comparison of LLMs

Example Report An example report generated by the Fiddler Auditor for text-davinci-003.

Installation

From PyPI

Auditor is available on PyPI and we test on Python 3.8 and above. We recommend creating a virtual python environment and installing using the following command

pip install fiddler-auditor

From source

You can install from source after cloning this repo using the following command

pip install .

Quick-start guides

Contribution

We are continuously updating this library to support language models as they evolve.

  • Contributions in the form of suggestions and PRs to Fiddler Auditor are welcome!
  • If you encounter a bug, please feel free to raise issues in this repository.

For step-by-step instructions follow the Contribution Guide.

Community