{"data":{"slug":"andrewnguonly-chatabstractions","name":"ChatAbstractions","tagline":"LangChain chat model abstractions for dynamic failover, load balancing, chaos engineering, and more!","github_url":"https://github.com/andrewnguonly/ChatAbstractions","owner":"andrewnguonly","repo":"ChatAbstractions","owner_avatar_url":"https://avatars.githubusercontent.com/u/7654246?v=4","primary_language":"Python","stars":84,"forks":5,"topics":[],"archived":false,"github_pushed_at":"2024-01-29T23:22:00+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/andrewnguonly-chatabstractions","markdown_url":"https://www.graphcanon.com/tools/andrewnguonly-chatabstractions.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/andrewnguonly-chatabstractions","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=andrewnguonly-chatabstractions","description":"LangChain chat model abstractions for dynamic failover, load balancing, chaos engineering, and more!","homepage_url":null,"license":"MIT","open_issues":4,"watchers":1,"ai_summary":null,"readme_excerpt":"# ChatAbstractions\n\nThis repo is a collection of chat model abstractions that demonstrates how to wrap (subclass) [LangChain's `BaseChatModel`](https://github.com/langchain-ai/langchain/blob/v0.0.350/libs/core/langchain_core/language_models/chat_models.py) in order to add functionality to a chain without breaking existing chat model interfaces. The use cases for wrapping chat models in this manner are mostly focused on dynamic model selection. However, other use cases are possible as well.\n\nSubclassing `BaseChatModel` requires implementing 2 methods: `_llm_type()` and `_generate()`.\n```python\nfrom typing import Any, List, Optional\n\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.schema import ChatResult\nfrom langchain.schema.messages import BaseMessage\n\n\nclass ChatSubclass(BaseChatModel):\n\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of chat model.\"\"\"\n        raise NotImplementedError\n\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n        **kwargs: Any,\n    ) -> ChatResult:\n        \"\"\"Add custom logic here.\"\"\"\n        raise NotImplementedError\n```\n\n## ChatDynamic\nThe implementation of `ChatDynamic` demonstrates the ability to select a chat model at runtime based on environment variable configuration. In the event of an outage or degraded performance by an LLM provider, this functionality (i.e. failover) may be desirable.\n\n```python\n# set environment variable DYNAMIC_CHAT_MODEL_ID=gpt-4\n\n# initialize chat models\ngpt_4_model = ChatOpenAI(model=\"gpt-4\")\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# specify all models that can be selected in the ChatDynamic instance\nchat_dynamic_model = ChatDynamic(\n    models={\n        \"gpt-4\": gpt_4_model,\n        \"gpt-3_5\": gpt_3_5_model,\n    },\n    default_model=\"gpt-4\",\n)\n```\n\nReading: [Dynamic Failover and Load Balancing LLMs With LangChain](https://medium.com/@andrewnguonly/dynamic-failover-and-load-balancing-llms-with-langchain-e930a094be61)\n\n## ChatLoadBalance\nThe implementation of `ChatLoadBalance` demonstrates the ability to select a method of load balancing (random, round robin, least rate limited) between LLM models. In the event of rate limiting or peak usage times, this functionality may be desirable.\n\n```python\n# initialize chat models\ngpt_4_model = ChatOpenAI(model=\"gpt-4\")\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# specify all models that can be selected in the ChatLoadBalance instance\nchat_load_balance_model = ChatLoadBalance(\n    models=[gpt_4_model, gpt_3_5_model],\n    load_balance_type=1,  # 0 - random, 1 - round robin, 2 - least rate limited\n)\n```\n\nReading: [Dynamic Failover and Load Balancing LLMs With LangChain](https://medium.com/@andrewnguonly/dynamic-failover-and-load-balancing-llms-with-langchain-e930a094be61)\n\n## ChatChaos\nThe implementation of `ChatChaos` demonstrates the ability to substitute normal LLM behavior with chaotic behavior. The purpose of this abstraction is to promote the [Principles of Chaos Engineering](https://principlesofchaos.org/) in the context of LLM applications. This abstraction is inspired by [Netflix's Chaos Monkey](https://github.com/Netflix/chaosmonkey).\n\n```python\n# initialize chat model\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# configure ChatChaos\nchat_chaos_model = ChatChaos(\n    model=gpt_3_5_model,\n    enabled=True,\n    cron=croniter(\"0 * * * *\"),\n    duration_mins=60,\n    ratio=1.0,\n    enable_malformed_json=False,\n    enable_hallucination=True,\n    enable_latency=False,\n    hallucination_prompt=\"Write a poem about the Python programming language.\",\n)\n```\n\nReading: [ChatChaos: The Good, the Bad, and the Ugly](https://medium.com/@andrewnguonly/chatchaos-the-good-the-bad-and-the-ugly-81f9612d7b00)\n\n## ChatNotDiamond\nThe implementation of `ChatNotDiamond` demonstrates the abi","github_created_at":"2023-11-08T23:46:11+00:00","created_at":"2026-07-11T10:45:04.797705+00:00","updated_at":"2026-07-11T10:45:15.47496+00:00","categories":[{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"},{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":716340652,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T10:45:05.461Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":893,"last_release_at":null},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":16,"high_count":0,"last_scan_at":"2026-07-11T10:45:06.376Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T10:45:05.993Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T10:45:05.993Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T10:45:05.993Z"}}}}