{"data":{"slug":"andrewnguonly-chatabstractions","name":"ChatAbstractions","tagline":"LangChain chat model abstractions for dynamic failover, load balancing, chaos engineering, and more!","github_url":"https://github.com/andrewnguonly/ChatAbstractions","owner":"andrewnguonly","repo":"ChatAbstractions","owner_avatar_url":"https://avatars.githubusercontent.com/u/7654246?v=4","primary_language":"Python","stars":84,"forks":5,"topics":[],"archived":false,"github_pushed_at":"2024-01-29T23:22:00+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/andrewnguonly-chatabstractions","markdown_url":"https://www.graphcanon.com/tools/andrewnguonly-chatabstractions.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/andrewnguonly-chatabstractions","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=andrewnguonly-chatabstractions","description":"LangChain chat model abstractions for dynamic failover, load balancing, chaos engineering, and more!","homepage_url":null,"license":"MIT","open_issues":5,"watchers":1,"ai_summary":"A collection of chat model abstractions that wrap LangChain's `BaseChatModel` to add functionalities like dynamic model selection, failover, and load balancing without breaking existing interfaces.","readme_excerpt":"# ChatAbstractions\n\nThis repo is a collection of chat model abstractions that demonstrates how to wrap (subclass) [LangChain's `BaseChatModel`](https://github.com/langchain-ai/langchain/blob/v0.0.350/libs/core/langchain_core/language_models/chat_models.py) in order to add functionality to a chain without breaking existing chat model interfaces. The use cases for wrapping chat models in this manner are mostly focused on dynamic model selection. However, other use cases are possible as well.\n\nSubclassing `BaseChatModel` requires implementing 2 methods: `_llm_type()` and `_generate()`.\n```python\nfrom typing import Any, List, Optional\n\nfrom langchain.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain.chat_models.base import BaseChatModel\nfrom langchain.schema import ChatResult\nfrom langchain.schema.messages import BaseMessage\n\n\nclass ChatSubclass(BaseChatModel):\n\n    @property\n    def _llm_type(self) -> str:\n        \"\"\"Return type of chat model.\"\"\"\n        raise NotImplementedError\n\n    def _generate(\n        self,\n        messages: List[BaseMessage],\n        stop: Optional[List[str]] = None,\n        run_manager: Optional[CallbackManagerForLLMRun] = None,\n        **kwargs: Any,\n    ) -> ChatResult:\n        \"\"\"Add custom logic here.\"\"\"\n        raise NotImplementedError\n```\n\n## ChatDynamic\nThe implementation of `ChatDynamic` demonstrates the ability to select a chat model at runtime based on environment variable configuration. In the event of an outage or degraded performance by an LLM provider, this functionality (i.e. failover) may be desirable.\n\n```python\n# set environment variable DYNAMIC_CHAT_MODEL_ID=gpt-4\n\n# initialize chat models\ngpt_4_model = ChatOpenAI(model=\"gpt-4\")\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# specify all models that can be selected in the ChatDynamic instance\nchat_dynamic_model = ChatDynamic(\n    models={\n        \"gpt-4\": gpt_4_model,\n        \"gpt-3_5\": gpt_3_5_model,\n    },\n    default_model=\"gpt-4\",\n)\n```\n\nReading: [Dynamic Failover and Load Balancing LLMs With LangChain](https://medium.com/@andrewnguonly/dynamic-failover-and-load-balancing-llms-with-langchain-e930a094be61)\n\n## ChatLoadBalance\nThe implementation of `ChatLoadBalance` demonstrates the ability to select a method of load balancing (random, round robin, least rate limited) between LLM models. In the event of rate limiting or peak usage times, this functionality may be desirable.\n\n```python\n# initialize chat models\ngpt_4_model = ChatOpenAI(model=\"gpt-4\")\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# specify all models that can be selected in the ChatLoadBalance instance\nchat_load_balance_model = ChatLoadBalance(\n    models=[gpt_4_model, gpt_3_5_model],\n    load_balance_type=1,  # 0 - random, 1 - round robin, 2 - least rate limited\n)\n```\n\nReading: [Dynamic Failover and Load Balancing LLMs With LangChain](https://medium.com/@andrewnguonly/dynamic-failover-and-load-balancing-llms-with-langchain-e930a094be61)\n\n## ChatChaos\nThe implementation of `ChatChaos` demonstrates the ability to substitute normal LLM behavior with chaotic behavior. The purpose of this abstraction is to promote the [Principles of Chaos Engineering](https://principlesofchaos.org/) in the context of LLM applications. This abstraction is inspired by [Netflix's Chaos Monkey](https://github.com/Netflix/chaosmonkey).\n\n```python\n# initialize chat model\ngpt_3_5_model = ChatOpenAI(model=\"gpt-3.5-turbo\")\n\n# configure ChatChaos\nchat_chaos_model = ChatChaos(\n    model=gpt_3_5_model,\n    enabled=True,\n    cron=croniter(\"0 * * * *\"),\n    duration_mins=60,\n    ratio=1.0,\n    enable_malformed_json=False,\n    enable_hallucination=True,\n    enable_latency=False,\n    hallucination_prompt=\"Write a poem about the Python programming language.\",\n)\n```\n\nReading: [ChatChaos: The Good, the Bad, and the Ugly](https://medium.com/@andrewnguonly/chatchaos-the-good-the-bad-and-the-ugly-81f9612d7b00)\n\n## ChatNotDiamond\nThe implementation of `ChatNotDiamond` demonstrates the abi","github_created_at":"2023-11-08T23:46:11+00:00","created_at":"2026-07-11T10:45:04.797705+00:00","updated_at":"2026-09-20T04:24:30.631217+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"chat-models","name":"chat-models"},{"slug":"dynamic-failover","name":"dynamic-failover"},{"slug":"langchain","name":"langchain"},{"slug":"load-balancing","name":"load-balancing"}],"trust":{"provenance":{"is_fork":false,"github_id":716340652,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-08T06:00:49.269Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":952,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":1},"security_summary":{"status":"findings","scanner":"osv@v1","low_count":16,"high_count":0,"last_scan_at":"2026-07-11T10:45:06.376Z","medium_count":0,"scan_profile":"deps","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-08T06:00:50.139Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-08T06:00:50.139Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-08T06:00:50.139Z"}},"decision_facts":{"hosting":null,"pricing":{"model":"freemium"},"requirements":{"min_ram_gb":2,"requires_docker":false},"constraints":{"min_ram_gb":2,"pricing_model":"freemium","requires_docker":false},"when_to_use":["- When you need dynamic model selection in your application to switch between different LLMs at runtime based on performance or outages.","- If you require a robust solution for load balancing across multiple language models during peak usage times to prevent rate limitations.","- For implementing chaos engineering principles to test and improve the resilience of your application involving language model interactions."],"when_not_to_use":["- This tool is not recommended if you are looking for static solutions where the chat model or load balancing policies do not change at runtime.","- If you require integration with a specific set of models that aren't compatible with LangChain's `BaseChatModel`, this might not be suitable.","- When your application has minimal risk of LLM provider outages and does not benefit from dynamic failover features."],"source":"enrich:decision_facts","observed_at":"2026-07-12T15:39:09.462Z"},"constraint_facets":{"min_ram_gb":2,"pricing_model":"freemium","requires_docker":false},"decision_summary":[{"label":"Pricing","value":"freemium"},{"label":"Requirements","value":"Min 2 GB RAM"},{"label":"Adopt for","value":"ChatAbstractions is a Python-based collection of abstractions wrapped around LangChain's `BaseChatModel` that enable dynamic failover, load balancing, and chaos engineering without altering existing chat interfaces."}]}}