{"data":{"slug":"fminference-flexllmgen","name":"FlexLLMGen","tagline":"Running large language models on a single GPU for throughput-oriented scenarios","github_url":"https://github.com/FMInference/FlexLLMGen","owner":"FMInference","repo":"FlexLLMGen","owner_avatar_url":"https://avatars.githubusercontent.com/u/125944572?v=4","primary_language":"Python","stars":9348,"forks":593,"topics":["deep-learning","gpt-3","high-throughput","large-language-models","machine-learning","offloading","opt"],"archived":true,"github_pushed_at":"2024-10-28T03:05:41+00:00","maintenance_label":"Archived","stars_delta_30d":-13,"url":"https://www.graphcanon.com/tools/fminference-flexllmgen","markdown_url":"https://www.graphcanon.com/tools/fminference-flexllmgen.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/fminference-flexllmgen","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=fminference-flexllmgen","description":"Running large language models on a single GPU for throughput-oriented scenarios.","homepage_url":null,"license":"Apache-2.0","open_issues":58,"watchers":10,"ai_summary":"FlexLLMGen is a Python-based tool for executing large language models on a single GPU, focusing on high-throughput scenarios. It supports deep learning frameworks like PyTorch and is designed for offloading tasks to improve performance.","readme_excerpt":"## Installation\nRequirements:  \n - PyTorch >= 1.12 [(Help)](https://pytorch.org/get-started/locally/)","github_created_at":"2023-02-15T21:18:53+00:00","created_at":"2026-07-11T23:11:04.937579+00:00","updated_at":"2026-09-20T04:29:38.11441+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"deep-learning","name":"deep-learning"},{"slug":"gpt-3","name":"gpt-3"},{"slug":"high-throughput","name":"high-throughput"},{"slug":"large-language-models","name":"large-language-models"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"offloading","name":"offloading"},{"slug":"opt","name":"opt"}],"trust":{"provenance":{"is_fork":false,"github_id":602270517,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:17:20.084Z","maintenance":{"label":"Archived","score":8,"methodology":"github_public_v1","releases_90d":0,"days_since_push":690,"last_release_at":null,"stars_delta_30d":-13,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:17:21.714Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:17:19.036Z"},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-09-18T08:17:19.036Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:17:19.036Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Requires PyTorch >= 1.12"],"min_ram_gb":null,"requires_docker":false},"constraints":{"min_ram_gb":null,"requires_docker":false},"when_to_use":["When you need to run large language models on a single GPU for high-throughput scenarios, FlexLLMGen can be an effective choice.","If your project requires offloading tasks to improve performance and you are working with PyTorch, FlexLLMGen is suitable.","For developers looking to enhance the throughput of their language model inference tasks on a single GPU, FlexLLMGen is a good option."],"when_not_to_use":["Avoid using FlexLLMGen if your project does not require high-throughput and you are working with frameworks other than PyTorch.","If your scenario demands multi-GPU setups for model execution, FlexLLMGen, which focuses on single GPU setups, may not be the best choice.","Do not use FlexLLMGen if your application does not benefit from task offloading or if you are working with models that do not align with the high-throughput design of this tool."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:54:07.608Z"},"constraint_facets":{"min_ram_gb":null,"requires_docker":false},"decision_summary":[{"label":"Requirements","value":"Requires PyTorch >= 1.12"},{"label":"Adopt for","value":"FlexLLMGen is a Python-based tool for executing large language models on a single GPU, focusing on high-throughput scenarios. It supports PyTorch and is designed for offloading tasks to improve performance."}]}}