“offloading”

petals

Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

FlexLLMGen

Running large language models on a single GPU for throughput-oriented scenarios.