Home/Computer Vision/NanoLLM

NanoLLM

dusty-nv/NanoLLM

Optimized local inference for LLMs using HuggingFace-like APIs

GitHub Website

GraphCanon updated Sep 20, 2026 · GitHub synced Sep 20, 2026

46views this month

382 stars67 forksLast push Oct 18, 2024 Python MIT

Decision brief

NanoLLM optimizes local inference for LLMs via HuggingFace-compatible APIs, supporting quantization and multimodal applications like vision, speech, RAG, and vector databases.

Good fit when

When building edge-ai solutions requiring optimized local inference
For multimodal AI projects needing integrations with speech and image models

Avoid when

In scenarios where a fully cloud-based solution is preferred over local inference
If the project does not benefit from multimodal or RAG capabilities

Observed Jul 17, 2026 · Source: enrich:decision_facts

NanoLLM alternatives Trust report

Verify the decision

Maintenance and security

Full trust report

Maintenance: Dormant (702d since push); As of Sep 20, 2026
Provenance: Not a fork · Personal account; As of Sep 20, 2026
Security (OSV): No lockfile; As of Jul 11, 2026

Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.

Install

pip install NanoLLM

PyPI

How it fits your stack(1)

Typed graph edges - alternatives, integrations, successors, and dependencies. Ranked by relationship type, not raw GitHub stars.

Integrates

headroom

NanoLLM provides optimized local inference, and Headroom could integrate with it to further reduce token usage through compression techniques.

integrates

Relationship graph

Optional deeper exploration of typed edges and category neighbours.

Similar tools

Same-category neighbours not already linked as typed edges.

inference

Unified production-ready inference API for various LLMs and models

9.6k★

oumi

Easily fine-tune, evaluate and deploy open source LLMs/VLMs

9.4k★

whichllm

Command-line tool to find and benchmark local LLM performance

6.7k★

Evidence and technical details

Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.

Overview

NanoLLM supports optimized local inference with HuggingFace-compatible interfaces for tasks such as quantization and multimodal applications including vision, language, speech, vector databases, and RAG.

Capability facts

Languages: python
Source: github.language · Sep 20, 2026

README

NanoLLM <a href="https://www.jetson ai lab.com" <img align="right" width="200" height="200" src="https://nvidia ai iot.github.io/jetson generative ai playground/images/JON Gen AI panels.png" </a Optimized local inference for LLMs with HuggingFace like APIs for quantization, visio...

For agents

This page has a .md twin and JSON over the API.

/tools/dusty-nv-nanollm.md GET /api/graphcanon/tools/dusty-nv-nanollm GET /api/graphcanon/graph?tool=dusty-nv-nanollm