GraphCanon updated 3w · GitHub synced 3w
Decision brief
llama3.java is a Java-centric tool for performing inference with Llama 3+ models without relying on external dependencies.
Good fit when
- Use llama3.java when you require language model inference capabilities fully implemented in Java, ensuring consistency within Java-based projects.
- Consider llama3.java if your project benefits from reduced setup complexity due to its lack of dependency on non-Java components.
Avoid when
- Avoid using llama3.java if your project needs specific features such as real-time chat integration that may be better supported by more specialized libraries.
- Do opt for a different tool if you prioritize performance metrics over the convenience of having an entirely Java-based solution, as competing tools might offer optimizations not found in llama3.java.
Observed Jul 14, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Slowing (91d since push)
- As of 3w
- Provenance
- Not a fork · Personal account
- As of 3w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/mukel/llama3.javaSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Offers LLM inference capabilities implemented entirely in Java, enabling users to perform language model inference without external dependencies.
Capability facts
- Languages
- java
Source: github.language · Jul 25, 2026
Categories
Tags
README
Llama3.java
Practical Llama 3, 3.1 and 3.2 inference implemented in a single Java file.
This project is the successor of llama2.java based on llama2.c by Andrej Karpathy and his excellent educational videos.
Besides the educational value, this project will be used to test and tune compiler optimizations and features on the JVM, particularly for the Graal compiler.
Features
- Single file, no dependencies
- GGUF format parser
- Llama 3+ tokenizer based on minbpe
- Llama 3+ inference with Grouped-Query Attention
- Support Llama 3.1 (ad-hoc RoPE scaling) and 3.2 (tie word embeddings)
- Support F16, BF16, F32 weights + Q4_0, Q4_1, Q4_K, Q5_K ,Q6_K, Q8_0 quantizations
- Fast matrix-vector multiplication routines using Java's Vector API
- Simple CLI with
--chatand--instructmodes. - GraalVM's Native Image support
- AOT model pre-loading for instant time-to-first-token
Interactive --chat mode in action:
Practical LLM inference in modern Java
Presented at Devoxx Belgium, 2024
Setup
Download pure Q4_0 and (optionally) Q8_0 quantized .gguf files from:
- https://huggingface.co/mukel/Llama-3.2-1B-Instruct-GGUF
- https://huggingface.co/mukel/Llama-3.2-3B-Instruct-GGUF
- https://huggingface.co/mukel/Meta-Llama-3.1-8B-Instruct-GGUF
- https://huggingface.co/mukel/Meta-Llama-3-8B-Instruct-GGUF
Or from unsloth https://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF
Optional: Pure quantizations
In the wild, Q8_0 quantizations are fine, but Q4_0 quantizations are rarely pure e.g. the token_embd.weights/output.weights tensor are quantized with Q6_K, instead of Q4_0.
A pure quantization can be generated from a high precision (F32, F16, BFLOAT16) .gguf source
with the llama-quantize utility from llama.cpp as follows:
./llama-quantize --pure ./Meta-Llama-3-8B-Instruct-BF16.gguf ./Meta-Llama-3-8B-Instruct-Q4_0.gguf Q4_0
Pick any of the supported quantizations: Q4_0, Q4_1, Q4_K, Q5_K, Q6_K, Q8_0.
Build and run
Java 21+ is required, in particular for the MemorySegment mmap-ing feature.
jbang is a perfect fit for this use case, just:
jbang Llama3.java --help
Or execute directly, also via jbang:
chmod +x Llama3.java
./Llama3.java --help
Optional: Makefile
A simple Makefile is provided, run make jar to produce llama3.jar.
Run the resulting llama3.jar as follows:
java --enable-preview --add-modules jdk.incubator.vector -jar llama3.jar --help
GraalVM Native Image
Compile with make native to produce a llama3 executable, then:
./llama3 --model Llama-3.2-1B-Instruct-Q8_0.gguf --chat
AOT model
For agents
This page has a .md twin and JSON over the API.