{"data":{"slug":"mukel-llama3-java","name":"llama3.java","tagline":"Llama 3+ inference in pure Java","github_url":"https://github.com/mukel/llama3.java","owner":"mukel","repo":"llama3.java","owner_avatar_url":"https://avatars.githubusercontent.com/u/1896283?v=4","primary_language":"Java","stars":815,"forks":94,"topics":["chatgpt","genai","gguf","huggingface","java","llama","llama3","llamacpp","llm","llm-inference","llms","openai","simd","transformers"],"archived":false,"github_pushed_at":"2026-04-24T16:38:39+00:00","maintenance_label":"Slowing","stars_delta_30d":-1,"url":"https://www.graphcanon.com/tools/mukel-llama3-java","markdown_url":"https://www.graphcanon.com/tools/mukel-llama3-java.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/mukel-llama3-java","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=mukel-llama3-java","description":"Llama 3+ inference in pure Java","homepage_url":null,"license":"MIT","open_issues":18,"watchers":27,"ai_summary":"Offers LLM inference capabilities implemented entirely in Java, enabling users to perform language model inference without external dependencies.","readme_excerpt":"# Llama3.java\n\nPractical [Llama 3](https://github.com/meta-llama/llama3), [3.1](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1) and [3.2](https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/) inference implemented in a single Java file.\n\n

\n \n

\n\nThis project is the successor of [llama2.java](https://github.com/mukel/llama2.java)\nbased on [llama2.c](https://github.com/karpathy/llama2.c) by [Andrej Karpathy](https://twitter.com/karpathy) and his [excellent educational videos](https://www.youtube.com/c/AndrejKarpathy).\n\nBesides the educational value, this project will be used to test and tune compiler optimizations and features on the JVM, particularly for the [Graal compiler](https://www.graalvm.org/latest/reference-manual/java/compiler).\n\n## Features\n\n - Single file, no dependencies\n - [GGUF format](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) parser\n - Llama 3+ tokenizer based on [minbpe](https://github.com/karpathy/minbpe)\n - Llama 3+ inference with Grouped-Query Attention\n - Support Llama 3.1 (ad-hoc RoPE scaling) and 3.2 (tie word embeddings)\n - Support F16, BF16, F32 weights + Q4_0, Q4_1, Q4_K, Q5_K ,Q6_K, Q8_0 quantizations\n - Fast matrix-vector multiplication routines using Java's [Vector API](https://openjdk.org/jeps/469)\n - Simple CLI with `--chat` and `--instruct` modes.\n - GraalVM's Native Image support\n - AOT model pre-loading for instant time-to-first-token\n\n**Interactive `--chat` mode in action:**\n

\n \n

\n\n## [Practical LLM inference in modern Java](https://www.youtube.com/watch?v=zgAMxC7lzkc)\n**Presented at Devoxx Belgium, 2024**\n
\n \n \n \n
\n\n## Setup\n\nDownload pure `Q4_0` and (optionally) `Q8_0` quantized .gguf files from:\n - https://huggingface.co/mukel/Llama-3.2-1B-Instruct-GGUF\n - https://huggingface.co/mukel/Llama-3.2-3B-Instruct-GGUF\n - https://huggingface.co/mukel/Meta-Llama-3.1-8B-Instruct-GGUF\n - https://huggingface.co/mukel/Meta-Llama-3-8B-Instruct-GGUF\n\nOr from unsloth\nhttps://huggingface.co/unsloth/Llama-3.2-1B-Instruct-GGUF\n\n#### Optional: Pure quantizations\n\nIn the wild, `Q8_0` quantizations are fine, but `Q4_0` quantizations are rarely pure e.g. the `token_embd.weights`/`output.weights` tensor are quantized with `Q6_K`, instead of `Q4_0`. \nA **pure** quantization can be generated from a high precision (F32, F16, BFLOAT16) .gguf source \nwith the `llama-quantize` utility from [llama.cpp](https://github.com/ggml-org/llama.cpp) as follows:\n\n```bash\n./llama-quantize --pure ./Meta-Llama-3-8B-Instruct-BF16.gguf ./Meta-Llama-3-8B-Instruct-Q4_0.gguf Q4_0\n```\n\nPick any of the supported quantizations: `Q4_0`, `Q4_1`, `Q4_K`, `Q5_K`, `Q6_K`, `Q8_0`.\n\n## Build and run\n\nJava 21+ is required, in particular for the [`MemorySegment` mmap-ing feature](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,long,long,java.lang.foreign.Arena)).\n\n[`jbang`](https://www.jbang.dev/) is a perfect fit for this use case, just:\n```\njbang Llama3.java --help\n```\nOr execute directly, also via [`jbang`](https://www.jbang.dev/):\n```bash \nchmod +x Llama3.java\n./Llama3.java --help\n```\n\n#### Optional: Makefile\n\nA simple [Makefile](./Makefile) is provided, run `make jar` to produce `llama3.jar`.\n\nRun the resulting `llama3.jar` as follows: \n```bash\njava --enable-preview --add-modules jdk.incubator.vector -jar llama3.jar --help\n```\n\n### GraalVM Native Image\n\nCompile with `make native` to produce a `llama3` executable, then:\n\n```bash\n./llama3 --model Llama-3.2-1B-Instruct-Q8_0.gguf --chat\n```\n\n### AOT model","github_created_at":"2024-04-25T09:09:02+00:00","created_at":"2026-07-11T11:44:32.734644+00:00","updated_at":"2026-08-25T06:01:39.730321+00:00","categories":[{"slug":"inference-serving","name":"Inference & Serving","url":"https://www.graphcanon.com/categories/inference-serving","markdown_url":"https://www.graphcanon.com/categories/inference-serving.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/inference-serving"}],"tags":[{"slug":"chatgpt","name":"chatgpt"},{"slug":"genai","name":"genai"},{"slug":"gguf","name":"gguf"},{"slug":"huggingface","name":"huggingface"},{"slug":"java","name":"java"},{"slug":"llama","name":"llama"},{"slug":"llama3","name":"llama3"},{"slug":"llamacpp","name":"llamacpp"}],"trust":{"provenance":{"is_fork":false,"github_id":791733442,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-25T06:01:39.006Z","maintenance":{"label":"Slowing","score":36,"methodology":"github_public_v1","releases_90d":0,"days_since_push":122,"last_release_at":null,"stars_delta_30d":-1,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:44:33.935Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-25T06:01:39.452Z"},"languages":{"value":["java"],"source":"github.language","observed_at":"2026-08-25T06:01:39.452Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-25T06:01:39.452Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use llama3.java when you require language model inference capabilities fully implemented in Java, ensuring consistency within Java-based projects.","Consider llama3.java if your project benefits from reduced setup complexity due to its lack of dependency on non-Java components."],"when_not_to_use":["Avoid using llama3.java if your project needs specific features such as real-time chat integration that may be better supported by more specialized libraries.","Do opt for a different tool if you prioritize performance metrics over the convenience of having an entirely Java-based solution, as competing tools might offer optimizations not found in llama3.java."],"source":"enrich:decision_facts","observed_at":"2026-07-14T18:01:48.561Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"llama3.java is a Java-centric tool for performing inference with Llama 3+ models without relying on external dependencies."}]}}