GraphCanon updated 2w · GitHub synced 2w
Decision brief
StableLM offers pre-trained language models for development and research with an emphasis on repeated-token training effects to improve performance.
Good fit when
- When targeting research into the impact of multi-epoch token repetition on model performance, as StableLM is specifically designed around this concept.
- In scenarios requiring creative or generative text tasks where a 3 billion parameter model suffices for the use case while offering flexibility and custom development.
Avoid when
- If your project requires the strictest data privacy guarantees since some models are under less restrictive licenses, limiting their usage in projects with such constraints.
- For applications needing larger language models than 13 billion parameters, as StableLM's largest model is StableVicuna-13B.
Observed Jul 14, 2026 · Source: enrich:decision_facts
Verify the decision
Maintenance and security
Full trust report- Maintenance
- Dormant (844d since push)
- As of 2w
- Provenance
- Not a fork · Organization account
- As of 2w
- Security (OSV)
- No lockfile
- As of 1mo
Public GitHub metadata and optional OSV scans. Signals, not a guarantee. Trust methodology.
Install
git clone https://github.com/Stability-AI/StableLMSimilar tools
Same-category neighbours. No typed graph edges are catalogued for this tool yet.
Evidence and technical details
Sourced facts, taxonomy, compatibility claims, README excerpt, and machine-readable endpoints.
Overview
Repo contains Stability AI's StableLM series of language models with details on model releases and technical documentation.
Capability facts
- Languages
- jupyter notebook
Source: github.language · Aug 1, 2026
Categories
Tags
README
StableLM: Stability AI Language Models
“A Stochastic Parrot, flat design, vector art” — Stable Diffusion XL
This repository contains Stability AI's ongoing development of the StableLM series of language models and will be continuously updated with new checkpoints. The following provides an overview of all currently available models. More coming soon.
News
September 29, 2023
- Released StableLM-3B-4E1T model under CC BY-SA-4.0.
August 5, 2023
- Released patched StableLM-Alpha v2 models with 3B and 7B parameters.
April 28, 2023
- Released StableVicuna-13B, our RLHF fine-tune of Vicuna-13B v0, which itself is a fine-tune of LLaMA-13B. Delta weights over the original Llama model is released under (CC BY-NC-SA-4.0).
April 20, 2023
-
Released initial set of StableLM-Alpha models, with 3B and 7B parameters. Base models are released under CC BY-SA-4.0.
-
Try to chat with our 7B model,
StableLM-Tuned-Alpha-7B, on Hugging Face Spaces.
Models
StableLM-3B-4E1T
Technical Report: StableLM-3B-4E1T
StableLM-3B-4E1T is a 3 billion (3B) parameter language model pre-trained under the multi-epoch regime to study the impact of repeated tokens on downstream performance. Given prior success in this area (Tay et al., 2023 and Taylor et al., 2022), we train on 1 trillion (1T) tokens for 4 epochs following the observations of Muennighoff et al. (2023) in "Scaling Data-Constrained Language Models" in which they find "training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data." Further inspiration for the token count is taken from "Go smol or go home" (De Vries, 2023), which suggests a 2.96B model trained for 2.85 trillion tokens achieves a similar loss to a Chinchilla compute-optimal 9.87B language model ($k_n = 0.3$).
| Size | StableLM-3B-4E1T | Training Tokens | Parameters |
|---|---|---|---|
| 3B | checkpoint | 4T | 2,795,443,200 |
Model Architecture
The model is a decoder-only transformer similar to the LLaMA (Touvron et al., 2023) architecture with the following modifications:
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
|---|---|---|---|---|
| 2,795,443,200 | 2560 | 32 | 32 | 4096 |
- Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022).
- Normalization: LayerNorm (Ba et al., 2016) with learned bias terms as opposed to RMSNorm (Zhang & Sennrich, 2019).
- Tokenizer: GPT-NeoX (Black et al., 2022).
Training Data
The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the HuggingFace Hub: Falcon Refined
For agents
This page has a .md twin and JSON over the API.