{"data":{"slug":"stability-ai-stablelm","name":"StableLM","tagline":"Stability AI Language Models","github_url":"https://github.com/Stability-AI/StableLM","owner":"Stability-AI","repo":"StableLM","owner_avatar_url":"https://avatars.githubusercontent.com/u/100950301?v=4","primary_language":"Jupyter Notebook","stars":15678,"forks":1001,"topics":[],"archived":false,"github_pushed_at":"2024-04-08T19:26:02+00:00","maintenance_label":"Dormant","stars_delta_30d":-6,"url":"https://www.graphcanon.com/tools/stability-ai-stablelm","markdown_url":"https://www.graphcanon.com/tools/stability-ai-stablelm.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/stability-ai-stablelm","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=stability-ai-stablelm","description":"StableLM: Stability AI Language Models","homepage_url":null,"license":"Apache-2.0","open_issues":27,"watchers":196,"ai_summary":"Repository for Stability AI's StableLM series of language models, including model checkpoints and updates.","readme_excerpt":"# StableLM: Stability AI Language Models\n\n\n<br/>*“A Stochastic Parrot, flat design, vector art” — [Stable Diffusion XL](https://clipdrop.co/stable-diffusion)*\n\nThis repository contains Stability AI's ongoing development of the StableLM series of language models and will be continuously updated with new checkpoints. The following provides an overview of all currently available models. More coming soon.\n\n## News\n\n*September 29, 2023*\n\n- Released StableLM-3B-4E1T model under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).\n\n*August 5, 2023*\n\n- Released patched StableLM-Alpha v2 models with 3B and 7B parameters.\n\n*April 28, 2023*\n\n- Released StableVicuna-13B, our RLHF fine-tune of [Vicuna-13B v0](https://huggingface.co/lmsys/vicuna-13b-delta-v0), which itself is a fine-tune of [LLaMA-13B](https://github.com/facebookresearch/llama). Delta weights over the original Llama model is released under ([CC BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)).\n\n*April 20, 2023*\n\n- Released initial set of StableLM-Alpha models, with 3B and 7B parameters. Base models are released under [CC BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/).\n\n- Try to chat with our 7B model, `StableLM-Tuned-Alpha-7B`, on [Hugging Face Spaces](https://huggingface.co/spaces/stabilityai/stablelm-tuned-alpha-chat).\n\n## Models\n\n### StableLM-3B-4E1T\n\n> Technical Report: [StableLM-3B-4E1T](https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo)\n\nStableLM-3B-4E1T is a 3 billion (3B) parameter language model pre-trained under the multi-epoch regime to study the impact of repeated tokens on downstream performance. Given prior success in this area ([Tay et al., 2023](https://arxiv.org/pdf/2205.05131.pdf) and [Taylor et al., 2022](https://galactica.org/static/paper.pdf)), we train on 1 trillion (1T) tokens for 4 epochs following the observations of [Muennighoff et al. (2023)](https://arxiv.org/abs/2305.16264) in \"Scaling Data-Constrained Language Models\" in which they find \"training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.\" Further inspiration for the token count is taken from \"Go smol or go home\" ([De Vries, 2023](https://www.harmdevries.com/post/model-size-vs-compute-overhead/)), which suggests a 2.96B model trained for 2.85 trillion tokens achieves a similar loss to a Chinchilla compute-optimal 9.87B language model ($k_n = 0.3$).\n\n| Size | StableLM-3B-4E1T                                                     | Training Tokens | Parameters    |\n|------|--------------------------------------------------------------------|-----------------|---------------|\n| 3B   | [checkpoint](https://huggingface.co/stabilityai/stablelm-3b-4e1t) | 4T              | 2,795,443,200 |\n\n#### Model Architecture\n\nThe model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:\n\n| Parameters     | Hidden Size | Layers | Heads | Sequence Length |\n|----------------|-------------|--------|-------|-----------------|\n| 2,795,443,200  | 2560        | 32     | 32    | 4096            |\n\n- **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).\n- **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)).\n- **Tokenizer**: GPT-NeoX ([Black et al., 2022](https://arxiv.org/abs/2204.06745)).\n\n#### Training Data\n\nThe dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon Refined","github_created_at":"2023-04-19T14:02:16+00:00","created_at":"2026-07-11T23:09:10.076825+00:00","updated_at":"2026-09-20T04:29:23.103658+00:00","categories":[{"slug":"llm-frameworks","name":"LLM Frameworks","url":"https://www.graphcanon.com/categories/llm-frameworks","markdown_url":"https://www.graphcanon.com/categories/llm-frameworks.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/llm-frameworks"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"checkpoints","name":"checkpoints"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"language-models","name":"language models"},{"slug":"pre-training","name":"pre-training"},{"slug":"stablelm","name":"stablelm"}],"trust":{"provenance":{"is_fork":false,"github_id":630023030,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:15:18.364Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":892,"last_release_at":null,"stars_delta_30d":-6,"open_issues_delta_30d":-1},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:15:19.972Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:15:17.253Z"},"languages":{"value":["jupyter notebook"],"source":"github.language","observed_at":"2026-09-18T08:15:17.253Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-09-18T08:15:17.253Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need a model that has been trained with a multi-epoch regime to study the impact of repeated tokens on performance, such as the StableLM-3B-4E1T model.","If you are looking for a model that is available under the CC BY-SA-4.0 license, allowing for broad use and modification with attribution.","When you require a decoder-only transformer model with specific architecture modifications, such as the use of Rotary Position Embeddings on the first 25% of head embedding dimensions."],"when_not_to_use":["If you need a model that is strictly under the Apache-2.0 license, as StableLM models are available under different licenses like CC BY-SA-4.0 and CC BY-NC-SA-4.0.","When you require a model that has not been trained with repeated tokens, as StableLM models like StableLM-3B-4E1T are specifically designed with this training method.","If your project requires a model that is not a decoder-only transformer with specific architecture modifications, such as the use of Rotary Position Embeddings."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:41:11.330Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"StableLM is a series of language models from Stability AI, featuring models like StableLM-3B-4E1T, which is a 3 billion parameter model trained on 4 trillion tokens over 4 epochs. The repository includes model updates, a"}]}}