{"data":{"slug":"microsoft-codebert","name":"CodeBERT","tagline":"CodeBERT series models for code pretraining in Python and programming languages","github_url":"https://github.com/microsoft/CodeBERT","owner":"microsoft","repo":"CodeBERT","owner_avatar_url":"https://avatars.githubusercontent.com/u/6154722?v=4","primary_language":"Python","stars":2787,"forks":497,"topics":[],"archived":false,"github_pushed_at":"2023-07-09T12:26:30+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/microsoft-codebert","markdown_url":"https://www.graphcanon.com/tools/microsoft-codebert.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/microsoft-codebert","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=microsoft-codebert","description":"CodeBERT","homepage_url":null,"license":"MIT","open_issues":86,"watchers":37,"ai_summary":"Provides pretrained CodeBERT series models for tasks involving natural language and programming languages like Python, Java, JavaScript, PHP, Ruby, Go.","readme_excerpt":"# Code Pretraining Models\n\nThis repo contains code pretraining models in the CodeBERT series from Microsoft, including six models as of June 2023.\n- CodeBERT (EMNLP 2020)\n- GraphCodeBERT (ICLR 2021)\n- UniXcoder (ACL 2022)\n- CodeReviewer (ESEC/FSE 2022)\n- CodeExecutor (ACL 2023)\n- LongCoder (ICML 2023)\n\n# CodeBERT\n\nThis repo provides the code for reproducing the experiments in [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://arxiv.org/pdf/2002.08155.pdf). CodeBERT is a pre-trained model for programming language, which is a multi-programming-lingual model pre-trained on NL-PL pairs in 6 programming languages (Python, Java, JavaScript, PHP, Ruby, Go). \n\n### Dependency\n\n- pip install torch\n- pip install transformers\n\n### Quick Tour\nWe use huggingface/transformers framework to train the model. You can use our model like the pre-trained Roberta base. Now, We give an example on how to load the model.\n```python\nimport torch\nfrom transformers import RobertaTokenizer, RobertaConfig, RobertaModel\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ntokenizer = RobertaTokenizer.from_pretrained(\"microsoft/codebert-base\")\nmodel = RobertaModel.from_pretrained(\"microsoft/codebert-base\")\nmodel.to(device)\n```\n\n### NL-PL Embeddings\n\nHere, we give an example to obtain embedding from CodeBERT.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModel\n>>> import torch\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/codebert-base\")\n>>> model = AutoModel.from_pretrained(\"microsoft/codebert-base\")\n>>> nl_tokens=tokenizer.tokenize(\"return maximum value\")\n['return', 'Ġmaximum', 'Ġvalue']\n>>> code_tokens=tokenizer.tokenize(\"def max(a,b): if a>b: return a else return b\")\n['def', 'Ġmax', '(', 'a', ',', 'b', '):', 'Ġif', 'Ġa', '>', 'b', ':', 'Ġreturn', 'Ġa', 'Ġelse', 'Ġreturn', 'Ġb']\n>>> tokens=[tokenizer.cls_token]+nl_tokens+[tokenizer.sep_token]+code_tokens+[tokenizer.eos_token]\n['<s>', 'return', 'Ġmaximum', 'Ġvalue', '</s>', 'def', 'Ġmax', '(', 'a', ',', 'b', '):', 'Ġif', 'Ġa', '>', 'b', ':', 'Ġreturn', 'Ġa', 'Ġelse', 'Ġreturn', 'Ġb', '</s>']\n>>> tokens_ids=tokenizer.convert_tokens_to_ids(tokens)\n[0, 30921, 4532, 923, 2, 9232, 19220, 1640, 102, 6, 428, 3256, 114, 10, 15698, 428, 35, 671, 10, 1493, 671, 741, 2]\n>>> context_embeddings=model(torch.tensor(tokens_ids)[None,:])[0]\ntorch.Size([1, 23, 768])\ntensor([[-0.1423,  0.3766,  0.0443,  ..., -0.2513, -0.3099,  0.3183],\n        [-0.5739,  0.1333,  0.2314,  ..., -0.1240, -0.1219,  0.2033],\n        [-0.1579,  0.1335,  0.0291,  ...,  0.2340, -0.8801,  0.6216],\n        ...,\n        [-0.4042,  0.2284,  0.5241,  ..., -0.2046, -0.2419,  0.7031],\n        [-0.3894,  0.4603,  0.4797,  ..., -0.3335, -0.6049,  0.4730],\n        [-0.1433,  0.3785,  0.0450,  ..., -0.2527, -0.3121,  0.3207]],\n       grad_fn=<SelectBackward>)\n```\n\n\n### Probing\n\nAs stated in the paper, CodeBERT is not suitable for mask prediction task, while CodeBERT (MLM) is suitable for mask prediction task.\n\n\nWe give an example on how to use CodeBERT(MLM) for mask prediction task.\n```python\nfrom transformers import RobertaConfig, RobertaTokenizer, RobertaForMaskedLM, pipeline\n\nmodel = RobertaForMaskedLM.from_pretrained(\"microsoft/codebert-base-mlm\")\ntokenizer = RobertaTokenizer.from_pretrained(\"microsoft/codebert-base-mlm\")\n\nCODE = \"if (x is not None) <mask> (x>1)\"\nfill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)\n\noutputs = fill_mask(CODE)\nprint(outputs)\n\n```\nResults\n```python\n'and', 'or', 'if', 'then', 'AND'\n```\nThe detailed outputs are as follows:\n```python\n{'sequence': '<s> if (x is not None) and (x>1)</s>', 'score': 0.6049249172210693, 'token': 8}\n{'sequence': '<s> if (x is not None) or (x>1)</s>', 'score': 0.30680200457572937, 'token': 50}\n{'sequence': '<s> if (x is not None) if (x>1)</s>', 'score': 0.02133703976869583, 'token': 114}\n{'sequence': '<s> if (x is not None) then (x>1)</s>', 'score': 0.018607674166560173, 'token': 172}\n{'sequence': '<s> if (x is not None) A","github_created_at":"2020-06-17T07:37:08+00:00","created_at":"2026-07-11T23:46:10.623003+00:00","updated_at":"2026-08-05T18:01:13.035828+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"code-pretraining","name":"code pretraining"},{"slug":"transformers-framework","name":"transformers framework"}],"trust":{"provenance":{"is_fork":false,"github_id":272909064,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-05T18:01:12.219Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1123,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:46:12.636Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-05T18:01:12.698Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-05T18:01:12.698Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-08-05T18:01:12.698Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":{"notes":["Install torch and transformers via pip before using CodeBERT for embedding generation or other tasks","Ensure Python and Hugging Face's transformers framework are available, as they form the core execution environment for utilizing this model"],"min_ram_gb":null},"constraints":{"min_ram_gb":null},"when_to_use":["When you need to work on tasks involving both programming and natural language processing across six different programming languages: Python, Java, JavaScript, PHP, Ruby, Go","For embedding generation from a combination of natural language and code inputs, enhancing cross-language understanding"],"when_not_to_use":["Avoid for direct mask prediction tasks without MLM (Masked Language Model) fine-tuning as CodeBERT is not natively equipped for such tasks unlike its variant designed with MLM capabilities","Do not consider it if your project requires pre-training models on a wider variety of programming languages beyond the six supported by this model"],"source":"enrich:decision_facts","observed_at":"2026-07-17T04:05:48.879Z"},"constraint_facets":{"min_ram_gb":null},"decision_summary":[{"label":"Requirements","value":"Install torch and transformers via pip before using CodeBERT for embedding generation or other tasks; Ensure Python and Hugging Face's transformers framework are available, as they form the core execution environment for utilizing this model"},{"label":"Adopt for","value":"CodeBERT is an advanced pre-trained model for programming and natural language tasks in multiple languages like Python and Java."}]}}