{"data":{"slug":"microsoft-codebert","name":"CodeBERT","tagline":"CodeBERT","github_url":"https://github.com/microsoft/CodeBERT","owner":"microsoft","repo":"CodeBERT","owner_avatar_url":"https://avatars.githubusercontent.com/u/6154722?v=4","primary_language":"Python","stars":2785,"forks":498,"topics":[],"archived":false,"github_pushed_at":"2023-07-09T12:26:30+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/microsoft-codebert","markdown_url":"https://www.graphcanon.com/tools/microsoft-codebert.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/microsoft-codebert","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=microsoft-codebert","description":"CodeBERT","homepage_url":null,"license":"MIT","open_issues":86,"watchers":37,"ai_summary":null,"readme_excerpt":"# Code Pretraining Models\n\nThis repo contains code pretraining models in the CodeBERT series from Microsoft, including six models as of June 2023.\n- CodeBERT (EMNLP 2020)\n- GraphCodeBERT (ICLR 2021)\n- UniXcoder (ACL 2022)\n- CodeReviewer (ESEC/FSE 2022)\n- CodeExecutor (ACL 2023)\n- LongCoder (ICML 2023)\n\n# CodeBERT\n\nThis repo provides the code for reproducing the experiments in [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://arxiv.org/pdf/2002.08155.pdf). CodeBERT is a pre-trained model for programming language, which is a multi-programming-lingual model pre-trained on NL-PL pairs in 6 programming languages (Python, Java, JavaScript, PHP, Ruby, Go). \n\n### Dependency\n\n- pip install torch\n- pip install transformers\n\n### Quick Tour\nWe use huggingface/transformers framework to train the model. You can use our model like the pre-trained Roberta base. Now, We give an example on how to load the model.\n```python\nimport torch\nfrom transformers import RobertaTokenizer, RobertaConfig, RobertaModel\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ntokenizer = RobertaTokenizer.from_pretrained(\"microsoft/codebert-base\")\nmodel = RobertaModel.from_pretrained(\"microsoft/codebert-base\")\nmodel.to(device)\n```\n\n### NL-PL Embeddings\n\nHere, we give an example to obtain embedding from CodeBERT.\n\n```python\n>>> from transformers import AutoTokenizer, AutoModel\n>>> import torch\n>>> tokenizer = AutoTokenizer.from_pretrained(\"microsoft/codebert-base\")\n>>> model = AutoModel.from_pretrained(\"microsoft/codebert-base\")\n>>> nl_tokens=tokenizer.tokenize(\"return maximum value\")\n['return', 'Ġmaximum', 'Ġvalue']\n>>> code_tokens=tokenizer.tokenize(\"def max(a,b): if a>b: return a else return b\")\n['def', 'Ġmax', '(', 'a', ',', 'b', '):', 'Ġif', 'Ġa', '>', 'b', ':', 'Ġreturn', 'Ġa', 'Ġelse', 'Ġreturn', 'Ġb']\n>>> tokens=[tokenizer.cls_token]+nl_tokens+[tokenizer.sep_token]+code_tokens+[tokenizer.eos_token]\n['<s>', 'return', 'Ġmaximum', 'Ġvalue', '</s>', 'def', 'Ġmax', '(', 'a', ',', 'b', '):', 'Ġif', 'Ġa', '>', 'b', ':', 'Ġreturn', 'Ġa', 'Ġelse', 'Ġreturn', 'Ġb', '</s>']\n>>> tokens_ids=tokenizer.convert_tokens_to_ids(tokens)\n[0, 30921, 4532, 923, 2, 9232, 19220, 1640, 102, 6, 428, 3256, 114, 10, 15698, 428, 35, 671, 10, 1493, 671, 741, 2]\n>>> context_embeddings=model(torch.tensor(tokens_ids)[None,:])[0]\ntorch.Size([1, 23, 768])\ntensor([[-0.1423,  0.3766,  0.0443,  ..., -0.2513, -0.3099,  0.3183],\n        [-0.5739,  0.1333,  0.2314,  ..., -0.1240, -0.1219,  0.2033],\n        [-0.1579,  0.1335,  0.0291,  ...,  0.2340, -0.8801,  0.6216],\n        ...,\n        [-0.4042,  0.2284,  0.5241,  ..., -0.2046, -0.2419,  0.7031],\n        [-0.3894,  0.4603,  0.4797,  ..., -0.3335, -0.6049,  0.4730],\n        [-0.1433,  0.3785,  0.0450,  ..., -0.2527, -0.3121,  0.3207]],\n       grad_fn=<SelectBackward>)\n```\n\n\n### Probing\n\nAs stated in the paper, CodeBERT is not suitable for mask prediction task, while CodeBERT (MLM) is suitable for mask prediction task.\n\n\nWe give an example on how to use CodeBERT(MLM) for mask prediction task.\n```python\nfrom transformers import RobertaConfig, RobertaTokenizer, RobertaForMaskedLM, pipeline\n\nmodel = RobertaForMaskedLM.from_pretrained(\"microsoft/codebert-base-mlm\")\ntokenizer = RobertaTokenizer.from_pretrained(\"microsoft/codebert-base-mlm\")\n\nCODE = \"if (x is not None) <mask> (x>1)\"\nfill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)\n\noutputs = fill_mask(CODE)\nprint(outputs)\n\n```\nResults\n```python\n'and', 'or', 'if', 'then', 'AND'\n```\nThe detailed outputs are as follows:\n```python\n{'sequence': '<s> if (x is not None) and (x>1)</s>', 'score': 0.6049249172210693, 'token': 8}\n{'sequence': '<s> if (x is not None) or (x>1)</s>', 'score': 0.30680200457572937, 'token': 50}\n{'sequence': '<s> if (x is not None) if (x>1)</s>', 'score': 0.02133703976869583, 'token': 114}\n{'sequence': '<s> if (x is not None) then (x>1)</s>', 'score': 0.018607674166560173, 'token': 172}\n{'sequence': '<s> if (x is not None) A","github_created_at":"2020-06-17T07:37:08+00:00","created_at":"2026-07-11T23:46:10.623003+00:00","updated_at":"2026-07-11T23:46:17.248312+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"python","name":"python"}],"trust":{"provenance":{"is_fork":false,"github_id":272909064,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:46:12.072Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1098,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:46:12.636Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-11T23:46:11.777Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-11T23:46:11.777Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-07-11T23:46:11.777Z"}}}}