{"data":{"slug":"lonepatient-bert-multi-label-text-classification","name":"Bert-Multi-Label-Text-Classification","tagline":"PyTorch implementation of a pretrained BERT model for multi-label text classification","github_url":"https://github.com/lonePatient/Bert-Multi-Label-Text-Classification","owner":"lonePatient","repo":"Bert-Multi-Label-Text-Classification","owner_avatar_url":"https://avatars.githubusercontent.com/u/35169745?v=4","primary_language":"Python","stars":921,"forks":207,"topics":["albert","bert","fine-tuning","multi-label-classification","nlp","pytorch","pytorch-implmention","text-classification","transformers","xlnet"],"archived":false,"github_pushed_at":"2023-04-18T10:47:17+00:00","maintenance_label":"Dormant","stars_delta_30d":-2,"url":"https://www.graphcanon.com/tools/lonepatient-bert-multi-label-text-classification","markdown_url":"https://www.graphcanon.com/tools/lonepatient-bert-multi-label-text-classification.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/lonepatient-bert-multi-label-text-classification","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=lonepatient-bert-multi-label-text-classification","description":"This repo contains a PyTorch implementation of a pretrained BERT model for multi-label text classification.","homepage_url":null,"license":"MIT","open_issues":41,"watchers":19,"ai_summary":"This repository provides a PyTorch-based approach to fine-tuning pretrained models like BERT for the purpose of classifying text into multiple labels. It is useful in natural language processing tasks where texts can belong to more than one category.","readme_excerpt":"## Bert multi-label text classification by PyTorch\n\nThis repo contains a PyTorch implementation of the pretrained BERT and XLNET model for multi-label text classification.\n\n###  Structure of the code\n\nAt the root of the project, you will see:\n\n```text\n├── pybert\n|  └── callback\n|  |  └── lrscheduler.py　　\n|  |  └── trainingmonitor.py　\n|  |  └── ...\n|  └── config\n|  |  └── basic_config.py #a configuration file for storing model parameters\n|  └── dataset　　　\n|  └── io　　　　\n|  |  └── dataset.py　　\n|  |  └── data_transformer.py　　\n|  └── model\n|  |  └── nn　\n|  |  └── pretrain　\n|  └── output #save the ouput of model\n|  └── preprocessing #text preprocessing \n|  └── train #used for training a model\n|  |  └── trainer.py \n|  |  └── ...\n|  └── common # a set of utility functions\n├── run_bert.py\n├── run_xlnet.py\n```\n### Dependencies\n\n- csv\n- tqdm\n- numpy\n- pickle\n- scikit-learn\n- PyTorch 1.1+\n- matplotlib\n- pandas\n- transformers=2.5.1\n\n### How to use the code\n\nyou need download pretrained bert model and xlnet model.\n\n<div class=\"note info\"><p> BERT:  bert-base-uncased</p></div>\n\n<div class=\"note info\"><p> XLNET:  xlnet-base-cased</p></div>\n\n1. Download the Bert pretrained model from [s3](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin) \n2. Download the Bert config file from [s3](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json) \n3. Download the Bert vocab file from [s3](https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt) \n4. Rename:\n\n    - `bert-base-uncased-pytorch_model.bin` to `pytorch_model.bin`\n    - `bert-base-uncased-config.json` to `config.json`\n    - `bert-base-uncased-vocab.txt` to `bert_vocab.txt`\n5. Place `model` ,`config` and `vocab` file into  the `/pybert/pretrain/bert/base-uncased` directory.\n6. `pip install pytorch-transformers` from [github](https://github.com/huggingface/pytorch-transformers).\n7. Download [kaggle data](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data) and place in `pybert/dataset`.\n    -  you can modify the `io.task_data.py` to adapt your data.\n8. Modify configuration information in `pybert/configs/basic_config.py`(the path of data,...).\n9. Run `python run_bert.py --do_data` to preprocess data.\n10. Run `python run_bert.py --do_train --save_best --do_lower_case` to fine tuning bert model.\n11. Run `run_bert.py --do_test --do_lower_case` to predict new data.\n\n### training \n\n```text\n[training] 8511/8511 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] -0.8s/step- loss: 0.0640\ntraining result:\n[2019-01-14 04:01:05]: bert-multi-label trainer.py[line:176] INFO  \nEpoch: 2 - loss: 0.0338 - val_loss: 0.0373 - val_auc: 0.9922\n```\n### training figure\n\n\n\n### result\n\n```python\n---- train report every label -----\nLabel: toxic - auc: 0.9903\nLabel: severe_toxic - auc: 0.9913\nLabel: obscene - auc: 0.9951\nLabel: threat - auc: 0.9898\nLabel: insult - auc: 0.9911\nLabel: identity_hate - auc: 0.9910\n---- valid report every label -----\nLabel: toxic - auc: 0.9892\nLabel: severe_toxic - auc: 0.9911\nLabel: obscene - auc: 0.9945\nLabel: threat - auc: 0.9955\nLabel: insult - auc: 0.9903\nLabel: identity_hate - auc: 0.9927\n```\n\n## Tips\n\n- When converting the tensorflow checkpoint into the pytorch, it's expected to choice the \"bert_model.ckpt\", instead of \"bert_model.ckpt.index\", as the input file. Otherwise, you will see that the model can learn nothing and give almost same random outputs for any inputs. This means, in fact, you have not loaded the true ckpt for your model\n- When using multiple GPUs, the non-tensor calculations, such as accuracy and f1_score, are not supported by DataParallel instance\n- As recommanded by Jocob in his paper <url>https://arxiv.org/pdf/1810.04805.pdf<url/>, in fine-tuning tasks, the hyperparameters are expected to set as following: **Batch_size**: 16 or 32, **learning_rate**: 5e-5 or 2e-5 or 3e-5, **num_train_epoch**: 3 or 4\n- The pretrained model has a limit for the sentence of input that its length","github_created_at":"2019-02-10T15:19:42+00:00","created_at":"2026-07-11T11:39:36.803224+00:00","updated_at":"2026-09-20T06:30:13.773584+00:00","categories":[{"slug":"evaluation-observability","name":"Evaluation & Observability","url":"https://www.graphcanon.com/categories/evaluation-observability","markdown_url":"https://www.graphcanon.com/categories/evaluation-observability.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/evaluation-observability"},{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"albert","name":"albert"},{"slug":"bert","name":"bert"},{"slug":"fine-tuning","name":"fine-tuning"},{"slug":"multi-label-classification","name":"multi-label-classification"},{"slug":"nlp","name":"nlp"},{"slug":"pytorch","name":"pytorch"},{"slug":"transformers","name":"transformers"},{"slug":"xlnet","name":"xlnet"}],"trust":{"provenance":{"is_fork":false,"github_id":169991845,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-19T18:25:40.141Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1250,"last_release_at":null,"stars_delta_30d":-2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:39:38.026Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-19T18:25:41.151Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-19T18:25:41.151Z"},"license_spdx":{"value":"MIT","source":"github.license","observed_at":"2026-09-19T18:25:41.151Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When PyTorch-based fine-tuning of BERT for multi-label text classification tasks is required.","If your project already utilizes Python and the MIT License aligns with your licensing requirements."],"when_not_to_use":["Avoid if TensorFlow is preferred over PyTorch for your workloads.","Not ideal if your text classification task only requires single-label outcomes rather than multi-label ones."],"source":"enrich:decision_facts","observed_at":"2026-07-12T11:53:22.520Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"Specific to Bert-Multi-Label-Text-Classification"}]}}