{"data":{"slug":"trustedllm-regavae","name":"RegaVAE","tagline":"A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling","github_url":"https://github.com/TrustedLLM/RegaVAE","owner":"TrustedLLM","repo":"RegaVAE","owner_avatar_url":"https://avatars.githubusercontent.com/u/131944026?v=4","primary_language":"Python","stars":15,"forks":1,"topics":[],"archived":false,"github_pushed_at":"2023-12-05T20:52:31+00:00","maintenance_label":"Dormant","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/trustedllm-regavae","markdown_url":"https://www.graphcanon.com/tools/trustedllm-regavae.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/trustedllm-regavae","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=trustedllm-regavae","description":"A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling","homepage_url":null,"license":null,"open_issues":0,"watchers":1,"ai_summary":"RegaVAE is a model that combines a Variational Auto-Encoder with a retrieval mechanism to enhance language modeling capabilities. It uses a Gaussian mixture distribution to improve the latent space representation.","readme_excerpt":"# RegaVAE\nThis is the official repo for our [paper](https://arxiv.org/abs/2310.10567): \n> RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling\n\n## Model Architecture\n\nArchitecture of RegaVAE. Based on the training data, we first train a VAE to construct a compact latent space, which ensures that the latent variable z contains both current and future information (see § 3.1 of the paper). We then build a retrieval database and then aggregate the retrieved information into the generator (see § 3.2 of the paper). VAE Encoder and Decoder parameters are the same in all steps. In order to ensure fairness, the Corpus data and the Source data in the training set are the same. $G$ represents the Gaussian mixture distribution, and $π$ is the corresponding parameter.\n\n## Datasets\nDownload three dataset from this [link](https://drive.google.com/drive/folders/1mcn6nqLDVvrGatKHbdbtDSj9PQI5Eu8S?usp=sharing). Unzip them and put them under the data directory.\n\n## Step1\nFirstly,\n```\ncd Step1\n```\n### Training\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nFor WP dataset,\n```\npython main.py --train_source_path ../data/writingPrompts/train.wp_source \\\n--train_target_path ../data/writingPrompts/train.wp_target \\\n--valid_source_path ../data/writingPrompts/valid.wp_source \\\n--valid_target_path ../data/writingPrompts/valid.wp_target \\\n--dataset_type wp \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nThe above are only the best adjusted hyperparameters. You can get a better Step1 model by passing other parameters. The model we trained is available at this [link](https://drive.google.com/drive/folders/1HmTqQmHSmP_VZUDV9ADM6QEHwE3SazDi?usp=sharing).\n\n## Step2\nFirstly,\n```\ncd Step2\n```\nStep2 here corresponds to Step2 and Step3 in the figure. Before training, please rename the model trained in Step 1 to model_epoch_-1.pth and add it to the model generation path. In addition, please download the file in this [link](https://drive.google.com/file/d/1JYqrDwsumzDsUHgwdKopVBUuAkSgILNu/view?usp=drive_link) to the Step2 folder.\n\n### Training\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing\n```\n\n### Test\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing \\\n--eval \\\n--eval_metrics\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing \\\n--eval \\\n--eval_metrics\n```\n\n### Generation\nFor Yelp dataset,\n```\npython main.py --generation \\\n--test_file ../data/yelp/yelp.test.txt \\\n --load_epoch -1 \\\n--top_k 50 \\\n--top_p 0.9\n```","github_created_at":"2023-10-15T11:02:05+00:00","created_at":"2026-07-11T23:06:13.291438+00:00","updated_at":"2026-09-20T04:28:58.878765+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"gaussian-mixture-model","name":"gaussian mixture model"},{"slug":"hyperparameters","name":"hyperparameters"},{"slug":"language-modeling","name":"language modeling"},{"slug":"latent-space","name":"latent space"},{"slug":"retrieval-augmented","name":"retrieval-augmented"},{"slug":"training-data","name":"training data"},{"slug":"variational-auto-encoder","name":"variational auto-encoder"},{"slug":"yelp","name":"yelp"}],"trust":{"provenance":{"is_fork":false,"github_id":705205809,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-09-18T08:11:50.304Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1017,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-09-18T08:11:52.924Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-09-18T08:11:49.074Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-09-18T08:11:49.074Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When you need to enhance language modeling capabilities by integrating retrieval mechanisms with a Variational Auto-Encoder.","If your project requires a model that can effectively represent both current and future information in the latent space.","For tasks that benefit from a Gaussian mixture distribution in the latent space to capture complex data structures."],"when_not_to_use":["If your project does not require the integration of retrieval mechanisms with a Variational Auto-Encoder.","When the datasets you are working with do not align with the specific datasets used in RegaVAE's training (Yelp, Yahoo, WP).","If your project does not benefit from the specific architecture of RegaVAE, which includes a retrieval database and Gaussian mixture distribution in the latent space."],"source":"enrich:decision_facts","observed_at":"2026-09-18T09:25:04.373Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"RegaVAE is a model that integrates a Variational Auto-Encoder with a retrieval mechanism to enhance language modeling, using a Gaussian mixture distribution to improve latent space representation."}]}}