{"data":{"slug":"trustedllm-regavae","name":"RegaVAE","tagline":"A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling","github_url":"https://github.com/TrustedLLM/RegaVAE","owner":"TrustedLLM","repo":"RegaVAE","owner_avatar_url":"https://avatars.githubusercontent.com/u/131944026?v=4","primary_language":"Python","stars":15,"forks":1,"topics":[],"archived":false,"github_pushed_at":"2023-12-05T20:52:31+00:00","maintenance_label":"Dormant","url":"https://www.graphcanon.com/tools/trustedllm-regavae","markdown_url":"https://www.graphcanon.com/tools/trustedllm-regavae.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/trustedllm-regavae","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=trustedllm-regavae","description":"A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling","homepage_url":null,"license":null,"open_issues":0,"watchers":1,"ai_summary":"RegaVAE is a retrieval-augmented framework that combines VAE with a retrieval mechanism to enhance language modeling capabilities by incorporating both current and future information in the latent space.","readme_excerpt":"# RegaVAE\nThis is the official repo for our [paper](https://arxiv.org/abs/2310.10567): \n> RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling\n\n## Model Architecture\n\nArchitecture of RegaVAE. Based on the training data, we first train a VAE to construct a compact latent space, which ensures that the latent variable z contains both current and future information (see § 3.1 of the paper). We then build a retrieval database and then aggregate the retrieved information into the generator (see § 3.2 of the paper). VAE Encoder and Decoder parameters are the same in all steps. In order to ensure fairness, the Corpus data and the Source data in the training set are the same. $G$ represents the Gaussian mixture distribution, and $π$ is the corresponding parameter.\n\n## Datasets\nDownload three dataset from this [link](https://drive.google.com/drive/folders/1mcn6nqLDVvrGatKHbdbtDSj9PQI5Eu8S?usp=sharing). Unzip them and put them under the data directory.\n\n## Step1\nFirstly,\n```\ncd Step1\n```\n### Training\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nFor WP dataset,\n```\npython main.py --train_source_path ../data/writingPrompts/train.wp_source \\\n--train_target_path ../data/writingPrompts/train.wp_target \\\n--valid_source_path ../data/writingPrompts/valid.wp_source \\\n--valid_target_path ../data/writingPrompts/valid.wp_target \\\n--dataset_type wp \\\n--per_gpu_train_batch_size 4 \\\n--cycle_annealing\n```\nThe above are only the best adjusted hyperparameters. You can get a better Step1 model by passing other parameters. The model we trained is available at this [link](https://drive.google.com/drive/folders/1HmTqQmHSmP_VZUDV9ADM6QEHwE3SazDi?usp=sharing).\n\n## Step2\nFirstly,\n```\ncd Step2\n```\nStep2 here corresponds to Step2 and Step3 in the figure. Before training, please rename the model trained in Step 1 to model_epoch_-1.pth and add it to the model generation path. In addition, please download the file in this [link](https://drive.google.com/file/d/1JYqrDwsumzDsUHgwdKopVBUuAkSgILNu/view?usp=drive_link) to the Step2 folder.\n\n### Training\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing\n```\n\n### Test\nFor Yelp dataset,\n```\npython main.py --train_file ../data/yelp/yelp.train.txt \\\n--valid_file ../data/yelp/yelp.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing \\\n--eval \\\n--eval_metrics\n```\nFor Yahoo dataset,\n```\npython main.py --train_file ../data/yahoo/yahoo.train.txt \\\n--valid_file ../data/yahoo/yahoo.valid.txt \\\n--per_gpu_train_batch_size 4 \\\n--load_epoch -1 \\\n--cycle_annealing \\\n--eval \\\n--eval_metrics\n```\n\n### Generation\nFor Yelp dataset,\n```\npython main.py --generation \\\n--test_file ../data/yelp/yelp.test.txt \\\n --load_epoch -1 \\\n--top_k 50 \\\n--top_p 0.9\n```","github_created_at":"2023-10-15T11:02:05+00:00","created_at":"2026-07-11T23:06:13.291438+00:00","updated_at":"2026-07-12T07:48:21.373568+00:00","categories":[{"slug":"model-training","name":"Model Training","url":"https://www.graphcanon.com/categories/model-training","markdown_url":"https://www.graphcanon.com/categories/model-training.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/model-training"}],"tags":[{"slug":"language-modeling","name":"language modeling"},{"slug":"variational-auto-encoder","name":"variational auto-encoder"},{"slug":"retrieval-augmentation","name":"retrieval-augmentation"}],"trust":{"provenance":{"is_fork":false,"github_id":705205809,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-07-11T23:06:19.722Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":949,"last_release_at":null},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T23:06:20.313Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-07-12T07:48:01.138Z"},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-07-12T07:48:01.138Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["When seeking to leverage both historical and future information in the latent space for improved language generation.","For tasks where incorporating external retrieved data can provide contextual benefits during the training of your model."],"when_not_to_use":["If traditional Variational Auto-Encoders (VAEs) without retrieval components suffice for your needs, as RegaVAE introduces complexity that may not be necessary in simpler scenarios.","When dataset requirements exceed available resources or when datasets with specific formatting are hard to obtain and adapt."],"source":"enrich:decision_facts","observed_at":"2026-07-12T07:48:21.170Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"RegaVAE brings a unique approach by integrating retrieval mechanisms with Gaussian Mixture VAEs to enhance language modeling."}]}}