{"data":{"slug":"dgarnitz-vectorflow","name":"vectorflow","tagline":"High volume vector embedding pipeline with support for multiple vector databases","github_url":"https://github.com/dgarnitz/vectorflow","owner":"dgarnitz","repo":"vectorflow","owner_avatar_url":"https://avatars.githubusercontent.com/u/126617947?v=4","primary_language":"Python","stars":704,"forks":51,"topics":["ai","data-engineering","embeddings","machine-learning","nlp","vectors"],"archived":false,"github_pushed_at":"2024-05-16T20:31:48+00:00","maintenance_label":"Dormant","stars_delta_30d":2,"url":"https://www.graphcanon.com/tools/dgarnitz-vectorflow","markdown_url":"https://www.graphcanon.com/tools/dgarnitz-vectorflow.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/dgarnitz-vectorflow","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=dgarnitz-vectorflow","description":"VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice. ","homepage_url":"https://www.getvectorflow.com/","license":"Apache-2.0","open_issues":15,"watchers":10,"ai_summary":"VectorFlow is a Python library designed to transform raw data into vector embeddings and store them in various vector databases.","readme_excerpt":"## Docker-Compose\n\nThe best way to run VectorFlow is via `docker compose`. If you are running this on Mac, please grant Docker permissions to read from your Documents folder [as instructed here](https://stackoverflow.com/questions/58482352/operation-not-permitted-from-docker-container-logged-as-root). If this fails, remove the `volume` section from the `docker-compose.yml`.\n\n---\n\n### 2) Run Docker-Compose\n\nMake sure you pull Rabbit MQ, Postgres, Min.io into your local docker repo. We also recommend running a vector DB in locally, so make sure to pull the image of the one you are using. Our `docker-compose` file will spin up qdrant by default and create two index/collections. If you plan to run Milvus or Weaviate, you will have to configure them on your own. \n\n```\ndocker pull rabbitmq\ndocker pull postgres\ndocker pull qdrant/qdrant | docker pull semitechnologies/weaviate\ndocker pull minio/minio\n```\n\nThen run:\n\n```\ndocker-compose build --no-cache\ndocker-compose up -d\n```\n\nNote that the `init` containers are running a script that sets up the database schema, vector DB and Min.io object store. These containers stop after the script completes. For qdrant, make sure to pull version 1.9.1 since that is the version the qdrant client python package is supposed to work with.\n\n---\n\n### Building the Individual Images with Docker Commands\nIf you want to use `docker build` and `docker run` to build and run individual images instead of `docker-compose` follow these steps:\n1. `cd src/`\n2. `docker build --file api/Dockerfile -t vectorflow_api:latest .` to build - don't forget the period at the end\n3. `docker run --network=vectorflow --name=vectorflow_api -d --env-file=../env_scripts/env_vars.env -p 8000:8000 vectorflow_api:latest` to run the api. you don't need the port argument to run the worker","github_created_at":"2023-07-25T16:13:07+00:00","created_at":"2026-07-11T11:32:40.337846+00:00","updated_at":"2026-08-23T00:00:51.513809+00:00","categories":[{"slug":"data-retrieval","name":"Data & Retrieval","url":"https://www.graphcanon.com/categories/data-retrieval","markdown_url":"https://www.graphcanon.com/categories/data-retrieval.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/data-retrieval"},{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"ai","name":"ai"},{"slug":"data-engineering","name":"data-engineering"},{"slug":"embeddings","name":"embeddings"},{"slug":"machine-learning","name":"machine-learning"},{"slug":"nlp","name":"nlp"},{"slug":"vectors","name":"vectors"}],"trust":{"provenance":{"is_fork":false,"github_id":670696350,"owner_type":"User","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-23T00:00:50.527Z","maintenance":{"label":"Dormant","score":18,"methodology":"github_public_v1","releases_90d":0,"days_since_push":828,"last_release_at":null,"stars_delta_30d":2,"open_issues_delta_30d":0},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:32:41.465Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-23T00:00:51.168Z"},"deploy":{"source":"dockerfile:docker-compose.yml","self_host":true,"observed_at":"2026-08-23T00:00:51.168Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language","observed_at":"2026-08-23T00:00:51.168Z"},"has_docker":{"value":true,"source":"dockerfile:docker-compose.yml","observed_at":"2026-08-23T00:00:51.168Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-23T00:00:51.168Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["- When your project requires handling large volumes of data that need to be transformed into vector embeddings efficiently.","- In scenarios where compatibility with various vector database systems is crucial for seamless integration."],"when_not_to_use":["- If your application only deals with small datasets and does not benefit from high-volume processing capabilities offered by VectorFlow.","- When the specific requirements of your project mandate using a single, particular vector database system as opposed to leveraging multiple options(VectorFlow provides)."],"source":"enrich:decision_facts","observed_at":"2026-07-12T17:17:41.396Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"VectorFlow is a Python library that supports high volume transformation of raw data into vector embeddings and storage in multiple vector databases."}]}}