{"data":{"slug":"qdrant-vector-db-benchmark","name":"vector-db-benchmark","tagline":"Framework for benchmarking vector search engines","github_url":"https://github.com/qdrant/vector-db-benchmark","owner":"qdrant","repo":"vector-db-benchmark","owner_avatar_url":"https://avatars.githubusercontent.com/u/73504361?v=4","primary_language":"Python","stars":368,"forks":153,"topics":["benchmark","vector-database","vector-search","vector-search-engine"],"archived":false,"github_pushed_at":"2026-08-21T14:35:52+00:00","maintenance_label":"Very active","stars_delta_30d":0,"url":"https://www.graphcanon.com/tools/qdrant-vector-db-benchmark","markdown_url":"https://www.graphcanon.com/tools/qdrant-vector-db-benchmark.md","api_url":"https://www.graphcanon.com/api/graphcanon/tools/qdrant-vector-db-benchmark","graph_url":"https://www.graphcanon.com/api/graphcanon/graph?tool=qdrant-vector-db-benchmark","description":"Framework for benchmarking vector search engines","homepage_url":"https://qdrant.tech/benchmarks/","license":"Apache-2.0","open_issues":35,"watchers":5,"ai_summary":"Provides tools and benchmarks to evaluate performance of various vector databases used in applications such as recommendation systems and semantic search.","readme_excerpt":"# vector-db-benchmark\n\n\n\n> [View results](https://qdrant.tech/benchmarks/)\n\nThere are various vector search engines available, and each of them may offer\na different set of features and efficiency. But how do we measure the\nperformance? There is no clear definition and in a specific case you\nmay worry about a specific thing, while not paying much attention to other aspects. This\nproject is a general framework for benchmarking different engines under the\nsame hardware constraints, so you can choose what works best for you.\n\nRunning any benchmark requires choosing an engine, a dataset and defining the\nscenario against which it should be tested. A specific scenario may assume\nrunning the server in a single or distributed mode, a different client\nimplementation and the number of client instances.\n\n## How to run a benchmark?\n\nBenchmarks are implemented in server-client mode, meaning that the server is\nrunning in a single machine, and the client is running on another.\n\n### Run the server\n\nAll engines are served using docker compose. The configuration is in the [servers](./engine/servers/).\n\nTo launch the server instance, run the following command:\n\n```bash\ncd ./engine/servers/<engine-configuration-name>\ndocker compose up\n```\n\nContainers are expected to expose all necessary ports, so the client can connect to them.\n\n### Run the client\n\nInstall dependencies:\n\n```bash\npip install poetry\npoetry install\n```\n\nRun the benchmark:\n\n```bash\n$ poetry shell\n$ python run.py --help\n\nUsage: run.py [OPTIONS]\n\n  Examples:\n\n  python3 run.py --engines \"qdrant-rps-m-*-ef-*\" --datasets \"dbpedia-openai-100K-1536-angular\" # Qdrant RPS mode\n\n  python3 run.py --engines \"*-m-*-ef-*\" --datasets \"glove-*\" # All engines and their configs for glove datasets\n\nOptions:\n  --engines TEXT                  [default: *]\n  --datasets TEXT                 [default: *]\n  --host TEXT                     [default: localhost]\n  --skip-upload / --no-skip-upload\n                                  [default: no-skip-upload]\n  --install-completion            Install completion for the current shell.\n  --show-completion               Show completion for the current shell, to\n                                  copy it or customize the installation.\n  --help                          Show this message and exit.\n```\n\nCommand allows you to specify wildcards for engines and datasets.\nResults of the benchmarks are stored in the `./results/` directory.\n\n## How to update benchmark parameters?\n\nEach engine has a configuration file, which is used to define the parameters for the benchmark.\nConfiguration files are located in the [configuration](./experiments/configurations/) directory.\n\nEach step in the benchmark process is using a dedicated configuration's path:\n\n* `connection_params` - passed to the client during the connection phase.\n* `collection_params` - parameters, used to create the collection, indexing parameters are usually defined here.\n* `upload_params` - parameters, used to upload the data to the server.\n* `search_params` - passed to the client during the search phase. Framework allows multiple search configurations for the same experiment run.\n\nExact values of the parameters are individual for each engine.\n\n## How to register a dataset?\n\nDatasets are configured in the [datasets/datasets.json](./datasets/datasets.json) file.\nFramework will automatically download the dataset and store it in the [datasets](./datasets/) directory.\n\n## How to implement a new engine?\n\nThere are a few base classes that you can use to implement a new engine.\n\n* `BaseConfigurator` - defines methods to create collections, setup indexing parameters.\n* `BaseUploader` - defines methods to upload the data to the server.\n* `BaseSearcher` - defines methods to search the data.\n\nSee the examples in the [clients](./engine/clients) directory.\n\nOnce all the necessary classes are implemented, you can register the engine in the [ClientFactory](./engine/clients/client_factory.py).","github_created_at":"2022-07-12T16:05:59+00:00","created_at":"2026-07-11T11:35:52.78278+00:00","updated_at":"2026-08-23T12:01:50.418718+00:00","categories":[{"slug":"vector-databases","name":"Vector Databases","url":"https://www.graphcanon.com/categories/vector-databases","markdown_url":"https://www.graphcanon.com/categories/vector-databases.md","api_url":"https://www.graphcanon.com/api/graphcanon/categories/vector-databases"}],"tags":[{"slug":"benchmark","name":"benchmark"},{"slug":"vector-database","name":"vector-database"},{"slug":"vector-search","name":"vector-search"},{"slug":"vector-search-engine","name":"vector-search-engine"}],"trust":{"provenance":{"is_fork":false,"github_id":513208282,"owner_type":"Organization","methodology":"github_public_v1","parent_repo":null,"near_duplicate_slugs":[]},"computed_at":"2026-08-23T12:01:49.619Z","maintenance":{"label":"Very active","score":96,"methodology":"github_public_v1","releases_90d":0,"days_since_push":1,"last_release_at":null,"stars_delta_30d":0,"open_issues_delta_30d":-10},"security_summary":{"status":"no_lockfile","scanner":null,"low_count":0,"high_count":0,"last_scan_at":"2026-07-11T11:35:54.263Z","medium_count":0,"scan_profile":"none","critical_count":0}},"capability_facts":{"scan":{"source":"repo_scan","observed_at":"2026-08-23T12:01:50.080Z"},"deploy":{"source":"dockerfile:Dockerfile","self_host":true,"observed_at":"2026-08-23T12:01:50.080Z","managed_saas":false},"languages":{"value":["python"],"source":"github.language+pyproject.toml","observed_at":"2026-08-23T12:01:50.080Z"},"has_docker":{"value":true,"source":"dockerfile:Dockerfile","observed_at":"2026-08-23T12:01:50.080Z"},"license_spdx":{"value":"Apache-2.0","source":"github.license","observed_at":"2026-08-23T12:01:50.080Z"}},"decision_facts":{"hosting":null,"pricing":null,"requirements":null,"constraints":null,"when_to_use":["Use this tool when you need precisely measured performance metrics of vector databases, especially in environments where decision-making is driven by nuanced data comparisons and analysis.","Consider leveraging it if your project involves developing or enhancing a recommendation system or another application requiring accurate semantic search capabilities."],"when_not_to_use":["Avoid this tool if you are looking to benchmark non-vector database types, as its focus specifically lies on vector databases used in specialized scenarios like the ones mentioned.","Do not use vector-db-benchmark when your project does not require deep analysis or comparison of vector search performance, as it might add unnecessary complexity."],"source":"enrich:decision_facts","observed_at":"2026-07-14T19:42:47.729Z"},"constraint_facets":null,"decision_summary":[{"label":"Adopt for","value":"vector-db-benchmark is a Python-based framework that focuses on benchmarking vector search engines critical for applications ranging from recommendation systems to semantic search."}]}}