Vectorstore

Qdrant-based vector storage for RAG. Embeddings, collections, and ingestion. Used for catalog and example-plan semantic retrieval when RAG is enabled.

        flowchart LR
    Q["User Query"] --> E["Embed (OpenAI)"]
    E --> S["Search Qdrant"]
    S --> T["Threshold Filter"]
    T --> P["Prune & Rank"]
    P --> R["Agent Context"]

    style Q fill:#2563eb,color:#fff
    style R fill:#059669,color:#fff
    

Overview

• Qdrant Store — Client wrapper. Collection management, point upsert, similarity search. Powers catalog_rag and example_plans_rag tools.

• Bootstrap — Initialize vectorstore at startup. Create collections, run migrations. Called when RAG is enabled.

Design Decisions

Why Qdrant for RAG?

Qdrant is a purpose-built vector database with native filtering, payload storage, and similarity search. For PathFinder’s use case (semantic search over ~2000 WDK searches and ~500 public strategies), Qdrant provides fast approximate nearest-neighbor search without the overhead of managing a general-purpose database extension (like pgvector).

Two-collection design

The vectorstore uses two collections: catalog (record types + searches) and example_plans (public strategies). Separating them allows independent ingestion cycles (catalog changes rarely; example plans grow over time) and different embedding strategies (catalog entries use search descriptions; plans use strategy summaries).

Incremental ingestion

The startup ingestion job checks existing point IDs before upserting, avoiding re-embedding unchanged data. This makes startup fast (typically < 5 seconds) while keeping the vectorstore up-to-date with catalog changes.

OpenAI embeddings

PathFinder uses text-embedding-3-small (1536 dimensions) for all embeddings. This model provides good quality at low cost and is compatible with Qdrant’s cosine similarity search.

Qdrant Store

Purpose: Qdrant client wrapper. Manages collections (e.g. catalog, example_plans), upserts points with embeddings, runs similarity search. Used for RAG retrieval when the agent explores the catalog or example plans.

Key methods: Collection operations, search, point upsert

veupath_chatbot.integrations.vectorstore.qdrant_store.stable_json_dumps(value)

Return type:

str

veupath_chatbot.integrations.vectorstore.qdrant_store.sha256_hex(text)

Return type:

str

veupath_chatbot.integrations.vectorstore.qdrant_store.context_hash(context)

Stable hash for (WDK-wire) contextParamValues.

Parameters:

context (JSONObject) – Context param dict from WDK wire format.

Returns:

SHA256 hex digest.

Return type:

str

veupath_chatbot.integrations.vectorstore.qdrant_store.point_uuid(key)

Deterministic UUID for a human-readable key.

Qdrant point IDs must be either an integer or UUID.

Parameters:

key (str) – Human-readable key.

Returns:

UUID string.

Return type:

str

class veupath_chatbot.integrations.vectorstore.qdrant_store.QdrantStore(url: str, api_key: str | None = None, timeout_seconds: float = 10.0)

Bases: object

url: str

api_key: str | None = None

timeout_seconds: float = 10.0

classmethod from_settings()

Return type:

QdrantStore

connect()

Yield the shared persistent AsyncQdrantClient.

The client is created lazily on first use and reused across all subsequent calls. It is NOT closed when the context manager exits; call close() during application shutdown instead.

Return type:

AsyncIterator[AsyncQdrantClient]

async close()

Close the shared client and release its connection pool.

Safe to call multiple times or when no client has been created.

async reset_collections(*names)

Delete collections if they exist (used before re-ingestion).

async ensure_collection(*, name, vector_size, distance='Cosine')

Create collection if missing; validate vector size if present.

async upsert(*, collection, points)

Upsert points.

Each point dict: {“id”: str|int, “vector”: list[float], “payload”: dict}

async get(*, collection, point_id)

Return type:

JSONObject | None

async search(*, collection, query_vector, limit=10, must=None, must_not=None)

Return type:

JSONArray

__init__(url, api_key=None, timeout_seconds=10.0)

async veupath_chatbot.integrations.vectorstore.qdrant_store.close_all_qdrant_stores()

Close every QdrantStore created via from_settings().

Called during application shutdown to release all Qdrant connection pools.

Bootstrap

Purpose: Vectorstore initialization at API startup. Creates collections if missing, runs migrations. Called when RAG is enabled and QDRANT_URL is set.

Key function: Startup entry point

Vectorstore startup/bootstrap helpers.

async veupath_chatbot.integrations.vectorstore.bootstrap.get_embedding_dim(model)

Return embedding dimension for model, using cache then OpenAI fallback.

Prefer this over calling embed_one() just to discover vector size.

Return type:

int

async veupath_chatbot.integrations.vectorstore.bootstrap.ensure_rag_collections()

Ensure core Qdrant collections exist.

This creates empty collections so that RAG lookups do not fail with 404 when ingestion has not been run yet.

Collections

Purpose: Qdrant collection definitions and schema. Defines the embedding dimensions, distance metrics, and payload indices for each collection (catalog, example_plans).

Qdrant collection name constants.

Keep these stable to avoid accidentally writing to multiple collections.

Dependent Vocab Cache

Purpose: Cache for dependent vocabulary lookups. Avoids repeated WDK calls for parameter vocabularies that depend on other parameter values.

async veupath_chatbot.integrations.vectorstore.dependent_vocab_cache.ensure_dependent_vocab_collection(store)

Create the dependent vocab cache collection if missing.

This collection is used for keyed lookup (site/rt/search/param/contextHash). We still store vectors to keep Qdrant schema consistent and allow optional similarity later.

async veupath_chatbot.integrations.vectorstore.dependent_vocab_cache.get_dependent_vocab_authoritative_cached(*, site_id, record_type, search_name, param_name, context_values, store=None)

Return authoritative dependent vocab, cached in Qdrant.

• Cache key is the WDK-wire encoded context values (json-string encoding for lists/dicts).

• On cache miss, calls WDK /refreshed-dependent-params (via existing client) and stores result.

Return type:

JSONObject

Ingestion

Purpose: Ingest data into the vectorstore. WDK catalog ingestion, public strategy ingestion, and shared utilities.

async veupath_chatbot.integrations.vectorstore.ingest.wdk_catalog.ingest_site(*, site_id, store, qdrant_client, embedder, concurrency, batch_size, skip_existing)

async veupath_chatbot.integrations.vectorstore.ingest.wdk_catalog.ingest_wdk_catalog(*, sites, reset=False, skip_existing=True, batch_size=64)

veupath_chatbot.integrations.vectorstore.ingest.wdk_catalog.main(argv=None)

async veupath_chatbot.integrations.vectorstore.ingest.public_strategies.ingest_site(*, site_id, store, embedder, llm_model, report_path, max_strategies, concurrency, skip_existing)

async veupath_chatbot.integrations.vectorstore.ingest.public_strategies.ingest_public_strategies(*, sites, reset=False, llm_model='gpt-4.1-nano', report_path=PosixPath('ingest_public_strategies_report.jsonl'), max_strategies_per_site=None, concurrency=None, skip_existing=True)

veupath_chatbot.integrations.vectorstore.ingest.public_strategies.main(argv=None)

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.backoff_delay_seconds(attempt)

Exponential backoff delay (capped).

Matches the ingest script’s prior behavior: min(8, 2 ** (attempt - 1)).

Parameters:

attempt (int) – Retry attempt number (1-based).

Returns:

Delay in seconds.

Return type:

int

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.truncate(s, *, max_chars)

Return type:

str

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.iter_compact_steps(step_tree)

Return type:

JSONArray

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.embedding_text_for_example(*, name, description, compact)

Return type:

str

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.simplify_strategy_details(details)

Return type:

JSONObject

veupath_chatbot.integrations.vectorstore.ingest.public_strategies_helpers.full_strategy_payload(details)

Return type:

JSONObject

veupath_chatbot.integrations.vectorstore.ingest.utils.parse_sites(value)

Parse a comma-separated list of site IDs or ‘all’ → None (meaning all).

Return type:

list[str] | None

async veupath_chatbot.integrations.vectorstore.ingest.utils.existing_point_ids(*, qdrant_client, collection, ids, chunk_size=256)

Return subset of ids that already exist in Qdrant.

If the collection doesn’t exist yet, returns an empty set.

Return type:

set[str]

async veupath_chatbot.integrations.vectorstore.ingest.utils.embed_and_upsert(*, store, embedder, collection, ids, texts, payloads)

Embed texts, pair with ids/payloads, and upsert to collection.

This is the shared core of both WDK and public-strategy indexing pipelines.