Strategy Services

Plan normalization, validation, serialization, and WDK snapshot handling. Bridges between the domain AST and the persistence/WDK layer.

Overview

• Plan Normalization — Coerce parameter values, fill defaults, resolve vocab terms for WDK compatibility. Called before save or push.

• Plan Validation — Validate against WDK constraints; structured errors.

• Serialization — Convert between domain AST and persistence format.

• WDK Snapshot — Build WDK-compatible step trees from the domain plan.

Plan Normalization

Purpose: Normalize plans for WDK compatibility. Coerce parameter values to the expected types, fill defaults, resolve vocabulary terms. Called before save or push to VEuPathDB.

Key function: canonicalize_plan_parameters()

Strategy plan normalization helpers.

Produce canonical JSON shapes for frontend consumption. Multi-pick becomes list[str], ranges become {min, max}, etc.

async veupath_chatbot.services.strategies.plan_normalize.canonicalize_plan_parameters(*, plan, site_id, load_search_details)

Canonicalize all search/transform node parameters using WDK specs.

load_search_details(record_type, search_or_transform_name, params) -> dict must return a WDK payload with expanded params (or raise).

Return type:

JSONObject

Plan Validation

Purpose: Validate plans against WDK constraints. Required parameters, valid search names, step structure. Returns structured validation errors with field paths.

Key function: validate_plan_or_raise()

Strategy plan validation helpers.

veupath_chatbot.services.strategies.plan_validation.validate_plan_or_raise(plan)

Parse and validate a strategy plan, raising typed ValidationError.

Parameters:

plan (JSONObject) – Plan or strategy dict.

Return type:

StrategyAST

WDK Conversion

Purpose: Pure WDK → AST conversion. Parses WDK strategy payloads into internal StrategyAST, extracts field values, and normalizes parameters.

WDK Wire Format and Parameter Coercion

WDK stores multi-pick parameter values as JSON-encoded strings (e.g. '["Plasmodium falciparum 3D7"]' rather than a native array). When strategies are synced from WDK via fetch_and_convert(), the ParameterNormalizer preserves this wire format in the stored plan.

The frontend step editor automatically coerces these JSON strings into native arrays when parameter specs load, so widgets (TreeBox, Select, etc.) can match values against their vocabulary options. This coercion runs once per editor mount via coerceParametersForSpecs in the useStepParameters hook.

WDK → AST conversion: parse WDK strategy payloads into internal AST.

Pure conversion functions (no I/O except _normalize_synced_parameters which fetches param specs from WDK).

Public API: - build_snapshot_from_wdk — full WDK strategy → (AST, steps_data, step_counts) - extract_wdk_is_saved — extract isSaved flag from WDK payload - parse_wdk_strategy_id — extract int ID from WDK list-strategies item - normalize_synced_parameters — enrich AST nodes with normalized param values

veupath_chatbot.services.strategies.wdk_conversion.extract_wdk_is_saved(payload)

Extract payload["isSaved"] with isinstance guard, defaults False.

Return type:

bool

veupath_chatbot.services.strategies.wdk_conversion.parse_wdk_strategy_id(item)

Extract integer WDK strategy ID from a list-strategies item.

WDK’s StrategyFormatter emits strategyId (JsonKeys.STRATEGY_ID) as a Java long (always an int in JSON).

Return type:

int | None

veupath_chatbot.services.strategies.wdk_conversion.extract_record_type(wdk_strategy)

Extract the record type (urlSegment) from a WDK strategy payload.

Return type:

str

veupath_chatbot.services.strategies.wdk_conversion.get_step_info(steps, step_id)

Look up a step object from the WDK steps dict.

Return type:

JSONObject

veupath_chatbot.services.strategies.wdk_conversion.extract_operator(parameters)

Return type:

str | None

veupath_chatbot.services.strategies.wdk_conversion.extract_estimated_size(step_info)

Extract the result count from a WDK step object.

Return type:

int | None

veupath_chatbot.services.strategies.wdk_conversion.build_node_from_wdk(step_tree, steps, record_type)

Recursively build a PlanStepNode tree from WDK stepTree + steps.

Return type:

PlanStepNode

veupath_chatbot.services.strategies.wdk_conversion.build_snapshot_from_wdk(wdk_strategy)

Convert a WDK strategy payload into an internal AST, steps list, and step counts.

The steps list is used only for parameter normalization enrichment — steps are derived from plan at read time and not persisted.

The step counts dict maps step IDs (strings) to their estimatedSize values from the WDK response, enabling zero-cost count display for WDK-linked strategies.

Return type:

tuple[StrategyAST, JSONArray, JSONObject]

async veupath_chatbot.services.strategies.wdk_conversion.normalize_synced_parameters(ast, steps_data, api)

Normalize parameters from WDK response using param specs.

Mutates AST nodes in place with normalized parameter values.

WDK Sync

Purpose: Fetch WDK strategies and sync into CQRS projections. Lazy detail fetching, isSaved sync, and projection upsert.

WDK sync: fetch WDK strategies and sync into CQRS projections.

Handles: - fetch_and_convert — fetch WDK strategy, convert to AST, normalize params - sync_to_projection — full sync flow: fetch + upsert into CQRS - upsert_projection — create-or-update a stream projection from WDK data - upsert_summary_projection — create-or-update from list summary data - plan_needs_detail_fetch — check if a projection needs WDK detail fetch - lazy_fetch_wdk_detail — lazy-load full WDK detail for summary-only projections - sync_is_saved_to_wdk — sync isSaved flag from projection to WDK

veupath_chatbot.services.strategies.wdk_sync.plan_needs_detail_fetch(projection)

Check if a WDK-linked projection needs its full detail fetched from WDK.

Returns True when the projection has a wdk_strategy_id but no plan data (i.e. it was synced with summary data only and the user is now opening it). Local strategies (no wdk_strategy_id) never need a WDK fetch.

Return type:

bool

async veupath_chatbot.services.strategies.wdk_sync.fetch_and_convert(api, wdk_id)

Fetch a WDK strategy and convert to internal AST.

Normalizes parameters best-effort (failures are logged and swallowed).

Returns:

Tuple of (StrategyAST, is_saved, step_counts). step_counts maps step IDs to estimatedSize values from the WDK response, enabling zero-cost count display.

Return type:

tuple[StrategyAST, bool, JSONObject]

async veupath_chatbot.services.strategies.wdk_sync.sync_to_projection(*, wdk_id, site_id, api, stream_repo, user_id)

Fetch a single WDK strategy and upsert into the CQRS layer.

Shared by open_strategy and sync_all_wdk_strategies.

Return type:

StreamProjection

async veupath_chatbot.services.strategies.wdk_sync.upsert_projection(*, stream_repo, user_id, site_id, wdk_id, name, plan, record_type, is_saved, step_count=0)

Upsert a WDK strategy into the CQRS layer (create or update stream projection).

Return type:

StreamProjection

async veupath_chatbot.services.strategies.wdk_sync.upsert_summary_projection(wdk_item, *, stream_repo, user_id, site_id)

Create or update a projection from WDK list summary data only.

Unlike sync_to_projection, this does NOT fetch the full strategy detail from WDK. It only stores metadata available from the list endpoint: name, recordClassName, estimatedSize, isSaved, leafAndTransformStepCount.

The plan field is left untouched (empty for new projections, preserved for existing ones). Full plan data is fetched lazily on first GET.

Returns the projection, or None if the WDK item has no valid ID.

Return type:

StreamProjection | None

async veupath_chatbot.services.strategies.wdk_sync.lazy_fetch_wdk_detail(*, projection, stream_repo)

Fetch full WDK strategy detail for a summary-only projection.

If the projection has a wdk_strategy_id but no plan data (created during sync-wdk to avoid N+1), fetches the full detail from WDK now and updates the projection. Returns the updated projection, or the original if no fetch was needed or the fetch failed.

Return type:

StreamProjection

async veupath_chatbot.services.strategies.wdk_sync.sync_is_saved_to_wdk(*, projection)

Sync the isSaved flag from a projection to WDK.

No-op if the projection has no wdk_strategy_id or site_id. Failures are logged and swallowed (non-critical sync).

WDK Step Counts

Purpose: Per-step result count computation. Uses anonymous reports for leaf-only strategies (fast) and temporary WDK compilation for complex strategies. Results are cached by plan hash.

Step count computation: get per-step result counts from WDK.

Supports two paths: - Leaf-only strategies: parallel anonymous reports (fast, no strategy creation) - Complex strategies: temporary WDK strategy compilation (slower)

Results are cached by plan hash to avoid redundant API calls.

veupath_chatbot.services.strategies.wdk_counts.plan_cache_key(site_id, plan)

Return type:

str

veupath_chatbot.services.strategies.wdk_counts.is_leaf_only_strategy(strategy_ast)

Check if all steps in the strategy are leaf (search) steps.

Return type:

bool

async veupath_chatbot.services.strategies.wdk_counts.compute_step_counts_for_plan(plan, strategy_ast, site_id)

Compute per-step result counts for a strategy plan.

For leaf-only strategies (all search steps, no combines/transforms), uses WDK’s anonymous report endpoint in parallel — no step or strategy creation needed. This is dramatically faster than full compilation.

For complex strategies (with combines/transforms), falls back to creating a temporary WDK strategy to get server-computed counts.

Results are cached by plan hash.

Return type:

dict[str, int | None]

Strategy Build

Purpose: High-level strategy build orchestration. Coordinates step creation and graph assembly.

Strategy build service: compile, create/update, and count extraction.

Encapsulates all business logic for building a strategy on WDK. The AI tool layer delegates to this module and only handles argument parsing and response formatting.

class veupath_chatbot.services.strategies.build.StrategyBuildAPI(*args, **kwargs)

Bases: StrategyCompilerAPI, StepDecoratorAPI, Protocol

Combined protocol: compile steps + decorate + strategy CRUD.

This is satisfied by the real StrategyAPI from the integrations layer.

async create_strategy(step_tree, name, description=None)

Return type:

JSONObject

async update_strategy(strategy_id, step_tree=None, name=None)

Return type:

JSONObject

async get_strategy(strategy_id)

Return type:

JSONObject

async get_step_count(step_id)

Return type:

int

class veupath_chatbot.services.strategies.build.SiteInfoLike(*args, **kwargs)

Bases: Protocol

Protocol for site metadata needed by the build service.

strategy_url(strategy_id, root_step_id=None)

Return type:

str

__init__(*args, **kwargs)

class veupath_chatbot.services.strategies.build.BuildResult(wdk_strategy_id, wdk_url, root_step_id, root_count, step_count, counts, zero_step_ids, compilation)

Bases: object

Outcome of a successful strategy build.

wdk_strategy_id: int | None

wdk_url: str | None

root_step_id: int

root_count: int | None

step_count: int

counts: dict[str, int | None]

zero_step_ids: list[str]

compilation: CompilationResult

__init__(wdk_strategy_id, wdk_url, root_step_id, root_count, step_count, counts, zero_step_ids, compilation)

class veupath_chatbot.services.strategies.build.StepCountResult(step_id, count)

Bases: object

Outcome of a step count lookup.

step_id: int

count: int

__init__(step_id, count)

exception veupath_chatbot.services.strategies.build.RootResolutionError(message, root_count=0)

Bases: Exception

Raised when a single root step cannot be resolved from the graph.

__init__(message, root_count=0)

veupath_chatbot.services.strategies.build.resolve_root_step(graph, explicit_root_step_id)

Resolve the root step from the graph.

Parameters:

• graph (StrategyGraph) – Strategy graph.

• explicit_root_step_id (str | None) – Optional explicit root step ID override.

Returns:

The resolved root PlanStepNode.

Raises:

RootResolutionError – When root cannot be determined.

Return type:

PlanStepNode

veupath_chatbot.services.strategies.build.create_strategy_ast(graph, root_step, strategy_name, description)

Create and validate a StrategyAST from graph state.

Record type is read from graph.record_type which must already be set (either from step creation or auto-resolution in build_strategy()).

Parameters:

• graph (StrategyGraph) – Strategy graph.

• root_step (object) – Root PlanStepNode.

• strategy_name (str | None) – Optional strategy name.

• description (str | None) – Optional description.

Returns:

Validated StrategyAST.

Raises:

ValueError – When record type is not set or validation fails.

Return type:

StrategyAST

async veupath_chatbot.services.strategies.build.create_or_update_wdk_strategy(api, compilation_result, strategy, existing_wdk_id)

Create a new WDK strategy or update an existing one.

If existing_wdk_id is provided, attempts to update first. Falls back to creating a new strategy if the update fails (e.g. 404 from WDK).

Returns:

The WDK strategy ID, or None if creation failed.

Return type:

int | None

veupath_chatbot.services.strategies.build.extract_step_counts(strategy_info, compiled_map)

Extract per-step result counts from a WDK strategy payload.

Parameters:

• strategy_info (JSONObject) – Raw WDK strategy dict (from api.get_strategy).

• compiled_map (dict[str, int]) – Mapping of local_step_id -> wdk_step_id.

Returns:

Tuple of (step_counts dict, root_count).

Return type:

tuple[dict[str, int | None], int | None]

async veupath_chatbot.services.strategies.build.build_strategy(*, graph, api, site, site_id, root_step_id=None, strategy_name=None, description=None)

Build or update a strategy on WDK.

Orchestrates: root resolution, AST creation, WDK compilation, create-or-update, step decorations, count extraction, and graph state mutation.

Record type is auto-resolved from leaf searches via the pre-cached SearchCatalog — callers never need to supply it.

Raises:

• RootResolutionError – If root step cannot be determined.

• ValueError – If validation or record type inference fails.

• Exception – On WDK API failures.

Return type:

BuildResult

async veupath_chatbot.services.strategies.build.get_result_count(api, wdk_step_id, wdk_strategy_id=None)

Get the result count for a built WDK step.

First tries to read estimatedSize from the strategy payload (cheaper), then falls back to a direct step count query.

Raises:

• TypeError – If strategy payload is malformed.

• Exception – On WDK API errors (propagated to caller).

Return type:

StepCountResult

async veupath_chatbot.services.strategies.build.build_strategy_for_site(*, graph, site_id, root_step_id=None, strategy_name=None, description=None)

Build a strategy using factory-resolved API and site info.

This is the entry point for the AI tool layer – it resolves the integration objects internally so callers don’t import from integrations.

Return type:

BuildResult

async veupath_chatbot.services.strategies.build.get_result_count_for_site(site_id, wdk_step_id, wdk_strategy_id=None)

Get result count using factory-resolved API.

This is the entry point for the AI tool layer.

Return type:

StepCountResult

Step Creation

Purpose: Create individual strategy steps with parameter validation and WDK integration.

Step creation business logic.

Extracts the validation-heavy step creation workflow from the AI tool layer so it can be tested and reused independently. All I/O dependencies (WDK client, discovery service) are injected via callbacks or explicit parameters.

class veupath_chatbot.services.strategies.step_creation.StepCreationResult(step, step_id, error)

Bases: object

Result of step creation: either a successfully added step or an error payload.

step: PlanStepNode | None

step_id: str | None

error: JSONObject | None

__init__(step, step_id, error)

veupath_chatbot.services.strategies.step_creation.coerce_wdk_boolean_question_params(*, parameters)

Extract left/right/operator from WDK boolean-question parameter conventions.

WDK boolean questions sometimes encode combines as a “boolean_question_*” search with bq_left_op_, bq_right_op_, bq_operator. Our graph model represents combines structurally; we translate these WDK boolean keys from the parameters dict.

Mutates parameters by removing any bq_* keys it consumes.

Parameters:

parameters (JSONObject) – WDK boolean-question parameters (may contain bq_left_op_, bq_right_op_, bq_operator).

Returns:

Tuple of (left_step_id, right_step_id, operator) or (None, None, None).

Return type:

tuple[str | None, str | None, str | None]

async veupath_chatbot.services.strategies.step_creation.create_step(*, graph, site_id, search_name=None, parameters=None, record_type=None, primary_input_step_id=None, secondary_input_step_id=None, operator=None, display_name=None, upstream=None, downstream=None, strand=None, resolve_record_type_for_search, find_record_type_hint, extract_vocab_options, validation_error_payload)

Create a new strategy step with full validation.

Step kind is inferred from structure: - leaf step: no inputs - unary/transform step: primary_input_step_id only - binary/combine step: primary_input_step_id + secondary_input_step_id (+ operator)

Returns:

StepCreationResult with either step/step_id or error.

Return type:

StepCreationResult

Auto Import

Purpose: Automatic import of WDK strategies into PathFinder.

Auto-import gene sets for WDK-linked strategy projections.

When strategies are synced from WDK, eligible projections automatically get a gene set created and linked. Once imported (or once the user deletes the auto-imported gene set), the projection is marked so re-syncs don’t recreate it.

async veupath_chatbot.services.strategies.auto_import.auto_import_gene_sets(projections, *, stream_repo, gene_set_service, site_id, user_id)

Create gene sets for eligible strategy projections.

For each eligible projection (has wdk_strategy_id, not yet imported, no existing gene set), creates a gene set and links it to the projection.

Returns the list of newly created gene sets.

Return type:

list[GeneSet]

async veupath_chatbot.services.strategies.auto_import.background_auto_import_gene_sets(*, site_id, user_id)

Run gene-set auto-import in a background task with its own DB session.

This avoids blocking the sync-wdk response while WDK API calls resolve gene IDs for each eligible strategy.

Auto Push

Purpose: Best-effort auto-push: sync a local strategy back to VEuPathDB WDK after mutations. Runs as a background task with per-strategy locking.

Best-effort auto-push: sync a local strategy back to VEuPathDB WDK.

Called after strategy mutations (CRUD updates, chat-driven changes) when the strategy has a wdk_strategy_id. Failures are logged but never propagate — the local save is the source of truth.

IMPORTANT: this runs as a background asyncio.Task, not inside the request’s DB session. It creates its own session to avoid the asyncpg InterfaceError: another operation is in progress that occurs when a fire-and-forget task shares a request-scoped connection.

async veupath_chatbot.services.strategies.auto_push.try_auto_push_to_wdk(strategy_id)

Push a strategy to WDK if it has a wdk_strategy_id.

Reads from the CQRS projection (stream_projections table).

This is a best-effort operation — any error is logged and swallowed.

If the WDK strategy no longer exists (404), the stale wdk_strategy_id is cleared so future pushes don’t keep failing.

Session Factory

Purpose: Create and restore strategy sessions. Loads strategy state from the database for chat context.

Helpers for hydrating in-memory strategy session context for agents.

veupath_chatbot.services.strategies.session_factory.build_strategy_session(*, site_id, strategy_graph)

Build a StrategySession from a persisted strategy graph payload.

This mirrors UI persistence semantics: prefer canonical plan if present/parseable, fall back to snapshot-derived steps + rootStepId hydration when plan is missing/invalid.

Parameters:

• site_id (str) – VEuPathDB site identifier.

• strategy_graph (JSONObject | None) – JSONObject | None.

Return type:

StrategySession

Step Builders

Purpose: Build WDK step payloads from the strategy plan AST. Creates the correct step structure for WDK API calls.

Strategy step construction helpers.

These functions are used to build the persisted step list from a Strategy AST. They intentionally do not depend on HTTP DTOs.

veupath_chatbot.services.strategies.step_builders.build_steps_data_from_ast(strategy_ast)

Return type:

JSONArray

Strategy Engine

Purpose: Core strategy execution engine. Graph integrity checks, step ordering, and execution helpers.

Shared base class for strategy tool implementations (service layer).

class veupath_chatbot.services.strategies.engine.base.StrategyToolsBase(session)

Bases: object

Base strategy tools class with shared state.

Declares method stubs for methods defined in ValidationMixin that are used by sibling mixins (GraphOpsMixin, IdMappingMixin). The real implementations live in ValidationMixin; these stubs exist solely so that mypy can verify attribute access in mixin classes without requiring the full MRO to be resolved.

__init__(session)

Strategy graph integrity helpers (service layer).

These utilities validate the working graph state (mutable session graph), not the compiled DSL AST (which is single-root by construction).

Design goals: - DRY: centralize root detection / validation logic. - Separation of concerns: keep logic out of AI prompt text and tool mixins.

class veupath_chatbot.services.strategies.engine.graph_integrity.GraphIntegrityError(code: str, message: str, step_id: str | None = None, input_step_id: str | None = None, kind: str | None = None)

Bases: object

code: str

message: str

step_id: str | None = None

input_step_id: str | None = None

kind: str | None = None

to_dict()

Return type:

JSONObject

__init__(code, message, step_id=None, input_step_id=None, kind=None)

veupath_chatbot.services.strategies.engine.graph_integrity.find_root_step_ids(graph)

Return root step IDs in sorted order.

Uses the incrementally-maintained graph.roots set (O(1)) instead of recomputing from scratch.

Parameters:

graph (StrategyGraph) – Strategy graph.

Returns:

Sorted list of root step IDs.

Return type:

list[str]

veupath_chatbot.services.strategies.engine.graph_integrity.validate_graph_integrity(graph)

Validate graph structure and single-output invariant.

Uses graph.roots for root detection and additionally checks for dangling input references.

Parameters:

graph (StrategyGraph) – Strategy graph to validate.

Returns:

List of integrity errors (empty if valid).

Return type:

list[GraphIntegrityError]

Helper methods for strategy tool implementations (service layer).

class veupath_chatbot.services.strategies.engine.helpers.StrategyToolsHelpers(session)

Bases: ValidationMixin, StepBuilderMixin, GraphOpsMixin, IdMappingMixin

Graph traversal, node/edge operations, serialization, and snapshots.

class veupath_chatbot.services.strategies.engine.graph_ops.GraphOpsMixin(session)

Bases: StrategyToolsBase

WDK ID <-> local ID mapping and record-type resolution.

class veupath_chatbot.services.strategies.engine.id_mapping.IdMappingMixin(session)

Bases: StrategyToolsBase

Step creation, parameter assembly, and vocabulary helpers.

class veupath_chatbot.services.strategies.engine.step_builder.StepBuilderMixin(session)

Bases: StrategyToolsBase

Validation and error-payload helpers for strategy tools.

class veupath_chatbot.services.strategies.engine.validation.ValidationMixin(session)

Bases: StrategyToolsBase