Vector Index Maintenance Under Continuous Ingestion

Practical context

A source-based guide to persistence and index maintenance boundaries.

Build document recall and decision-history prototypes with explicit source ownership.

Implementation and research boundary

The durable store integrates snapshots, WAL recovery and HNSW search. Writer ownership is explicit: the API reports a busy writer rather than assuming unrestricted concurrent mutation.

Recovery and chain checks must be exercised on the actual storage path. These components do not establish a continuously available distributed index, unlimited ingest throughput or a published recall-under-load result.

Separate durability from retrieval freshness

A durable record, an in-memory index entry and a checkpoint are different observations. The durable store combines snapshot and WAL recovery with its search index, while explicit writer ownership prevents treating it as an unrestricted multiwriter service. An application should define when a successful insert becomes searchable and which recovery boundary it promises. Inspect the actual API result instead of inferring a durability guarantee from a successful UI message.

Proposed repeatable workload

Use a deterministic sequence of insertions and a fixed set of exact nearest-neighbor queries. Check recall after selected insertion batches against an exhaustive baseline, retaining record identifiers and insertion order. Repeat after checkpoint and reopen. On copied scratch data, interrupt between declared stages to evaluate recovery behavior, and attempt a second writer to verify the expected busy-writer outcome. Preserve the failing operation and error if any stage refuses work; do not silently remove errors from throughput statistics.

Maintenance decisions and acceptance

Evaluate compaction and retention separately from indexing, because removing a segment can change what history remains available. Measure storage size, recovery time and query behavior with the same workload and source revision. A passing test should demonstrate the expected record set and ordering or declared approximation tolerance, not just that the process restarted. Traceability also requires stable application source references through replacement and deletion. These experiments would support a bounded maintenance claim; the source alone does not establish continuous availability or recall quality at a production ingest rate.

Evaluation before adoption

Start with a synthetic or authorized corpus. Check recovery, source identifiers, update behavior and retrieval relevance. Treat application permissions, data retention and deployment evidence as additional requirements.