Collapse the multi-strategy entity engine into one benchmarked chunk retrieval path, derive entities from retrieved chunks, and update consumers, docs, and clippy fixes across the workspace.
Replace anyhow in embedding production code with EmbeddingError, move
RetrievalStrategy into common config, and deduplicate Surreal test setup
via common::test_utils.
Unify embedding config, build providers from system settings, and fail
startup when index builds error or time out. Move Surreal assets under
common/db so embeds exclude crate source, and read storage via streams.
Relocate SurrealDB serde helpers out of storage types so they can be
reused broadly, and align retrieval-pipeline test setup with configured
embedding dimensions.
Align text chunk embedding identity with knowledge entities (chunk id as record id, UNIQUE chunk_id index, dimension validation), make cascade deletes transactional, and improve text content patch/search reliability with tests.
Use UPSERT for analytics counters, enforce message ownership in SQL,
return
NotFound when patch_title updates nothing, scope file dedup by user_id
with a
composite unique index, and expand tests for auth, ordering, and edge
cases.
Parse content before building file payloads to move shared metadata when
possible, add create_all_and_add_to_db for concurrent stores, and extend
tests for batch persistence and payload edge cases.
Run FastEmbed inference on spawn_blocking, propagate Surreal take
failures,
add AppError::internal and typed ingest/embedding parse errors, and take
owned file lists in ingestion payload construction.
- replaced write!() loops with serde_json::to_string in 4 re-embedding methods
- standardized SQL building to use write!() with proper error propagation
- eliminates manual f32 vector string building (memory waste + loop risk)
- Use with_capacity for chunk_by_source, results, per_entity_traces,
and selected_chunks in assemble() where bound is known
- Pre-allocate tokens/terms vectors in normalize_fts_query and
extract_keywords based on input length
- Pre-allocate neighbor_ids, seen, and ordered in graph expansion
based on relationship count
- Move headless_chrome PDF rasterization from async context to
spawn_blocking, keeping tokio worker threads responsive.
- Switch RerankerPool from tokio::sync::Mutex to std::sync::Mutex
and run TextRerank::rerank inside spawn_blocking, since the
rerank call is CPU-bound with no .await points.
Move FlexibleIdVisitor, deserialize_flexible_id, and four datetime serde helpers
from repeating inside every macro expansion into a shared
common/src/storage/types/serde_helpers.rs module.
14 macro invocations × 6 items = ~84 fewer redundant function definitions.
Fragile cross-module imports (file_info::deserialize_flexible_id etc.)
are updated to point to the canonical module.
Use strict template response handling and sanitized template user context, then add an in-process conversation archive cache with mutation-driven invalidation for chat sidebar renders.
Per Stark