starred/minne
1
0
Fork 0
mirror of https://github.com/perstarkse/minne.git synced 2026-08-30 23:27:12 +02:00
Code Issues Packages Projects Releases 10 Wiki Activity
434 Commits 4 Branches 19 Tags
744482f2c853305fd511fb0ab06a1bd32c35e0ca
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Per Stark 744482f2c8 fix: knowledge entity suggestions simplification 2026-05-31 20:23:40 +02:00
Per Stark 6c3475ca0e chore: ingestion-pipeline refactor, sort technical debt, rustfmt 2026-05-31 19:48:41 +02:00
Per Stark e9d8654324 chore: refactor retrieval pipeline to chunk-first RRF with derived entities and slimmer eval surface. 
Collapse the multi-strategy entity engine into one benchmarked chunk retrieval path, derive entities from retrieved chunks, and update consumers, docs, and clippy fixes across the workspace.
 2026-05-30 22:19:08 +02:00
Per Stark a8e30192ba chore: harden api-router errors and add router integration tests while slimming html handlers. 2026-05-30 15:18:12 +02:00
Per Stark ec80a4e540 chore: improve html-router auth, caching, and analytics while centralizing search labels in common. 
small fix
 2026-05-29 15:03:55 +02:00
Per Stark 920d7b5efb chore: centralize embedding errors, retrieval strategy, and test DB helpers. 
Replace anyhow in embedding production code with EmbeddingError, move
RetrievalStrategy into common config, and deduplicate Surreal test setup
via common::test_utils.
 2026-05-29 14:44:23 +02:00
Per Stark d90319f3b0 chore: harden common storage bootstrap and slim embedded db assets 
Unify embedding config, build providers from system settings, and fail
startup when index builds error or time out. Move Surreal assets under
common/db so embeds exclude crate source, and read storage via streams.
 2026-05-29 14:44:23 +02:00
Per Stark 964d57ec97 test: cover system settings sync, validation, and ingestion prompts 
Add tests for embedding provider sync, patch isolation, typed backend
serde, and DB-backed ingestion prompts.
 2026-05-29 14:44:23 +02:00
Per Stark 544a790e34 chore: harden system settings and unify prompt usage 
Validate settings updates, use typed embedding backends, and route
ingestion through DB-stored prompts so admin edits take effect.
 2026-05-29 14:44:23 +02:00
Per Stark f625a7e0a9 chore: move serde helpers to common utils 
Relocate SurrealDB serde helpers out of storage types so they can be
reused broadly, and align retrieval-pipeline test setup with configured
embedding dimensions.
 2026-05-29 14:44:23 +02:00
Per Stark b80f942cb5 chore: harden text chunk embeddings and text content storage 
Align text chunk embedding identity with knowledge entities (chunk id as record id, UNIQUE chunk_id index, dimension validation), make cascade deletes transactional, and improve text content patch/search reliability with tests.
 2026-05-29 14:44:23 +02:00
Per Stark 2e2a26f5f1 chore: harden knowledge graph storage and clear common clippy warnings 
Enforce stable 1:1 entity embeddings, relationship endpoint auth, and
user-scoped deletes; align schemas/migrations and resolve common crate
clippy findings.
 2026-05-29 14:44:23 +02:00
Per Stark 4071314e6a chore: harden analytics, conversation access, and per-user file dedup 
Use UPSERT for analytics counters, enforce message ownership in SQL,
return
NotFound when patch_title updates nothing, scope file dedup by user_id
with a
composite unique index, and expand tests for auth, ordering, and edge
cases.
 2026-05-29 14:44:23 +02:00
Per Stark baf3e8b172 chore: optimize ingest payloads and add parallel task batch store 
Parse content before building file payloads to move shared metadata when
possible, add create_all_and_add_to_db for concurrent stores, and extend
tests for batch persistence and payload edge cases.
 2026-05-29 14:44:23 +02:00
Per Stark 1e0dba72c8 chore: harden common errors, fastembed blocking, and ingest ownership 
Run FastEmbed inference on spawn_blocking, propagate Surreal take
failures,
add AppError::internal and typed ingest/embedding parse errors, and take
owned file lists in ingestion payload construction.
 2026-05-29 14:44:23 +02:00
Per Stark 1e25705377 chore: improved error handling 2026-05-28 19:58:14 +02:00
Per Stark 4f02fcb853 chore: rename get_id to id, add doc comments, pre-allocate format_history 2026-05-27 18:06:16 +02:00
Per Stark 9ccf8dde25 chore: lowercase all error messages and add # Errors doc sections 
- Fix err-lowercase-msg: normalize all #[error(...)] display strings to
  lowercase (AppError, FileError, ApiErr) and update affected tests
- Fix err-doc-errors: add # Errors sections to 25+ fallible public
  functions across db.rs, store.rs, embedding.rs, indexes.rs,
  ingestion_task.rs, and ingest_limits.rs
 2026-05-27 14:59:48 +02:00
Per Stark e63498ecee chore: resolve remaining uninlined_format_args clippy warnings 2026-05-27 14:34:37 +02:00
Per Stark b03290abd5 chore: fix and reduce clippy allows in knowledge_entity.rs 
- rm duplicate 'document' match arm (match_same_arms)
- .get(0) -> .first() (get_first)
- for entity in all_entities.iter() -> &all_entities (explicit_iter_loop)
- 2x error!("{}", err_msg) -> error!("{err_msg}") (uninlined_format_args)
- 2x test format!()/assert!() positional -> inlined (uninlined_format_args)
- removed 6 now-unnecessary allow attributes
 2026-05-27 14:28:08 +02:00
Per Stark 81624850c0 chore: add must_use to 27 non-Result public functions 
- constructors: KnowledgeEntity, TextChunk, Scratchpad, IngestionTask,
  Conversation, KnowledgeRelationship, Message, TextContent,
  KnowledgeEntityEmbedding, TextChunkEmbedding
- accessors: Theme::as_str, Theme::initial_theme, TaskState::as_str,
  TaskState::display_label, StorageManager::backend_kind,
  StorageManager::local_base_path, EmbeddingProvider::backend_label,
  EmbeddingProvider::dimension, EmbeddingProvider::model_code
- queries: TaskState::is_terminal, IngestionTask::can_retry,
  KnowledgeEntityType::variants, StorageManager::resolve_local_path,
  resolve_base_dir, IngestionTask::lease_duration
- helpers: Message::format_history
- builders: StorageManager::with_backend
 2026-05-27 14:23:56 +02:00
Per Stark 99db54b2e7 fix: replace manual embedding serialization with serde_json 
- replaced write!() loops with serde_json::to_string in 4 re-embedding methods
- standardized SQL building to use write!() with proper error propagation
- eliminates manual f32 vector string building (memory waste + loop risk)
 2026-05-27 14:13:19 +02:00
Per Stark cd0d95abaa fix: revoke_api_key sets NONE, remove unused bind, lowercase error msgs 
- fix bug where revoke_api_key set literal 'test_string_nullish' instead of NONE
- remove unused table_name bind in update_timezone
- lowercase ~16 error messages across 4 crates
 2026-05-27 13:56:32 +02:00
Per Stark e2284b1e69 chore: removed anyhow from apperror for improved error handling 2026-05-27 13:33:02 +02:00
Per Stark 76fcdcd6ce chore: index slicing and lowercase errors 2026-05-27 12:41:26 +02:00
Per Stark cd0b0368a0 chore: tightening and removing super fn 2026-05-27 11:23:39 +02:00
Per Stark 48985a61d3 chore: clippy and nix fmt 2026-05-27 11:23:08 +02:00
Per Stark 056f116885 perf: avoid small own clones and intermediate Vec allocations 
- Derive Copy on 6 small enums (MessageRole, TaskState, StorageKind, EmbeddingBackend, PdfIngestMode, KnowledgeEntityType)
- Change create_ingestion_payload files param from Vec<FileInfo> to &[FileInfo]
- Remove 5 intermediate Vec allocations (4 embedding serialization + 1 format_history) using write! loop
- Remove 7 unnecessary .clone() calls exposed by Copy derive
 2026-05-27 10:28:08 +02:00
Per Stark ffea0ef126 fix: html-router dependency of json-stream-parser 2026-05-27 09:59:26 +02:00
Per Stark dd9ee22670 refactor: json-stream-parser aligned to clippy standard 2026-05-27 09:07:38 +02:00
Per Stark 017d6c5ba9 chore: additional clippy fixes after rebasing 2026-05-27 07:37:18 +02:00
Per Stark 12f989b3a1 fix: pin surrealdb 2026-05-26 20:21:40 +02:00
Per Stark 82e0619850 clippy: evaluations crate 2026-05-26 20:21:25 +02:00
Per Stark 1bf93f0125 perf: pre-allocate collections with known capacity in hot paths 
- Use with_capacity for chunk_by_source, results, per_entity_traces,
  and selected_chunks in assemble() where bound is known
- Pre-allocate tokens/terms vectors in normalize_fts_query and
  extract_keywords based on input length
- Pre-allocate neighbor_ids, seen, and ordered in graph expansion
  based on relationship count
 2026-05-26 20:21:25 +02:00
Per Stark b8f7c826b4 perf: offload blocking calls to spawn_blocking 
- Move headless_chrome PDF rasterization from async context to
  spawn_blocking, keeping tokio worker threads responsive.
- Switch RerankerPool from tokio::sync::Mutex to std::sync::Mutex
  and run TextRerank::rerank inside spawn_blocking, since the
  rerank call is CPU-bound with no .await points.
 2026-05-26 20:21:25 +02:00
Per Stark fe68194e4f lint: inherit workspace clippy config in json-stream-parser and evaluations 
Both crates were missing the [lints] workspace = true directive,
bypassing workspace clippy rules (unwrap_used, expect_used, etc.).
 2026-05-26 20:21:25 +02:00
Per Stark d76f86f56f refactor: simplify and improve testing for initialization 2026-05-26 20:21:24 +02:00
Per Stark 5ce7a76c75 clippy: adhere to pedantic clippy, uniform test error handling 2026-05-26 20:21:13 +02:00
Per Stark e0068ebe26 chore: remove unused clap dep and fix test_session_table name 
- Remove clap dependency from retrieval-pipeline (RetrievalStrategy
  already has FromStr/Display; evaluations uses clap directly)
- Rename session table from test_session_table to session
 2026-05-26 20:14:29 +02:00
Per Stark 7c718712c9 refactor: replace Box<dyn Error> with anyhow::Result 
- ingestion_pipeline::run_worker_loop returns anyhow::Result<()>
- api_router::ApiState::new returns anyhow::Result<Self>
- html_router::HtmlState::new_with_resources is infallible, returns Self
- main/server/worker binary entry points return anyhow::Result<()>
 2026-05-26 20:14:11 +02:00
Per Stark 9affc0b05b refactor: extract serde helpers from stored_object! macro 
Move FlexibleIdVisitor, deserialize_flexible_id, and four datetime serde helpers
from repeating inside every macro expansion into a shared
common/src/storage/types/serde_helpers.rs module.

14 macro invocations × 6 items = ~84 fewer redundant function definitions.
Fragile cross-module imports (file_info::deserialize_flexible_id etc.)
are updated to point to the canonical module.
 2026-05-26 20:12:54 +02:00
Per Stark d0d00469e9 chore: devenv inconsistency, spawn server manually in dev 2026-02-15 18:31:43 +01:00
Per Stark 611d7217e6 release: 1.0.2 2026-02-15 11:57:04 +01:00
Per Stark 87459e1189 test: minio to devenv, improved testing s3 and relationships 2026-02-15 08:52:56 +01:00
Per Stark 6c5297ec6e chore: dep updates & kv-mem separation to test feature 
docker builder update
 2026-02-15 08:51:48 +01:00
Per Stark f5d72b66b0 test: add admin auth integration coverage 2026-02-14 23:11:35 +01:00
Per Stark 93ac56ebc7 feat: caching chat history & dto 2026-02-14 19:43:34 +01:00
Per Stark c6e499e5dc fix: harden html responses and cache chat sidebar data 
Use strict template response handling and sanitized template user context, then add an in-process conversation archive cache with mutation-driven invalidation for chat sidebar renders.
 2026-02-14 17:47:14 +01:00
Per Stark ffae16bc84 fix: simplified admin checking 2026-02-13 23:04:01 +01:00
Per Stark d1b3e9b23a fix: name harmonization of endpoints & ingestion security hardening 2026-02-13 22:36:00 +01:00
Per Stark 545a5dc9f3 fix: redact ingestion payload logs and update changelog 2026-02-13 12:06:18 +01:00
Per Stark 91f3ab231c fix: parameterize storage-layer queries and add injection tests 2026-02-12 21:42:46 +01:00
Per Stark f3d4625a3e fix: border in navigation 2026-02-12 20:39:36 +01:00
Per Stark 822eaa5688 fix: browser back navigation from chat windows 
addenum
 2026-02-12 20:32:06 +01:00
Per Stark 3b58582963 fix: references bug 
fix
 2026-02-11 22:02:40 +01:00
Per Stark 45da7bacf3 release: 1.0.1 2026-02-11 15:39:28 +01:00
Per Stark ebba45c129 docs: updated domain name 2026-02-11 15:17:03 +01:00
Per Stark 6d996f7775 fix: gracefully handle old users 2026-02-11 07:50:19 +01:00
Per Stark 99fdb0fc8d docs: updated readme 2026-01-18 18:48:53 +01:00
Per Stark 0887d2b07f dev: devenv processes 2026-01-18 18:45:30 +01:00
Per Stark 599709c4cd fix: edge case when deleting content 
nit
 2026-01-18 18:45:21 +01:00
Per Stark 2f069ef7b8 design: better dark mode 2026-01-17 23:31:05 +01:00
Per Stark 60087e10fb refactor: additional responsibilities to middleware, simplified handlers 
fix
 2026-01-17 21:07:25 +01:00
Per Stark 3c16c17472 theme: obsidian-prism 2026-01-17 08:45:47 +01:00
Per Stark a9a37b2468 feat: s3 storage backend 2026-01-16 23:38:47 +01:00
Per Stark 8e9978540e feat: add user theme preference 
- Add theme field to User model (common)
- Create migration for theme field
- Add theme selection to Account Settings (html-router)
- Implement server-side theme rendering in base template
- Update JS for system/preference theme handling
- Remove header theme toggle for authenticated users
 2026-01-16 13:54:07 +01:00
Per Stark 336de2c0fb docs: addenum 2026-01-14 22:24:23 +01:00
Per Stark 5b16e88991 refactor: extendable templates 
refactor: simplification

refactor: simplification
 2026-01-13 22:18:00 +01:00
Per Stark 534d0f8c31 fix: allow for multiple templates directories 2026-01-12 21:25:12 +01:00
Per Stark f782812f21 fix: updating models in admin view 2026-01-12 21:01:53 +01:00
Per Stark 8664abdf01 release: 1.0.0 
fix: cargo dist
 2026-01-11 20:35:01 +01:00
Per Stark 07421fdb18 fix: schemafull and textcontent 2026-01-02 15:41:22 +01:00
Per Stark b933ba35cd docs: more complete and correct 2025-12-24 23:36:58 +01:00
Per Stark 58e560be9d chore: wording 2025-12-22 23:03:33 +01:00
Per Stark 97ce77c574 fix: never block fts, rely on rrf 2025-12-22 22:56:57 +01:00
Per Stark 589257e721 fix: migrating embeddings to new dimensions 
changing order
 2025-12-22 22:39:14 +01:00
Per Stark b4b2ab8974 fix: ordering of index creation 2025-12-22 21:59:35 +01:00
Per Stark f95e14df25 docs: evaluations instructions and readme refactoring 2025-12-22 18:55:47 +01:00
Per Stark a4d6aa88fa fix: migrations 
schemafull
 2025-12-22 18:32:08 +01:00
Per Stark 97b8c39065 fix: admin page sorted 2025-12-21 21:35:52 +01:00
Per Stark 0bd91d16c1 Merge branch 'main' into benchmarks 2025-12-20 23:09:16 +01:00
Per Stark ca8b37f0dc changelog 2025-12-20 23:03:06 +01:00
Per Stark 86270de873 tidying stuff up, dto for search 2025-12-20 22:30:31 +01:00
Per Stark 90bac299a3 passed wide smoke check 2025-12-10 13:54:08 +01:00
Per Stark 032630c083 faster index creation 2025-12-09 21:32:23 +01:00
Per Stark 8121e04125 retrieval simplfied 2025-12-09 20:35:42 +01:00
Per Stark 192e6480e0 benchmarks: fin 2025-12-08 21:57:53 +01:00
Per Stark e77f2d51e8 beir-rff 2025-12-08 20:39:12 +01:00
Per Stark 1fccf3ab59 dataset: beir 2025-12-04 17:50:35 +01:00
Per Stark 6c458c97ce retrieval: hybrid search, linear fusion 2025-12-04 12:48:59 +01:00
Per Stark a56eb3ed7c release: 0.2.7 2025-12-04 12:25:46 +01:00
Per Stark c60efb2af7 benchmarks: ready for hybrid revised 2025-12-03 11:38:07 +01:00
Per Stark 5e5053039a fix: removed stale embeddings handler 2025-11-29 20:07:48 +01:00
Per Stark 38cb2e5e24 fix: all tests now in sync 2025-11-29 18:59:08 +01:00
Per Stark abf0223bd2 ndcg fix 2025-11-29 16:24:09 +01:00
Per Stark 888c055fee refactored to clap, mrr and ndcg 2025-11-28 21:26:51 +01:00
Per Stark 06344cc108 fix: index creation at init 2025-11-26 21:49:20 +01:00
Per Stark 05bdaac672 evals: v3, ebeddings at the side 
additional indexes
 2025-11-26 15:15:10 +01:00
Per Stark 6611bf3645 retrieval-pipeline: v1 2025-11-19 12:58:27 +01:00
Per Stark de47dc7a2f fix: add dockerfile changes related to retrieval-pipeline 2025-11-18 22:51:48 +01:00