Synthadoc: Beyond Keyword Search -How Combines BM25 and Vector Search to Build a Smarter Domain Wiki

Synthadoc v0.2.0 introduces a range of advanced features aimed at enhancing the efficiency and effectiveness of domain-specific knowledge management systems. Here's a summary of key enhancements:

1. Query Decomposition: Complex queries are broken down into simpler sub-queries that can be processed in parallel, improving response time and accuracy.

2. Vector Re-ranking (Optional): Synthadoc now supports semantic re-ranking using BAAI/bge-small-en-v1.5 via fastembed. This feature enhances the relevance of search results by considering the context and meaning beyond keyword matching.

3. Knowledge Gap Detection: The system can identify gaps in knowledge based on three signals (BM25 pool size, vector top candidates, and purpose.md scope) and automatically generate targeted ingest suggestions to fill these gaps.

4. Web Search Decomposition: Broad search topics are split into focused queries using Tavily, with URL deduplication and a cap to manage the volume of ingested data efficiently.

5. Per-Model Cost Tracking: Detailed cost tracking per token and operation is provided, allowing users to monitor and control their spending on LLM services like OpenAI or Anthropic.

6. **Query Audit Trail

Read the full article at DEV Community

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.