Managed vs Direct Vector Search in Databricks: The Hidden Normalization That Changes Everything

The output you're seeing from the Direct Access Path reveals a critical difference in how vectors are handled compared to the Managed Delta Sync Index. Specifically, the Direct Access path does not automatically normalize the vectors before storing them or during querying.

Key Observations:

1. Query Vector Magnitude:

   • The query vector q_raw is returned by the embedding model and has a norm of approximately 24 (as observed in previous experiments with Managed Delta Sync Index).

2. Stored Vectors' Norms:

   • When you retrieve stored vectors from the Direct Access Index, their norms are also around 24.

3. Query Results:

   • The similarity scores returned by the Direct Access Index are extremely low (~0.002) because both the query vector and the stored vectors have large magnitudes that are not normalized.

Verification Steps:

To verify this, you can re-embed each document retrieved from the index, normalize both the query vector and the document vectors to unit length, and then compute the cosine similarity manually.

Here's how you can do it step-by-step:

1. Re-Embed Documents:
   • Re-compute embeddings for the documents returned by the Direct Access

Read the full article at Towards AI - Medium

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.