Based on the provided information and code snippets, here's a summary of how the project works for building a semantic search system over YouTube video transcripts:
Step 1: Preprocessing
Extracting Transcript Chunks
- metadata.py: Fetches metadata (title, thumbnail URL) from YouTube using their oEmbed API.
- transcript_extractor.py (not shown but implied): Likely extracts the transcript text from the YouTube video.
Processing and Embedding
-
embedder.py: Handles embedding of the transcript chunks. It uses OpenAI's
text-embedding-3-smallmodel to create embeddings for each chunk, normalizes them, and stores these in a FAISS index.python1def build_index(chunks: list[dict], client: OpenAI) -> tuple[faiss.Index, list[dict]]: 2 texts = [c["text"] for c in chunks] 3 embeddings = get_embeddings(texts, client) 4 5 # Normalize for cosine similarity 6 norms = np.linalg.norm(embeddings, axis=1, keepdims=True) 7 embeddings = embeddings / (norms + 1e-10) 8 9 dim = embeddings.shape[1] 10 index = fa
Read the full article at Towards AI - Medium
Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.

![[AINews] The Unreasonable Effectiveness of Closing the Loop](/_next/image?url=https%3A%2F%2Fmedia.nemati.ai%2Fmedia%2Fblog%2Fimages%2Farticles%2F600e22851bc7453b.webp&w=3840&q=75)



