ShadowSpeak
Overview
Learners of English mostly meet pronunciation as isolated dictionary audio or synthetic speech. What is missing is how words and phrases sound in real conversation: natural rhythm, connected speech, and situational meaning.
A huge amount of real speech already exists on YouTube, with timestamped subtitles. If those transcripts are indexed at sentence level, any phrase becomes a doorway to real examples: the exact moment, from a real speaker, in context.
ShadowSpeak treats video transcripts as a searchable pronunciation corpus. The learner asks, the system finds the moment, the player plays it, and AI explains it, grounded in the real transcript.
Design principles
- Evidence first: The transcript and the source video are the truth. AI only explains what was retrieved.
- Two ways to ask: Exact phrase for precision, Meaning / situation for discovery. The two are never blurred.
- Fail soft: If the AI service fails, the learner still gets the clip.
Local-first growth (planned) The collection is designed to grow with demand. The system always searches its own index first. Only when no relevant example exists does it discover new videos, ingest their full transcripts, and index every segment, so later searches for other phrases in the same videos are served locally at no extra discovery cost.
The Learner Journey
- Ask: Type a word, a phrase, or a situation such as "ask politely for help", and choose exact or meaning mode.
- Find: See matching clips with the transcript excerpt and a link to the source video.
- Listen and repeat: Play the exact moment and replay it for repetition practice.
- Understand: Open an explanation in Indonesian: meaning in context, usage, register, sentence pattern, and clearly labeled AI-generated examples.
How the System Works
| Stage | What happens | Technology |
|---|---|---|
| Ingest | Raw timestamped cues are validated, then grouped into overlapping segments (split at gaps over 2 seconds, capped at 30 seconds) so phrases that span subtitle lines are still found. | FastAPI, SQLModel, JSON/CSV import |
| Index | Each segment is embedded as a 2048-dimension vector. Text and vectors are stored together in one transaction, and identical content is never processed twice. | OpenRouter embeddings, PostgreSQL + pgvector |
| Retrieve | Exact: phrase matching over normalized transcript text. Meaning: the query is embedded, candidates come from a vector index, and results are re-ranked by cosine similarity. | SQL, pgvector HNSW, optional Pinecone |
| Shape results | At most 10 clips, at most 3 per video, overlaps removed, and short matches padded so each clip is at least 10 seconds when the transcript allows. | Application use cases |
| Play | The selected clip plays at its exact timestamps through YouTube. No media files are stored or re-encoded. | YouTube player, Svelte 5 |
| Explain | On demand, the backend sends the clip and its context to an LLM with structured output, validates the response, caches it, and returns it. | OpenRouter LLM, server-side schema validation |
Applying the Technology
Finding phrases in fragmented subtitles: Subtitles break mid-sentence, so searching raw lines misses phrases. The system rebuilds sentence-level segments from cues and searches normalized text. A phrase like "could you" is found even across a line break, and a synonym never passes as an exact match.
Searching by meaning: Embeddings place the learner's intent and every transcript segment in the same vector space. pgvector keeps vectors next to relational data in a single PostgreSQL database. A half-precision HNSW index provides speed, and full-precision re-ranking preserves accuracy. Embeddings are versioned by model so vectors from different models are never mixed.
AI as explainer, not authority: Retrieval decides what is shown; the model only explains it. The quote, video ID, and timestamps always come from the database, never from the model. The model returns a strict JSON structure that is validated on the server, and any invented example is labeled as AI-generated. A failed explanation never blocks the clip.
Reliable and affordable by design: Explanations are generated only for the clip a learner opens, then cached. Ingestion is transactional and idempotent. Rate limits (Redis-backed in production) protect embedding and LLM spend, and the AI feature can be switched off through configuration.
Clean delivery: Both codebases use layered, provider-agnostic architecture, so embedding providers and vector stores can be swapped without touching business logic. A same-origin proxy keeps credentials server-side, and the full stack runs with Docker Compose.
Outcomes
For learners
- One continuous flow from a question to real audio in context.
- Explanations in Indonesian tied to the exact clip being watched.
For product quality
- Every quote, timestamp, and source link is traceable to stored transcript data, and AI-written examples are always labeled.
- A failed explanation never blocks playback.
- Two search modes with distinct guarantees: exact means exact, meaning means discovery.
For engineering and delivery
- The full stack (API, database with vector index, frontend) starts with a single Docker Compose command and is configured for Vercel with Neon Postgres.
- Automated tests, including real-PostgreSQL integration tests, cover ingestion, search, session isolation, and concurrency.
- Swappable providers for embeddings (fixture or OpenRouter) and vector storage (pgvector or Pinecone).
Success metrics (defined for the pilot)
Search-to-play conversion, replay usage, no-match rate, time to first new result, and cost per successful search. Quality targets include at least 95% top-3 precision for exact matches (design target, not yet measured).