Fig. 03 · Personal project
MindFrame
An AI pipeline that turns a topic into a narrated, captioned explainer video - timed to the real voice and rendered deterministically.
- Project
- MindFrame
- Role
- Designer and developer
- Period
- 09/2026 - present
- Context
- Personal project
- Stack
- TypeScript · Gemini · ElevenLabs · Remotion · React · Zod · Next.js
- Outcomes
- Topic to finished 60-90 s vertical explainer with narration, captions and synced animation
- Every caption and animation timed from the real voice, never invented by the model
- Failed or revised runs reuse cached narration instead of paying for it again
Problem
Short educational videos are slow to make: someone has to write the script, record narration, draw diagrams and line every caption and animation up with the voice. Generative video models can produce something quickly, but not something accurate, editable or in sync. MindFrame turns a topic and a few reference notes into a 60-90 second vertical explainer where the AI decides what to say and show, and deterministic code decides exactly when.
Constraints
- Accuracy before polish. AI drafts can be wrong, so nothing is narrated until a person has reviewed and approved the plan.
- Sync must be exact. Captions and animations have to land on the words that trigger them, within about 200 ms.
- Paid calls are expensive. Narration costs money per request, so retries, revisions and failures must not regenerate speech that is still valid.
- Reproducible output. The same approved plan and audio must always render the same video.
Architecture
The figure follows a topic from planning to a checked video, and a failed run back to its cached work.
- 01
A topic becomes a structured storyboard
Gemini returns a schema-constrained plan - scenes, narration and visual cues - and every visual must come from a trusted component registry. Invalid output gets one bounded repair attempt; the model never writes code that runs.
- 02
Nothing is narrated before it is approved
A reviewer approves the exact plan hash. Any change after that invalidates the approval, so paid narration only ever runs on reviewed content.
- 03
Timing comes from the voice, not the model
ElevenLabs returns character-level timing for each scene. The compiler maps cues onto real audio and converts them to frames; a cue that cannot be matched exactly stops the run instead of being guessed.
- 04
Rendering is deterministic
Remotion draws approved React components at compiled frames into a vertical 1080 x 1920 video. Rendering makes no network calls and uses no randomness, so the same timeline always produces the same frames.
- 05
Checked before it is called ready
Automated checks confirm codec, dimensions, frame rate, a 60-90 second duration and an audio stream, and a human reviews the video before it is marked ready.
- 06
Failures resume without paying twice
Every stage writes hashed artifacts atomically under a run lock. An injected render failure resumed with no provider calls, and a visual-only revision reused all seven narration clips.
Read the diagram as text
Components
- Creator
- Someone turning a topic and a few reference notes into a short educational video.
- Pipeline orchestrator
- TypeScript stage runner - preflight, plan, approve, speech, compile, render, validate - with checkpoints, run locks and resume.
- Gemini (planner) (external, not owned)
- Returns a schema-constrained storyboard - scenes, narration and visual cues - validated again by the application.
- Approval gate
- A reviewer approves the exact plan hash; any later change to the plan invalidates the approval.
- ElevenLabs (voice) (external, not owned)
- Narrates each scene and returns character-level timing used to place every caption and animation.
- Timeline compiler
- Maps narration cues onto real audio timing and converts them to exact frames; ambiguous cues are rejected, never guessed.
- Remotion renderer
- Deterministic React components drawn at compiled frames into a 1080 x 1920, 30 fps MP4 - no network calls or randomness.
- Run artifacts
- Plan, approval, audio, alignment, timeline and output per run, written atomically and hashed so they can be reused safely.
- Media QA
- Checks codec, dimensions, frame rate, 60-90 s duration and audio presence before a video can be called ready.
Flows
- A topic becomes a storyboard: Creator → Pipeline orchestrator → Gemini (planner).
- Only approved narration is voiced: Pipeline orchestrator → Approval gate → ElevenLabs (voice).
- Timing comes from the real voice: ElevenLabs (voice) → Timeline compiler → Remotion renderer.
- The render is checked: Timeline compiler → Remotion renderer → Media QA.
- A failed run resumes from cache: Pipeline orchestrator → Run artifacts → Timeline compiler → Remotion renderer.
Key decisions & trade-offs
The AI plans; trusted code renders
Gemini returns a structured storyboard that may only use visual components from a trusted registry, with typed parameters. It never generates code that runs. Letting the model write React components would have allowed any visual, but it would have made rendering unsafe and impossible to reproduce; a fixed component catalogue limits variety in exchange for output that is safe, testable and identical on every render.
Timing from the voice, not from the model
The model only names the phrase a visual should react to. Exact times come from the character-level alignment ElevenLabs returns with the audio, mapped to frames at 30 fps. Asking the model for timestamps would have been simpler, but they would drift from the real narration; when a phrase cannot be matched exactly, the run stops rather than guessing.
Approval bound to a hash
Approval records the hash of the exact plan that was reviewed, and any later change invalidates it. That keeps paid narration from ever running on content nobody has checked - one early AI draft overstated a fact, and review caught it before narration.
Hashed artifacts make every stage resumable
Each stage writes its outputs atomically under a run lock and records content hashes. Speech is reused only when its text, voice and settings match; renders only when every dependency matches. This made recovery cheap: an injected render failure resumed with no provider calls, and a visual-only revision reused all seven narration clips.
Results
- Three narrated explainers produced end to end - the water cycle, RAM versus storage, and binary search - at 1080 x 1920 with captions and synced diagrams.
- Conversational and operator revisions create new, separately approved versions while preserving the previous export.
- Render failures resume from saved artifacts without new provider calls; 23 automated tests and recovery scripts cover the pipeline.
- A Next.js application foundation is in place for the creator-facing workflow.
What I’d do next
- Build the four-step web workflow on top of the existing contracts, with per-user storage and durable background jobs.
- Grow the component catalogue so more kinds of topics can be illustrated well.
- Add automatic, bounded speech retries and a fact-checking step before approval.