⟶ ALPHA ENGINEER Build In Public

AI Tools to Cut Raw Footage Into Reels — Why Most Whiff on B-Roll

If you have ever been handed a folder of unlabelled client footage and told to make Reels out of it, you already know the auto-highlight tools do not save you. They are not broken. They are built on an assumption your footage violates, and once you see the assumption you can stop shopping and start solving the right problem.

The complaint that keeps repeating

Spend ten minutes in the small-business and social-marketing forums and the same post cycles through every few weeks: "Is there any tool that can go through raw footage and find the good moments?" The replies recommend the usual auto-editors. Then, reliably, somebody who has actually tried them says the quiet part:

"auto-highlight tools are built for long podcasts and tend to whiff on short b-roll"

That single line is the whole diagnosis, and it is worth spelling out because it changes what you should be looking for.

The assumption you are violating

Nearly every mainstream auto-editor is a transcript-first system. The pipeline is: transcribe the audio, score the sentences, keep the high-scoring ones, cut on word boundaries. That design is excellent for a two-hour podcast or a talking-head webinar, where the words are the content and there is a dense signal to score.

Now feed it b-roll. Handheld shots of a storefront. A slow pan across a plated dish. Someone walking through a door. The transcript is empty, or it is thirty seconds of traffic noise and a half-audible "okay, again". There is nothing to score. The model is not making bad decisions — it has no input at all.

This is why the fix is never a different transcript-based tool. Anything that leads with speech-to-text will fail in the same place. What you need is a system that looks at the pictures.

So I tested exactly that

I run a small self-built pipeline for my own short-form output. Its first stage is deliberately vision-first: a multimodal model watches the video and returns timestamped segments describing what is on screen, with the transcript as one optional field among several. I had never pushed it with genuinely dialogue-free material, so I built the worst case on purpose.

Stage 1 — what the vision model returned

Trimmed from the actual output file:

"language": "no voiceover",
"segments": [
  { "start": 0, "end": 8, "speech": "", "energy": "low",
    "visual": "seated by a warm window reading,
               then looks up and smiles" },

  { "start": 26, "end": 35, "speech": "", "energy": "mid",
    "visual": "in an apron, absorbed in painting
               at a studio desk, then a smile to camera" },

  { "start": 44, "end": 53, "speech": "", "energy": "mid",
    "visual": "jogging lightly along a green park path" }

  ... 8 segments total, every one with speech: ""
]

Two things matter here. First, every single speech field is an empty string — this is precisely the input a transcript-first tool would be working from, and it is nothing. Second, the visual and energy fields are populated for all eight segments, which is a usable signal where the transcript had none.

It also found the cuts by itself

I never told it there were eight clips. It returned eight segments — the correct count — with boundaries at 0/8/17/26/35/44/53/62/71 seconds against true splice points of 0/9.04/18.08/27.13/36.17/45.21/54.25/63.29/72.33.

Being honest about that number: every boundary sits 1.0–1.3 seconds early, and the error grows monotonically. That is a sampling/rounding drift (the model answers in whole seconds), not a scene it got wrong. Shot detection: 8 of 8. Frame-accuracy: not there, and you would not want to trust it for a hard cut without a nudge.

Stage 2 — the edit decisions

The segment list then goes to a reasoning model that returns a cut plan. It kept 7 cuts totalling 36 seconds, and the interesting part is not the trimming, it is the reordering and the rejection:

Cut	Source time	Role	Stated reason
1	26–29.5s	hook	most engaging shot — pulled to the front
2	0–5s	body	establishes the calm tone
3	8–13s	body	strong light, good atmosphere
4	17–22s	body	contrast shot
5	44–49s	body	motion — lifts the pace, avoids a flat middle
6	53–58s	body	brings the emotion back down
7	62–69s	cta	quiet ending, room for the call to action
—	35–44s	dropped	not selected at all

The hook is the fourth clip in the timeline, promoted to position one. One of the eight scenes was thrown away entirely. The stated logic for the running order was quiet → motion → quiet, so the middle would not sag. That is the actual job people are asking for when they say "find the good moments" — not just trimming dead air, but choosing and sequencing.

The clock

Stage	Time
Vision analysis (72s of footage)	22s
Cut planning	22s
Render + burn-in subtitles	14s
Total, unattended	58 seconds → a 35.5s vertical cut

API cost for the run was a couple of US cents. I pulled a frame at 1.5s to confirm the hook overlay had actually burned in rather than silently failing, which is the check I would recommend to anyone automating a render step.

The workflow, whether or not you write code

The transferable part is the order of operations, not my particular stack.

If you are hiring the tools rather than building

Ask one question before you pay for anything: does it look at the frames, or only at the words? Vendors describe this in their own way — "scene detection", "visual understanding", "multimodal" — but the test is simple. Feed it sixty seconds of your silent footage during the trial. If it returns nothing useful, that is the architecture answering, and no amount of settings will change it.

Where I landed

The recurring forum complaint is not a gap in the market for a better transcript tool. It is people bringing visual material to a linguistic system. Once the first stage of your pipeline is something that genuinely watches the footage, "go through this dump and find the good bits" turns into a solvable, and surprisingly fast, problem — 58 seconds unattended in my run, on the least helpful input I could construct.

Whether that holds up across a real 50-clip client batch is the next thing worth measuring, and I will publish that number too.