AI Video Search Evaluation: Build a 30-Query Test Set

Evaluate AI video search with a fixed, representative media set and 30 queries written before testing. Cover visual meaning, cinematic language, metadata filters, ambiguity, known failures, and source-file handoff. Define relevance in advance, keep the footage and result depth constant, and report misses as well as successes. This produces workflow evidence—not a universal accuracy claim.

What This 30-Query Evaluation Produces

The output is a repeatable pilot that answers five practical questions:

• Does the system return a usable moment for real editorial requests?
• Does it point to the right file, scene, shot, or time range?
• Can verified metadata restrict the search without being confused with visual meaning?
• What kinds of queries fail, and how much review do false positives create?
• Can a selected result reconnect to the original media and move into the next workflow step?

This is a procurement and workflow test, not a scientific benchmark. NIST's TREC program describes retrieval evaluation around test collections, topics, and relevance judgments; TRECVID applies shared evaluation to video retrieval research. A 30-query internal pilot borrows the discipline of fixing inputs and judgments, but its findings apply only to the tested footage, queries, settings, and date.

Prerequisites: Freeze the Test Before Opening a Demo

Create a test manifest before evaluating any product. Record:

Item	What to fix	Why it matters
Media corpus	Files, versions, duration, languages, genres, and rights status	Prevents candidates from seeing different footage
Search unit	File, fixed window, scene, shot, frame, or transcript segment	Defines what a returned result means
Query set	Exact wording of all 30 queries	Prevents rewriting weak queries after seeing results
Relevance judgment	Known target and acceptable alternatives for each query	Makes “good result” reviewable
Result depth	For example, inspect the first 10 results for every query	Keeps review effort comparable
Settings	Model/index version, filters, language, and tuning	Makes reruns interpretable
Reviewer	Role, domain knowledge, and conflict resolution	Controls subjective variation
Test date	Date indexed and date searched	Product behavior changes over time

Use representative, rights-cleared media rather than an entire archive. Include ordinary footage, difficult material, duplicates, long takes, silent B-roll, interviews, poor lighting, multilingual speech, and at least a few items that should not match any query. Do not let a vendor quietly replace the corpus with a prepared demo library.

Define Relevance Before Search

For every query, write a target note before running it:

• Relevant: directly satisfies the request and can be used after normal editorial review.
• Partially relevant: contains the right subject or action but misses an important constraint such as shot size, timing, location, or rights filter.
• Not relevant: looks plausible but does not satisfy the request.
• No valid result expected: the corpus intentionally contains no qualifying footage. Returning nothing can be correct.

Relevance is task-specific. Person walking may be relevant to a broad archive request but not to low-angle tracking shot of a speaker walking onto a stage. Keep the written criteria beside the query so reviewers judge the same task.

The 30-Query AI Video Search Test Set

Replace names, projects, territories, and known targets with values that exist in your authorized test corpus. Preserve the six-category structure so the pilot tests more than attractive semantic-search examples.

#	Category	Example query	What it tests	Pre-test judgment
1	Visual	person opening a cardboard package	Subject plus action	Known untagged target
2	Visual	hands assembling a small device	Fine hand-object interaction	Known target and near-misses
3	Visual	crowd celebrating after a score	Multi-person action and context	Known target
4	Visual	rainy street with reflections at night	Weather, setting, and appearance	Known target
5	Visual	speaker writing on a whiteboard	Person, object, and action	Known target
6	Visual	empty office after working hours	Absence and scene context	Valid result may be subjective
7	Cinematic	wide establishing shot of a harbor	Shot size and scene function	Known wide shot
8	Cinematic	slow handheld follow shot through a market	Camera movement and pace	Known tracking sequence
9	Cinematic	static close-up with shallow depth of field	Framing and optical appearance	Known target
10	Cinematic	aerial push toward a coastline at golden hour	Viewpoint, motion, and lighting	Known target or no valid result
11	Cinematic	low-angle reaction shot after the announcement	Angle, editorial role, and temporal context	Known reaction moment
12	Cinematic	symmetrical product shot on a dark background	Composition and subject presentation	Known target
13	Hybrid	Project Atlas interview, filmed in 2025, subject smiling	Metadata plus visual meaning	Verified project/date filter and visual target
14	Hybrid	Brand A product close-up approved for paid social	Rights filter plus visual search	Only approved assets may qualify
15	Hybrid	Europe-cleared drone footage of a city at night	Territory plus scene	Rights field must control eligibility
16	Hybrid	Camera B shots of the presenter walking on stage	Camera metadata plus action	Known Camera B target
17	Hybrid	Episode 4 kitchen scene with an overhead shot	Production structure plus shot language	Known episode/scene target
18	Hybrid	Customer interview mentioning onboarding, status=approved	Transcript meaning plus workflow state	Approved transcript segment
19	Ambiguous	something that feels premium	Subjective visual concept	Acceptable examples defined in advance
20	Ambiguous	the moment before things go wrong	Narrative inference	Expect disagreement; record it
21	Ambiguous	fast energy without camera shake	Combined positive and negative constraint	Known alternatives
22	Ambiguous	not an interview, a person speaking to camera	Negation and adjacent concepts	Known non-interview target
23	Ambiguous	Apple	Entity ambiguity: fruit, brand, or object	Mark ambiguity, do not guess intent
24	Failure	snowmobile crossing a desert	No-result behavior	No valid result expected
25	Failure	licensed for global broadcast forever	Unsupported legal inference	Must require authoritative metadata
26	Failure	identify the person who signed the release	Identity and consent boundary	Must not infer from appearance
27	Workflow	quiet close-up after the interview answer	Temporal precision and surrounding context	Known shot and adjacent context
28	Workflow	best product demonstration for a 15-second edit	Retrieval plus human editorial judgment	Several candidates, no single truth
29	Workflow	source clip for the selected harbor shot	Source-file reconnection	Exact file path and time range required
30	Workflow	add three approved market shots to an edit-ready collection	Multi-result selection and downstream handoff	Collection must preserve source references

Queries 13–18 require trustworthy structured fields. Video Metadata vs Semantic Search explains why rights, ownership, approval, and production IDs must remain authoritative filters rather than model guesses. Queries 24–26 deliberately test whether a system refuses unsupported conclusions.

Run the Test Consistently

For each candidate system:

1. Index the same media and document any excluded or failed files.
2. Run the exact query text with the same permitted metadata filters.
3. Inspect the same result depth and save the ranked results.
4. Judge each result as relevant, partial, or not relevant using the prewritten rule.
5. Confirm the result's file, time range, and surrounding context.
6. Complete the downstream action: open source, create a select, or export to the intended editing workflow.
7. Record query rewrites, vendor tuning, crashes, indexing errors, and human interventions.

If a query is rewritten, retain both versions. The original measures natural usability; the revised query measures what is possible after learning the system's preferred language. Give every candidate the same tuning allowance.

Score Relevance and Workflow Fit

Use a scorecard that exposes misses and review effort rather than collapsing everything into one “accuracy” number.

Metric	Calculation or observation	Interpretation
Success@10	Queries with at least one relevant result in the first 10 / 30	Can the user find a usable candidate?
First relevant rank	Rank position of the first relevant result	How much result review is needed?
No-result correctness	No-result queries correctly returning no qualifying result / planned no-result queries	Does the system avoid confident noise?
Constraint pass rate	Hybrid queries where all hard metadata constraints are satisfied / hybrid queries	Are filters enforcing eligibility?
Temporal precision	Result points to the usable moment rather than only the containing file	How much timeline scrubbing remains?
Source-link success	Selected results that reopen the correct original and time range / selections tested	Can retrieval survive handoff?
False-positive review time	Minutes spent rejecting plausible but unusable results	Reveals hidden labor
Workflow completion	Tasks completed without an external workaround / workflow tasks attempted	Tests the whole job, not only search

Do not compare scores unless the corpus, query set, relevance rules, result depth, and reviewers are comparable. If two reviewers disagree, preserve both judgments and resolve them with a documented rule. A small pilot is useful for deciding the next test; it is not proof of performance across every archive.

Diagnose Failure Patterns

Group failures instead of treating every miss as the same problem:

• Corpus failure: the needed media was never indexed or could not be decoded.
• Granularity failure: the right file appears, but the useful moment is buried inside it.
• Visual-semantic failure: the system misses the subject, action, composition, or relationship.
• Transcript/audio failure: speech is absent, mistranscribed, or matched too literally.
• Metadata failure: fields are missing, stale, mapped incorrectly, or not enforced.
• Query ambiguity: reasonable reviewers interpret the request differently.
• Boundary failure: the system presents inferred identity, rights, or consent as fact.
• Handoff failure: the result cannot reconnect to the correct original or editing timecode.

This diagnosis tells the team whether to change the corpus, metadata governance, indexing unit, model, query interface, or workflow integration. It also prevents a metadata problem from being mislabeled as an AI-model problem.

Where ShotAI Fits

ShotAI's public semantic video search documentation describes natural-language retrieval across a user's indexed footage. Its shot-level management page describes shot-level assets, source references, collections, and editing-workflow export. Those capabilities make ShotAI a candidate for the visual, cinematic, temporal-precision, and source-handoff parts of this test.

ShotAI is not a public-web reverse video search engine, a legal rights authority, or a complete enterprise DAM/MAM for permissions, retention, approval, and distribution. Keep verified business metadata in the appropriate system of record and test the actual integration. The 30-query set does not guarantee a result or establish superiority; it gives every candidate the same questions.

For broader selection criteria, use the AI video search tools test framework and the video asset management buyer's guide.

Pilot Checklist

• Freeze representative, rights-cleared media.
• Write all 30 queries and expected judgments before testing.
• Include valid, ambiguous, hybrid, and no-result cases.
• Keep result depth, settings, and tuning allowance constant.
• Save ranked outputs and reviewer decisions.
• Verify source file, time range, and surrounding context.
• Complete the downstream workflow instead of stopping at search.
• Report limitations, interventions, and test date with every score.

FAQ

Why use exactly 30 queries?
Thirty is a manageable pilot size that can cover six distinct failure and workflow categories. It is not an industry standard or enough for a universal benchmark; expand it after the pilot reveals important gaps.

Should every query have a known correct result?
No. Include ambiguous requests and queries with no valid result. A useful system should surface uncertainty and avoid inventing matches when eligibility or evidence is absent.

How many results should reviewers inspect?
Choose a fixed depth before testing, such as the first 10, based on realistic user behavior. Apply the same depth to every system and disclose it with the findings.

Can metadata and semantic search be scored together?
They can be tested in one workflow, but judge them separately. Metadata filters establish eligibility; semantic ranking orders the eligible footage by meaning.

What makes a source-file handoff successful?
The selected result must reopen the correct original asset at the intended time range and retain enough identity for the next editing, review, or export step.

Disclosure

This evaluation guide is published by ShotAI. It does not report a ShotAI benchmark or claim that ShotAI wins this test. Teams should run the protocol on representative footage and publish their own conditions and limitations.

Sources

• NIST: Text REtrieval Conference (TREC) Overview
• NIST: TREC Data—Test Collections and Evaluation Resources
• NIST: TREC Video Retrieval Evaluation (TRECVID)
• Radford et al.: Learning Transferable Visual Models From Natural Language Supervision
• Luo et al.: CLIP4Clip—An Empirical Study of CLIP for End-to-End Video Clip Retrieval
• ShotAI: Semantic Video Search
• ShotAI: Shot-Level Management