Choose the extraction output before the analyzer

Start with the downstream consumer. An accounts-payable practice record may require vendor, invoice number, currency, dates and line totals; a searchable training video may require a timestamped transcript and scene descriptions. These are not interchangeable schemas. The model should return explicitly absent values as unknown, not invent them to fill a mandatory field. Define field types, allowed empty values, source-reference requirements and a validation plan before configuring the analyzer.

Create a simple synthetic invoice with an ambiguous date and a small two-row table. Mark the correct values by hand and note which page or visual region supports each. If a field cannot be verified from the image, annotate that as expected uncertainty. For an audio example, prepare a transcript with one speaker correction and a clear time range. These labeled fixtures provide a way to evaluate the service rather than relying on the apparent plausibility of generated JSON.

For the invoice fixture, include a document number printed twice, once in the header and once in a purchase-order reference box. A naive extractor could choose the wrong one while satisfying the schema. Mark the intended invoice number and the supporting page region in the reference answer. Include a missing-vendor case in which “unknown” is expected. This separates extraction correctness from JSON completeness and makes review useful when a field appears plausible but belongs to the wrong part of the document.

Distinguish OCR, layout and structured fields

OCR recognizes visible text from an image, while layout analysis identifies reading order, tables and page relationships. A field extractor then assigns meaning such as “invoice total” or “due date.” Correctly reading every character does not guarantee the right value is mapped to a label. A scanned form may contain two dates or repeated rows that require different validation. Preserve original evidence so a human can inspect why a model chose a particular value.

In a two-page purchase order, a table header may appear only on the first page while rows continue onto the next. Flattening the entire document to one text string can detach a quantity from its unit or product. Test whether the selected representation preserves the table structure and page identity. Keep raw recognized text and normalized business fields separate; a reviewer must be able to see both. A well-formed schema alone cannot tell whether a value came from the wrong row.

Pin a supported API version and modality

Microsoft Content Understanding documentation distinguishes generally available capabilities from newer preview features. The 2025-11-01 GA API is the documented production baseline for supported standard analysis, while 2026-06-01-preview introduces additional evaluation features, including agentic document analysis. Historical pro mode from 2025-05-01-preview was retired. These versions are not drop-in synonyms; supported modality, field methods and request limits can differ.

Do not use document-only agentic preview for an audio timeline simply because it has an impressive name. Record endpoint, API version, analyzer type, region, supported input formats and maturity state for the exact task. If a Microsoft Learn sample shows older syntax, reconcile it against current reference documentation before running it. A successful preview demo is not enough to promise general availability or contract terms to a production customer.

Create two configuration records instead of one general statement that a feature is “available”: one for a standard document analyzer with GA API, and another for an experimental document-only preview analyzer. Record supported field methods and whether the preview handles the number of inputs your task needs. A multi-file case may work under an older architecture but not a replacement preview with a one-input limit. Versioned tests reveal these distinctions before they become operational surprises. The exam asks for recognition of supported approaches, not blind reuse of retired API syntax.

Design a simple schema and validate it in code

An extraction schema should distinguish text actually present in the source from categories or summaries inferred by a model. For a synthetic receipt, choose supplier, item list, subtotal, currency and tax. Use nested objects for repeating lines rather than unrelated parallel arrays. Require source page or span for a value that could trigger a financial action. A generated opinion or classification must not be labeled as though the document directly asserted it.

Validate the output using deterministic rules: correct data types, allowed currencies, plausible date ranges, line-item arithmetic and presence of required source references. If the invoice total does not match its lines, report an exception rather than silently changing the printed total. If a date is ambiguous across locales, retain the raw string and ask for clarification. A confidence value from one model is not automatically a calibrated probability for every business field.

Preserve source locations and review uncertainty

An extracted value should be accompanied by the original file identifier and, where supported, a page number, text span, bounding region or related structural element. A reviewer needs to navigate to the source instead of trusting a summarized label. Distinguish a field missing from the document from a recognition failure and from a conflict between two visible values. These conditions require different corrective actions.

Design one intentionally low-quality page with a rotated number and one blank signature field. A successful pipeline might leave the amount unconfirmed and route it to review. Record the initial machine value, reviewer correction, reason and model/analyzer version without overwriting the original evidence. For a risk-bearing invoice, do not automatically approve payment based on an extracted number even if all syntax checks pass. Document understanding supports human and application decisions; it does not replace business authority.

Design the reviewer screen around the disputed field. Show the selected value, page image, bounding box or text span and any alternative candidate. Do not ask the person to hunt through an entire document to find the error. When an invoice total disagrees with computed line items, display both and the failed arithmetic rule. Record who corrected it and what evidence justified the change. This lets a later evaluator distinguish an OCR misread from a business-rule exception instead of hiding both under “low confidence.”

Analyze audio without losing the timing

Audio content can be turned into timed speech segments, topics or structured fields when supported by the selected analyzer. Speaker separation, multilingual recognition and background-noise handling must be verified for the exact model and input. A fluent combined transcript can misattribute a deadline to the wrong person. Store segment times and identifiers alongside the words and keep uncertainty visible when overlapping voices make attribution unreliable.

Test a fictional support call in which one speaker first states an incorrect date and then corrects it. The extraction should preserve the correction and its source time rather than choosing whichever sentence reads more confidently. Evaluate critical names and numbers, not only average word error rate. Consent and retention apply to both audio and the derived transcript, which can be easier to search and share than the original file. Never upload real private recordings to an open test project.

Treat video summaries as evidence-linked interpretations

Video adds visual frames, motion and possibly spoken narration. A timestamp can connect a generated event description to a source interval, but the model may infer a cause not actually visible. A speaker saying that a safety guard is in place does not prove the guard can be seen. Preserve the spoken statement and observed visual features separately when they conflict, and let an authorized reviewer inspect the original segment.

Choose a short permitted training clip or create a synthetic storyboard. Ask for a list of observable scene changes with timestamps, then a separate summary of speaker instructions. Include one deliberate contradiction between words and visuals. The expected result is a visible disagreement or uncertainty, not a fabricated reconciliation. Also test a silent clip and one with small on-screen text. These cases help candidates distinguish extraction, description and inference in a multimodal workflow.

A training clip may show a technician turning a switch while the narration describes a different step. Build one test where the order of visual actions is clear and another where the switch is obscured. Ask for a time-coded action list and a separate transcript rather than a single fluent story. A safe analyzer should say which action is observed and where evidence is missing. That distinction is especially important if someone might use the summary as proof of completed maintenance.

Secure and evaluate the whole evidence lifecycle

Derived text, structured fields, images, source coordinates and summaries should inherit the underlying document or media access constraints. A user with permission to view a summarized topic may not be authorized to read the entire recording. Do not replicate sensitive extracted content into broad debug logs or open search indexes. Track original checksum, analyzer version, validation rules, reviewer decisions and downstream actions in an access-controlled audit trail.

Evaluate a varied small test corpus by document family and modality: legible PDFs, noisy scans, absent fields, short audio with correction, and a clip with conflicting narration. Report field-level correctness, unsupported-value invention, timing alignment, review effort and task outcomes separately. At the end, identify one failure where the right behavior is to stop or request review. The useful result is attributable evidence, not the largest possible JSON response.