Practice Exams:

Microsoft AI-103: Hybrid Search and Reranking

AI & Machine Learning

A user can describe a problem without using any of the words stored in the answer. Another user might ask for an exact error code that must not be paraphrased at all. Those two requests should not be forced through one retrieval strategy. Azure AI Search offers lexical, vector and hybrid search, with optional semantic ranking, but the features are useful only when their ranking behavior matches the application’s questions. The engineering challenge is to build a query plan that keeps exact matches discoverable, retrieves genuinely related concepts and gives the generator a defensible evidence set.

On this page
  1. Understand what each retrieval signal can and cannot find
  2. Start with a query set that exposes opposing failures
  3. Build a hybrid query with complementary fields
  4. Use semantic ranking after candidate retrieval
  5. Filter for correctness and authorization before ranking becomes persuasive
  6. Decide how many results the model actually needs
  7. Diagnose ranking failures with a controlled experiment
  8. Measure relevance alongside latency and cost
  9. Keep the search strategy distinct from the overall RAG product

Understand what each retrieval signal can and cannot find

Lexical search evaluates words and their relationship to indexed text. In Azure AI Search, common full-text relevance scoring is based on BM25, which considers how terms occur across documents. This is often a strong choice for error numbers, model names, product codes and formal policy labels. If the question includes ERR-412, losing that identifier can be much worse than missing a conceptually similar paragraph. A realistic Microsoft AI-103 retrieval experiment compares vector, keyword and semantic signals on the same difficult queries before choosing a combined ranking strategy.

Vector search compares a query representation with indexed embeddings. It can connect a user’s phrase such as “the laptop keeps disconnecting from the office network” with documentation titled “Intermittent VPN session failures.” That is valuable when the vocabulary of the user and author differs. But nearest-vector ranking is a similarity measurement, not a guarantee of correct context, date or authorization. Two products can share language while requiring different troubleshooting procedures.

Hybrid search runs lexical and vector retrieval as complementary signals and combines ranked results. Microsoft documents reciprocal rank fusion as the mechanism for merging the parallel rankings in Azure AI Search.

An item that performs well under both signals often deserves attention; an exact identifier that performs poorly as an embedding can remain visible through the text query. This does not mean hybrid is automatically better for every workload; evaluate it against your question set.

Start with a query set that exposes opposing failures

Build a small test collection with deliberately different query classes. Include exact identifiers, acronym-heavy technical questions, natural-language paraphrases, multi-part requests, short ambiguous questions and documents that share vocabulary but answer a different topic. Attach relevance judgments to each question rather than grading results by whether an answer sounds credible.

For example, an operations index might contain a runbook for SRV-184 and another for a similarly named storage feature. A lexical query should place the exact runbook high; a vector query may instead favor the longer conceptual description.

In a second test, “How do I rotate a service’s credentials without downtime?” should retrieve a runbook with “zero-interruption secret replacement” even if the first wording does not appear. Neither test can be replaced by a random collection of everyday chatbot prompts.

Record expected documents, query context, tenant or department restriction and acceptable rank position. Evaluate whether the right evidence appears at position 1, among the first several results, or not at all. The threshold depends on how many passages the downstream model can effectively use. The important question is whether the query layer returns source material that can support a justified response, not whether it emits a large result set.

Build a hybrid query with complementary fields

A searchable index should keep human-readable text and the vector representation tied to a stable source identity. Text fields carry the terms that BM25 can examine. Vector fields carry embeddings generated from an appropriate model. Structured fields such as document type, product, environment, permission group and effective date support filtering rather than similarity. If those fields are missing, the query must solve with ranking what should have been resolved by business rules.

For an enterprise support question, the query plan might include a keyword search over title, summary and body text; a vector query over the chunk embedding; a mandatory product-family filter; and an active-document condition.

Start with one vector signal unless the use case demonstrates a need for more. Multiple vector fields and large candidate sets increase resource use and make evaluation harder when the problem is simply that the right product filter was absent.

Azure AI Search hybrid queries bring ranked streams together using RRF. Because different scoring systems contribute to the combined rank, do not compare a raw BM25 score directly with a cosine similarity value and pretend they are on one universal scale. Examine the returned ranking behavior and the query’s debug or diagnostic information where supported. Tune with observed relevance judgments rather than with guesses about a perfect score.

Use semantic ranking after candidate retrieval

Semantic ranking is a second-stage relevance process, not a substitute for bringing the correct documents into the first candidate set. Microsoft’s semantic ranker can reorder text-based and hybrid query results using language understanding of the document content and the query. If the relevant source never entered that initial set, a reranker cannot recover it by magic.

A useful experiment compares three configurations on the same labeled questions: lexical-only, lexical-plus-vector hybrid, and hybrid with semantic reranking. Keep filters and corpus version fixed. If reranking improves natural-language troubleshooting questions but harms exact product-code lookups, consider query-type-specific handling rather than globally forcing the same plan for every request.

Semantic ranking may also provide extractive captions or answers for search experiences. Those are useful for inspecting why a document appears relevant, but they should not automatically be treated as a complete final answer to a high-stakes operational question. The application still needs to evaluate context, permissions and whether a retrieved excerpt contains the full precondition for an action.

Filter for correctness and authorization before ranking becomes persuasive

A high ranking does not make a result appropriate for the user. A document tagged “production” should not be used for a question explicitly scoped to a testing environment. A previous version of a warranty policy should not outrank the active one solely because the wording looks closer. Define tenant, version, region, product and status filters according to the application’s real business rules.

Permission enforcement is even more important. A search service credential that can access a shared index is not proof that each end user can read every document. Microsoft documents several approaches to document-level access control, including native permission-aware mechanisms and security-filter patterns. Some capabilities depend on API version or preview status. Choose a supported implementation and test both positive and negative access cases; never rely on a text instruction to the model to hide retrieved private content.

Also distinguish filtering from boosting. “Only documents this caller may read” is a mandatory exclusion rule. “Prefer newer troubleshooting notes when both are applicable” may be a ranking preference. Treating both as soft relevance weights creates subtle data exposure and factual errors. The difference should be explicit in the query builder and test evidence.

Decide how many results the model actually needs

A top-k value is a product configuration, not an exam answer. Too few candidates increase the chance of missing a supporting passage; too many bring duplicated, contradictory or loosely related text into the prompt. If five returned chunks all come from the same duplicated document, the effective evidence diversity may be only one. Use source-aware deduplication and consider whether passages from different versions or departments must be kept separate.

Try an evidence-packing policy. One short answer might need one authoritative chunk and its title. A comparison request may need two different documents and a source timestamp. A multi-step troubleshooting request could require a procedure section plus a safety prerequisite. The generation layer should receive enough context to support the claim, not all retrieved text that fits into a token budget.

When a result is near the cut-off, ask whether it adds a new condition, exception or source. If it repeats a better passage, exclude it. Retrieval quality improves when the model sees fewer contradictions and can trace each material assertion to a specific source. An interactive search interface may benefit from a longer result list than a short generated answer; these are different consumers of the same index.

Diagnose ranking failures with a controlled experiment

Suppose users ask why a service returns HTTP 429, but the highest result is a general networking page. The correct answer may be in a rate-limit document that uses the phrase “throttling response.” Begin by confirming the correct document was indexed and its metadata is current.

If it is missing, changing the query will not solve ingestion. If it exists but fails keyword search, inspect analyzers and terminology. If it passes vector search but fails hybrid ranking, inspect how the candidate streams are combined.

If the correct passage appears at position eight but the final prompt includes only five, experiment with candidate depth and semantic reranking. If it appears prominently yet the generated answer ignores it, the defect has moved from retrieval to evidence packing or generation. This separation prevents the team from “fixing” a model prompt when the actual issue is a dropped chunk.

A practical test record includes query, expected document IDs, corpus snapshot, retrieval configuration, rank positions, filters, latency and an investigator’s decision. Keep poor-result examples, because improvements on average can hide regressions on exact identifiers or restricted topics. A small bank of hard questions maintained by real support engineers often reveals more than a large synthetic set of polite questions.

Measure relevance alongside latency and cost

Vector and semantic search incur different compute and pricing considerations, and the optimal plan depends on index size, region, model choices and the application’s latency target. Azure AI Search also has vector index quota and storage rules that vary by service characteristics. Adding more chunks or larger embeddings changes capacity requirements even when the application interface looks the same.

Measure retrieval latency by stage and track the share of questions with an acceptable source in the first results. For security-sensitive use cases, also check denied-result leakage under different authenticated users. If a reranker improves relevant-at-k only slightly but doubles user-perceived delay, decide whether the improvement justifies the cost on the questions that need it. Do not hide a slow search behind a fast model demo.

Index changes deserve controlled comparison. An embedding-model upgrade may require re-embedding previously indexed content and ensuring the query vectors use compatible representations. A lexical analyzer change can affect exact-code lookups. Build a test index or an isolated environment, compare performance and quality, and retain a rollback path before changing the production query plan.

Keep the search strategy distinct from the overall RAG product

The query layer answers a narrow question: which authorized, current passages most likely support this request? It should not be responsible for defining the organization’s policy, inventing missing facts or deciding what the model is allowed to disclose. A broader RAG architecture must handle ingestion, source ownership, freshness, grounded response format and continuous operations. Those subjects belong together, but they should not collapse into a single article or an untestable “search quality” setting.

Begin with lexical and vector baselines, introduce hybrid retrieval where the questions justify it, and test semantic reranking against a labeled set. Apply mandatory filtering before celebrating relevance gains. Then look at the actual evidence the model receives, not only the order returned by the search API. The best retrieval plan is the simplest one that reliably puts the right, permitted source in front of the application when it matters.

Continue learning

Related guides

Azure AI Search for RAGDesign Azure AI Search indexes, retrieval filters, and grounding paths.Chunking Strategies for RAGDesign candidate passages before tuning retrieval ranks.Grounding with Enterprise DataApply evidence and ownership rules to answer generation.Testing AI Prompts on AzureTest the response layer after retrieval has been validated.

Related Posts

• Your Ultimate Guide to Crushing the Microsoft AI-102 Exam

• Is the Microsoft Azure AI Engineer Badge Worth Your Time and Effort?

• Think Smart, Build Smarter: Mastering AI-900 with Microsoft Azure

• Microsoft AI-300: Feature Stores Solve Coordination Before Performance

• Microsoft AI-300: Serving GenAI: Balancing Latency, Throughput, and Cost

• Microsoft AI-300: Reproducibility Is the First Test of Production ML

• Microsoft AB-620: Give an AI Agent a Job Before More Tools

• Microsoft AB-100: Integration Design for AI-Powered Business Processes

• Microsoft AI-103: Managed Identities for AI Apps

• Microsoft AI-103: Multimodal Vision Workflows