Microsoft AI-103: Managing Secrets in Azure AI Apps
An AI prototype can work with a key pasted into an environment file, but that convenience becomes a production liability when notebooks are shared, workloads scale or tools reach more sensitive systems. An agent may call a model endpoint, search index, database and external service, each with its own credential and permission scope. Microsoft recommends managed identities for supported Azure services and Azure Key Vault for secrets that cannot be eliminated. Secure implementation requires an inventory of credential owners, least privilege, rotation and safe failure behavior. A secret stored in a vault is not automatically safe if every developer or agent can retrieve and misuse it.
On this page
Eliminate credentials before storing them
Inventory every authentication path and determine whether Microsoft Entra ID or another supported workload identity can replace a static key. A managed identity can avoid distributing a durable secret for many Azure resource interactions, provided the required data-plane or control-plane roles are explicitly assigned. Do not assume that obtaining a token grants access to all resources in the subscription. Scope permissions to the necessary service and operation, and confirm token audience, tenant and policy during testing. A Microsoft AI-103 secrets-management design should minimize credential exposure while still supporting rotation, auditing, and recovery.
For external systems that still require an API key or password, record the business justification and owning service. Separate user-delegated access from application-level credentials. An agent acting on behalf of a user may need explicit consent for a tool action even when the application has a broad integration credential. Storing that credential in a secret manager does not authorize every proposed use of it.
Keep secret values out of model context
Never paste secrets into system prompts, retrieval documents, test transcripts or tool messages for a model to “remember.” The model should select among preauthorized operations while trusted application code attaches credentials after validation. If a tool requires an API key, the application should retrieve it securely and prevent the model from reading or returning the value. Redact sensitive headers and payloads from logging, error messages and distributed traces.
Review indirect exposure paths. A failed HTTP call might log a full request URL containing a token, or a notebook might display a connection string in output saved to a shared workspace. Restrict prompt and trace retention according to data sensitivity. A security test should attempt to coax the assistant into echoing environment variables or tool configuration and confirm that the application boundary prevents it even if the generated text is persuasive.
Store unavoidable credentials in Key Vault
Use a vault configured with appropriate network restrictions, audit logging, deletion protection and role-based permissions where applicable. Grant the workload only the specific secret-read rights it needs and restrict modification or deletion to a separate administrative process. A secret version should be associated with an owner, purpose, expiry policy and rotation procedure. Avoid granting a shared “get every secret” role to an application that calls one downstream API.
Plan runtime access carefully. A managed identity authenticating to Key Vault still needs appropriate permissions and network reachability; missing private DNS or a denied role assignment can look like an application failure. Cache secrets only within an approved lifetime and invalidate them after rotation. The application should fail safely if a secret is unavailable rather than falling back to an undocumented hardcoded credential or another tenant’s key.
Rotate without turning maintenance into an outage
Secrets have different rotation patterns. Some systems support two simultaneously valid credentials, enabling a client to switch before retiring the old one; others require a coordinated single-key change and may have a short risk window. Azure Key Vault rotation can use events and automation for supported scenarios, but engineers must design the downstream update and verify service continuity. Setting a new vault version does not automatically update every process already holding the old value.
Suppose an application must temporarily use a third-party API key because that external system does not support the organization’s workload identity. Put the secret in a managed secret store under a narrow application identity, and keep the key out of notebooks, prompts, connector definitions and diagnostic URLs. A rotation plan should determine whether the provider permits two simultaneously active keys or requires a brief switchover. When dual credentials are supported, issue the replacement, deploy consumers, verify successful authenticated calls and only then revoke the previous value. If dual credentials are not available, prepare a maintenance and rollback procedure that reflects the actual outage risk. Track which workloads depend on the secret so an unused test worker cannot silently continue using the retired key.
Test failure handling when the key is denied or expired. The application should return a controlled service error rather than inviting a model to invent a new credential, retry indefinitely or expose the token in exception logs. Investigate access traces that show unusual use after rotation, and document the incident response if a secret is suspected compromised. Prefer a managed identity for Azure-owned resources wherever supported and review Key Vault access and data-plane roles separately from broad resource management permissions. Credential security is both a design and an operating discipline: a secret moved from source code to a vault without least privilege and effective rotation remains an avoidable liability.
Test rotation in staging and record what happens to running agents, queued workflows and long-lived clients. A tool call may begin with one credential and finish after its expiry; the application should recognize authentication errors and use controlled refresh logic where safe. Do not retry a side-effecting transaction blindly after a credential-related timeout. Use idempotency keys and authoritative readback to decide whether the business action was completed.
Bound access for multi-tool agents
An agent that can choose among tools should not inherit one powerful service account that can execute all their operations. Use tool-specific identities or narrowly scoped integration credentials when the platform supports them. Verify the caller’s current authorization before invoking each tool. A retrieved page telling the agent to send a report externally is untrusted data, not permission to use an outbound messaging key. Restrict agent tool endpoints, arguments and allowable data movement independently of the model’s proposed action.
Review service principals and managed identities periodically. Remove permissions for retired tools and experiments, and investigate unusual token requests or secret-read activity. A former demo endpoint can remain an attractive attack surface if its key is never retired. Credential inventory should include non-production environments, evaluation jobs and CI/CD systems because they often contain secrets that production governance overlooks.
Detect compromise and prepare a response
Monitor Key Vault access and relevant service authentication failures, but avoid collecting secret values in the same monitoring system. Define escalation for an unexpected bulk secret read, exposed key in source control, suspicious external tool request or unexplained API use. Revoking a compromised credential may disrupt dependent workloads, so keep an incident plan with identified owners and secure replacement paths.
After rotation or revocation, verify that old credentials no longer authenticate and that intended workflows use the new identity. Inspect logs, notebooks and indexes for leaked values under the security policy; removing a key from code does not remove historical copies. If a model may have processed an exposed token, treat it as compromised rather than assuming the model will forget it. Secure secret management is measured by reduced standing access and reliable recovery, not merely by the existence of a vault resource.
Keep credential decisions part of release QA
Before publishing a new AI feature, review its data flows, identities, secret dependencies, approved domains, rotation behavior and failure states. Test a denied identity and an expired key, not only the happy path. Verify that model-generated tool arguments cannot influence credential selection or expand privileges. Record which credentials change with a release and how to reverse the update safely.
A production Azure AI application should be able to explain which identity performs each action and why it holds the necessary rights. Managed identities reduce stored secret exposure, while Key Vault protects unavoidable credentials under explicit lifecycle controls. Both are only effective when the application enforces authorization and keeps secret material out of prompts, traces and untrusted content.