Microsoft AI-103: Secrets Management for AI Apps
Secrets management for AI applications starts with a simple priority: do not create a secret when identity-based authentication can solve the same problem. Managed identities and Microsoft Entra ID can remove API keys and client secrets from many Azure-to-Azure connections. When a workload still needs a secret—for a third-party API, legacy service, or connection that cannot use Entra ID—Azure Key Vault provides controlled storage, access, and rotation.
Microsoft Foundry also supports Key Vault-backed connections for scenarios that require stored connection secrets. Current documentation notes important operational limits around bring-your-own Key Vault connections, including the need to preserve the vault and its secrets while dependent Foundry connections exist.
Secrets therefore belong inside Azure AI engineering as part of identity, deployment, and operational design.
Prefer identity over stored credentials
A managed identity gives an Azure resource its own Entra identity without requiring developers to distribute a reusable password or key.
Managed identity is especially useful for calls from Functions, App Service, Container Apps, virtual machines, and other Azure-hosted workloads into Azure OpenAI or related Azure services.
The credential still needs RBAC. Secretless does not mean permissionless.
Use Key Vault for secrets that must exist
Some external systems still require API keys, shared secrets, or certificates. Store those values in Key Vault rather than source code, environment files committed to repositories, or agent instructions.
Retrieve secrets at runtime through the application’s managed identity where possible. This keeps the secret store and the application identity separate.
Do not copy Key Vault values into long-lived configuration merely to make deployment easier; that defeats the lifecycle benefit.
Keep secrets out of prompts and memory
A prompt is not a secret store. Anything placed in model context can appear in traces, debugging output, evaluation datasets, or model responses if another control fails.
Agents should call server-side tools that use credentials internally rather than receiving the credential as text.
Prompt injection becomes far more dangerous if the model has access to credentials that should have remained outside its context.
Separate runtime and deployment permissions
The identity that deploys infrastructure may need permission to create connections or assign roles. The runtime service usually needs far less.
Azure RBAC should separate scope from role so an application can read one required secret or invoke one service without gaining broad administrative control.
This also makes incident response easier because runtime compromise does not automatically grant deployment authority.
Rotate secrets with a migration path
Rotation is not only generating a new value. The application needs a way to accept the new credential, update configuration, validate connectivity, and retire the old value without unnecessary downtime.
For third-party services, understand whether multiple active keys are supported. If not, the release may require a tighter cutover.
Record the owner, rotation interval, and dependent services for every long-lived secret.
Foundry connections need lifecycle ownership
Foundry connections can depend on secrets stored in Key Vault. Deleting the underlying vault or required secret can break dependent project or resource connections.
Current Microsoft guidance also notes limits around migration and bring-your-own vault connections. Treat the vault as production infrastructure rather than a convenience resource that can be replaced casually.
Connection ownership should therefore include both the logical connection and the secret lifecycle behind it.
Do not leak credentials through logs
HTTP headers, connection strings, exception objects, and debug output can expose credentials if logging is too broad.
Redact authorization headers and secret values at the logging boundary. Use request IDs and nonsecret metadata for troubleshooting.
AI observability should help reconstruct requests without turning the telemetry store into another secret repository.
Use separate secrets by environment
Development, test, and production should not share the same third-party API key merely because it is easier. Separate credentials reduce blast radius and make usage attribution clearer.
Production secrets should not be available to local developer code unless there is a specific operational need.
Use environment-specific Key Vault instances or strict secret naming and RBAC boundaries appropriate to the organization’s architecture.
Secrets management is a reduction exercise
The best secret is often the one the architecture no longer needs. Replace keys with Entra authentication, replace direct client access with server-side tools, narrow RBAC, and remove unused connections.
For current AI-103 work, the durable pattern is to minimize secret creation, keep unavoidable secrets in Key Vault, separate runtime from deployment authority, and keep credentials outside prompts, logs, and model memory.
Secret inventory is the first operational step. List every API key, connection string, certificate, client secret, and third-party token the application depends on, along with owner, environment, purpose, rotation method, and expiration. Unknown secrets are difficult to rotate and even harder to revoke during an incident.
Applications should fetch secrets lazily enough to avoid unnecessary exposure but efficiently enough not to call Key Vault on every small operation. A common pattern is to retrieve a secret through managed identity and cache it in process for a short controlled period. The cache should respect rotation needs and should never be written to logs or local disk.
Certificates and signing keys need lifecycle planning just like passwords. Track expiration, renewal, trust-chain changes, and the services that depend on them. Automated renewal is valuable only when downstream consumers actually pick up the new certificate before the old one expires.
Development workflows need safe substitutes. Do not solve local convenience by giving every engineer direct access to production secrets. Use development credentials, mock services, or isolated nonproduction vaults. Production access should be limited to people and workloads with a real operational reason.
Private networking can also reduce secret exposure by keeping Key Vault and dependent services on approved network paths, but networking does not replace identity. A private vault with broad read permissions is still overexposed.
Emergency rotation should be rehearsed. The team should know how to revoke a compromised credential, update the dependent application, validate recovery, and confirm that the old secret is no longer accepted. That process should not be invented for the first time during an incident.
Secrets often enter CI/CD through deployment variables. Protect pipeline logs, restrict variable visibility, and prefer workload identity federation or managed service connections where available. A secure runtime is undermined if the deployment pipeline exports the same secret into plain text during release.
For AI applications specifically, inspect evaluation and tracing systems for accidental credential capture. Tool arguments, HTTP headers, and exception messages can flow into telemetry. Redaction should happen before data reaches observability storage.
Credential discovery should include transitive dependencies. An AI application may call a tool that calls another service with its own secret. The first application does not directly hold that credential, but the business workflow still depends on it. Include tool backends, connectors, build pipelines, and monitoring integrations in the secret inventory.
Use secret versioning carefully. Key Vault can retain multiple versions of a secret, which helps rollback, but clients should not pin an old version forever unless that is intentional. Most applications should resolve the current version and rely on a controlled rotation process.
Certificate-based authentication can reduce some password risks, but private keys are still secrets. Store them securely, restrict export, and rotate before expiration. A certificate is not automatically safer if the private key is copied into every container image.
Where third-party APIs support short-lived tokens, prefer those over permanent keys. A broker or backend can exchange its managed identity or another trusted credential for a short-lived token and keep the long-lived credential out of the application tier.
Private networking and Key Vault firewall controls can reduce exposure, but they also create operational dependencies for CI/CD and recovery. Test that deployment runners, break-glass paths, and secondary regions can reach the vault before an incident.
Secret rotation events should be visible in telemetry. A sudden wave of authentication failures immediately after rotation may indicate one workload did not reload the new value. Correlating failures to the secret version or rotation time can shorten recovery.
For applications that still need API keys to Azure AI endpoints, centralize their use behind a gateway or backend rather than distributing the key to every client. Browser or mobile applications should not receive a high-value service key that can be extracted and reused outside the product.
Unused secrets should be removed rather than left “just in case.” Every active credential increases the attack surface and creates future rotation work. A periodic credential inventory should confirm that each secret still has an owner and a live dependency.