Designing Agent Observability and FinOps in Foundry

1. Study log: the dashboard is green and the bill is not

This entry designs the telemetry, cost controls and failure handling for an agent in production. It covers the AB-100 objective group “Analyze, monitor, and tune AI-powered business solutions.”[1] The queue called this topic Azure AI Foundry. Microsoft Learn now calls the product Microsoft Foundry, so I use that name.

I haven’t deployed an agent, connected Application Insights or sent a single request to a model for this entry, which is why the title says designing. Everything below comes from Microsoft Learn, plus a PowerShell script that checks a design file and replays six synthetic failure scenarios through a retry policy. The design, the scenarios and every name in them are synthetic. No tenant, subscription, gateway or model was involved.

Three facts from the documentation shaped the design.

First, the rate limiter and the invoice count different things. Azure OpenAI checks tokens per minute (TPM) against an estimate made when the request arrives, and that estimate includes max_tokens. It “isn’t the same as the token count used for billing.”[13] An agent that sets max_tokens to 4,000 and gets 200 tokens back is billed for the 200, but the rate limiter counted the full 4,000 plus the prompt.[13]

Second, there’s no built-in hard stop on spend. The Foundry cost page says: “While OpenAI has an option for hard limits that prevent you from going over your budget, Azure OpenAI doesn’t currently provide this functionality.” Budget alerts can start automation through action groups, but that “requires additional custom development.”[19] If you want a ceiling, you build it, usually as a token quota at a gateway.

Third, the model under your agent can change without a deploy on your side. Standard, Global Standard and Data Zone Standard deployments get automatic upgrades when a version retires, and model router can update its set of underlying models, “which could affect the overall performance of the model and costs.”[20][21] Your agent’s behavior can drift even when nobody touched its instructions.

Default approach	AI-hardened approach
Watch the Azure OpenAI metrics and call it monitoring.	Connect Application Insights to the Foundry project so traces, the agent dashboard and alerts have a data source. Agents without it “don’t have health metrics, cost tracking, or drill-down traces.”[9]
Log full prompts and tool results for everyone who can read logs.	Route prompts, outputs and tool arguments to the protected sensitive-content table and grant Privileged Monitoring Data Reader only to the people who need it.[7]
Evaluate once before launch.	Run continuous evaluation on sampled live traffic and scheduled evaluation on a fixed test dataset to catch drift.[3][4]
Wrap every call in a retry library on top of the SDK’s own retries.	Use one retry layer. Microsoft’s guidance sets `max_retries=0` on the SDK when you add your own retry, “to avoid double-retrying.”[13]
Retry anything that failed.	Retry 429 and 5xx with `retry-after-ms` or jittered backoff.[13] Treat a 403 from a gateway token quota as “stop,” because it won’t clear until the quota period resets.[14][16]
Rely on an Azure budget to stop spend.	Put a token quota in front of the model and keep budgets for alerting, because Azure OpenAI has no hard spend limit.[14][19]
Let deployments auto-upgrade and hope.	Choose `versionUpgradeOption` on purpose, alert on retirement notices in Service Health and rerun the golden dataset after every model change.[3][20]

Blueprint note. As of October 29, 2026, Microsoft’s study guide publishes only the “Skills measured as of November 16, 2026” list, with Plan at 35–40%, Design at 30–35% and Deploy at 20–25%.[1] The exam page says the English exam updates on November 16 and localized versions about eight weeks later.[2] This objective group sits in the Deploy group. The change log doesn’t list it, and it marks the Deploy group’s share of the exam as decreased.[1] If you test before November 16, the October 14 weights in the AB-100 blueprint entry apply. I use the November 16 objective names because that’s the only list Microsoft shows now, and I don’t mix the two sets of weights.

2. Architecture and prerequisites

The telemetry architecture

The agent in this design is the IT triage agent from the AB100-02 entry, rebuilt as a Foundry agent that calls models through an API Management AI gateway. Every name is synthetic.

                        Users / Copilot Studio front end
                                     |
                                     v
                         +------------------------+
                         |  zk-triage-agent       |  Foundry agent
                         |  (Foundry project)     |  server-side tracing on
                         +-----------+------------+
                                     |  model calls
                                     v
                 +-------------------------------------------+
                 |  API Management AI gateway (StandardV2)   |
                 |  llm-token-limit: TPM + monthly quota     |
                 |  llm-emit-token-metric                    |
                 |  priority pool + circuit breakers         |
                 +------+-----------------------------+------+
                        | priority 1                  | priority 2
                        v                             v
          +---------------------------+   +---------------------------+
          | zk-foundry-eus2           |   | be-swc-std                |
          | gpt-prod-ptu (provisioned)|   | standard deployment,      |
          |   spillover ->            |   | second region             |
          | gpt-prod-std (standard)   |   +---------------------------+
          +---------------------------+
                        |
     traces, metrics, evaluation results, token metrics
                        v
   +--------------------------------------------------------------+
   | Application Insights appi-zk-agents-prod                      |
   |  - Foundry Monitor tab and Agent Monitoring Dashboard         |
   |  - Agents (preview) view and Grafana dashboards               |
   |  - protected sensitive-content table                          |
   |  - alerts: latency, run success rate, token usage, eval score |
   +-------------------------------+------------------------------+
                                   |
                                   v
   Foundry Control Plane (fleet view)    Cost Management budgets
   Service Health alert: Azure OpenAI    action group ag-zk-finops

How the pieces connect, with the source for each:

Foundry has three observability capabilities: evaluation, monitoring and tracing. Monitoring and tracing are integrated with Azure Monitor Application Insights.[3]
Server-side tracing for Foundry agents is automatic once the project has a connected Application Insights resource. No code changes, and the portal shows traces for the past 90 days.[6] Tracing is generally available for prompt and hosted agents. Workflow and external agents are in preview.[5]
The Agent Monitoring Dashboard reads from the same Application Insights resource. “Retention and billing follow your Application Insights configuration.”[4]
Application Insights also has an Agents (Preview) view that pulls together Foundry, Copilot Studio and third-party agents, with prebuilt Grafana dashboards.[8]
Foundry Control Plane aggregates the metrics and logs from each connected Application Insights resource into a fleet view, and it needs agents to log with the OpenTelemetry semantic conventions for generative AI.[9]
Model deployments emit platform metrics such as ModelAvailabilityRate, TimeToResponse, InputTokens, OutputTokens and ProvisionedUtilization automatically. Logs need a diagnostic setting.[12]
The AI gateway in API Management adds token rate limits and quotas, token metrics, load-balanced backend pools and circuit breakers in front of the models.[15]

What each signal answers

Question	Signal	Where
Is the agent slow?	Latency. “Latency above 10 seconds might indicate model throttling, complex tool calls, or network issues.”[4]	Agent Monitoring Dashboard
Is it failing?	Run success rate. “A rate below 95% warrants investigation into failed runs.”[4]	Agent Monitoring Dashboard
Where did a bad answer come from?	Spans for agent invocation, planning, tool execution and memory, with tool arguments and results.[5]	Traces in Foundry and Application Insights
Is quality slipping?	Continuous evaluation on sampled live traffic and scheduled evaluation on a test dataset.[3][4]	Monitor tab, Evaluations
Is the model throttled?	`ModelRequests` split by `StatusCode`, plus `ProvisionedUtilization`. At 100% or more, provisioned calls get 429.[12]	Azure Monitor metrics
Who is spending the tokens?	Token metrics per consumer from `llm-emit-token-metric`, and gateway quota headers.[16][25]	Application Insights custom metrics
What did it cost?	Cost Management, filtered to the Azure OpenAI service tier. Portal estimates exclude discounts and provisioned throughput.[19]	Cost Management
Did the model change?	Service Health retirement notices and the deployment’s `versionUpgradeOption`.[20]	Service Health, deployment settings

Roles and settings I verified

Need	What the documentation says
View the agent dashboard	Application Insights connected to the project, and Log Analytics Reader on it. Privileged Monitoring Data Reader if tables are protected.[4]
Continuous evaluation	The project managed identity needs the Foundry User role, which was previously named Azure AI User.[4]
Protect sensitive trace content	Register the `protectGenAISensitiveData` feature flag, set the table as protected and grant Privileged Monitoring Data Reader. Needs Log Analytics Contributor and a subscription role that can register preview features.[7]
View model metrics	Monitoring Reader. Monitoring Contributor to create diagnostic settings.[12]
Set token limits in Foundry Control Plane	An AI gateway configured, and API Management Service Contributor or Owner.[14]
Fleet view	Log Analytics Reader on the Application Insights resource and Cost Management Reader.[9]
Cost views in Foundry	Cost Management Reader, plus Foundry User on the project.[19]
Spillover	A provisioned managed deployment and a standard deployment in the same Foundry resource, with matching model and version, and Cognitive Services Contributor or higher.[18]
Copilot Studio agent telemetry	An Application Insights connection string under Settings > Advanced, with optional switches for conversation details and sensitive activity properties.[23]

Two cost notes. Agent playground evaluations “are enabled by default for all Foundry projects and are included in consumption-based billing,” and you turn them off by clearing the evaluators in the metrics settings.[3] And trace data stored in Application Insights may cost money of its own.[5]

3. Build steps and where each one fails

These are the steps the design assumes, in order. I haven’t run them. Each one names the point where the documentation says it can go wrong.

Telemetry

Connect Application Insights to the Foundry project. In the Foundry portal, go to Manage > Project details > Connected resources and add an Application Insights connection, or use Connect from the traces page.[6][9] Failure point: agents on resources without Application Insights have no health metrics, cost tracking or drill-down traces.[9]
Decide what goes in the traces before you turn on content capture. Prompts, model responses, system instructions, tool definitions, tool arguments and tool results are all sensitive attributes.[7] Route them to the dedicated table, set it as protected and grant read access by role.[7] Failure point: tracing setup tells you not to store secrets, credentials or tokens in prompts, tool arguments or span attributes, and protecting the table doesn’t stop someone from logging them in the first place.[6]
Instrument anything Foundry doesn’t trace for you. Custom agents need OpenTelemetry with the generative AI semantic conventions, sent to the same Application Insights resource.[4][9] Failure point: those conventions “have Development status and might change,” so a dashboard built on attribute names can break on an SDK update.[5]
Connect the Copilot Studio front end too, if you have one. Copilot Studio sends agent telemetry to Application Insights from Settings > Advanced, and its events land in the customEvents table.[23] Failure point: test-canvas traffic is logged as well. Filter on the designMode custom dimension to keep it out of production numbers.[23] The Copilot Studio Monitor page already leaves out test panel activity, keeps analytics for up to 360 days and keeps session transcripts for 28 days.[24]

Evaluation and drift

Pick evaluators that fit the agent. The agent evaluators cover task completion, task adherence, intent resolution, tool call accuracy, tool selection, tool input accuracy, tool output utilization and tool call success, among others.[10] Failure point: several tool evaluators have limited support for built-in tools such as Azure AI Search, Bing grounding, SharePoint grounding, Code Interpreter, Fabric data agent and web search.[10] The triage agent calls custom tools, so the design keeps tool call accuracy. An agent built on those built-in tools would need a different set.
Turn on continuous evaluation for live traffic. Assign Foundry User to the project managed identity first, then create the rule from the agent’s Monitor tab or in code.[4] Failure point: evaluation runs above the hourly limit are skipped, and “the default limit is 100 runs per hour.”[4] At higher traffic you’re sampling whether you planned to or not.
Schedule an evaluation on a fixed test dataset. Microsoft lists “scheduled quality and safety evaluation using test datasets to detect system drift” as a production practice.[3] In the portal, that’s a Scheduled recurring evaluation with Dataset as the data source.[4] Failure point: evaluation run creation is rate-limited at the tenant, subscription and project levels, and a rejected request comes back with a retry-after header.[11] Treat your own evaluation pipeline like any other API client.
Choose each deployment’s upgrade behavior. For standard deployment types, versionUpgradeOption takes OnceNewDefaultVersionAvailable, OnceCurrentVersionExpired or NoAutoUpgrade.[20] Failure point: NoAutoUpgrade means the deployment “stops working at retirement,” and a retired model returns 410 Gone.[20] Provisioned deployments aren’t auto-upgraded at all, so somebody has to own the migration.[20]
Alert on retirement notices. GA models get a retirement notice at least 60 days ahead, sent to subscription owners with active deployments.[20] The design adds a Service Health alert so the operations team gets it as well. Microsoft says to select Azure OpenAI Service as the service name for those alerts.[20]

Gateway, retry and fallback

Put the models behind the AI gateway. Use a backend pool with priority groups. API Management only sends to a lower priority group when every backend in the higher group has a tripped circuit breaker.[17] Failure point: the circuit breaker isn’t supported in the Consumption tier, a backend can have only one breaker rule, and tripping is approximate because gateway instances don’t share state.[17]
Make the breaker understand 429. Azure OpenAI can return a Retry-After value that “can be large (for example, 1 day).” Microsoft’s advice is to write circuit breaker rules that handle 429 and accept the Retry-After duration.[17] Failure point: a breaker that only counts 5xx keeps sending traffic to a throttled backend.
Enable spillover on the provisioned deployment. Spillover sends overflow from a fully used provisioned deployment, for example 429s when PTUs run out, to a standard deployment.[18] Failure point: the standard deployment must be in the same resource and match the model and version, or spillover can’t be turned on.[18]
Set one retry layer in the agent’s client. Use retry-after-ms when the 429 carries it, otherwise exponential backoff with random jitter.[13] The OpenAI Python SDK already retries twice by default with backoff, so a custom layer sets max_retries=0.[13] Failure point: two layers multiply attempts during an outage, when the backend can least absorb them.
Size max_tokens to the expected answer. “If you expect responses of about 200 tokens, don’t set max_tokens to 4,000.”[13] Failure point: the TPM estimate includes max_tokens, so an oversized value throttles you early.[13]

FinOps

Set token limits per project. In Foundry Control Plane, Token management sets a TPM limit and a token quota per project through the AI gateway.[14] In API Management the same idea is the llm-token-limit policy with a counter key, a rate and a quota period from hourly to yearly.[16] Failure point: exceeding the rate returns 429 Too Many Requests, exceeding the quota returns 403 Forbidden, and concurrent requests can temporarily go past the limit.[14][16] Clients need to tell those two codes apart.
Emit token metrics with useful dimensions. llm-emit-token-metric sends token counts to Application Insights with dimensions you choose. Default dimension names such as Subscription ID, User ID and Backend ID can be used without a value.[25] Failure point: you get at most 5 custom dimensions per policy, API Management tracks only 100 unique values per dimension and 1,000 active time series per namespace, and anything past those limits “is silently discarded.”[25] A per-user dimension in a large tenant hits 100 values quickly, so the design uses the agent and the API Management subscription instead.
Write prompts that can be cached. Caching needs prompts of at least 1,024 tokens with identical first 1,024 tokens, and cache hits show as cached_tokens.[22] On supported models, cached reads are billed at a discount for standard deployments.[22] Failure point: the first 1,024 tokens must match exactly, so a timestamp or user name at the top of the prompt defeats the cache.[22]
Budget and reconcile. Create Cost Management budgets with alert thresholds and filters, and reconcile against Cost Management or the invoice, not portal estimates.[19] Failure point: Foundry’s cost estimates exclude discounts and provisioned throughput, “HTTP status codes alone don’t determine whether usage is billed,” and Azure Monitor log and alert costs don’t show up scoped to the Foundry resource.[19]

Which signals to alert on

The Agent Monitoring Dashboard supports alerts on latency, token usage, evaluation scores and red team findings, all in preview.[4] Run success rate isn’t on that list, so the design adds it as an Azure Monitor alert on the same Application Insights data, which matches Microsoft’s post-production guidance to use Azure Monitor alerts.[3] The thresholds are 10 seconds of latency and a 95% run success rate because those are the numbers the dashboard page uses to describe when to look closer.[4] They aren’t SLAs. Tune them once you have your own baseline.

4. Engineering deliverable: retry, fallback and telemetry design review

The deliverable is the architecture above plus a design file, a scenario file and a PowerShell script. The script reads both files, checks the design against the rules in section 3 and then replays each synthetic failure through the retry policy. Every scenario has to end in Success, FailFast or Fallback, and match the outcome you expected. A scenario that runs out of responses while the policy still wants to retry is a design error.

It reads local JSON only. It never connects to Azure, Foundry, API Management, Application Insights or Copilot Studio.

The design file

resilience-design.json describes the triage agent’s production setup. All names are synthetic, and the model is a placeholder, not a real model name or version.

{
  "schemaVersion": 1,
  "workload": "zk-it-triage",
  "apim": {
    "tier": "StandardV2"
  },
  "telemetry": {
    "appInsights": "appi-zk-agents-prod",
    "logAnalyticsReaders": [
      "sg-zk-agent-ops"
    ],
    "protectedSensitiveTable": true,
    "privilegedReaders": [
      "sg-zk-agent-sec"
    ],
    "copilotStudio": {
      "logConversationDetails": false,
      "logSensitiveActivityProperties": false
    }
  },
  "agents": [
    {
      "name": "zk-triage-agent",
      "platform": "Foundry",
      "appInsightsConnected": true,
      "expectedResponsesPerHour": 60,
      "continuousEvaluation": {
        "enabled": true,
        "projectIdentityRole": "Foundry User",
        "maxHourlyRuns": 100
      },
      "scheduledEvaluation": {
        "enabled": true,
        "dataSource": "Dataset",
        "dataset": "golden-triage-v3",
        "schedule": "Daily 09:00"
      },
      "alerts": {
        "latencySeconds": 10,
        "runSuccessRatePercent": 95
      },
      "timeBudgetSeconds": 30
    }
  ],
  "deployments": [
    {
      "name": "gpt-prod-ptu",
      "foundryResource": "zk-foundry-eus2",
      "type": "ProvisionedManaged",
      "versionUpgradeOption": null,
      "migrationOwner": "sg-zk-agent-ops",
      "model": "chat-model-a",
      "version": "v1"
    },
    {
      "name": "gpt-prod-std",
      "foundryResource": "zk-foundry-eus2",
      "type": "GlobalStandard",
      "versionUpgradeOption": "OnceCurrentVersionExpired",
      "model": "chat-model-a",
      "version": "v1"
    }
  ],
  "spillover": {
    "from": "gpt-prod-ptu",
    "to": "gpt-prod-std"
  },
  "serviceHealthAlert": {
    "service": "Azure OpenAI Service",
    "actionGroup": "ag-zk-agent-ops"
  },
  "gateway": {
    "pool": [
      {
        "backend": "be-eus2-ptu",
        "priority": 1,
        "weight": 1,
        "kind": "AzureOpenAI"
      },
      {
        "backend": "be-swc-std",
        "priority": 2,
        "weight": 1,
        "kind": "AzureOpenAI"
      }
    ],
    "circuitBreakers": [
      {
        "backend": "be-eus2-ptu",
        "statusCodeRanges": [
          [
            429,
            429
          ],
          [
            500,
            599
          ]
        ],
        "acceptRetryAfter": true,
        "tripDuration": "PT1M"
      },
      {
        "backend": "be-swc-std",
        "statusCodeRanges": [
          [
            429,
            429
          ],
          [
            500,
            599
          ]
        ],
        "acceptRetryAfter": true,
        "tripDuration": "PT1M"
      }
    ]
  },
  "retry": {
    "sdkMaxRetries": 0,
    "customRetry": true,
    "retryOn": [
      429,
      500,
      502,
      503,
      504
    ],
    "honorRetryAfter": true,
    "maxRetries": 4,
    "baseDelayMs": 1000,
    "maxDelayMs": 8000,
    "jitter": true,
    "maxTokens": 600,
    "expectedOutputTokens": 250
  },
  "finops": {
    "tokenQuotas": [
      {
        "project": "zk-triage-prod",
        "deployment": "gpt-prod-std",
        "tokensPerMinute": 20000,
        "quotaTokens": 30000000,
        "period": "Monthly"
      }
    ],
    "budget": {
      "scope": "rg-zk-agents-prod",
      "thresholdsPercent": [
        50,
        80,
        100
      ],
      "actionGroup": "ag-zk-finops"
    },
    "playgroundEvaluationsOff": true
  }
}

Three values in that file are my design choices, not Microsoft numbers: the 30-second time budget per agent turn, the 1-second base delay with an 8-second cap and four retries, and the 250-token expected answer.

The scenario file

scenarios.json lists what the model or gateway returns on each attempt, and the outcome the design should produce.

{
  "schemaVersion": 1,
  "scenarios": [
    {
      "name": "clean-429-with-header",
      "responses": [
        {
          "status": 429,
          "retryAfterMs": 2000
        },
        {
          "status": 200
        }
      ],
      "expect": "Success"
    },
    {
      "name": "429-no-header-twice",
      "responses": [
        {
          "status": 429
        },
        {
          "status": 429
        },
        {
          "status": 200
        }
      ],
      "expect": "Success"
    },
    {
      "name": "server-errors-exhaust-retries",
      "responses": [
        {
          "status": 500
        },
        {
          "status": 502
        },
        {
          "status": 503
        },
        {
          "status": 500
        },
        {
          "status": 500
        }
      ],
      "expect": "Fallback"
    },
    {
      "name": "gateway-quota-exhausted",
      "responses": [
        {
          "status": 403
        }
      ],
      "expect": "FailFast"
    },
    {
      "name": "long-retry-after",
      "responses": [
        {
          "status": 429,
          "retryAfterMs": 86400000
        }
      ],
      "expect": "Fallback"
    },
    {
      "name": "bad-request-no-retry",
      "responses": [
        {
          "status": 400
        }
      ],
      "expect": "FailFast"
    }
  ]
}

The rules

Rule	Checks	Source
T1	Application Insights exists and every agent is connected to it	[6][9]
T2	Someone holds Log Analytics Reader	[4]
T3	Sensitive content goes to a protected table with at least one Privileged Monitoring Data Reader	[7]
T4	Copilot Studio sensitive activity logging is a deliberate choice	[23]
E1, E2	Continuous evaluation is on, the project identity holds Foundry User, and expected traffic vs the hourly run cap	[4]
A1	Latency and run success rate alerts have thresholds	[3][4]
D1	Each deployment’s upgrade behavior is known, and provisioned deployments have a migration owner	[20]
D2	If any deployment can change version under the agent, a scheduled evaluation runs on a fixed dataset	[3][20]
D3	A Service Health alert covers Azure OpenAI retirement notices	[20]
G1	API Management tier supports the backend circuit breaker	[17]
G2	The pool has at most 30 backends and at least two priority groups	[17]
G3	Every Azure OpenAI backend has one breaker that trips on 429 and accepts Retry-After	[17]
G4	Spillover goes from provisioned to standard, in the same resource, with the same model and version	[18]
R1	Not both SDK retries and a custom retry layer	[13]
R2	The policy honors `retry-after-ms`	[13]
R3	No retries on 400, 401, 403 or 404	[14][16] for 403. The other codes are a design choice
R4	Backoff has jitter and a sane cap	[13]
R5	`max_tokens` isn’t more than 4x the expected answer (my threshold)	[13]
R6	Each agent has a positive time budget	Design choice
F1	A hard stop exists: a gateway token quota or budget automation	[14][19]
F2	Token limits are positive and use a valid quota period	[16]
F3	The budget has alert thresholds	[19]
F4	Playground evaluations are off in production, since they’re billed by default	[3]
S1, S2	Every scenario reaches a defined outcome, and the outcome matches `expect`	Design choice

The script

Test-AgentResilienceDesign.ps1 needs PowerShell 7.2 or later. It supports -WhatIf, writes an audit log, skips the write when the inputs and outputs haven’t changed, and only rolls back files it wrote that nobody edited since. The jitter uses a seeded random generator, so the same inputs and seed always produce the same retry schedule.

#Requires -Version 7.2
<#
.SYNOPSIS
  Reviews an AB-100 observability and FinOps design for a production agent (telemetry, evaluation,
  drift controls, gateway fallback, retry policy, token quotas) and replays synthetic failure
  scenarios through the retry policy to prove every path ends in a defined state.
.NOTES
  Study aid. Reads local JSON files only. Never connects to Azure, Foundry, API Management,
  Application Insights or Copilot Studio. All names in the sample files are synthetic.
  The jitter uses a seeded random generator so the same inputs produce the same schedule.
#>
[CmdletBinding(SupportsShouldProcess)]
param(
    [Parameter(Mandatory)] [string] $DesignPath,
    [Parameter(Mandatory)] [string] $ScenarioPath,
    [string] $OutputDirectory = './resilience-review-output',
    [ValidateRange(0, [int]::MaxValue)] [int] $Seed = 42,
    [switch] $Force,
    [switch] $Rollback
)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
$manifestFile = Join-Path $OutputDirectory 'manifest.json'
$auditPath = Join-Path $OutputDirectory 'audit.log'

function Get-Sha256([string] $Path) { (Get-FileHash -LiteralPath $Path -Algorithm SHA256).Hash.ToLowerInvariant() }
function Write-Audit([string] $Action, [string] $Detail) {
    if (-not (Test-Path -LiteralPath $OutputDirectory)) { return }
    $line = [ordered]@{ utc = [DateTime]::UtcNow.ToString('o'); user = [Environment]::UserName; action = $Action; detail = $Detail }
    Add-Content -LiteralPath $auditPath -Value ($line | ConvertTo-Json -Compress)
}
function Read-Json([string] $Path) {
    if (-not (Test-Path -LiteralPath $Path)) { throw "File not found: $Path" }
    try { Get-Content -LiteralPath $Path -Raw | ConvertFrom-Json -AsHashtable }
    catch { throw "Invalid JSON in ${Path}: $($_.Exception.Message)" }
}

# Rollback: remove only files this script wrote and nobody changed since.
if ($Rollback) {
    if (-not (Test-Path -LiteralPath $manifestFile)) { Write-Output "Nothing to roll back in $OutputDirectory."; return }
    $m = Get-Content -LiteralPath $manifestFile -Raw | ConvertFrom-Json
    $kept = 0
    foreach ($f in $m.files) {
        $p = Join-Path $OutputDirectory $f.name
        if (-not (Test-Path -LiteralPath $p)) { continue }
        if ((Get-Sha256 $p) -ne $f.sha256) { Write-Warning "Kept $($f.name): changed since this script wrote it."; $kept++; continue }
        if ($PSCmdlet.ShouldProcess($p, 'Remove generated file')) { Remove-Item -LiteralPath $p; Write-Audit 'remove' $f.name }
    }
    if ($kept -eq 0 -and $PSCmdlet.ShouldProcess($manifestFile, 'Remove manifest')) { Remove-Item -LiteralPath $manifestFile; Write-Audit 'remove' 'manifest.json' }
    return
}

$design = Read-Json $DesignPath
$scen = Read-Json $ScenarioPath
if ($design['schemaVersion'] -ne 1 -or $scen['schemaVersion'] -ne 1) { throw 'Design and scenario files must both use schemaVersion 1.' }
foreach ($key in 'telemetry', 'agents', 'deployments', 'gateway', 'retry', 'finops') {
    if (-not $design.ContainsKey($key)) { throw "Design is missing the '$key' section." }
}

$findings = [System.Collections.Generic.List[object]]::new()
function Add-Finding([string] $Severity, [string] $Rule, [string] $Target, [string] $Message) {
    $findings.Add([pscustomobject][ordered]@{ Severity = $Severity; Rule = $Rule; Target = $Target; Message = $Message })
}
function Get-Value($Table, [string] $Key) { if ($null -ne $Table -and $Table.ContainsKey($Key)) { $Table[$Key] } else { $null } }

# ---- Telemetry (T) ----
$tel = $design['telemetry']
if (-not (Get-Value $tel 'appInsights')) { Add-Finding 'Error' 'T1' 'telemetry' 'No Application Insights resource. Dashboards, traces and alerts all read from it.' }
if (@(Get-Value $tel 'logAnalyticsReaders').Count -eq 0) { Add-Finding 'Error' 'T2' 'telemetry' 'Nobody holds Log Analytics Reader, so nobody can open the monitoring dashboard.' }
if ((Get-Value $tel 'protectedSensitiveTable') -ne $true) { Add-Finding 'Warning' 'T3' 'telemetry' 'Prompts, outputs and tool arguments are not routed to a protected table.' }
elseif (@(Get-Value $tel 'privilegedReaders').Count -eq 0) { Add-Finding 'Error' 'T3' 'telemetry' 'Protected table has no Privileged Monitoring Data Reader. Nobody can read sensitive spans.' }
$cs = Get-Value $tel 'copilotStudio'
if ((Get-Value $cs 'logSensitiveActivityProperties') -eq $true) { Add-Finding 'Warning' 'T4' 'copilotStudio' 'Log sensitive Activity properties is on. Confirm the Application Insights data owner approved it.' }

foreach ($a in $design['agents']) {
    $n = $a['name']
    if ($a['appInsightsConnected'] -ne $true) { Add-Finding 'Error' 'T1' $n 'Agent has no Application Insights connection: no health metrics, cost tracking or traces.' }
    $ce = Get-Value $a 'continuousEvaluation'
    if ((Get-Value $ce 'enabled') -eq $true) {
        if ($ce['projectIdentityRole'] -ne 'Foundry User') { Add-Finding 'Error' 'E1' $n 'Continuous evaluation needs the project managed identity to hold Foundry User.' }
        $cap = if ($ce['maxHourlyRuns']) { [int]$ce['maxHourlyRuns'] } else { 100 }
        if ([int]$a['expectedResponsesPerHour'] -gt $cap) { Add-Finding 'Info' 'E2' $n "Expected $($a['expectedResponsesPerHour']) responses/hour, cap $cap evaluation runs/hour. Runs above the cap are skipped." }
    } else { Add-Finding 'Warning' 'E1' $n 'No continuous evaluation on live traffic.' }
    $al = Get-Value $a 'alerts'
    if (-not (Get-Value $al 'latencySeconds') -or -not (Get-Value $al 'runSuccessRatePercent')) { Add-Finding 'Error' 'A1' $n 'Latency and run success rate alerts must both have thresholds.' }
    if (-not $a['timeBudgetSeconds'] -or [int]$a['timeBudgetSeconds'] -le 0) { Add-Finding 'Error' 'R6' $n 'timeBudgetSeconds must be a positive number.' }
}

# ---- Drift (D) ----
$needsGolden = $false
foreach ($d in $design['deployments']) {
    $opt = $d['versionUpgradeOption']
    if ($d['type'] -like 'Provisioned*') {
        if (-not (Get-Value $d 'migrationOwner')) { Add-Finding 'Warning' 'D1' $d['name'] 'Provisioned deployments are not auto-upgraded. Name an owner for version migrations.' }
    } elseif ($opt -eq 'NoAutoUpgrade') { Add-Finding 'Warning' 'D1' $d['name'] 'NoAutoUpgrade: the deployment stops working when the version retires.' }
    elseif ($opt -in 'OnceNewDefaultVersionAvailable', 'OnceCurrentVersionExpired' -or $d['type'] -eq 'ModelRouter') { $needsGolden = $true }
    else { Add-Finding 'Error' 'D1' $d['name'] "Unknown versionUpgradeOption '$opt'." }
}
foreach ($a in $design['agents']) {
    $se = Get-Value $a 'scheduledEvaluation'
    $ok = (Get-Value $se 'enabled') -eq $true -and (Get-Value $se 'dataSource') -eq 'Dataset' -and (Get-Value $se 'dataset')
    if ($needsGolden -and -not $ok) { Add-Finding 'Error' 'D2' $a['name'] 'A deployment can change model version under this agent, but no scheduled evaluation runs on a fixed test dataset.' }
}
$sh = Get-Value $design 'serviceHealthAlert'
if ((Get-Value $sh 'service') -ne 'Azure OpenAI Service' -or -not (Get-Value $sh 'actionGroup')) { Add-Finding 'Warning' 'D3' 'serviceHealth' 'No Service Health alert for Azure OpenAI Service retirement notices.' }

# ---- Gateway and fallback (G) ----
$gw = $design['gateway']
if ((Get-Value $design['apim'] 'tier') -eq 'Consumption') { Add-Finding 'Error' 'G1' 'apim' 'Backend circuit breaker is not supported in the Consumption tier.' }
$pool = @($gw['pool'])
if ($pool.Count -gt 30) { Add-Finding 'Error' 'G2' 'pool' 'A backend pool holds at most 30 backends.' }
if (@($pool | ForEach-Object { $_['priority'] } | Sort-Object -Unique).Count -lt 2) { Add-Finding 'Error' 'G2' 'pool' 'Only one priority group, so there is no fallback tier.' }
foreach ($b in $pool) {
    $cb = @($gw['circuitBreakers'] | Where-Object { $_['backend'] -eq $b['backend'] })
    if ($cb.Count -eq 0) { Add-Finding 'Error' 'G3' $b['backend'] 'No circuit breaker, so the pool never fails over from this backend.'; continue }
    if ($cb.Count -gt 1) { Add-Finding 'Error' 'G3' $b['backend'] 'Only one circuit breaker rule is allowed per backend.' }
    $has429 = @($cb[0]['statusCodeRanges'] | Where-Object { $_[0] -le 429 -and $_[1] -ge 429 }).Count -gt 0
    if ($b['kind'] -eq 'AzureOpenAI' -and (-not $has429 -or $cb[0]['acceptRetryAfter'] -ne $true)) { Add-Finding 'Error' 'G3' $b['backend'] 'Azure OpenAI backend: the breaker must trip on 429 and accept Retry-After.' }
}
$sp = Get-Value $design 'spillover'
if ($sp) {
    $from = @($design['deployments'] | Where-Object { $_['name'] -eq $sp['from'] })[0]
    $to = @($design['deployments'] | Where-Object { $_['name'] -eq $sp['to'] })[0]
    if (-not $from -or -not $to) { Add-Finding 'Error' 'G4' 'spillover' 'Spillover names a deployment that is not in the design.' }
    elseif ($from['type'] -notlike 'Provisioned*' -or $to['type'] -like 'Provisioned*') { Add-Finding 'Error' 'G4' 'spillover' 'Spillover goes from a provisioned deployment to a standard one.' }
    elseif ($from['foundryResource'] -ne $to['foundryResource']) { Add-Finding 'Error' 'G4' 'spillover' 'Spillover target must be in the same resource as the provisioned deployment.' }
    elseif ($from['model'] -ne $to['model'] -or $from['version'] -ne $to['version']) { Add-Finding 'Error' 'G4' 'spillover' 'Spillover target must run the same model and version as the provisioned deployment.' }
}

# ---- Retry policy (R) ----
$r = $design['retry']
if ([int]$r['sdkMaxRetries'] -gt 0 -and $r['customRetry'] -eq $true) { Add-Finding 'Error' 'R1' 'retry' "SDK retries ($($r['sdkMaxRetries'])) plus custom retry: every failure is retried twice over. Set the SDK max_retries to 0." }
if ($r['honorRetryAfter'] -ne $true) { Add-Finding 'Error' 'R2' 'retry' 'Policy ignores retry-after-ms / Retry-After.' }
$bad = @($r['retryOn'] | Where-Object { $_ -in 400, 401, 403, 404 })
if ($bad.Count) { Add-Finding 'Error' 'R3' 'retry' "Retries non-transient status codes: $($bad -join ', '). A 403 from the gateway token quota will not clear until the quota period resets." }
if ($r['jitter'] -ne $true) { Add-Finding 'Error' 'R4' 'retry' 'Backoff has no jitter, so clients retry in lockstep.' }
if ([int]$r['maxDelayMs'] -lt [int]$r['baseDelayMs']) { Add-Finding 'Error' 'R4' 'retry' 'maxDelayMs is smaller than baseDelayMs.' }
# Threshold below is this design's own rule, not a Microsoft number.
if ([int]$r['maxTokens'] -gt 4 * [int]$r['expectedOutputTokens']) { Add-Finding 'Warning' 'R5' 'retry' "max_tokens $($r['maxTokens']) is over 4x the expected output. The rate limiter counts max_tokens toward TPM." }

# ---- FinOps (F) ----
$fin = $design['finops']
if (@(Get-Value $fin 'tokenQuotas').Count -eq 0 -and -not (Get-Value (Get-Value $fin 'budget') 'actionGroup')) { Add-Finding 'Error' 'F1' 'finops' 'No hard stop. Azure OpenAI has no built-in hard spend limit: add a gateway token quota or budget automation.' }
foreach ($q in @(Get-Value $fin 'tokenQuotas')) {
    if ([long]$q['tokensPerMinute'] -le 0 -or [long]$q['quotaTokens'] -le 0) { Add-Finding 'Error' 'F2' $q['project'] 'Token limits must be positive.' }
    if ($q['period'] -notin 'Hourly', 'Daily', 'Weekly', 'Monthly', 'Yearly') { Add-Finding 'Error' 'F2' $q['project'] "Unknown quota period '$($q['period'])'." }
}
$bud = Get-Value $fin 'budget'
if (-not $bud -or @(Get-Value $bud 'thresholdsPercent').Count -eq 0) { Add-Finding 'Warning' 'F3' 'finops' 'No budget alert thresholds.' }
if ((Get-Value $fin 'playgroundEvaluationsOff') -ne $true) { Add-Finding 'Info' 'F4' 'finops' 'Agent playground evaluations are on by default and billed. Turn them off in production projects if nobody uses them.' }

# ---- Replay synthetic scenarios through the retry policy (S) ----
$rng = [System.Random]::new($Seed)
$budgetMs = [long]([int]@($design['agents'])[0]['timeBudgetSeconds']) * 1000
$schedule = [System.Collections.Generic.List[object]]::new()
$outcomes = [ordered]@{}
foreach ($s in $scen['scenarios']) {
    $waited = [long]0; $retries = 0; $outcome = 'Unresolved'; $attempt = 0
    foreach ($resp in $s['responses']) {
        $attempt++; $code = [int]$resp['status']; $delay = [long]0
        if ($code -ge 200 -and $code -lt 300) { $action = 'Success'; $outcome = 'Success' }
        elseif ($code -eq 403) { $action = 'FailFast: quota exhausted, route to fallback answer'; $outcome = 'FailFast' }
        elseif ($code -notin $r['retryOn']) { $action = 'FailFast: not retryable'; $outcome = 'FailFast' }
        elseif ($retries -ge [int]$r['maxRetries']) { $action = 'Fallback: retries exhausted'; $outcome = 'Fallback' }
        else {
            $ra = Get-Value $resp 'retryAfterMs'
            if ($r['honorRetryAfter'] -eq $true -and $null -ne $ra) { $delay = [long]$ra }
            else {
                $capped = [math]::Min([double]$r['maxDelayMs'], [double]$r['baseDelayMs'] * [math]::Pow(2, $retries))
                $delay = if ($r['jitter'] -eq $true) { [long]($capped * (0.5 + 0.5 * $rng.NextDouble())) } else { [long]$capped }
            }
            if ($waited + $delay -gt $budgetMs) { $action = "Fallback: wait of $delay ms breaks the $budgetMs ms budget"; $outcome = 'Fallback'; $delay = 0 }
            else { $waited += $delay; $retries++; $action = "Retry in $delay ms" }
        }
        $schedule.Add([pscustomobject][ordered]@{ Scenario = $s['name']; Attempt = $attempt; Status = $code; Action = $action; DelayMs = $delay; WaitedMs = $waited })
        if ($outcome -ne 'Unresolved') { break }
    }
    $outcomes[$s['name']] = $outcome
    if ($outcome -eq 'Unresolved') { Add-Finding 'Error' 'S1' $s['name'] 'Scenario ended without a defined outcome.' }
    elseif ($s['expect'] -and $s['expect'] -ne $outcome) { Add-Finding 'Error' 'S2' $s['name'] "Expected $($s['expect']), policy produced $outcome." }
}

$errorCount = @($findings | Where-Object Severity -eq 'Error').Count
$report = [ordered]@{
    workload = $design['workload']; seed = $Seed; errors = $errorCount
    warnings = @($findings | Where-Object Severity -eq 'Warning').Count
    outcomes = $outcomes; findings = @($findings)
}

# ---- Output with idempotency ----
$inputHash = (@($DesignPath, $ScenarioPath) | ForEach-Object { Get-Sha256 $_ }) -join ''
$inputHash += ":seed=$Seed"
$outNames = @('resilience-report.json', 'retry-schedule.csv')
$skipWrite = $false
if (Test-Path -LiteralPath $manifestFile) {
    $old = Get-Content -LiteralPath $manifestFile -Raw | ConvertFrom-Json
    $intact = @($old.files | Where-Object { $p = Join-Path $OutputDirectory $_.name; (Test-Path -LiteralPath $p) -and (Get-Sha256 $p) -eq $_.sha256 }).Count -eq @($old.files).Count
    if ($old.inputSha256 -eq $inputHash -and $intact) { Write-Output 'No changes: output already matches these inputs.'; $skipWrite = $true }
    elseif (-not $Force) { throw "Output in $OutputDirectory came from different input or was edited. Re-run with -Force." }
}
if (-not $skipWrite -and $PSCmdlet.ShouldProcess($OutputDirectory, "Write report with $($findings.Count) findings and $($schedule.Count) schedule rows")) {
    if (-not (Test-Path -LiteralPath $OutputDirectory)) { New-Item -ItemType Directory -Path $OutputDirectory | Out-Null }
    $report | ConvertTo-Json -Depth 6 | Set-Content -LiteralPath (Join-Path $OutputDirectory 'resilience-report.json') -Encoding utf8
    $schedule | Export-Csv -LiteralPath (Join-Path $OutputDirectory 'retry-schedule.csv') -NoTypeInformation -Encoding utf8
    $files = @($outNames | ForEach-Object { [ordered]@{ name = $_; sha256 = Get-Sha256 (Join-Path $OutputDirectory $_) } })
    [ordered]@{ inputSha256 = $inputHash; files = $files } | ConvertTo-Json -Depth 4 | Set-Content -LiteralPath $manifestFile -Encoding utf8
    Write-Audit 'write' "findings=$($findings.Count) errors=$errorCount rows=$($schedule.Count)"
}
$schedule | Format-Table -AutoSize | Out-String -Width 160 | Write-Output
$findings | Sort-Object Severity, Rule, Target | Format-Table -AutoSize -Wrap | Out-String -Width 160 | Write-Output
if ($errorCount -gt 0) { throw "$errorCount design error(s). See $(Join-Path $OutputDirectory 'resilience-report.json')." }

Run it from the folder that holds the two JSON files:

./Test-AgentResilienceDesign.ps1 -DesignPath ./resilience-design.json -ScenarioPath ./scenarios.json -WhatIf
./Test-AgentResilienceDesign.ps1 -DesignPath ./resilience-design.json -ScenarioPath ./scenarios.json
./Test-AgentResilienceDesign.ps1 -DesignPath ./resilience-design.json -ScenarioPath ./scenarios.json -Rollback

The retry schedule it produced

This is retry-schedule.csv from the good design with the default seed of 42, unchanged except for the table format:

Scenario	Attempt	Status	Action	DelayMs	WaitedMs
clean-429-with-header	1	429	Retry in 2000 ms	2000	2000
clean-429-with-header	2	200	Success	0	2000
429-no-header-twice	1	429	Retry in 834 ms	834	834
429-no-header-twice	2	429	Retry in 1141 ms	1141	1975
429-no-header-twice	3	200	Success	0	1975
server-errors-exhaust-retries	1	500	Retry in 563 ms	563	563
server-errors-exhaust-retries	2	502	Retry in 1523 ms	1523	2086
server-errors-exhaust-retries	3	503	Retry in 2337 ms	2337	4423
server-errors-exhaust-retries	4	500	Retry in 5050 ms	5050	9473
server-errors-exhaust-retries	5	500	Fallback: retries exhausted	0	9473
gateway-quota-exhausted	1	403	FailFast: quota exhausted, route to fallback answer	0	0
long-retry-after	1	429	Fallback: wait of 86400000 ms breaks the 30000 ms budget	0	0
bad-request-no-retry	1	400	FailFast: not retryable	0	0

The schedule tests the policy’s logic, and the delays come from the seeded jitter. The 429 with a 2,000 ms retry-after-ms waits exactly that long. The two 429s without a header back off with jitter. Five server errors in a row use up four retries and then fall back. The 403 from the token quota fails fast instead of retrying into a quota that won’t reset this month. The 1-day Retry-After goes straight to fallback because waiting would break the 30-second budget, and this is where the circuit breaker and the priority-2 backend take over.

What I tested

All runs used the synthetic files above on Linux with PowerShell 7.4.6, except the rows that name 7.6.7.

Test	Result
`-WhatIf` on the good design	Printed `Write report with 0 findings and 13 schedule rows` and created no output folder.
Normal run on the good design	Exit 0 with no findings. All six scenarios matched `expect`. Wrote `resilience-report.json`, `retry-schedule.csv`, `manifest.json` and `audit.log`.
Second run, same inputs	`No changes: output already matches these inputs.`
Edited `retry-schedule.csv`, run again	Stopped with “came from different input or was edited. Re-run with -Force.”
Same, with `-Force`	Rebuilt the output. Exit 0.
`-Rollback -WhatIf`	Listed three removals and removed nothing.
Edited the report, then `-Rollback`	Removed the untouched CSV and kept the edited report with a warning. The manifest stayed because a tracked file remained.
`-Force`, then a clean `-Rollback`	Only `audit.log` remained, with 7 JSON lines covering every write and removal.
Standard deployment on a different version than the provisioned one	1 error: G4, the spillover target must run the same model and version.
Broken design	16 errors, 5 warnings and 2 Info findings across 22 rules. T2, F2, G2 and R6 weren’t triggered by this variant. With retries on 403, no jitter, SDK and custom retries both on and `retry-after-ms` ignored, the five-error scenario failed fast on 502 (S2) and the 1-day Retry-After scenario ran out of responses still retrying (S1).
Invalid JSON in the design file	Stopped with `Invalid JSON in broken.json` and created no output folder.
Same seed twice	Identical `retry-schedule.csv` hashes. Seed 7 produced a different schedule.
PowerShell 7.6.7	Same schedule for the good design, and the same 16 errors and 5 warnings for the broken one.
PSScriptAnalyzer 1.25.0	No errors or warnings. 39 Information findings, all `PSAvoidUsingPositionalParameters`.

Not tested: anything in Azure. That includes Application Insights ingestion, the Agent Monitoring Dashboard, continuous and scheduled evaluation, Service Health alerts, API Management policies and circuit breakers, spillover, Foundry Control Plane token limits, the OpenAI SDK’s retry behavior, Cost Management budgets and real billing.

5. Public-objective concepts to retain

The objective names come from the November 16, 2026 skills list:[1]

“Recommend the process and tools required for monitoring agents.” Foundry’s three capabilities are evaluation, monitoring and tracing, with Application Insights underneath.[3] Copilot Studio has its own Monitor page and can send telemetry to Application Insights.[23][24] Foundry Control Plane gives the fleet view, and Application Insights has an Agents view across Foundry, Copilot Studio and third-party agents.[8][9]
“Monitor agent performance and metrics.” Latency, run success rate, token usage and evaluation scores on the Agent Monitoring Dashboard,[4] plus model metrics such as ModelAvailabilityRate, TimeToResponse and ProvisionedUtilization.[12]
“Interpret telemetry data for performance and model tuning.” A 429 from the deployment means TPM or RPM ran out, and the TPM estimate counts max_tokens.[13] A 403 from the gateway means a token quota ran out.[14][16] ProvisionedUtilization at 100% means provisioned calls get 429s.[12] Low cached_tokens on long prompts can mean the start of the prompt keeps changing.[22]
“Apply AI-based tools to analyze and identify issues and perform tuning.” Built-in agent evaluators, continuous evaluation on live traffic and scheduled evaluation on a test dataset to catch drift.[3][4][10]
“Analyze backlog and user feedback of AI and agent usage.” Copilot Studio’s Monitor page summarizes engagement, sentiment and feedback, with a customer comments summary in preview.[24]
FinOps facts worth memorizing: no hard spend limit on Azure OpenAI,[19] token quotas at the gateway,[14][16] spillover from provisioned to standard,[18] estimates that exclude discounts and provisioned throughput,[19] and playground evaluations billed by default.[3]
Drift: auto-upgrades for standard deployment types, manual migration for provisioned, NoAutoUpgrade stopping at retirement, and model router updates that can change performance and cost.[20][21]

6. Next step in the study journey

Next is the last AB-100 entry: what the exam asks of an agentic AI solutions architect, mapped across this series, before the series moves on to AB-731. This entry assumed a finished agent. For how it got to production, see the Copilot Studio ALM entry. For the tools this agent calls and how to review them, see the MCP and computer-use entry. For environment and data policy basics, see the AB-900 agent lifecycle entry. The AB-100 certification guide covers whether the Expert credential fits your career.

Sources

Designing Agent Observability and FinOps in Foundry

Top 10 Intune and Autopilot Enrollment Errors: Causes and Fixes