ServiceNow AI Control Tower Runtime Observability: A UK AI Steward Playbook

Runtime observability is the Steward control — not another dashboard tile

ServiceNow's August & September 2026 AI Control Tower (AICT) wave ships runtime monitoring and evaluations: continuous telemetry from ServiceNow and external agents, scored by LLM-as-a-judge against quality and safety metrics, with results on the Monitor dashboard. Through the Traceloop acquisition (integrated into AICT), Stewards can observe external agents — not only Now-platform agents — via Trace Collectors or the Traceloop SDK.

This playbook is the production gate for observability. It deliberately does not rehash Kill Switch (credential revoke / runtime stop across Okta, GCP and Bedrock) — that is a separate IR control covered in our Kill Switch UK CISO playbook. Observability tells you what is degrading; Kill Switch contains what is compromised.

Enterprise security and observability

What shipped (facts for CAB / AI Steward packs)

Why UK estates should care

UK AI Stewards and platform owners are already inventorying agents. Without runtime evaluations:

1. Quality drift and tool-path failures show up as user complaints, not Steward tickets.
2. External agents (CrewAI, LangChain, hyperscaler runtimes) remain blind spots if you only watch Now Assist.
3. DPIA / audit packs need trace-level evidence and judge reasoning — not screenshots of a happy path.

Observability closes that gap: sessions → traces → spans, scored and rolled up to system and portfolio quality/safety scores on Monitor.

UK production gate — numbered steps

1. Patch, plugin and role before any metric toggle

1. Confirm instance is on Zurich Patch 11 or Australia Patch 4+.
2. Install / verify AI Native Experience for AI Control Tower.
3. Assign sn_ai_observe.ai_observability_admin (or full sn_ai_governance.ai_steward) to named Stewards — not a shared admin account.
4. Plan the new AICT UI cutover; most observability tiles live there.

2. Inventory first — mark agents managed

1. Populate AI inventory (manual entry, Service Graph connectors, or Shadow AI detection).
2. Mark pilot agents managed — evaluations only run on managed assets.
3. Prefer connectors that bring the metadata Trace Collectors need (provider, region, resource IDs).
4. If you use domain separation, verify Steward visibility and MID scope per domain before go-live.

3. Choose ingest path: Trace Collector vs Traceloop SDK

For collectors you need: validated MID Server, cloud credentials, region (AWS), and managed AI asset records. For SDK: valid x-snc-observability-token and a user with sn_ai_observe.ai_data_sender.

4. Select metrics, sampling and weighting

1. Open Rules and Templates → Evaluations.
2. Enable quality and safety metrics for ServiceNow and external agents separately (lists differ).
3. Set sampling above 0% (default 5%); raise for pilots, then tune for cost/assists.
4. Optionally edit metric templates so quality/safety weights sum to 100%.
5. Remember: assists for evaluation are billed at 500 assists/month per managed AI asset (vendor FAQ) — model that into CAB cost notes.

5. Prove Monitor before you claim production readiness

1. Generate real traffic on the pilot agent.
2. Confirm sessions/traces/spans land (sn_ai_observe_ai_session / _trace / _span).
3. Open Monitor: portfolio scores, ranked systems, top action items, session drill-down with LLM judge reasoning.
4. Build a Workflow Studio path from low scores on sn_ai_observe_ai_trace → AI task / incident (in-product threshold triggers are limited today).
5. Document DPIA delta: what prompt/tool content enters evaluation, retention, and who can see traces (Steward role sees broadly — plan ACLs if needed).

What good looks like in CAB evidence

• Patch level + AI Native Experience confirmed.
• Named Steward with observability admin role.
• Managed inventory for pilot agents (internal + at least one external path if in scope).
• Working Trace Collector or Traceloop SDK with a successful scored session on Monitor.
• Sampling and assist-cost note signed by product owner.
• Explicit boundary: observability ≠ Kill Switch; both owners and runbooks linked.

Adjacent AICT items (do not conflate)

• Kill Switch — contain by revoking credentials / stopping runtime (IR).
• AI Gateway MCP pause — pause MCP servers at the gateway boundary.
• Post-runtime OWASP security evaluations — complementary security scoring, not a substitute for quality Monitor.
• Otto / Guided Setup — accelerate Day-0 wiring; they do not replace metric ownership.

Closing

Treat AICT runtime observability as the UK Steward operating system for production agents: inventory → managed → traces → evaluated → Monitor → remediate. Wire Traceloop or Trace Collectors once, rehearse a scored session in non-prod, then promote with assist-cost and DPIA notes in the CAB pack — and keep Kill Switch as a separate, rehearsed containment control.