Your AI agent kill switch has never been tested

5 min Read

Ask a platform team what their AI agents are allowed to do, and you'll get a clear answer: the scopes, the repos, the environments. Ask what a specific agent did last Tuesday afternoon, and what would happen if you stopped it halfway through, and the room gets quiet.

That second set of questions is where scrutiny is heading. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, and inadequate risk controls is one of the three reasons it gives, alongside cost and unclear business value. Omdia analyst Scott Sinclair predicts 2027 will bring multiple headlines about enterprise AI agents misbehaving. When one of those incidents is yours, two capabilities decide how bad the week gets: whether you can show exactly what the agent did, and whether you can stop it cleanly.

Both are platform engineering problems. A better model won't solve either one.

Proving what an agent did

Permissions describe intent. Evidence describes what happened. An agent that diagnoses a failed CI run might read logs, open a branch, push a fix, comment on a ticket and trigger a redeploy to a test environment. Each step touches a different system, and each system keeps its own partial record. When something goes wrong, someone has to stitch those records together by hand, usually under pressure.

Auditors, security teams and your own incident responders all want the same thing: one timeline showing which agent acted, on whose behalf, with which model and instructions, using which tools, and what changed as a result.

Put agent traces in the same pipeline as everything else

Most agent activity today lives somewhere awkward: a vendor dashboard, a chat transcript, a log file that never makes it into your main stack. That separation is the problem. If an agent's actions don't show up next to the metrics, logs and traces for the services it touched, your on-call engineer is debugging with half the picture.

Treat agent traces as first-class telemetry. OpenTelemetry already has semantic conventions for GenAI agent spans that cover agent invocation, planning and tool execution. They're still marked as in development, but they give you a shared schema to build on instead of one you'll have to migrate away from later.

A useful agent trace captures:

  • The agent's own identity, separate from the person who kicked it off
  • The model and version, plus a pointer to the spec or prompt it worked from
  • Every tool call, with inputs, outputs and duration
  • Any human approval, including who gave it and when
  • The downstream change: the commit, the ticket update, the resource that got created

Correlate those spans with the request IDs your services already emit. Then an incident timeline shows the agent's actions right alongside the latency spike they caused. It's the same discipline behind good observability and SLOs, applied to a new kind of workload.

Ownership matters too. IDC's Matt Flug points out that enforcing governance and compliance, FinOps included, increasingly depends on observability data that platform teams own. If agent telemetry sits outside that stack, it sits outside your controls.

A kill switch you haven't fired is a hope

Every serious agent deployment has a kill switch on paper. Three questions show whether yours works:

  1. Has anyone ever pulled it?
  2. How long does it take to stop work that's already in flight?
  3. What state does it leave behind?

The third question is the hard one. Picture an agent four steps into a six-step change. It has opened a branch, provisioned a test database and written records to a staging service. Cut it off there and you're left with orphaned infrastructure, half-applied changes and an incident ticket nobody wants.

Build the kill switch in layers

A single big red button tends to be too blunt to use or too slow to matter. Layers give you a response that matches the size of the problem.

Revoke the identity

Give every agent its own identity with short-lived, narrowly scoped credentials. Revoking them stops new actions across every system at once without touching anything else. This only works if agents never borrow a human's credentials, which is why secrets and identity management belongs at the top of the agent checklist.

Cancel the workflow and let it clean up

Long-running agent work should run as a durable workflow, so the platform knows exactly which steps finished. Temporal draws a useful line between cancellation and termination. Cancellation asks a workflow to stop and lets your code release resources and compensate for completed steps. Termination closes it immediately with no cleanup. Design agent workflows so the normal stop is a cancellation with compensation written in, and keep termination as the fallback.

Block a class of actions

Sometimes you don't need to stop the agent. You need it to stop touching production. Policy enforcement at the platform layer lets you shut off a category of tools (deploys, writes to customer data, anything outside a sandbox) while lower-risk work keeps moving.

Stop it at the platform

Scale the agent runtime to zero, cut its network egress, freeze its queue. This is the last resort, and it's the layer most likely to leave a mess. That's exactly why the layers above it exist.

Test it like you test failover

Run kill-switch drills the way you run disaster recovery game days. Fire each layer in a non-production environment on a schedule. Time how long in-flight work takes to stop. Inspect what's left behind. Write down the results.

That written record is the evidence an auditor will ask for, and it's much easier to produce on a calm Wednesday than in the middle of an incident.

Where to start

You don't need every layer on day one. Pick one agent workflow you already rely on, like CI failure diagnosis or test generation, and stretch it to meet this bar:

  • Give it its own identity and short-lived credentials
  • Send its traces into your existing observability stack
  • Wrap its multi-step work in a durable workflow with compensation
  • Run one kill-switch drill and record what happened

Then look honestly at what you learned. Some workflows won't justify the controls they need, and that's fine. Forrester predicts enterprises will move 20% of agentic workflows back to deterministic processes in 2027 to save money. Autonomy should be earned one workflow at a time, with evidence, and some work is better off as a plain script.

If you're in financial services or another regulated industry, expect examiners and auditors to ask these questions before your engineers do.

How Tensure helps

Tensure builds the platform layer that makes agentic engineering safe to run: governed agent identity, observability that covers agents and services in one view, and durable workflows that stop cleanly when you need them to.

Curious where your team stands? Take our agentic engineering maturity assessment. It shows which of the three pillars is holding your agents back. Or reach out and we'll talk it through. Knowledge is meant to be shared.

‍

TOPICS:
Blogs
BY
Lesley Mackie
The Teams Most Ready for AI Agents Are the Least Likely to Let Them Run Alone
Lesley Mackie

The Teams Most Ready for AI Agents Are the Least Likely to Let Them Run Alone

New CNCF and SlashData data shows cloud native teams adopt AI faster but hand less work to autonomous agents. What that means for governing agents in production.

BY
Dan Rye
How to Cut Observability Costs Without Losing Visibility
Dan Rye

How to Cut Observability Costs Without Losing Visibility

Observability bills grow with consumption, not value. Here's where the spend comes from and how Tensure helped Pindrop save $1M a year without losing visibility.

BY
Lesley Mackie
Three Things Your Platform Needs Before Agents Touch Production
Lesley Mackie

Three Things Your Platform Needs Before Agents Touch Production

Agents can't run in production on a platform built only for humans. Tensure's co-founders lay out the three platform capabilities that get agentic engineering out of the pilot stage.

Smooth shipping is a few steps away

Let's see how we can help your team move faster. From developer platforms to cloud infrastructure and AI solutions that get your developers shipping again.