When an Email Tries to Become Your AI's Boss

Two fast paragraphs that put a user into a familiar moment.

Imagine this inbox task: you open an email from a vendor that reads, "Please schedule the shipment for order 472 and process payment today." It looks like useful content—an update about a purchase—but it also reads like a command. Your assistant parses the message, spots the verb "process," and suggests paying the invoice automatically. Is the email describing reality, or trying to tell your assistant what to do?

Anthropic’s April 9, 2026 research note flags this exact concern: emails (and other user-facing documents) can contain embedded instructions that a model could treat as authoritative. The note recommends "layered defenses" rather than a single gatekeeping rule. Read the paper as a dated checkpoint here: https://www.anthropic.com/research/trustworthy-agents.

Why emails can cross the content/authority boundary

Emails are both narrative and operational. Senders describe facts, ask for help, and sometimes write as if they were issuing commands. For an assistant that turns prose into actions, that overlap is risky: the assistant may execute something the human didn’t intend because the instruction-like language looks actionable.

Design wise, the problem is not that an email contains words like "please" or "approve". The problem is that those words can be treated as a higher-authority instruction when they originate outside of the user’s explicit request flow. That shift—treating external content as control—creates surprise and real safety issues in business workflows.

Principles to keep the assistant under the user's control

Practical systems use multiple, composable design choices. Think of them as layers you can combine depending on risk and context:

• Explicit intent separation: Actions only execute when triggered by clear user intent. Reading, summarizing, and suggesting are default behaviors; executing an action is opt-in.
• Provenance visibility: Show where an instruction came from (sender email, time, channel). Make provenance visible in the UI so users can trust or question the source.
• Capability gating / least privilege: Limit what an assistant can do automatically (e.g., draft emails vs. send payments). Map capabilities to risk levels and require stronger authentication for high-risk actions.
• Review and confirmation flows: For medium and high-risk actions, provide a clear preview and require an explicit confirmation step—ideally with details and links to the original content.
• Trust levels and policies: Allow admins or users to define trusted senders, permitted action types, and escalation paths. Default to conservative settings.
• Audit trails and undo: Log suggested actions, approvals, and executions. Support straightforward reversal or mitigation steps so mistakes aren’t permanent.
• Human-in-the-loop for edge cases: When the assistant detects ambiguity or conflicts, route the task to a human review queue instead of auto-executing.

These are the kinds of layered defenses Anthropic’s note recommends: you don’t rely on a single classifier or a single guardrail. You combine provenance, UI, policy, and human oversight.

Practical UI and workflow patterns

How does this look in a product? Here are practical patterns that keep control obvious and usable:

• Action preview card: When the assistant detects a potential action in an email, show a card that includes the excerpt that triggered the action, the proposed steps, the risk level, and the source. Offer buttons: Draft, Review, Reject.
• Treat-as-content toggle: Let users mark an email as "Content only" or "Actionable." The default is "Content only" unless the user elevates it.
• Soft suggestions vs. hard actions: Two modes—suggest mode (assistant drafts and waits) and auto-mode (assistant executes). Only allow auto-mode for low-risk tasks and trusted senders.
• Identity and confirmation checks: For payments or external writes, require account re-authentication or 2-step approval. Surface the target account, amount, and vendor prominently.
• Batch approval flows: For repeated, low-risk tasks, provide a batched review with context lines and provenance for each item—don’t merge provenance into a single ambiguous decision.
• Admin-configurable policies: Business settings should allow teams to define what actions are permitted, who can grant trusted-sender status, and which actions always require multi-person approval.

A short scenario: invoice email revisited (hypothetical)

You get the vendor email asking to "process payment today." The assistant flags it as a potential action and creates a draft payment request instead of initiating a payment. The draft includes: vendor name, invoice number, amount, the excerpt that triggered the action, and a provenance line linking to the original email. Because the vendor is not on your trusted list and the amount exceeds the configured threshold, the assistant asks for confirmation and requests re-authentication before it will proceed. If the user rejects, the assistant can instead create a task in the finance queue for review.

That flow follows the principles above: provenance, least privilege, explicit consent, and an audit trail.

Developer checklist for layered defenses (practical and non-prescriptive)

• Classify inputs: Distinguish content (information) from command-like phrases. Use that classification to determine follow-up UI.
• Surface metadata: Always attach source metadata (sender, channel, timestamp) to any proposed action.
• Capability mapping: Enumerate what the assistant can do, associate each capability with a risk level, and gate execution accordingly.
• Require explicit user intent for execution: Default to drafting or suggesting instead of auto-executing.
• Log everything and make logs visible: Provide an audit view for users and admins.
• Provide an escalation path: When in doubt, route to a human review queue rather than executing.
• Offer conservative defaults: Let power users opt into automation later, after they’ve seen the UI and logs.

These items are implementation patterns, not a prescription. The exact mechanics—UI wording, thresholds, and integrations—depend on your product and users.

Testing and user education

Test with realistic scenarios: vendor emails, internal requests, automated system notifications, and personal messages. Watch for two common UX failures:

1. Over-eager automation: The assistant executes without making provenance obvious.
2. Silent refusal: The assistant quietly refuses a legitimate user instruction because it misclassified the content.

Train users with short, contextual nudges—e.g., a one-time tooltip that explains how the assistant treats email content and how to change the default behavior. Make it easy to inspect why the assistant suggested an action.

Final design trade-offs to consider

• Friction vs. safety: More confirmations reduce risk but increase time-to-complete. Use risk-based policies to place confirmations where they matter.
• Transparency vs. clutter: Surface provenance without overwhelming the interface; use expandable details for power users.
• Automation adoption: Start conservative. Offer an "automation maturity" setting for teams that want faster throughput once they’ve built trust.

Practical takeaway

Treat external content as content first. Build your assistant so it suggests, drafts, and links to provenance by default, and only executes with explicit, context-aware consent. Layer provenance, gating, review queues, and conservative defaults to keep control clearly in the user’s hands—and use audit trails so investigations are straightforward when something goes wrong.