Designing trust: how I cut task time 60% and errors from 15% to under 3% in a high-stakes AI compliance tool

A system-level behavioral redesign that replaced a rejected Appian (low-code platform) prototype with a human-in-the-loop platform built for high-stakes regulatory work.

Role
Product designer (UX strategy and end-to-end design)
Duration
4 months
Client
Under NDA
Status
Shipped

6 min read · Jump to:Key decisionFinal designImpact

Problem
Regulation SMEs rejected an AI pilot that was technically accurate because they didn’t trust its output and it didn’t fit how they worked.
My role
Led UX strategy and end-to-end product design, partnering daily with the engineering and AI lead, PM, client product owner and SMEs.
Key decision
Turned a UI facelift into a behavioral redesign and chose accuracy over speed: human review on every AI line item, plus Excel export.
Outcome
The client awarded the contract for the platform.
64 → 87System Usability Score (SUS)vs. pilot
15% → <3%Error ratewith active human review
60%Faster task completionvs. original Excel workflow

The final review screen: SMEs check and mark each AI-extracted obligation.

Every new regulation kicks off a slow, manual review where mistakes are costly

Every time the FDIC or SEC releases a new regulation document, subject matter experts (SMEs) at large financial institutions go through a grueling, manual workflow:

Regulation document released

The FDIC or SEC publishes a new rule

Broken down into obligations

SMEs extract each requirement by hand

Map to internal policies

Each obligation matched to the bank’s controls

Create implementation plan

Plan the changes needed to comply

Every new regulation repeats this cycle, mostly by hand.

The cost of mistakes is high: in 2024 alone, the SEC collected $8.2 billion in compliance-related penalties. Regulations change constantly, and these institutions operate across overlapping global jurisdictions.

The AI pilot was accurate, but SMEs still rejected it

Our engineering team trained an AI model to break large documents into granular “obligations” and map them automatically. Instead of manual extraction, SMEs only needed to review and correct the AI’s work. They didn’t adopt it.

Accuracy didn’t earn adoption: the pilot broke SMEs’ trust and their workflow

I analyzed 15 recorded sessions of regulation change analysts working cases in their current Excel workflow and in our pilot, then ran contextual inquiries with SMEs.

Pilot application review screen
The pilot SMEs were testing.

Despite the model’s high accuracy, the pilot failed for four reasons

  1. 1

    It didn’t fit their existing mental models or workflows.

  2. 2

    They completely mistrusted the automated output.

  3. 3

    They felt a lack of agency to properly correct what the AI generated.

  4. 4

    It felt like it was slowing them down rather than speeding them up.

I mapped each trust risk to a specific design response

Trust risk
What I designed

Automation bias

Trusting the system’s output blindly over personal judgment, leading to overlooked errors.

Mandatory “Mark as Reviewed”

Every AI line item needs an explicit human check before a case moves on.

Complacency

Vigilance drops when automation seems highly accurate over long periods.

Reviewer + Approver flow

A second SME approves each case and can send specific items back with comments.

Loss of situational awareness

Losing the big picture of the regulatory landscape when you don’t extract it yourself.

Traceability + mapping view

Each obligation links to its exact source text, and an alluvial chart shows how obligations map to controls.

Skill degradation

Fear of losing expertise when work shifts from active production to passive monitoring.

Excel integration

SMEs can export AI output into their familiar Excel layout and keep working as active editors.

That became two workflow changes I proposed to the team

Proposal 1: Excel integration

AI parses doc, then structured CSV, then uploaded back to system after edits
  • Download AI output as .CSV to augment the existing workflow instead of replacing it
  • Deep links to exact source document sections for easier referencing

Proposal 2: Human-in-the-loop

AI parses doc, reviewer, approver, next phase
  • A second SME as reviewer on every case to catch errors
  • Mandatory “Mark as Reviewed” on every AI line item (active monitoring)

I chose accuracy over speed

I A/B tested a speed-focused workflow against an active-monitoring workflow with SMEs. Both beat the original manual workflow on time, and the speed-focused one was fastest, but it produced more than five times the errors. In a domain where one missed obligation can mean penalties, the lower error rate mattered more.

MeasureSpeed focusActive monitoring Chosen
Error rate15%<3%
Case completion~80% faster~50% faster
User confidenceLowHigher

5 SMEs per workflow. Case completion speed is compared to the original manual workflow.

The final platform keeps SMEs in control of every AI output

With the strategy validated, we designed and built the full web platform from scratch, centered on one clear review flow that keeps SMEs in charge of every AI output.

Redesigned review screen
The pilot review screen

Drag the slider to compare the pilot (left) with the redesign (right).

Before: a cluttered pilot with no visual hierarchy: case details crammed into side panels and AI obligations buried in a dense, generic grid, so nothing told SMEs where to look first.
After: a clean, minimal UI that directs attention to the work: a clear case header and stage stepper, then each AI obligation as a line item SMEs review and mark off, with reviewed items hidden on demand.

A scannable table replaced hard-to-scan cards

The pilot used a card layout. SMEs pick and track cases by keywords, jurisdictions and mandates, so I moved to a table with quick filters and auto-sorting by status.

Case management table with callouts for quick filters, tabular scanning and auto-sort by status
Case management. Overdue and due-soon cases sort to the top with semantic status colors.

Four patterns keep humans meaningfully in the loop

Traceability

Every AI obligation shows the exact path to its source. Clicking it opens the original PDF with the source text highlighted.

Active monitoring

A mandatory “Mark as Reviewed” check on each line item tracks progress and counters drops in vigilance.

Excel integration

AI output downloads as a CSV structured like SMEs’ existing workflow, so adoption starts somewhere familiar.

Reviewer → Approver flow

Approvers comment on and flag specific obligations, returning them to the reviewer with precise context.

An alluvial chart shows obligation-to-control mapping at a glance

Each obligation can map to multiple internal compliance controls, and vice versa. The chart gives a macro view of mapping density, and clicking a ribbon filters the work items.

Interactive alluvial chart of obligations mapped to controls
Mapping review. Coverage is summarized up top (fully, partially and not covered) before SMEs approve.

A design system built for dense enterprise data

I built a custom design system tailored to high-density data: workflow states, status tags and coverage levels that stay legible in long tables.

Workflow stepper components with default, active and completed states Status tags, risk coverage tags, notes and case activity components
Top: stage stepper in default, active and completed states. Bottom: status and coverage tags, notes and case activity.

Plus a reference library and case history

Reference library: a network information architecture to explore regulations, policies, controls and their relationships. Case management: past cases and a manager overview.

Left: reference library. Right: case management.
See 5 more screensObligation detail, control mapping, tasks, risks and controls

The redesign won the client contract

The client awarded the contract after final testing. Pivoting from a cosmetic facelift to a behavioral redesign moved every metric that mattered:

15% → <3%Error rate

Human-in-the-loop principles improved error catching.

64 → 87System Usability Score (SUS), vs. pilot

UI/UX changes raised the product’s usability rating.

60% fasterTask completion, vs. original workflow

AI augmentation of the workflow plus platform efficiency.

What I’d bring to the next AI product

Measure the right metrics

The platform was sold on speed, but SMEs judged it on trust and risk.

Design for the human-as-editor shift

As AI automates work, users move from creators to curators of output, and UX has to optimize for that.

Design the transition from legacy tools

Forcing experts to abandon familiar software causes rejection, however good the new tool is. Adoption needs to be designed.