Test, Fix, and Trust Your Automation

Start with an example

A staffing agency automated its candidate follow-up emails. For three months it worked perfectly. Then someone updated the job application form and renamed the “First Name” field to “Preferred Name.”

The automation didn’t crash. It just quietly kept running, with nothing in the name spot. For two days, dozens of candidates received emails that began: “Hi ,”

Nobody noticed until a candidate posted a screenshot online. The fix took five minutes. The embarrassment took much longer to fade.

This lesson is about catching problems like that before customers do, and fixing them quickly when they slip through.

Why automations break

Automations are chains. When one link changes, the whole chain can fail, sometimes loudly with an error and sometimes silently with bad output. Most failures come from four causes:

Cause	What happens	Example
Missing or changed data	A field is blank, renamed, or in the wrong format	“Hi ,”, or a date written as text
Changed apps	A connected app updates, or a login expires	The email connection stops working after a password change
Prompt problems	The AI misunderstands or gets an unexpected input	A gibberish message gets a polite reply
Limits and errors	You hit a usage limit, a spending cap, or a temporary outage	The workflow stops halfway through the month

Knowing these four causes turns troubleshooting from guesswork into a checklist.

Step 1: Test with a plan

Don’t test by randomly clicking around. Write a test plan: a list of inputs and what should happen for each one.

Good test plans include four types of cases:

1. Normal cases: typical, everyday inputs. “Do you offer weekend appointments?”
2. Edge cases: unusual but possible. A blank optional field, a 2,000-word message, a name with an apostrophe like O’Brien, an email in all capital letters.
3. Messy cases: typos, gibberish, or two questions in one. “hi do u do repairs and also how much”
4. Risky cases: angry, urgent, or sensitive messages. “This is the third time you’ve messed up my order.”

#	Test input	Should happen	Did happen	Pass?
1	Normal service question	Friendly draft with booking link
2	Blank name field	Draft starts “Hi there,”
3	Angry complaint	Flagged NEEDS HUMAN, owner alerted

Fill in “Did happen” as you run each test.

Step 2: Read the run history

Every automation tool keeps a run history (also called a task log or execution history). It’s the most useful troubleshooting tool you have. For each run, it shows:

• Which step succeeded or failed
• The error message, if any
• The exact data each step received and produced

When something goes wrong, open the run history and work through the steps in order. Find the first step where the data looks wrong. That’s almost always where the problem starts.

Step 3: Fix one thing at a time

When you find a problem:

1. Fix one thing.
2. Re-run your entire test plan, not just the case that failed.
3. Then move on to the next fix.

Changing several things at once makes it impossible to know which change helped, and it can create new problems you can’t trace.

Common fixes

Problem	Fix
Blank fields break the message	Add a fallback, like “Hi there” if Name is empty, or a filter that stops the run
AI invents details	Add a rule to the prompt: “Use only the information provided. Never state prices.”
AI replies to gibberish	Add a filter: only continue if the message is longer than 10 characters
Connection expired	Reconnect the account in the tool’s settings
Hit a usage limit	Check your plan, or raise your spending cap carefully

Step 4: Watch it after launch

Testing before launch catches most problems, not all. After going live:

• Turn on error alerts. Most tools can email you when a run fails.
• Spot-check samples. Read a few outputs every day for the first two weeks, then weekly.
• Re-test after any change. If anyone edits the form, the prompt, or a connected app, run your test plan again.

Step 5: Earn trust in stages

Don’t jump from “just built” to “fully automatic.” Move through stages, and use numbers to decide when to advance:

Stage	What it means	Move on when…
1. Testing	Fake data only	All test cases pass
2. Drafts only	Real data; a person approves every output	Fewer than 1 in 10 drafts need real edits for 2–4 weeks
3. Limited auto-send	Low-risk cases send automatically; others stay as drafts	Error rate stays near zero and spot-checks look good

Some workflows should never reach Stage 3, especially anything involving health, money, legal matters, or upset people. That’s not a failure; it’s good judgment.

A worked example: Rashad’s troubleshooting

Rashad built a workflow that turns new online orders into thank-you emails and adds each order to a shipping spreadsheet. One Monday, he gets an error alert.

He opens the run history:

• Trigger (new order): ✓ succeeded
• Add row to spreadsheet: ✓ succeeded
• AI step: ✗ failed: “Usage limit reached”

Diagnosis: His AI provider account hit its monthly spending cap because a weekend sale brought in three times the usual orders.

His fixes, one at a time:

1. He raises the cap a modest amount after checking that the cost per email is still less than a cent.
2. He re-runs the failed orders from the run history so those customers get their emails.
3. He adds a calendar reminder to check usage before big sales.

Then he re-runs his test plan. Everything passes. Total time from alert to fix: about 20 minutes, and no customer was ever confused.

Think deeper: testing something that never answers the same way twice

Normal software is easy to test: same input, same output, pass or fail. AI doesn’t work that way. It might word an answer differently every time, so a test that looks for an exact sentence fails even when the answer is fine.

So professionals test with criteria instead of exact answers:

• Did it include the booking link?
• Did it stay under 120 words?
• Did it avoid quoting a price?
• Did it flag the emergency message?

A person can check these, a simple rule can check some (like word count), and increasingly a second AI is asked to grade the first one. That grader needs checking too, but it makes testing hundreds of cases practical.

A saved set of test cases with criteria is called an evaluation set, or “evals.” Teams rerun them every time they change a prompt, switch models, or update an app. That catches regressions: things that used to work and quietly stopped, often because fixing one problem created another.

After launch, teams keep watching. They track numbers over time, set alarms (like “investigate if more than 20% of drafts need editing for a week”), and keep logs of what went in and what came out.

Pilot, measure, then expand. It’s slower than switching everything on at once, but it replaces hope with evidence.

Questions to sit with:

• How do you decide whether to trust a new person at work or school? How is that like testing an AI system?
• If an AI is right 98% of the time, is that good enough? What would the other 2% need to be for you to say yes?

Practice: write and run a test plan

1. List eight test cases for your workflow: 3 normal, 2 edge, 2 messy, and 1 risky.
2. Write what should happen for each.
3. Run all eight and record what did happen.
4. Find the first failure and use the run history to see which step went wrong.
5. Fix one thing, then re-run all eight.
6. Decide which trust stage your workflow is ready for, and why.

Next, you’ll set the rules for where people stay in charge, and how to measure whether your automation is truly working.