Six Hard Lessons From Building With AI Agents

In this episode of the Vernon Richard show, the hosts discuss their experiences with AI tools and agents, focusing on the challenges and lessons learned from using these technologies in coding and software engineering. They explore best practices for utilizing AI effectively, the importance of context in interactions with AI, and the future of AI agents in the workplace. The conversation highlights the balance between leveraging AI for efficiency while maintaining control and understanding of the underlying processes.

Links to stuff we mentioned during the pod:

00:00 - Intro
01:17 - Welcome
01:30 - TANGENT BEGINS... All kinds of egregious waffling follows. Skip to the actual content at 08:34
01:31 - Rich VS Tree Stump
01:57 - What on earth did Rich need the pulley for?
02:26 - Vern's nerdy confession and pulley confusion
02:52 - Does Rich live next door to Tony Stark?!
03:22 - What to do when you need a steel RSJ
03:35 - We admit defeat. 03:36 - Welcome to Rich's Garden Adventures Podcast!
07:25 - What has Vern been up to?
08:34 - We attempt to segue into the episode at last!
08:35 - TANGENT ENDS...
08:51 - Rich’s POC: using agents to help build AI tools
09:45 - The Replit disaster: vibe coding meets deleted production data 11:12 - Sociopathic assistants and the case for AI gaslighting 11:55 - Vernon wants his team experimenting with AI tools
12:50 - Rich explains the context for his latest AI adventures
13:18 - Rich’s bench project and “putting the engineering hat on” 
15:22 - Setting up the stack and staying in control 
16:53 - A familiar story: things were going fine until they weren’t 
17:00 - Ask vs Edit vs Agent mode in Copilot explained 
19:06 - The innocent linting error that spiralled out of control 
21:16 - Stuck in a loop: “I didn’t know what it was doing, but I let it keep going” 
22:11 - The fateful click: “I’m going to reset the DB” 
23:10 - The aftermath: no data, no damage… but very nearly 
23:33 - Security wake-up call: agents are acting as you 
24:39 - You can’t fix what you don’t know it broke 
25:52 - Can you interrupt an agent mid-task? 
27:14 - When agents get “are you sure?” moments 
28:15 - Tea breaks as a dev strategy: outsourcing work to agents 
29:24 - Jason Aborn vs Keith & Maaike: where Rich sits on the AI enthusiasm spectrum 
30:41 - Tip1. The first of Rich’s 6 agent tips: commit after every interaction
32:12 - Why trusting the “keep all” button is risky 
34:01 - Writing your own commits vs letting the agent do it 
35:26 - When agents lose the plot: reset instead of fixing 
36:55 - “You’re insane now, GPT. I’m giving you a break.” 
37:54 - Tip 2: Make the task as small as possible 
39:59 - The middle ground between 'ask' and full agent delegation 
41:12 - Tip 3: Ask the agent to break the task down for you 
43:36 - The order matters: why you shouldn’t start with the form UI 
44:33 - Vernon compares it to shell command pipelines 
45:09 - It can now open browsers and run Playwright tests (!) 
46:23 - Star Trek and the rise of the engineer-agent hybrid 
47:57 - Tips 4–6: Test often, review the code, use other models 
49:39 - Pattern drift and the importance of prompt templates 
50:51 - Vernon’s nemesis: m dashes, emojis, and being ignored by GPT 
51:48 - Context engineering vs prompt engineering 
52:43 - When codebases get too big for agents to cope 
53:40 - Why agents sometimes act dumber than your IDE 
54:32 - The danger of outsourcing good practices to AI 
54:48 - Spoilers: Rich’s upcoming keynote at TestIt 
55:01 - Agents don’t ask why — they just keep going 
56:42 - Goals vs loops: when failure isn’t part of the plan 
58:32 - The question of efficiency: is training agents worth it? 
59:47 - Rich’s take: we’ll buy agents like we buy SaaS 
61:08 - Meredith Whittaker’s warning: what agents really need access to 
63:38 - Code secrets, browser history, and encrypted chaos 
64:39 - Rich’s final verdict: value through engineering know-how 
66:04 - Teams with poor habits aren’t ready for AI 
66:42 - Outro: next time… how to help your team explore AI tools