Speaker: Bob Walker (_BobJWalker_)
See live blog table of contents for more posts
Relearning the same lesson
- AI horror stores – clickbait/sensationalism. But outages real
- AWS – agent decided to rebuild prod
- AI ant deletes company’s database
- Not same exact mistake – rhyme with each other
- Sharing Cook county voter database
- Problem: People making direct changes to production
Non deterministic users
- AI and People are both non deterministic users.
- Both do things that are so baffling that you can’t explain it..
- Why did you do that?
- I decided it was the best approach I could take.
- Both can make same mistakes and burn down production
Why AI is like an intern
- Book smart
- Doesn’t understand consequences of production
- Like tellling an intern not to speed
AI philosophy
- Terminator: when told not to kill someone, shot in knee instead
- AI is an amplifier. If you don’t care about something, it won’t and vice versa.
AI Usage types
- Cyrborgs – AI is pairing partner
- Centaurs – AI is research assistant
- Self automators – AI is an Oracle
Causing Prod Problems
- Did show of hands for who caused a prod incident. Almost everyone. Commented if didn’t raise hand, lying or hasn’t happened yet.
- “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” – IBM
Humans
- “Humans will be involved every step of the way”. But humans caused problems too
- AI PRs can be too big for a human to review.
- Spent time trying to automate to get people out of the deploy to prod workflow. Solution isn’t to go back.
- OSS – more/bigger PRs
AI is getting better
- It is, but not at same rate as it did at beginning
- Models now tuned to benchmarks. AI companies helped right benchmarks
- When same problems happen over and over again, does good job. When something novel, tends to fall apart
Where AI fits in software delivery (LLMs and AI agents)
- Want deterministic process (ex: build) between non deterministic users and production
- Includes CI/CD pipelines, automated runbooks for common production incidents, creating/deploying infrastructure, logging/alerting
- AI and people contribute to the pipeline – ex: writing, approving part
Permissions
- AI should have different permissions than people
- ex: branch protection
Process Changes
- The process doesn’t change every day.
- Monthly at most
AI capabiilities
- AI can invoke deterministic processes via MCP/CLI/API via token
- Automate common decisions so non deterministic users aren’t making them.
Sliding scales of trust
- “I don’t trust my developers” – shouldn’t trust AI. ex: spotify. Inconvenience if don’t but not a material impact on lives
- “I fully trust my developers” – should trust AI. Have enough automation and guardrails in place to prevent making the wrong choice. Often not working for bank/insurance companies.
- Every company is different and changes over time. Also depends on company politics and policies
Takeways
- Non deterministic users shouldn’t make production changes
- More human involvement isn’t a panacea
- Automate deterministic decision making for non-deterministic users
- Non deterministic users use and contirbute to deterministic processes
Action Items
- Create AI skills or context boundaries as first line of defense. This is like a chain link fence to protect Fort Knox
- Table top work case scenarios. – ex: maliciuos insider
- Establish clear boundaries, create deterministic processes/runbooks, create deterministic approvals for common decisions, build contributions guidelines for AI or people
- AI on behalf of a person uses different API keys/tokens (and different permissions – ex: can deploy but not make changes)
My take
Bob said a few times that purposely vague/vendor agnostic. This session was a great start to the morning. AI course but focused on a perspective. I really like the idea of humans and AI being non-deterministic users as a commonality. I also like trust being situational. Also excellent summary!