Speaker: Avindra Fernando
See live blog table of contents for more posts
Survey
- Most have ai agent writing code
- Most jave markfown file agent reads
- About half have written a kskill
- A handful have a playbook/automated workflow for skill
- A few have tested whether skill triggers
Reusable intelligence in repo
- agents.md, claude.md
Skills
- Prompt tells an agent what to do once
- A skill tells it how to work here, every time
- Create skill when see action being repeated over and over
Flow
- Context
- Skills
- Automate skills, make playbooks
- Prove did work
- Maintain codebases
When people say “prompt file” it could mean
- Context file (always gets loaded). Not helpful when file gets too long. Overwhelms the agent
- Skill – loaded on demand
- Subagent – roles – ex: planner, tester, reviewer
- Command – playbook combining context/skill/agent
- Hook – systematic gates so can restrict access
Learnings
- Agent will figure out the how. The why is important
- Want agent to stick with rules
- Train like a new human teammate, would give them the why
- Specify hooks – things absolutely don’t want it to do
Open Skills
- Standard format
- Name, description, body
Other notes
- Grill me – takes a vague idea and interviews you to refine it. need to know when to end
- Found a place it didn’t trigger because nothing run
Agent types
Can choose whether to check in with human after each step.
- Planner agent – turns story into a plan. Does not do any coding. Good place to use latest models.
- Implementer/coder agent – takes plan and implements it. No creativity. If problem with plan, that’s on you. Can use less expensive agent.
- Tester agent – run tests
- Reviewer agent – code review. Do not fix; just identify issues.
- Also 3rd party agents can reuse
Playbooks
- Agents for planning, implementing, testing and reviewing
- See what generating – give format want plan output in
Failures
- Capture why
- Refine skill, figure out if wrong skill files, etc
- If agent makes mistake, add to agent’s md file so doesn’t make same mistake
- Surprising changes in cost
Future of reviews
- Different tools, not looking at code
- New people will be used to a higher level of abstraction.
War stories
- The undocumented deployment – wiped out config. Lesson is to create the instructions when inherit an undocumented codbase
- The queue that woke up – mail queue had dormant key for years. When restarted, all the emails went out, lucky not on prod.
- When the reviewer goes wrong – memoized when not right answer and caused infinite re-render
Loop
- Prompt engineering optimizes a message
- Loop engineering optimizes the pipeline
My take
Avidra has been linking tweets about my live blog so cool he gets a post in it! I missed a few bits of this as I had a moment of panic (I booked the wrong flight home and couldn’t put it out of my mind to fix it later). Overall very good. I learned a bunch and am inspired to make skills/playbooks. Particularly enjoyed the demo.