Speaker: James Ward
See the table of contents
General
- LLMs are knowledgeable translators
- Agents are LLMs with integrations in a loop. Continue until achieve users goal or give up.
Travel
- Pitch “The AI can book your travel for you”
- Not there yet.
- [personally, I’m not ready to give AI my finances. Also, I like tp pick my flight, not outsource that. I didn’t even have a human admin assistant book mine in the past]
Inference API
Using AWS Converse
- Text > LLM > Text
- Lext > LLM > JSON > Object
- Text > LLM > Stream
- Text + Text > LLM
- Text _ Image > LLM > Te/xt
- Image > LLM > Text
System promps
- extra information
- goal
- like an additional message
Integrations/tool calling
- ex: a clock to see the time
- Flow: send message and list of tools the LLM can use. LLM takes result from tool and has enough to formulate response
- Most models support tool calling; older ones didn’t always.
- Can set default tools or tools by call
- http://www.javadocs.dev/mcp for MCP and https://www.javadocs.dev for UI. Can get latest version, Javadoc
- Challenge: by default on load gets all tool metadata from all MCP servers which wastes tokens. Also hard to figure out which tool use if overlapping descriptions.
- Can deal with using semantic search across tooos, tool groups (idea from Embabel) so agent sees subset of tools that it can use, Embabel Unfolding tools
Skills
- Markdown file with sections for pieces of data
- Can selectively load as needed
Reusable skills
- https://skills.sh
- https://www.skillsjars.com
Memory
- LLM is stateless
- Memory provides means to keep track of messages such as database or hosted services like Amazon Bedrock Agent Core
- Short term – ex: message windowing. Keeping track of all messages doesn’t capture all nuggets like name in long run
- Long term – ex: LLM Compaction/Extraction
My take
This session was right after lunch and in a dark room. Super glad that James presents with a lot of energy making it easy to pay attention. Code was lcear and made it easy to understand the APIs. It was sufficiently different than the Hack Haus session which I appreciate