Speaker: Kelly Morrison
For more, see the 2024 DevNexus Blog Table of Contents
HIstory
- Fairly recent. GPT created in 2018. Number parameters increasing exponentially
- Microsoft CoPilot released in 2021. Uses Codex; a specialized model off GPT3 for creating code. Trained on billions of lines of GitHub code and can learn from a local code base
- Amazon released CodeWhisperer in 2022. Can generate code for 15 languages. Specialized for AWS Code Deployment
Basic Example
- Asked ChatGPT to write a Java 17 Spring boot rest API for stats in a MongoDB with JUnit 5 tests cases for the most common cases
- Looks impressive on first pass, but then find problems
- Hard coded info
- Used Lombok instead of Java 17 records
- Code doesn’t compile
Complicated Example
- Asked ChatGPT to write an entire enterprise app for selling over 10K crafts with a whole bunch of requirements like OpenID, Sarbanes Oxley, etc
- Didn’t try. Instead came back with a list of things to consider in terms of requirements
What AI can/can’t do
- Can do “Ground level” work.
- Still need humans for large orchestratoin – ex: architects
- Can do more self without junior devs
- Garbage in, garbage out. Trained on public code in GitHub. Not all good/correct. Some obsolete.
- Humans better at changing frameworks, working with CSS (does it look nice), major architectural changes, understanding impact of code when requirements change
Hallucinations
- Doesn’t understand. Asks as mime/mimic/parrot
- If can’t find answer, will give answer that looks like what you want even if made up. Example where made up up a kubectl option
- Not enough training data on new languages/technologies. More hallucinations when less training data
- Mojo created May 2023. Likely to get Python examples if ask for Mojo. However, it is a subset of Python with some extra things
Security Concerns
- Learns from what you enter so can leak data
- Almost impossible to remove something in a LLM. ex: passwords, intellectual propery, trade secrets
- Some companies forbid using these models or require anonymous air gapped use. Translate something innocuous into what actually want
Debugging
- Can human understand AI generated code well enough to debug
- GPT and Copilot can sometimes debug code, but have to worry about security
Pushback
- Law – ChatGPT made up cases
- Hollywood strike – copying old plots/scripts/characters
- Unclear if generated output can be copyrighted. For now, not copyrightable but could change.
- Some software is too important to risk hallucinations 0 ex: plane, car (although Telsa getting there), pacemakers, spacecraft, satellites
- Lack of context – other software at compnay, standards, reuse, why use certain technologies, securities
- Lack of creativity – need to determine problem to solve or new approaches
What AI does well
- Low level code gen (REST APIs, config, database access)
- Code optimization
- Greenfield development
- Generateing docs or tests
- Basically the kin of tasks you hand off to a junior developer [I disagree that some of these are things you hand off]
Career Advice
- Focus on architecture, not code
- Don’t just learn a langauge or framework.
- Learn which langauges are best in different situations
- Learn common idioms
- Look at pricing, availability of libraries and programmers
- Learn which architectures should be implemented in different languages
- Learn how to create great prompts for code generation
- Learn how to understand, follow, test, and debug AI generated code
Book recommendations
- Building Evolutionary Architectures
- Domain Driven Design
- Fundamentals of Software Archicture
- Head First Software Architecture
More skills
- Types or architecutures – Layered, event driven, microkernel, microservices, space based, client/server, broker, peer to peer, etc
- Determine requirements – domain experts don’t know enough about software to specify. Can be bridge between AI and domain experts
Mentoring junior developers
- Teach how write high quality prompts.
- Remind to ask for security, test cases, docs, design patterns, OWASP checks
- Show to spot and deal with hallucinations
- Help to understand and debut AI written code
- Help learn architecture by explaining why choices made
- Ensure code reviews are held
- Precommit git hooks to test code
- Use AI to help generate unit tests
ArchUnit
- archunit.org tests architecuture.
- Can add own architecture rules.
- ex: never use Java Util Logging or Joda Time
- ex: fields should be private/static/final
- ex: no field injection
- ex: what layers are allowed to call
- Can include “Because” reason for each rule
- Ensures AI doesn’t sneak in something that goes against conventions
My take
Good examples. I was worried about the omission of “where to senior devs” come from but there were examples like changing frameworks so not entirely ignored. Good examples from the ecosystem as well. Good list of skills to focus on.