[devnexus 2024] breaking ai: live coding and hacking apps wih generative ai

Speaker: Micah Silverman

For more, see the 2024 DevNexus Blog Table of Contents


  • AI is not a silver bullet
  • Treat AI like a junior dev and verify everything produced
  • New iteration of copy/paste. More code that got from Stack Overflow
  • We don’t do thorough code reviews of libary code. Loo at security, number commiers, when last release
  • Different because our name now on commit
  • More challenging to maintain visiblity
  • Backlog ever growing

Common use in Dev

  • Addig coents
  • Summarizing code
  • Writing Readme
  • Refactorin code
  • Providing templates
  • Pair programming
  • Generating code (the new stack overflow)


  • 92% software dev using AI in some form
  • Those who use AI are 57% faster
  • Those who use AI are 27% more likely to complete task
  • 40% of co pilot generated code conains vulnrabiliies. Same stat without AI s o this is what we trained it on
  • Those using AI wrote less secure code but believed ore secure. We trust AI too much. Junior devs get more scrutinity than senior devs
  • [Some of these stats aren’t austation. Ex: early adopters]

Using AI well

  • Good starting point
  • Don’t use without review


  • Hallucinations – including supporting “evidence” even where wrong. Gave addition example
  • ChatGPT got worse in math over a few months. 98% to 2%. Now some math specific AIs
  • ”ChatGPT is confidentally wrong” – Eelko de Vos
  • First defamation lawsuit – ChatGPT made up case law
  • AI doesn’t know when wrong

AI and Code

  • Asked for Express app taking name as a request parameter. Tried a bunch of times and name parameter never sanizitzed so cross site scripting vulnerabilities. Ideally wouldn’t auto generate vulnerabilities
  • Can give bad advice – asked if code was safe from NoSQL inection. GPT and Bard said safe. Was not safe.
  • Samsung put all code in ChatGPT and leaked code, keys, trade secrets. Became part of training data. ChatGPT says got better about dealing with secrets.
  • Terms of services of ChatGPT say can use anything as training data

Sample Conference App

  • Using CoPilot
  • Spring Boot app in IntellIJ

Basic example

  • Gave co-pilot comments/prompts to create code
  • Showed generating code to read a file that doesn’t exist due to a typo

JPA SQL injection example

  • Showed not the most common way to write JPA here in 2024; usually don’t need direct query
  • Showed copilot offers prompt to get result
  • Tried adding to prompt to protect against SQL injection. Got a naive regex sanitizer
  • Then tried requesting named parameters in the query. After that did promps for setting parameter and getting result. All was well.


  • Tried getting file then wih a file separator
  • Successfully got constant defined in file so some context sensitivity. But ignored file separator reques fro propt
  • Then requesting saving the file and successfully geerated code to write it
  • Requested getting a person and successfully called already written getPerson() method
  • Then set image name
  • Co pilot offered to write prompt to save person.
  • Then requested adding the message and added attribute to model
  • However, has path traversal issue
  • Showed BurpSuite monitoring as run example. “Send to repeater” keeps cookies and such while letting alter request.
  • Changed file name to ../image/snyklogo.png”. If works will replace logo with uploaded pic
  • Showed Synk IDE extension which also note path traversal issue
  • Tried asking to sanitize input against path traversal an got check for two dots. Not good enough but a first pass
  • Tried asking for whitelist to protect against directory traeversal. Checked directory name prefix which is better but also not a whitelist
  • Then tried requesting to validae that there is not a path traversal using the normalize method. Did what requested including the prefix check from previous prompt but Micah noted that’s what did in the past

What can do

  • Use tool to scan code and tell when makes mistake
  • Learn. Ex Snyk has a lesson on prompt injection. – Getting AI to tell info it shouldn’t


  • Currently have Math aware AIs. Maybe will have security aware AI in future

My take

Good mix of background and live code. I like that Micah didn’t assume security knowlege while keeping it engaging for people who are familiar. I had never seen BurpSuite so that was a happy bonus in the demo.

[devnexus 2024] More tales from the Dark Side: How AI is the bad guys new friend[devnexus 2024] dark tales ai

Speaker: Stevel Poole


For more, see the 2024 DevNexus Blog Table of Contents


  • Supply chain
  • Now we are all attack vectores


  • We also use wifi
  • How many use VPN?
  • Easy to spoof wifi
  • Only need battery, raspberry pi and a few more things
  • Would you notice a box on the wall?


  • Plug in Mac laptop charger at conference
  • If leave unattended, someone could add hardware
  • Any USB has problem
  • USB data cable and power cable look same

Hotel rooms

  • Hidden camera
  • In some countries during cold war, used human cherography to influence where sit
  • Becoming more common
  • More people are pass thru to company now


  • Getting better
  • More targetting. Can know how company does things. Or knowing boss;’ namePhishing -> Spear Phishing -> Personalized Attacks
  • Moving towards more organized and long term attacks

Adding AI

Bad things can do

  • Deepfake nude generator
  • Deepfake phishing grew by three thousand percent in 2023

Why now

  • Not hard to do a reasonable fake. USB acceleration is sixty bucks
  • Huggingface.co has lots of models
  • Models and data avaialble to you and bad guys

Other problems

How Protect

  • Paper on identifying mouth inconsistencies for lip synching
  • Text/numbers wrong
  • Find anomalies from lack of training data – this is going to be an arms race. Once AI knows wrong, can do better next time.
  • Be more suspicious
  • Secure supply chain – all the pieces involved in creating and delivering software
  • Control AI tools in process
  • Look at where models came from and decide if safe. Will have to prove where got it from
  • Consider how train AI and when retrain it
  • Government wants a SBOM, automated supply chain, evidence of software integrity and regular aduit
  • SBOM (software bill of materials) don’t find malicious code but ensure you know what have

My take

Demos were great. Security has changed a lot. Good emphasis on depending on how much money you spend at it. It’s scary, but supposed to be. Need to think about what else I can do in my own life.

Someone challenged saying the grandparent scam sounds fake and nothing like the person. Steve didn’t get to reply, but it’s not a fare analogy. The grandparent same isn’t targeting (at least not much). Some targeting you specifically will have audio/bideo of you to base it off of. And then we are back to the 7 seconds is enough.

[2023 kcdc] cve 101: the unfolding of a zero day attack

Speaker: Theresa Mammarella

Twitter: @t_mammarella

For more, see the table of contents.


  • Annual cost of cyber crime predicting to top 8 trillion. Only US and China have more than that as GDP


  • Vulnerability – weakness/flaw in system
  • Threat – attack vector, potential action
  • Risk – probably frequency of that loss.
  • Goal of cybersecurity is to minimize risk. Can’t control intent to do harm so focus on vunlerability


  • CVE – Common Vulnerabilities and Exposures
  • Format CVE-xxxx-yyyyy. xxxx = year came out. yyyy = identifier
  • CVSS scoring – how bad is it on a scale of 0-10. Ten is worst
  • CVSS score has three parts – basic (exploitability, impact), temporal, environmental. Good description here
  • Basic is the one we see on the CVE
  • CVE can be rejected. The number is used and cannot be reused. Example. Something thought found a vulnerability. Investigation was flawed and not an actual issue. Story about it here.

How to talk about

  • Private disclosure – organization can choose when/whether to fix/share
  • Coordinated/responsible disclosure – best practice – agreed upon time frame
  • Full/public disclosure – share everything
  • Best to report via company website, security.md file, security files on server, github private vulnerability reporting

Zero day vulnerability


  • log4jshell – remote code loading. Was reported responsibility but incomplete fix so zero days on those CVEs
  • Could be as simple as a bounds check. For OpenSSL. Announced something big coming and get ready. When announced learned it only affected OpenSSL 3 (not 2) and high, not critical so boy who cried wolf situation.

Security Practices for Developers

  • Insider threat includes poor training
  • A lot more developers than info security. Increasingly harder for security teams to keep up.
  • Cost of finding and fixing bugs increases over time
  • Does this touch the internet? take untrusted input/ handle sensitive data?
  • OWASP Top 10. Updated in 2021 to add insecure design, software/data integrity failures and server side request forgery (SSRF). Some merged such as injection.
  • Starting OWASP Top 10 for Large Language Model Applications. A draft version is available
  • mitre/hipcheck – scorecard for supply chain risk. Similarly, Sonatype security rating and OpenSSF Scorecard
  • Open source dependency management. Embedded in many projects. 90% of app is open source on average. North Korea attacked many apps including Putty

Attack types

  • Typosquatting – look alike domain with one or two wrong characters
  • Open source repo attackes – attempt to get maleware/weakness added into depednecy source
  • Build tool attacks
  • Dependency confusion – different version that shows up as latest


  • Sometimes third party projects. ex: OpenSSF Scorecard
  • NPM and PyPI often have supply chain attacks. Maven Central more so
  • Scanning tools to find issues can be helpful
  • You are responsible when things go wrong

My take

Good talk. Covered concepts and good real life examples. I learned a few things like the OWASP Top 10 for LLMs. Appreciated the shout out to “the Java people in the front row” when talking about log4j. I added a few links in my blog that weren’t in the original presentation for things I wanted to learn more about.