[2026 kcdc] how my team got worse with ai: The hidden tax of generated code

Posted on by
Reply

Speaker: Fernando Cargnelutti

See live blog table of contents for more posts

“Faster“

  • “AI will make us 4x faster” – claim comes from people selling the tools. Independent measurement shows gap between that and reality
  • Moved QA engineers to separate team and less of them
  • Also removed Solution Architects, they now do development
  • No more ScrumMasters. Do Kanban but without work in progress
  • Fired 1 developer per team

Quotes between teammates

  • “I wouldn’t create a PR for such a change”
  • “That PR is 3 weeks old; bad for metrics”
  • “We don’t merge MRs just because they are interesting”
  • “I merge changes when results in better quality not because of metrics”
  • “I don’t think this needs more discussion; it’s already been merge”

Laws of system

  • A building architect gets gravity for free.
  • In software, we write the gravity.
  • We craft the laws of the system

Legacy

  • Program is the theory held by the team
  • The code is a byproduct
  • Legacy = code whose theory died even if no line of code has changed
  • Process of writing code develops knowledge of what doing
  • If tests inherit the wrong theory, you have local coherence but global failure

Bugs

  • Most expensive bug is the one in the requirements
  • Now we write the requirements

Vs Compilers

  • 99% of our code is written by AI but 100% of their code is written by compilers
  • A compiler is physics; A LLM is a guess
  • Compilers automate below the theory line; AI writes where the theory lives
  • Source code is a design document
  • Compiled code is disposable. Can generate it again.
  • Prompt isn’t always persisted. But incomplete even if it is saved because non-deterministic translator
  • Commit still attributed to you.

Code gen

  • In past, we used UML to generate the plumbing
  • Had to fill in middle
  • Didn’t work that well; abandoned

Writing GitHub automation script

  • Created code, documentation, repos
  • To do by hand, read doc, understand API/auth, consider a lot of tradeoffs
  • If AI writes code, you don’t feel the tradeoffs
  • Struggle manufacturers understanding
  • Seniors doubt AI output because they walked the path. They know what costs. Juniors accept fast; they never traveled it.

Mitre 2025 study

  • Did study with using chat (and pasting in code)
  • Predicted 20% faster than non-AI group
  • Said 20% faster
  • When measured were 19% slower

Repeated study in 2026

  • Now speedup of 18% with same developers
  • New developers had speedup of 4%
  • Realized can’t trust numbers because nobody wanted to write code without AI.
  • All agree write code faster with AI. 18% is not 4x
  • Now depending on a tool that takes away judgment

Skill formation

  • On quiz of material on tasks just finished. 67% with handcoding, 50% with AI
  • Predict bigger difference when measure in agentic environment.

Experiment

  • Randomly took 3 developers and an old PR with almost new AI and a new one with more than 60% AI
  • Predict won’t be able to explain new PR as well as old one
  • Found remember the 8 month old PR, can explain it better, remember file structure, trust it more.
  • Walked through code at a glance for handwritten code. Minutes of reading to explain 40% of the new AI PR
  • Trust 6/10 on PR that couldn’t explain half of.
  • One developer remembered the 3 week PR way better. The structure was biult by AI. She dealt with a bug and spent time on it. She did not remember the code structure as AI did that part.

Other stats

  • 81% more duplication
  • Went from 22% to 3.8% on refactoring
  • 441% incrase in PR review time
  • 243% incrase incidents per PR
  • 31% PRs merged with zero review
  • Time saved in creation was re-allocated to auditing.
  • “We’re accumulating code faster than we are accumulating trust”

Urgency and AI

  • 4x belief
  • roles/buffers removed
  • urgency lands on devs
  • AI becomes only way to keep face
  • struggle removed
  • understanding evaporates
  • defects and report create more urgency

Future analysis

  • Look at feasibility of a feature
  • Engineer asked AI vs understanding
  • Does it affect design? What could go wrong? How long will it take?

Distance

  • False belief of knowing scales with delegation.
  • Dev who worked on
  • Dev who merged it
  • Dev who asked to analyze
  • Management

What it costs

  • AI answering human conversations
  • Production incident
  • Can’t explain code form 3 weeks ago
  • Having AI analyze instead of humans

What can do

  • Choose friction where appropriate
  • One minute rule – if can’t explain in under a minute and defend why approach is acceptable, don’t merge
  • Architecture audits – review the theory not the syntax
  • Manual design first – write the conceptual how by hand
  • Real WIP limits
  • Harness is team infrastructure. version/review/maintain like code
  • “Frictionless teams produced understanding free teams”

My take

This session started 8 minutes late due to issues with the projector. Nobody said anything though. Would have been nice to either give an intro or at least say waiting. I did appreciate a few references to confirming we would be on time to lunch. The content itself was good. I like the perspective and it had good depth. I really liked the difference based on experiences in understanding. And the stats.

[2026 kcdc] the undersea infrastructure

Posted on by
Reply

Speaker: Richard Campbell

See live blog table of contents for more posts

History

  • 1800s – submarine cable to transfer electricity between France and England
  • Then telegraph which is electricity pulses.
  • 1866 – reliable cable
  • :From two months to two minutes” increase in data transfer
  • British empire spread cables around the world to connect to their empire
  • World war 2 – transport fuel under English Channel (operation PLUTO). Reclaimed a lot at end since steal is expensive
  • Oil exploration in water and pipe oil/natural gas to land. Stable so rare to need to build more. Piples 3-5 feet across; covered in concrete
  • 1950s – TAT – transatlantic telephone line – handle 35 phone calls in parallel. Brought back up because wire expensive.
  • 2020 – most recent copy wire laid.

Repeaters

  • Need repeaters to boost signal for coper wires
  • Repeater 9 feet long
  • Need every 60-70 km.

Fiber optic

  • 90% of data transfers under the sea
  • Starlink is about 1% of internet traffic. Equivalent to 2 undersa cables. Last mile technology.
  • Fiber is useful/complex glass
  • Layers of an indoor cable – Fiber core, cladding (rptection), coasting, tight buffere, stright memebers, cable jacket. Less than one milimeter across
  • Underseas cable – package denser. More cables inside buffer. Water protected. Also need to use repeaters
  • Fibers don’t vary that much. Way they are shelled does
  • Need armor in some places like near short and volcano
  • Weight – 500 pounds per km without armor.
  • Multimode fiber – uses lasers
  • Single mode fiber – usually used in a building (inside/short range)
  • Phone company used to own. Now big tech. Cables are mostly pooled. Allow using each others. Meta has own vs using pool

Wind turbines

  • For power
  • These are rare.
  • Usually ancored to shallow floor
  • Floating turbines experimental

Example: Google Durant cable from Reston VA to France

  • 1 cable form google can carry terrabytes of data
  • Have to get permission at both ends
  • Estimated to last 20 years
  • Map ocean floor to ensure stable
  • Need to lay cable precisely (with in meeter of mapped location)
  • If hurricaine, cut cable, put buoy on it and continue. Similarly if longer than the Atlantic. 10K is limit without connecting
  • Now bury 4 meters deep at ends
  • Can take 3 months to run the cable. Slow and precise
  • Transmission station at end to connect to internet
  • Challenge to try to make sure cables don’t overlap

Restrictions/Repairs

  • Cables on sea maps
  • Not allowed to go near them; dangerous
  • Can cut with anchor of ship
  • Specifically designed ships to cut out/replace part of cable
  • Insurance pays for timely repair
  • 100-200 calble fixes a year
  • Small number of repair ships in the world. Only work i safe. Can’t risk ship

Baltic Sea

  • Shallow. Can’t mkae deep enough to make safe
  • Gas explosion in sea – from keeping pressurized
  • Continuous monotring so can detect attempts real time

Hunga Tonga erruption

  • Tonga went offline at same time
  • Large steam exposion affecting most of country
  • 100km of cabe gone
  • Cell phones stopped working because have to authenticate to tower which was in Japan
  • Did banking by USB key. Did reconciliation a year later. Long time to identify problem and orger new cable.
  • Borrowed some StarLink

Drones

  • Easier to monitor
  • Autonomous ships

Technology advances

  • Increased density
  • Smaller repeaters
  • New routes ex: Tokyo to London

My take

I like the otter and sea lion video in the pre-show. While I don’t expect to ever use this, it was really interesting and I’m glad I came. It was really interesting seeing what’s in a fiber optic cable. Also great pictures on the topic of cables. Interesting stories of issues in wartime.

[kcdc 2026] would you trust an intern with production

Posted on by
Reply

Speaker: Bob Walker (_BobJWalker_)

See live blog table of contents for more posts

Relearning the same lesson

  • AI horror stores – clickbait/sensationalism. But outages real
  • AWS – agent decided to rebuild prod
  • AI ant deletes company’s database
  • Not same exact mistake – rhyme with each other
  • Sharing Cook county voter database
  • Problem: People making direct changes to production

Non deterministic users

  • AI and People are both non deterministic users.
  • Both do things that are so baffling that you can’t explain it..
  • Why did you do that?
  • I decided it was the best approach I could take.
  • Both can make same mistakes and burn down production

Why AI is like an intern

  • Book smart
  • Doesn’t understand consequences of production
  • Like tellling an intern not to speed

AI philosophy

  • Terminator: when told not to kill someone, shot in knee instead
  • AI is an amplifier. If you don’t care about something, it won’t and vice versa.

AI Usage types

  • Cyrborgs – AI is pairing partner
  • Centaurs – AI is research assistant
  • Self automators – AI is an Oracle

Causing Prod Problems

  • Did show of hands for who caused a prod incident. Almost everyone. Commented if didn’t raise hand, lying or hasn’t happened yet.
  • “Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?” – IBM

Humans

  • “Humans will be involved every step of the way”. But humans caused problems too
  • AI PRs can be too big for a human to review.
  • Spent time trying to automate to get people out of the deploy to prod workflow. Solution isn’t to go back.
  • OSS – more/bigger PRs

AI is getting better

  • It is, but not at same rate as it did at beginning
  • Models now tuned to benchmarks. AI companies helped right benchmarks
  • When same problems happen over and over again, does good job. When something novel, tends to fall apart

Where AI fits in software delivery (LLMs and AI agents)

  • Want deterministic process (ex: build) between non deterministic users and production
  • Includes CI/CD pipelines, automated runbooks for common production incidents, creating/deploying infrastructure, logging/alerting
  • AI and people contribute to the pipeline – ex: writing, approving part

Permissions

  • AI should have different permissions than people
  • ex: branch protection

Process Changes

  • The process doesn’t change every day.
  • Monthly at most

AI capabiilities

  • AI can invoke deterministic processes via MCP/CLI/API via token
  • Automate common decisions so non deterministic users aren’t making them.

Sliding scales of trust

  • “I don’t trust my developers” – shouldn’t trust AI. ex: spotify. Inconvenience if don’t but not a material impact on lives
  • “I fully trust my developers” – should trust AI. Have enough automation and guardrails in place to prevent making the wrong choice. Often not working for bank/insurance companies.
  • Every company is different and changes over time. Also depends on company politics and policies

Takeways

  • Non deterministic users shouldn’t make production changes
  • More human involvement isn’t a panacea
  • Automate deterministic decision making for non-deterministic users
  • Non deterministic users use and contirbute to deterministic processes

Action Items

  • Create AI skills or context boundaries as first line of defense. This is like a chain link fence to protect Fort Knox
  • Table top work case scenarios. – ex: maliciuos insider
  • Establish clear boundaries, create deterministic processes/runbooks, create deterministic approvals for common decisions, build contributions guidelines for AI or people
  • AI on behalf of a person uses different API keys/tokens (and different permissions – ex: can deploy but not make changes)

My take

Bob said a few times that purposely vague/vendor agnostic. This session was a great start to the morning. AI course but focused on a perspective. I really like the idea of humans and AI being non-deterministic users as a commonality. I also like trust being situational. Also excellent summary!