jeff williams on dev ops and portfolios at app sec usa

Imagine application security is health care
We have a lot of doctors and patients. Think of app security as a public health issue. How do you stamp out SQL Injection as a disease. Can’t scale up patient level tehniques (penetration tests) to wipe out completely. New tools to instrument body to provide continuous real time monitoring. Annual checkups are reactive. That’s discovering a problem way too late and many people don’t even go that often. One day your phone will know you are sick before you do.

Tools are failing us

  • Static analysis tools – request.getParameter() and conn.executeStatement() calls aren’t explicit anymore. They are within your framework. Unless the tool knows your framework, it wont fin issues.
  • Penetration testing – spending time figuring out how to get tools to intercept traffic, doesn’t go fast enough to work with DevOps and Agile.

Showed the diagram that we climbing the wrong hill. We need “Continuous AppSec”


  • Portfolio Scale: The right defenses for every application are present, correct and used properly. (you can’t look at thousands of apps manually
  • DevOps speed – applicaton security happens continuously and in real time

An example – clickjacking

  • Want to design a sensor so that if a vulnerability gets injected, we get notified. Could look at headers.
  • Datasource: HTTP, analysis technique: passive, expriment style: positive,environment: test
  • Choose based on speed, accuracy, feedback , scalability, ease of use and cost
  • Rather than an annual PDF per app, you can get data for the entire porfolio. Store in data warehouse so can get dashboard

Jeff wrote a tool: check your headers. Can check headers for common issues against a public website.

Other uses

  • Access control – could have tool generate matrix from code rather than rely on old doc. Help identify missing ones. Ask developers to produce sensors. Can feed into Sonar dashboard. [need standards or per app]
  • Known vulnerable libraries – Run DependencyCheck during every build (OWASP tool)
  • CSRF – run tests trhough ZAP, use ZEST to check CSRF token, get results via ZAP REST API
  • Check correctness of controls – tool don’t even try to validate correct. Write code test first so your tests cover this. Jeff used JUnit.
  • Injection – this is a data flow problem – for example, build an IAST sensor
  • Security intelligence – can gather any information such as outbound connections

What sensors you need

  • most likely to have at least one vulnerability of: identifiaion and authentication, session management. Static analysis companies find XSS because they check for that well
  • Figure out what most important to you


  • Can’t do security testing without a model
  • Testers don’t know model so waste time testing SQL in an app without SQL
  • Risk of not testing things that are needed

Nice Venn diagram to show not testing what we care about. Plus spotty even within the areas that do overlap.

Continuous Application Security
Map business concerns, defense strategies, actual defenses and sensors. Should start with business concerns not sensors. That say a new threat/business priority becomes a sensor and you get data back on it in the dashboard. Real time data, iteration.

Get started by building a sensor and creating a dashboard with Excel. Then can switch to point where can show covers business concerns and can add more based on business.

Business case to do: build sensor ones,get benefits forever. Now, a pen tester IS the sensor and is a very expensive one.

Detecting attack is not the same as detecting vulnerabilities.

Most of the industry is looking at compliance. Want to get to monitoring. Thn can have a strategy and optimization. [Like the levels used by CMM and many other processes.]

We can never improve if our only metric i whether we are doing what everyone else is doing. This wouldn’t work to stamp out a disease and shouldn’t b our approach.

My take
I always like seeing Jeff Williams speak. He is a great speaker and usually presents a unique spin on an idea orr a new idea. I liked the heallth care hook here.

Leave a Reply

Your email address will not be published. Required fields are marked *