the bad things happen when you’re not looking – ryan huber – qcon

See the live blog table of contents. Gist is posted at (github login required)

Ryan was the first security employee at Slack. He is doing an experiment where red slides means don’t take pictures or tweet about the slide. I really like that idea. It makes speaker intent clear.

How find out about a problem

  • Don’t want to find out from Brian Krebs that you’ve been breached
  • Don’t want hackers to tell you something strange is going on. They are done at that point and are showing off
  • Even worse – don’t notice

General Notes

  • Time to detect is important metric
  • Credential theft is biggest/one of the biggest
  • Goal – watch as many things as possible, but don’t be a dashboard. Want as little as possible on the dashboard. If it is mostly empty, things will get noticed when they are there.
  • Bad model – NetCool – train people to acknowledge all alerts and they miss things because bad habit
  • The defender’s advantage – if the attackers don’t know what you are looking for/trip wire, they dont know what to avoid
  • “Zero days are not invisibility cloaks” – other boxes can pick up on it
  • The hypothetcial malicious insider – a former security team member has a lot of knowledge. And an insider with credentials has access
  • Don’t overwhelm users. Confirm bulk actions in bulk not one at a time.
  • Canaries – need to validate monitoring, recording, etc.
  • Do table top red team exercises if not doing real ones.

Slack Security

  • Setup reliable logging platform
    • RELP (reliable event logging protocol)
    • steamstash/logstash -> Elastic search (Splunk is superior but costs more)
    • Two weeks of data is about 2 terrabytes of logged data. Almost never sits on disk
  • auditd – part of Linux. Run auditctl commands and kernel looks for matching events.
  • audisp – works with auditd to transform data
  • osquery – Facebook project for system monitoring using SQL
  • ElastAlert – yelp project to pick up on ElasticSearch events. Does queries on a timer against Elastic Search.
  • AlertCenter – have SecurityBot looking at alerts. Security bot posts to Slack asking user to type “acknowledge” on phone to confirm action. That way, know have phone and not just Slack account. If no reply in X hours, goes to Pagerduty. Automated triage to avoid flood of data. Instead of security team looking at all alerts, whole company is helping. This means the security team responds to less than 5 alerts a day.


  • Listeners – specific events
  • Time awake – nobody is awake for 24 hours. Trigger an alert when this happens
  • GeoIP – Doesn’t work perfectly. T-Mobile has feature that can travel abroad without paying roaming. This works by routing some traffic through Texas so your location keeps jumping between Texas and aboard
  • IPs – less unique IPs than you’d think. Worth looking at when user comes from new IP.

Leave a Reply

Your email address will not be published. Required fields are marked *