how containers have panned out – adrian trenaman – qcon

Posted on by
Reply

For more QCon posts, see my live blog table of contents. Adrian is from Gilt.

History

  • No off the shelf software to run a flash sale business. Therefore Gilt has to do something custom.
  • Started with Ruby on Rails in 2007. Didn’t scale well enough
  • Moved to Java in 2011
  • Moved to microservices in 2015
  • In a 30 day period, moved bulk of Gilt to Amazon

Problems

  • Isolation problem – nobody should be able to take down someone else’s work
  • A noon outage in 2013 – what happened
  • Impedance mismatch problems. “Developers often think of machines as something that’s all theirs, magically provided by the hardware fairy.”

Machines for Gilt Japan

  • Run 20-40 containers per machine.
  • Load balancer between two racks of three boxes each.
  • Separate machines for the database and email.
  • From developer’s point of view, a machine is a machine.

What did Gilt Japan learn

  • Scalable by time of day
  • Solves impedance mismatch – developers see “a machine”
  • Limits damage one person can do
  • Infra/Devops engineer embedded into engineering team
  • Outstanding potential problems
    • Static infrastructure
    • Resource hogging

Docker topology

  • Dark canary – only for internal use
  • Canary – First prod install. Let it run for a while (ex through a noon cycle for Gilt)
  • Release – Once happy with canary, roll it out to other nodes
  • Gilt has a lot of read only traffic which limits damage you can do and reduces need for staging environment.
  • Gilt has one container per host/EC2 instance
  • Want to have as few moving parts/risk points in deployment process
  • “We could solve this now, or just wait six months and Amazon wil provide a solution”

Projects

  • ION Roller
    • Immutable deployment – Destroy original cluster when done with this process for Docker upgrades.
    • Slow to setup/tear down environments.
    • Can be expensive for continuous deployment
    • Open source, but in house.
  • Nova
    • Uses yaml to deploy
    • No Docker registry. Base images are on Docker. Releases aren’t needed on there so go straight to Amazon
    • Less boilerplate
    • Immutable deployment on mutable infrastructure. Docker container is immutable.
  • Fighting bit rot, chaos-monkey style
    • Don’t want things to run forever in Prod.
    • What if there is a security vulnerability
    • Every day, kill oldest AMI randomly. This forces latest AMI with fixes and fail early.
    • Doesn’t solve vulnerability in Docker container. Would need new release with new base image for that. Hasn’t happened to Gilt yet.
  • Sundial
    • For running batch jobs
    • Automatically reschedules if fail
    • Define a process – group of tasks with dependencies between them

EC2

  • Less configuration
  • Automatic rollout
  • Integrations
  • IAM roles are at instance level, not container level

Using Docker as a local build platform

  • Different projects use different versions of build tools
  • Docker can be used as a versioned build container.
  • A year from now, will still have everything need to run code

Lessons

  • Containers let separate what deploy from how.where deploy it
  • Still the wild west on how containers are deployed
  • Seek immutability in the container, not in the stack
  • The competitive advantage for Gilt is to be able to deploy quickly/frequently/safely to production and therefore can innovate faster. Gilt lets engineers deploy whenever they want without asking permission.

reactive programming for java developers – rossen stoyanchev – qcon

Posted on by
Reply

For more QCon posts, see my live blog table of contents.

General notes

  • Spring 5 has reactive features.
  • Maybe a third of the audience is doing something with reactive, a little more have read about it (that’s me) and a lot haven’t heard of it. Which is good given the title of this talk!
  • Moore’s law slowing down. No more free lunch from the hardware
  • Now apps more distributed/independent/cloudbased compared to 10 years ago
  • Now we expect latency
  • Thread now has to do more async things. Hit limits of scale. imperative logic more complicated.

Other approach

  • Async/non-blocking
  • Use very few threads. node.js has a single thread. Must be careful because blocking those few threads blocks the whole app
  • Reactive programming – think async. Not everything is a callback

Comparing styles

  • Imperative
    • call a method, get a result, do something with that result
  • Async
    • return a Future instead of a value. Don’t throw a checked exception because my happen in a different thread. They are asynchronous in that you can get a result later. But when you eventually call feature.get(), it throws different checked exceptions.
    • In Java 8, can return a CompletableFuture and call future.whenComplete((user, throwable) -> {})
    • If doing a findAll(), the future/completable future approach doesn’t give you a callback/return anything until all of them are ready. You can’t stream or could run out of memory.
    • Async results as stream – get one notification per data item and another for completion or error
  • Declarative
    • Focus on what, not how
    • Declare what should happen rather than focusing on callbacks
    • Java 8 stream API uses this style. It is meant for collections and pull based/usable once
    • Hot streams – latency sensitive streams, data need to be pushed for you
    • Cold streams – pull based

Reactive libraries

  • Stream like API
  • Can be used for hot/cold streams
  • Project Reactor – Similar to ReactiveX, easy to bridge to Java 8 streams. (ReactiveX is like XUnit – commonality for different languages)

Abstractions

  • Flux – sequence of 0 to n – equivalent of Java 8 stream – can convert to/from Java 8
  • Mono – sequence of 0 or 1 – can convert to/from CompletableFuture

Reactive streams spec

  • Back pressure – producers must not overwhelm consumers. Communicates if downstream isn’t ready for more items. Ex: a slow client like a mobile device isnt able to handle more data and server isn’t blocking
  • Small API – only 4 interfaces
  • Rules
  • Interoperability across reactive components so can compose chain

Java 9

  • Reactive streams included (the four interfaces)
    • Publisher
    • Subscriber
    • Subscription
    • Processor
  • Use a reactive library. Don’t implement the four interfaces yourself
  • subscribe() triggers the flow of data
  • “push” was missing until now. Want in JDK to have foundation
  • Classes
    • java.util.concurrent.Flow – the four interfaces
    • SubmissionPublisher – bridge to reactie streams
    • Tie-ins to CompletableFuture and Stream

Reactor

  • GA release scheduled for July
  • Currently called 2.5. Might changed to 3.0

Reactive Spring MVC

  • Apps annotate controllers even now.
  • Return a Flux type.
  • Spring MVC itself needs to change a lot – called Spring Web Reactive
  • The servlet API assumes blocking. There are async workarounds. Servlet 4.0 might support reactive spring integration, but probably just the basics. Using own bridge to Servlet 3.1 in meantime.
  • Can still use Jetty/Tomcat. They are non-blocking behind the scenes.
  • Don’t need servlet container. Can use Netty.
  • HandlerMapping – change to return a Mono so can be non-blocking
  • Focusing on REST/microservices scenarios

Other reactive efforts

  • MongoDB – reactive driver
  • Couchbase – reactive driver
  • Thymeleaf – split templates into chunks and throttle to respect backpressure

unprivileged containers – jessie frazelle – qcon

Posted on by
Reply

For more QCon posts, see my live blog table of contents.

Today

  • Docker typically runs as a privileged user.
  • Containers are meant to limit the damage from a compromise. The world an attacker can see inside the container is a limited one)
  • Want unprivileged containers so don’t need sudo/privileged access to launch container in the first place.

Chrome sandbox on Linux

  • uses Seccomp, Namspaces, Apparmor.
  • doesn’t need to be run as root.
  • each tab is in its own namespace – process only knows about itself
  • if Chrome can do this, why not Docker

General notes

  • cgroups (Controlgroups) limit what resources a process can use and how much.
  • Each time you docker build something it spawns a new container. Just blocking things wholesale would cause issues here.
  • I had trouble following what was current/future in the examples.

Future

  • Won’t need to run as root
  • Can customize sandboxes from defaults, better UX for dealing with security policies.
  • “postgres should maintain a postgres profile”

Impression: A lot of this was recorded demos (show typing commands as graphic/video that plays out.) For the namespaces, it was helpful seeing the examples. For the Docker part, some of it went over my head. I only know a little Docker. And my system admin Linux isn’t strong enough to understand the implications of everything she brought up either. I still go something out of it though. And learned things that would be interesting to read more about.