[QCon 2019] The Trouble With Memory

Kirk Pepperdine

For other QCon blog posts, see QCon live blog table of contents

General

  • Slow database queries, inefficient app code and too many database queries are most reported problems
  • Once drill down, over 70% of all Java apps are bottlenecked on memory churn. It’s not reported because hard to observe
  • Tend to put logging around past problems.
  • If apply instrument to a system, it will always tell you something. And then you act on it
  • Cheapar to predict than react

Common libraries

  • Logback
  • Marshalling Json, SQL
  • Caching products
  • Hibernate

Memory

  • Java heap has generations
  • Hopefully people have moved to G1GC
  • Everything happens in the free list

Problems

  • Large number of temporary objects quickly fills Eden
  • Causes frequent young cycles. Causes premature promotion which means will go to tenured too early
  • Heap becomes more fragment
  • Allocation is quick. No cost to collect if objects die quickly. However, still slow if you do something quick enough times.
  • Large live data set size. Data consistently live in your heap. Increases time to copy/compact. Likely have less space to copy to. Think about Windows defragmenter. [Do people still have to do that?]
  • Memory leak from unstable live data. JVM will terminate if you are lucky.
  • Out of memory – 98% of recent time spent in GC with less than 2% of heap recovered. If don’t meet that criteria, app is just really slow, but don’t get the out of memory error.

Escape analysis

  • Test applied to a piece of data. What is the visibility/scope.
  • If scoped locally, only thread that created it can see it.
  • If passed to method, partial escape.
  • If data scoped so multiple threads can see it (Ex: static), full escape.

Demo

  • Showed GC log. Want to see low pause times
  • Showed allocation rates. Problem if too high
  • In Visual VM, looked at profiler. Check filters to ensure not filtering the bottleneck out of your profile
  • Sort by # allocated objects to see frequency. It doesn’t take longer to allocate a large object than a small one.
  • Take a snapshot and look at trace
  • “Stop thinking” – explore what is shown without assuming
  • Time to look at the code from the stack trace that is creating all the objects
  • Escape analysis code
  • Run jitwatch to see allocations. Can see if direct/inline allocation. Can see when bytecode eliminates an allocation
  • Profiler is lying to you.
  • Performance differs in test vs prod environment

Q&A

  • How know the performance problem is the int[] in the demo? Went through profiler to show stack trace. Used BigInteger which uses up a lot more memory than a long
  • Absolute number for GC allocation rate? Sparc? Number seem to hold regardless of hardware. Should focus on the CPU going forward.
  • <missed question> – try to find mutable state that is not shared

My impression

This was great. I learned a lot and it kept my attention. I really liked the demo.

bye world maker faire from FIRST

Norm and I organized the FIRST robotics presence at Maker Faire for the past 9 years. Given that Make Magazine has gone out of business, last year was the last one.

I think some of the Mini Maker Faire events will live on. The NY and San Mateo ones were quite a production to put on. NY hadn’t even opened the call for Makers yet so it makes sense for the event not to happen.

As a way of saying goodbye, I’m posting some of memories from the event.

Maker Faire was held rain or shine. We learned that it can in fact be too hot for a robot. Luckily teams innovated with a sunhat.

The first year our “rain plan” was “it better not rain.” Luckily, we got more sophisticated by the year it was really rainy and windy overnight.

Little kids loved the robots and we won a ribbon almost every year. Two years ago, Norm had the great idea to have each team there on Sunday take a picture with the ribbon.

A shout out to every participating team over the years. As we grew from a small space to a big attraction, we couldn’t have done it without you. Thank you for influencing lots of people and spreading the reach of FIRST. I’m proud to say that I helped *make* a *space* for FIRST robotics exhibitors.

creating a chromebook recovery disk

In the past, I have used the Chromebook utility to create a recovery image USB. This time, it hung at 0% on writing to disk. I tried with two different USBs. Then I started searching for a workaround

A reddit post led me in the right direction. This URL downloads a conf file with the direct URLs to download the image for each Chromebook. I searched the file for my model (Acer Chromebook 15 (CB3-531)) and downloaded the zip file. It was about the size of a CD (655MB)

Then I chose the gear and the local image option. I picked the bin.zip file. It still wouldn’t load. I wound up just copying the zip (and the unzipped cpgz) to the USB. (still under 3GB). That way I have it in case it is needed. My mother would have to bring the ChromeBook to best buy anyway so they can make the recovery disk if needed. We’ve never needed it so I don’t expect to!

I’m now wondering why it needs a 8GB disk when the image is far smaller. i also note that in 2012, it said you needed an 8GB disk and i used a 4GB one. So inflation?