This is part of my live blogging from QCon 2015. See my QCon table of contents for other posts.

people aren’t logical. need to understand people; not just technology

with kids, you are creating a culture in your family. it’s about what you do, not what you say. modeling behavior. nice to see it played back to you as kids grow.

culture matters because:

• shapes how people work togther
• makes comanies great
• to blame when people not doing right things anymore
• powerful tool for new solutions

people need to be empowered and responsible

nobody should be allowed to treat others badly

culture in tech is creating shared expections for behavior

when something bad happens, does someone say not ok. if not, it becomes ok. culture is the accumulation of lessons like this

No manual on culutre. So learn to recognize, reason about and affect culture around you. In time, you gain the confidence to develop culture.

Conway’s law – structure of the architecture comes to reflect the organization of the company.

rules not written down, but we all know about the culture where we work

culture happens in real time – boundaries between people/groups, unexpected challenges, discovering the rules, rituals that reinforce, fear/anger/surprise/punishment/reward, stories are repeated

leaders set culture whether good or bad. if you set the culture, you became the leader

Once show dialog in team, info starts to flow in different ways.

“need to know” means “you aren’t going to know”

If had an API for culture:

Upper bound (ideal)

What can you do?

• What is your personal culture
• Choose positive culture over other rewards
• Ask about culture on interview
• Develop confidence to speak up when group and personal culture conflict.Thereare other places to go
• Choose people for culturl fit. Model values you want to establish. Avoid ambiguity

Q & A

• How handle public situations? People can only change so fast. “Let’s talk about this later.” Discuss one on one. Explain why a larger issue as group. Values group embodies are in conflict with your action. Choice of bringing closer to those values or looking for something new to do. Can progogate message via someone else on team in a softer/less direct way.
• What do if team culture is in conflict with org culture? Doesn’t matter; have to be prepared to leave as leader to endorse values – those values are that important. Will cause conflict with values/goals of organization.
• Culture takes time to develop. What if planting wrong seeds and waiting 6 months to find out? Takes time for person, but can still get team results faster. If inherit group, can take time. Will probably lose most of those people
• When culture improves, people still have bitterness about past. How deal with that? Harder when not starting from scratch. Bigger organization limits how much one person can change. Even if CEO. Hard to change. Person has to want to change.
• Fluid org structure. How switch teams when different culture on different teams? More about adapting over time. In startup, rate of change is so high, that changes daily/weekly. Don’t want people who will be upset if job/job title changes.
• How balance against getting super talented engineers? So hard to find tech talent that becomes a moot question. Better to get lesser skilled engineer who will be productive and a good fit. Want someone who will pair rather than impeed team work.
• What ask at interview to judge culture? Look at body language – eye contact, body language, flushing skin, talk to team members not just managers. Or ask, what would you do if X or tell me a story about X.
• How “infect” organization to change? Person who cares ,know what is important, talks about it, etc. If people want it, they infect others as well. If in conflict with org values, need top down change

This is part of my live blogging from QCon 2015. See my QCon table of contents for other posts.

Startup 2WebTechnologies made product XL2Web. It took a complex Excel spreadsheet – many worksheets/formulas and make reusable. Converted to a middle layer format and then use Java to produce an in memory version. Made UI in Excel to change field variables and re-calculate server side.

In 2004/2005, Google approached and asked what if could drive emulator through actual web browser client. Made a prototype for IE 6 Service Pack 1 because “when a big company approaches you as a startup you usually do what they want”. It worked and Google acquired the 3 person company. Did server side calculations because couldn’t figure out how to do everything in JavaScript at the time.

Then wondered what would happen if connected a second web client at the same time to the in memory spreadsheet.Got startedwith atomic broadcasting (multiple clients receiving events from a single server). Used a hanging GET to the server so server could always respond to clients. Determined save based on a base state plus a series of mutations until got to a savepoint. Realized needed autosafe. Side effect was that users could access any revision that ever existed.

Mutations must be commutative so multiple changes by multiple users are consistetnt. Used operational tranformations which is a mathematical approach to solve for non-commutativity. Must apply messages in certain order. All peers must apply transformations from their messaging order.

Collaborative undo – If A changes from 6 to 7 and B changes from 7 to 8. A un-does. What should happen. Mix of 6, 7 and 8in audience. 7 means a single undo stack for all collaborators. Users didn’t like this at all. 8 uses log omission. Make it like A’s mutation never happened which invalidates snapshots. Crashed when user undid many edits. Then went with 6 which means revert to A’s state before change. Client has local undo log. 6 is what the user expected to happen.

Scaling – delares master server for all writes to each document. If that server goes down, fail over to a differnet primary. Multiple servers can’t handle non-communtative messages at the same time. Could shard by chapter. Can’t small unit of non-commutativity. Once your unit of consistency is too large for one server, it’s no longer a unit of consistency

Google forms is easier because each line is isolated. Easier to aggregrate.

Google hangouts has 15 person limit to guarantee consistency.

Q&A

• What db given size of data? BigTable like thing
• How deal with latency when different users from different parts of world? Latency has to give to guarantee consistency. Locale caching/async helps
• If client doesn’t receive echo, what happens? Will try to revert mutation if don’t get confirmationin 20 seconds. Then pop up a Google Docs crashed message. Get billions of queries a day. So a one in a billion thing happens 20 times a day. Rather user see a crash message than a chance of data loss.
• How handle sub-sharding a document? Google doesn’t sub-shard chapters in a doc. It’s that they could. Would lose functions like double sace the entire doc.
• Can share docwit any other Google user or send to someone outside and anyone can see it. Can withdraw sharing link? Has enthropy and Google satisfied. [you can make the doc private though, right?]
• How often send? At most once a second. Can throttle based on network use?
• On undo? Now set to new value. Would have to update undo stack

This is part of my live blogging from QCon 2015. See my QCon table of contents for other posts.

Immutable Infrastructure

Never change what’s on server- no upgrading packages, no installing new packages.

We upgrade by creating a new server and deploy to it. Keep old server around just in case.

We do this to avoid configuration drift between “identical” servers. Caused by provisioning servers at different time or manual operations. Normally deal with drift by following careful instructions or automation with many edge cases. For example with parallel ssh, what if one server is down/unreachable or repo partially down. Now have different states.

Rolling back is hard because of transitive dependencies. With immutable servers have old versions of everything so rolling back is easy – switch to old server.

Reprovision servers regularly even if “nothing changed” so know can and always usig recent packages. Also, manual changes get reverted so find out while still remember problem/solution.

Improvement: golden image – Create multiple identical servers from same image. Allows to keep past versions around. Downsides: small changes are cumbersome. Even if automate, takes time. But the time a human is involved is small. Can save time with intermediate golden image. Deploy from checkpoint rather than from scratch

How to debug

Workarounds for dealing with debugging tools

• allow drift but tag as “re-imaging” and schedule self destruct. This lets you install debug tools or whatever you need.
• Install debug tools on all servers. Have feature switch to enable/disabled

How to store data

Need to retain cross server reprovisioning. One option is to use AWS or outsource. Another option is to externalie files such as putting on SAN. Or can use containers.

Containers
Can build from scratch orincremental. Simple DSL to break down the build into steps.

A Docker file is a glorified shell script. [looks like DOS it has commands like RUN git clone repo

Docker recognizes what parts haven’t changed so doesn’t re-install all the infrastructure for an incremental change such as new app code. But still get benefit of two copies – old one is still around.

Containers can share directories, logs, backups, metrics, etc since on same machine.

Can make container read only to enforce immutablity. Easier security because can’t install. Can make exception for data volumes and prevent execution in that area.

Q&A

• How to think correctly about config in container? Some in container. Some in app. What about middle ground? Can use command line params if small amount of config. Or environment variables. Or if lots of config, use a volume or repository of config and pass URL. Another approach is dynamic DNS. (use virtual names and then inject real URLs later. Let’s have redis point to different location for dev/qa/prod)
• What kinds of workloads is container tech suited for? Think any workload can be containerized. Harder to do on desktop such as Open Office. Gets easier over time because automate as more people need it.
• What is best host to run a container on? The one your ops team knows best.
• For an app with core data storage, do you need to bring apps down when update database? Yes, a tiny amount of down time. Does break existing connections/transactions. Have that when do manually too; just a longer one.

Impressions: I like the mix of lecture vs demo. And that it wasn’t very docker specific.