Author: Baron Schwartz (xaprb)

Approaching the Unacceptable Workload Boundary

layout: true


class: title
background-image: url(action-balance-fun-305250.jpg)
background-size: cover

Approaching the Unacceptable Workload Boundary
Baron Schwartz • SREcon18 Americas

class: img-right, bigger

Logistics & Stuff

Slides are at

Ask questions anytime.

Please get in touch: @xaprb or



class: bigger


What happens as systems get bigger and more heavily loaded?

* What is a system’s operating domain?

* How is load defined?

* Where is the load limit? How can you see it coming?

* How does the system behave near this limit?

* Can you measure and model this behavior?

background-image: url(nature-3258924-1280.jpg)
class: title


The Operating Domain


class: center, img-300h

Operating Domain and Failure Boundaries

Rasmussen’s model describes an operating domain bounded by economic risk, effort, and
safety. The system’s operating state is a point within the domain, always moving

background-image: url(rasmussens-model.jpg)

class: img-450h, center

The Actual Boundaries Are Unknown

class: img-450h, center

We Draw Limits Where We Think It’s Safe

Margin of Error
Overdraft Protection

class: img-450h, center, two-column

The Buffer Zone Is Nonlinear



We think the gradient looks like this.

It really looks more like this.


class: bigger

Complex Systems Run In Degraded Mode

Richard Cook lists 18 precepts of system failure in How Complex Systems
Precepts 4) and 5) are especially relevant.

4) Complex systems contain changing mixtures of failures latent within them.
The complexity of these systems makes it impossible for them to run without
multiple flaws being present.

5) Complex systems run in degraded mode.
A corollary to the preceding point is that complex systems run as broken systems.


Systems can and do function beyond their load limits.

class: title
background-image: url(gears-1236578-1280.jpg)


System Load


class: bigger

What Is The Definition Of Load?

There’s no one right answer to this question, but there’s a useful answer
for this discussion.

Load is the sum of task residence times during an observation interval
\(T\). This is equivalent to average concurrency of tasks queued or in

N = \frac{\sum_{}^{}{R}}{T}

You can prove this with Little’s Law.

class: bigger

Load, Utilization, And Queueing

Load (concurrency) is related to utilization and queue length, but it’s not
the same.

* Concurrency is the number of requests in process simultaneously.

* Average concurrency is an average over an observation interval \(T\).

* Utilization is the fraction of \(T\) that was busy.

* Queue length is the instantaneous or time-averaged number of tasks waiting
to be serviced.

class: bigger

Utilization, Queue Length, & Concurrency

By Little’s Law, utilization and queue length are types of concurrency.

Utilization is the concurrency of in-service tasks.

* Queue length is the concurrency of queued tasks.

class: two-column, bigger

What Is The Load Limit?

If the load limit were defined in terms of utilization, queueing theory could
tell us where the load limit will be.

But it can’t: load can be infinite, utilization ranges 0-1.

Plus it’s impractical:
* The “hockey stick” queueing curve is hard to use
* The “knee” is unintuitive



This is appealing because utilization has a clear limit: it can’t be more than

So we need to translate the problem to a different domain, where the units
match. Scalability is the answer.

class: title
background-image: url(snow-3260088-1280.jpg)




class: bigger

What’s the Definition of Scalability?

There’s a mathematical definition of scalability as a function of

I’ll illustrate it in terms of a parallel processing system that uses
concurrency to achieve speedup.

It’s practical, easy to use, and matches the domain well.

I’ll show how the equation is composed piece by piece, but don’t sweat the math.

class: bigger, img-center

Linear Scaling

Suppose a clustered system can complete X tasks per second with no

With parallelism, it divides tasks and executes subtasks
concurrently, completing tasks faster.

Faster completion also means increased throughput.

* Tasks per second is throughput.
* Throughput is a function of concurrency.

class: bigger, img-center

Linear Scaling

Ideally, throughput increases linearly with concurrency.

* Linear scaling is the ideal.
* Another way to say this is that the system’s output is a linear function of

class: two-column, bigger

The Linear Scalability Equation

The equation that describes ideal scaling is

X(N) = \frac{\lambda N}{1}

where the slope is \(\lambda=X(1)\).



– X is throughput
– N is concurrency, which is the workload
– Lambda is the system’s output when there’s no parallelism
– Really important to note that N is the independent parameter, the driver

class: center, bigger

But Our Cluster Isn’t Perfect

Linear scaling comes from subdividing tasks perfectly.

What if a portion isn’t subdividable?

class: two-column,bigger

Amdahl’s Law Describes Serialization

X(N) = \frac{\lambda N}{1+\sigma(N-1)}

Amdahl’s Law describes throughput when
a fraction \(\sigma\) can’t be



class: bigger

Amdahl’s Law Has An Asymptote

X(N) = \frac{\lambda N}{1+\sigma(N-1)}

Parallelism delivers speedup, but there’s a limit:

\lim_{N \to \infty}{X(N)} = \frac{1}{\sigma}

e.g. a 5% serialized task can’t be sped up more than 20-fold.

If 5% of the work is serialized, infinite concurrency will still result in tasks
taking 5% as long as non-parallelized tasks.

class: img-center, bigger

What If Workers Coordinate?

Suppose the parallel workers also have dependencies on each other?

class: two-column, bigger, img-center, img-300h

How Bad Is Coordination?

\(N\) workers = \(N(N-1)\) pairs of interactions, which is
\(\mathcal{O}(n^2)\) in \(N\).





class: two-column, bigger

The Universal Scalability Law

X(N) = \frac{\lambda N}{1+\sigma(N-1)+\kappa N(N-1)}

The USL adds a term for crosstalk, multiplied by the \(\kappa\)

Now there’s a point of diminishing returns!



Crosstalk is also called coordination or coherence.

class: bigger, img-center

You Already Know This

You’ve seen lots of benchmarks with diminishing returns.


By the way, pay attention to the axis scale, it’s log-scaled by powers of two.
If you scale the X-axis linearly you’ll get the shape of the curve on the
previous slide.

class: img-center, bigger, img-300h

The USL Describes Behavior Under Load

The USL explains the highly nonlinear behavior we know systems exhibit near
their saturation point.

– Serialization (red) grows slowly, but crosstalk (blue) grows rapidly.
– This is why systems get so unpredictable near their limits.
– Near and above the point of diminishing returns, systems exhibit high variance
and get unpredictable.

class: bigger

A Summary Of The USL

The Universal Scalability Law defines throughput as a function of concurrency.

It explains how and why systems don’t scale linearly with load.

class: bigger

What is the USL Good For?

Armed with the USL, you are ready to:

Measure and model nonlinear behavior.
Predict the onset of nonlinearity.
Design better systems.

It’s easy. Let’s see how!

class: title
background-image: url(compass-2946958_1280.jpg)


How To Measure, Model, And Predict


class: bigger

What To Measure

You can’t measure serialization & crosstalk directly.

Instead, measure throughput and concurrency.

Then fit the USL model to the data to estimate the parameters.

class: center, middle



Throughput is so trivially easy to measure in most systems that I won’t talk
about it. But there’s two easy ways to measure concurrency.

class: bigger

How To Measure Concurrency, Pt. 1

Many systems have a metric of concurrency already.
Look for a metric of things actively working.

MySQL: SHOW STATUS LIKE ‘Threads_running’

Apache: active worker count

It works well to poll this e.g. 1x/sec, then average these into 1- or
3-minute averages.

class: bigger

How To Measure Concurrency, Pt. 2

If there’s no metric of concurrency, you can sum up latencies and divide by
the duration.

N = \frac{\sum_{}^{}{R}}{T}

– Again, in my experience it’s good to use averages over a moderately long window like 1-5 minutes.
– You want to end up with dozens to hundreds of data points.

class: img-center, img-450h

Plot Your Data

Simply scatterplot your data and eyeball it for sanity.

Source data in “row16.csv” file. If you’re reading this note and you’re not a
VividCortex employee, sorry, I can’t give you access to this data.

class: bigger, img-450h

Plug The Data Into The Model

Paste the data into the Excel
model I built.

You can do it in R, or gnuplot, or even with JavaScript in Plotly. Lots of
options. This is an easy one.

class: bigger

Interpreting The Results

What does the output mean?

Shows whether your system has more serialization or crosstalk.

– Shows the estimated max load where it’ll stop scaling.

– Helps you predict nonlinearity.

class: bigger, img-center

Paypal’s NodeJS vs Java Benchmarks

Paypal’s NodeJS vs Java benchmarks are a good example!

class: bigger, img-300h, img-center

Bringing It Back To The Operating Domain

The USL is one way to understand what happens near this boundary.

class: bigger, two-column

What Happens Here?

– When the system approaches workload limits it gets twitchy.
– You may be able to see this approaching before it gets bad.
– Simply scatterplotting throughput vs concurrency is super useful!



class: two-column, bigger

You Don’t Need To Do Any Modeling!

Let’s take another look at this data. What jumps out?



class: two-column, bigger

What If You Had Only The First Part?

– I would model and project out to the right.
– I’d see “hmm, it’s leveling off.”
– I’d say “don’t count on much more than you see now.”



class: two-column, bigger

Think Differently About Outlying Points

– Given all the data, I mentally cluster it into two parts.
– If the high-end outliers deviate, it’s nonlinear already.
– Those points are evidence that the system is struggling there.
– You don’t need to model anything to see that.



class: center, two-column, bigger


Some Resources

I wrote a

I created an Excel

These slides are at



How To Write Exciting Conference Proposals

Most conference proposals are too boring, even when the speakers and topics are
great. This is a pity. I think something about the process of submitting to a
CfP sets a trap for most speakers. This post is my advice for avoiding that

TL;DR: Your proposal should focus on your story about what you’ve done personally
and what you’ve learned. Your story, not the topic. And, don’t tell us anything
about the importance of the topic or how high the stakes are.

I’ve written twice before
(1, 2) about
how to write better conference proposals. But while recently reviewing another
thousand or so proposals, I suddenly understood something simple that separates
the boring ones from the exciting ones. And I realized that I make the same
mistake myself when I submit proposals!

Because I only see this pattern myself when I’m reviewing, not when I’m writing
proposals, this time I’ll try to help you experience what it’s like to review
conference proposals. Remember, your first audience is the review committee, who
are reading hundreds of proposals.

What follows are real conference proposal abstracts. I have lightly edited where
necessary to avoid identifying people explicitly. If you see one of your
proposals below, please don’t take offense or jump to any conclusions about
implied quality of your talk or expertise. Your proposal is hopefully serving a
valuable purpose as part of this blog post.

Read through the following, spending no more than 8-10 seconds on each:

The rate of business change is accelerating. Organisations must build the right things better and faster to survive. Agile, Continuous Delivery and Lean help, but we need practices to ensure the “right things” flow into these processes. This talk explores design sprints in the enterprise to: Bridge the business-IT divide; Build the right thing; Increase organisational agility.

Databases require capacity planning (and to those coming from traditional RDBMS solutions, this can be thought of as a sizing guide). Capacity planning prevents resource exhaustion. Capacity planning can be hard. This talk has a heavier leaning on MySQL, but the concepts and addendum will help with any other data store.

The popularity of containers has driven the need for distributed systems that can provide a secure substrate for container deployments. In this talk, we will go over how Acme has been working on multiple components to secure every part of the container platform – from the newly announced LinuxKit all the way up to SwarmKit – and how all of these blocks fit together!

The Continuous Delivery (CD) movement has been around for more than a decade, and its lessons are even more relevant when embracing container technology. The talk will provide guidance and specific technical solutions on how to implement CD best practices on a reference implementation based on Git, Docker, Kubernetes and Terraform.

With ever increasing demands for fast business change how can we ensure our digital channels have the increasingly exacting standards of performance our customers (and business owners) expect? What does this look like in an age of DevOps and Continuous Delivery? We’ll take you through our experiences as we build a strategy for shifting left and automating performance analysis.

End-to-end engineering ownership – or DevOps – is the only way to scale operations in a micro-services world. Achieving this in large engineering organizations requires the right mindset, architecture and processes. The Acme Small Business Group (SBG) is on the journey to turning it’s 1000 engineers into full lifecycle engineers who own their services from cradle to grave.

Creating a fast & reliable pipeline poses numerous hurdles that cause most dev teams to fall back to slow release cycles, low levels of test & platform coverage & straight up buggy apps in production. To keep re-work & technical debt down to a minimum, defects must be identified & addressed as early as possible & frequently across multiple environments & under realistic conditions.

Do you see any similarities in these proposals? I selected only 7, but already
the pattern emerges, and most of them feel very alike in some ways:

They point to significant changes or important circumstances, to create
context for the talk.
They mention specific challenges that arise in these circumstances.
They refer to the urgency of learning what the speaker will share.
They pose questions, but don’t answer them, as a way of showing what the
audience will learn from the talk.

On reflection, all of these are very natural things to do, and seem like good
things to do for that matter. And I think conferences usually ask speakers to
make sure proposals include this information! Unfortunately, though, the result
is mind-numbing when you’re reading through proposals and trying to decide which
to vote for (or which to attend, if you’re the audience).

What’s going wrong? Although well-intentioned, the effort to illustrate the context
and importance of the topic ends up having a very different effect:

The first several sentences of most proposals focus on obvious,
well-established facts that don’t need to be stated. The speaker is trying to
frame the topic and its importance, but it ends up being equivalent to humans
need oxygen to live or something similarly banal.
The speaker tries to indicate that the subject matter will be widely
applicable and everyone will find it relevant, but it ends up coming across as
regurgitation of best practices.

As soon as I realized this, I was able to pinpoint two specific things about
clear, brisk proposals that make them more fresh and exciting. To illustrate
this, here are two proposals on a similar topic, hiring:

Today hiring the best people for the job is getting difficult and keeping them is another struggle. Not forgetting that people and plants are similar in that in order to become their best they need to have the right environment. This talk is about how to build a workplace (culture) that attracts talent and enables them to give their best.

Eric Schmidt’s quote, “Hiring is the most important thing you do,” is easy to
say and very hard to implement. In this session, we’ll share the highs and
lows of our journey to rebuild our recruiting program from scratch and create
an approach that scales to the needs of our business and increase our annual
hiring rate by 400% at the same time.

What’s different?

One is about the topic and the other is about the speaker’s experience.

One starts with context and the other starts with a challenge.

After I realized this, it became clear to me that a lively proposal is usually a
personal story about learning from an experience, and a boring one is usually
about a topic and best practices.

Here’s another lively one:

Our company did releases every 4 weeks by IT Ops at early hours 2 years ago.
We transformed our way of working in less than 2 years to teams deploying at
all hours, with the responsibility of monitoring their applications during
business hours. How do you handle 400 IT-professionals with different skill
sets? We created a guide for DevOps in our organization and want to share our

The general tone of this proposal is “we did something, learned something, and
we want to share it with you.” I’m instantly excited to attend this session!

What’s my advice to people writing conference proposals? Tell a story about
what you have actually done yourself, and don’t worry about setting the context,
motivation, importance, or urgency, unless it’s non-obvious. Stories always
win, and everyone knows that the pace of business change is accelerating. People
want to hear your story, your mistakes, and your lessons learned.

Also, remember that busy reviewers will only give your proposals a few seconds
of attention.

What am I not saying in this post? I’m not saying that writing needs to be
high quality. When I realized that I didn’t really care a lot about passive
voice and bad grammar while reviewing, I felt guilty that I’ve emphasized the
need for “good” writing in the past. I’m not saying it doesn’t help. It does:
clear, active, direct, compelling writing always helps. But it’s less important
than the other points I’ve emphasized in this post. Some of the exciting
proposals have pretty egregious writing mistakes, and I don’t care! A run-on
sentence and passive voice is entirely forgivable as long as the speaker’s
experience comes through clearly.

I know it’s hard. It’s so hard that I don’t follow this advice myself. But I’ll
try harder in the future. I hope this helps.

PS: You’ll note that not all of the first seven “bad” proposals are wholly
boring. Some of them do point to the speaker’s experience and story. I chose
those seven almost sequentially from a conference, skipping only a couple that
seemed exceptional. They’re not bad, they’re just ordinary, and many of them
will be great talks even if the proposals are hard to read. My point is simply
that when you read 350 ordinary proposals in a row, a non-ordinary one leaps out
and screams for a high rating.

Pic Credit

Monitoring with Artificial Intelligence and Machine Learning

Artificial intelligence and machine learning (AI and ML) are so over-hyped today
that I usually don’t talk about them. But there are real and valid uses for
these technologies in monitoring and performance management. Some companies have
already been employing ML and AI with good results for a long time.
VividCortex’s own adaptive fault detection uses ML, a fact we don’t generally

AI and ML aren’t magic, and I think we need a broader
understanding of this. And understanding that there are a few types of ML use
cases, especially for monitoring, could be useful to a lot of people.

I generally think about AI and ML in terms of three high-level results they
can produce, rather than classifying them in terms of how they achieve those

1. Predictive Machine Learning

Predictive machine learning is the most familiar use case in
monitoring and performance management today. When used in this fashion, a data
scientist creates algorithms that can learn how systems normally behave. The
result is a model of normal behavior that can predict a range of outcomes for
the next data point to be observed. If the next observation falls outside the
bounds, it’s typically considered an anomaly. This is the basis of many types of
anomaly detection.

Preetam Jinka and I wrote the book on using anomaly detection for
Although we didn’t write extensively about machine learning, machine learning is just
a better way (in some cases) to do the same techniques. It isn’t a fundamentally
different activity.

Who’s using machine learning to predict how our systems should behave? There’s a
long list of vendors and monitoring projects. Netuitive, DataDog, Netflix,
Facebook, Twitter, and many more. Anomaly detection through machine learning is
par for the course these days.

2. Descriptive Machine Learning

Descriptive machine learning examines data and determines what it means, then
describes that in ways that humans or other machines can use. Good examples of
this are fairly widespread. Image recognition, for example, uses descriptive
machine learning and AI to decide what’s in a picture and then express it in a
sentence. You can look at to see this
in action.

What would descriptive ML and AI look like in monitoring? Imagine diagnosing a
crash: “I think MySQL got OOM-killed because the InnoDB buffer pool grew larger
than memory.” Are any vendors doing this today? I’m not aware of any. I think
it’s a hard problem, perhaps not easier than captioning images.

3. Generative Machine Learning

Generative machine learning is descriptive in reverse. Google’s software
famously performs this technique, the results of which you can see on their

I can think of a very good use for generative machine learning: creating
realistic load tests. Current best practices for evaluating system performance
when we can’t observe the systems in production are to run artificial benchmarks
and load tests. These clean-room, sterile tests leave a lot to be desired.
Generating realistic load to test applications might be commercially useful.
Even generating realistic performance
is hard and might be useful.

Photo Credit

Remembering Alan And Harry

Harry Weller died suddenly this week. Harry was one of the best venture capital investors in history. He led NEA’s investment in my company, VividCortex. Harry was an advocate and mentor to me, but he was more: in both life and death, he reminded me of Alan Rimm-Kaufman, a boss and mentor who died a handful of years ago.

Alan Rimm-Kaufman

Harry isn’t the first person to believe in me and create an amazing opportunity before dying unexpectedly. Before Harry, there was Alan Rimm-Kaufman, who shaped my career tremendously.

I joined Alan in 2006 at the Rimm-Kaufman Group, a bootstrapped startup he founded in Charlottesville, Virginia. Alan had an infectious love of life and was unstoppably energetic. He’d enter a room with his face lit up, bubbling with an idea he couldn’t resist sharing. He was unable to stand still. He hopped around like a soccer goalie.

Alan would start a thought and someone else would venture a related idea. Alan would pounce on it. “Yes!” he’d say. With Alan, you had zero doubt that he believed in you. I sometimes wondered if the ideas were all his anyway, and he just withheld parts of them, letting us complete the thoughts he’d already had so we’d believe in ourselves too.

One day I said I might want to write a book someday. Alan didn’t even blink. “Oh yeah,” he said, as he bounced up and down on my yoga ball. “Oh, YEAH! You’ll be published in less than 18 months.” It wasn’t a question. It was foretold. When the opportunity to coauthor High Performance MySQL appeared a short time later, Alan switched me to half-time work with full-time benefits and gave me a private office to work in. He told me to shut the door in the morning and write until lunchtime. I don’t remember how long this went on—perhaps a year.

Alan also sent me to conferences, where Google tried to recruit me. I didn’t want to move to Silicon Valley. Alan thought I was crazy and sternly told me to go interview with them. The day before I left, he handed me a letter full of reasons I’d be better off not taking the job. It was classic Alan: ten hilarious statistics like the number of poisonous snake species in California versus Virginia. I didn’t take a job at Google, but I got to know many wonderful people there.

Then I broke Alan’s heart anyway: I left to work at Percona. When I walked into his office he knew. I remember him doing his best to be happy for me.

While I got absorbed into my new life, working all hours of the day and night, Alan’s leukemia returned. I later found out it was a big part of why he was so energetic. After it went into remission the first time, he’d attacked life and all of the crazy dreams he had for himself and his family with renewed vigor, no holds barred. This time, the disease was not to be beaten. It was looking hopeful, and he was feeling good, he wrote me in emails. But then, suddenly he was gone in a matter of days.

When I re-read my emails, I realized Alan was asking me to come see him before he died, but I had been too busy to pay attention.

I was devastated at Alan’s death. It was the first time anyone I cared about had died, and Alan had been like a father to me. But far worse, I was consumed with guilt at being too selfish to go see him, wracked with regret at taking him for granted. It took me a long time to forgive myself. Eventually, I realized that the regret itself was a waste. It was just a form of self-hatred so I could feel better for punishing myself. When I recovered enough to see this, I realized Alan would have wanted nothing more than for me to stop wallowing in guilt and live fully as he had. That was part of the reason I left my job to found VividCortex.

Harry Weller

My first contact with Harry was when he reached out to me in early 2013. At the time, I was fundraising for VividCortex, and many potential investors were highly skeptical. It was discouraging, because some of the more deeply technical investors, those who’d run databases themselves, were more skeptical than those without domain knowledge. There were lots of helpful and supportive venture capitalists, but there was also meeting after meeting after meeting of “there’s no problem here. This isn’t a business. This is niche at best. Nobody has pain that this solves.” Some were polite about it, some barely concealed their feelings.

In just a few words, Harry proved that he got it in a way few others had done. He knew there was a need and opportunity, even though the timing wasn’t right. Over the following couple of years, we stayed in touch frequently. He never wrote more than a sentence or two, but he always did something helpful: an introduction, an encouragement, something. I sent him occasional updates that would fit onto an iPhone screen.

We brought the company through an existential crisis, then we hit cash-flow positive and the business’s fundamental strength became obvious in the numbers. At each milestone, Harry wrote me a few words. When we won a huge deal, for example, Harry just wrote “can u call me” and his phone number. When I got him on the phone he celebrated our success. He cheered me more than I cheered myself.

One day his message was simply “is there a dollar amount of investment that gets me X% of the company with X% postmoney option pool? Hello from a ship in the Atlantic.” And that was how it started. I had a pitch deck, I had financials, a roadmap, a slide full of logos in a quadrant with us at the top right corner, all that jazz. But he didn’t need it. We found a time between two board meetings, and I drove to his house and waited to meet him in person for the first time. He arrived ravenously hungry, and started pitching me on why we should work together while he made a smoothie, which he shared with me.

Like Alan, he was like a kid who couldn’t stand still. He paced restlessly, interrupting himself midsentence to call other partners at NEA and tell them how big this could be, then hanging up and resuming his thoughts. Like Alan, he told me to stop explaining why this was a good idea, he didn’t need convincing. It wasn’t a question of whether we’d make it. Hah! His biggest fear was that we wouldn’t execute fast enough to outrun the competition. He asked a few questions, did some mental math that was startlingly accurate, summed up the business in a few sentences, and told me if we didn’t fund up and sweep the market, our raw efficiency would become obvious to other investors and they’d fund someone to outgun us. “I’m telling, you, I’ve seen this again and again,” he said. “You’ve built a great business, and you’re not going to be able to hide it. In enterprise infrastructure software, my one worry is that you can’t stay ahead of the pack that will come after you. And make no mistake. They will come after you.”

I wasn’t even officially fundraising, but I took his investment and we hit the ground running. Harry frequently called me out of the blue. When I called him I usually got voicemail, then a call back a short while later. I have no idea how he did it. He was on 14 boards and leading the East Coast practice at the world’s largest venture firm, among 99 other things. How can one person have so much bandwidth?

I didn’t have enough time with Harry to get to know him well, not even a year, and only a couple of hours all together. I was still coming to terms with some of his ways, forming opinions about how he operated. He was teaching me to lead more strongly, and the way he did it was brilliantly simple and effective: he told me what the job needed and that if I didn’t do it, a new CEO would. His point was unmistakeable: there is no room in a fast-moving company to leave authority or responsibility vague or let it be owned by the group. It had the effect of pressuring everyone to meet a higher level of performance or leave. I’m still trying to learn how to do that myself.

If I tell the unvarnished truth, I was sometimes nervous of Harry’s motives, because I’d never worked with someone who took pains to sniff out a power vacuum and feint at it to force me to claim and defend it myself. But in the last couple months, I was honestly beginning to believe he was doing it just to make me be my best.

Then I got the call.

Alan and Harry’s Legacy

Alan and Harry were intensely focused on all of life, and accomplished extraordinary things. They live on through family, work, religious community, and many other ways including me. I am part of their legacy, and I try to honor them by doing what they did: greet each day with joy, optimism, and energy. Put the pedal to the metal and don’t even think about the brakes.

I was looking forward to seeing Harry soon, and telling him what’s taking shape right now. But I don’t need to, because I already know what he’d do. He’d cheer me, because it’s risky and gutsy and crazy and of course it will work! He’d tell me damn the torpedoes and full speed ahead, he’d tell me to run, run, run! and do it before anyone else does. I know Alan would say exactly the same thing.

Words can never express my gratitude to Alan and Harry. Words are puny. You know what’s sufficient? Actions, and actions alone.

PS: To the countless others who’ve also believed, supported, helped, and encouraged me: thank you, it means more than I can say, but watch me, and I’ll put my gratitude into action; and someday I hope I can do as you’ve done, too.

Just Enough Better

What makes a product, project, or application become popular? People talk a lot about attributes such as stickiness, virality, addictive qualities, and gamification.

I don’t pretend to have a universal answer, but many of the successful things I’ve been involved with worked well because they were just enough better for people to care a lot.

I could cite a lot of examples from my personal career and work, but one in particular stands out to me because it has evolved a few times and has found more success each time for different reasons. I’m referring to the book High Performance MySQL, which is very successful by technical book standards.

The first edition was a hit for a few reasons. First, it was timely; people wanted the book because there was a lot of demand for and interest in using MySQL for the explosion of Web 2.0. Secondly, the MySQL database was pretty rudimentary, with a lot of limitations and performance issues, and Jeremy had experience using it at scale where performance mattered. Thirdly, in my opinion the audience was largely uneducated about relational databases. This was not a book for Oracle experts learning MySQL; it was a book for PHP programmers who didn’t know much about database fundamentals. Jeremy and Derek hit a few sweet spots. The book wasn’t a masterpiece, but it was five out of five stars for its audiences and their needs, because the alternative was a couple of blog posts here and there, plus the MySQL documentation.

The second edition was even more popular than the first. MySQL had spread far beyond the web use case, although that was still core, and was being used at far larger scale and under much more demanding performance scenarios. There was a huge audience for whom the first edition was now outdated and didn’t have enough breadth. Peter and Vadim brought the experience to address that. They’d solved far more than a handful of types of performance problems; they’d solved hundreds. There was practically nothing you could ask about performance that they didn’t have some experience with. Different workloads? Check. Widely varying hardware? Check. And so on. Where the second edition excelled was in presenting example solutions and worked-through narratives for a huge variety of problems. It was just enough better than the first edition that people bought it in spades.

What could a third edition offer? In hindsight, the second edition was pretty chaotic. All that information was scraped together and edited into as much of a coherent whole as possible, but it couldn’t teach much beyond the scope of the specific stories. It taught use cases and tips and tricks, but not a mindset and method. In the third edition I wanted to pull back the curtain and teach the reasoning that led to all of the successful solutions to difficult problems I’d seen with Peter, Vadim, and others. If I’d written about the problems and solutions themselves, it could have filled several thousand pages. There were other improvements over the second edition, but my focus was on teaching much more by teaching much less.

As a result, the third edition continues to sell extremely well, even as MySQL has seen several major new versions and there have been drastic changes to the technology landscape in general (hardware, cloud, blended relational and multi-model data access, etc). It’s no more of a masterpiece than the first edition, but it’s better in the right ways.

What would make a fourth edition just enough better than the third edition? I’m not at all sure that there will ever be a need for a fourth edition. I’ve seen many different books about databases and performance. Some of them barely talk about databases or performance at all. Some of them are about other things, like architecture of large-scale applications. I’ve been inspired by many of these books, but I’ve never thought, “aha! If I applied this to a MySQL-specific context, this would be the backbone and unifying concept of a fourth edition of High Performance MySQL!”

Maybe it’s just run its course. Maybe it’s good enough that there’s never going to be a just-enough-better MySQL-specific book.

Or maybe it will eventually be updated superficially just to meet demand for new MySQL versions, without doing much else. (Would that book sell well? I’m not sure.)

Who knows? All I know right now is that I don’t hear people telling me en masse that there’s a gap between what they need, and what they get from the book. I could be wrong, but it seems that the third edition has remained “good enough” for a larger group of people and for longer than before, and as long as that remains true, there’s no problem to solve. It’s hard to create something “just enough better” when there’s already something good enough.

And solving a problem, with something that’s not only good, but better, seems to be the threshold you have to cross.

Photo Credit

Help Me Make This Blog More Accessible

Dear Reader,

Hi! I probably don’t know you, but I’d like to make this blog better for you. If
you’re disabled—for example, perhaps you use a screen reader, a special
device, or have other needs that I might not anticipate, I’d really love your
feedback on what I can do to make my content easier for you to read and enjoy.

A couple of things I’ve been thinking about recently:

If you use a screen reader, does the navigation or other stuff get in the way?
If you use some device to do something like, I dunno, convert my RSS feed to
podcasts and listen in the subway, how well does that work?
What should I do if I want to include a complicated equation? Use an image,
use something like MathJax, mark it as skippable for screen readers?
If I include images, does it matter what I name them? How good of an
experience is the image I included above?
What can I do to make links easier to use? Should I have meta tags like “next”
and “previous” and so on? Should I always add link titles or something?

I’m going to link to a couple of recent articles and ask you what I could have
done better to make them easier for people-who-are-not-like-me. Here: article
1, article


Photo Credit

What Does The Universal Scalability Law Reveal About MySQL?

In the last couple of weeks, there have been a few blog posts about benchmarks
comparing the performance of various versions of MySQL and variants such as
MariaDB. There’s also been some analysis of the results using formal models such
as Neil Gunther’s Universal Scalability Law.

What can the Universal Scalability Law (USL) teach us about the performance
characteristics of these systems, as revealed by the benchmarks? To find out,
I’ll examine one particular benchmark, MariaDB 10.1 and MySQL 5.7 performance
on commodity hardware.

The context for this benchmark, in a nutshell, is that MySQL 5.7 was just
released as GA, and the published performance results are spectacular, from the
point of view of throughput on large servers. Although this is good, showing
that MySQL can scale to do lots of work on large servers, the MariaDB
benchmarkers wanted to understand how a simple benchmark would run on a typical
commodity server.

The benchmark compares several versions of MySQL. Neil
tweeted an analysis
of the benchmark with the USL:

Be careful with that chart, because the horizontal axis is log-scaled, not
linear. What can we learn from that? First of all, if you’re familiar with the
USL at all you’ll instantly recognize that the system isn’t behaving as the USL
predicts. A much better way to model this is to use only the first few
datapoints to predict what might happen if the system didn’t encounter the
limitation we can see beginning at 8 connections. Neil did this

Neil’s commentary, which I’ll paraphrase for clarity, is If you can remove the
bottleneck at 118k queries per second, the USL predicts that the system will hit
a ceiling around 250k queries per second. Can MySQL or MariaDB get there? If
not, why not? With the USL, it’s no longer OK to just measure, you have to
EXPLAIN the data.

The explanation is simple. The server has 4 cores and 8

Intel Xeon E5 E5–1630v3 4/8t
3,7 / 3,8 GHz 64 GB RAM
DDR4 ECC 2133 MHz 2 ×2TB SOFT

This server can run only 8 queries simultaneously. After that, adding
more connections trying to run queries is not going to improve things.

The USL doesn’t include parameters that model the fixed size of underlying
hardware and other resources, and as a result, it’s very common to find some
type of resource saturation that creates a ceiling it doesn’t predict. This is a
perfect example.

As I discuss in my new 50-page ebook on the
Universal Scalability Law,
it’s rather pointless to try to use the USL to assess behavior beyond a
saturation point such as the one you can see in this benchmark. That’s why
fitting the USL to the data up to and including 8 connections is the right

Beyond the saturation point, all you can see is whether, under increased queue
length, the server stays efficient or wastes effort. Most servers lose
efficiency. To find out exactly what causes this, please read the ebook I linked

The charts above are all of several versions of the server analyzed together. I
think it’s a better idea to analyze only a single version of the server. Let’s
look at the results for MySQL 5.6.27 only, because on this benchmark it beat the
other versions:

clients qps
1 24456
2 45314
4 78024
8 126892
16 129029
32 127780
64 125526
128 124158
256 116337

And the USL analysis of this data up to 8 clients:

There’s such a small kappa coefficient that it should be ignored, however,
sigma is about 7.4%, which is quite significant and limits performance very
seriously. If you’re familiar with the USL this provides concrete ideas about
what needs to be changed, and defines how far you can scale this server on this
type of configuration. When I was at Percona in the 2009-2011 timeframe we made
a business out of finding and alleviating those kinds of bottlenecks,
which is what led to XtraDB and eventually to Percona Server.


When a system hits an obvious ceiling, don’t try to fit the USL to the data
beyond that point. Especially when you know the hardware explains the
behavior simply.
What happens beyond that point does indicate something about the server’s
ability to go beyond saturation without completely spitting its cereal. We
see essentially a flatline here, but in older versions of MySQL we used to
see serious retrograde scalability.
The USL’s parameters point you in the right direction to find and fix
scalability problems.
There are lots of weird and contradictory results for lots of benchmarks.
Benchmarketing is everywhere. Learn about the Universal Scalability
Law and
innoculate yourself against lots of brain twisters.

photo credit

Don’t Miss PGConfSV, Silicon Valley’s Newest PostgreSQL Conference

If you haven’t heard about PGConfSV yet, it’s a
conference for the Silicon Valley PostgreSQL community and beyond, featuring
leading PostgreSQL performance and scalability experts. It’s happening November
17-18 at the South San Francisco Conference Center. I encourage everyone in the
area to attend, since this is likely to be the best Postgres conference held
thus far in the Silicon Valley.

I also urge you to buy your tickets before they sell out! Of course, the earlier
you buy, the more you save, too. (Use SeeMeSpeak for a 20% discount).

I’ll be at the conference along with some of my colleagues. I’m pretty excited
about this for a few reasons. Allow me to ‘splain why?

First, I’ve loved PostgreSQL for a long
time although fate has
mostly seen me work in the MySQL community. I’ve attended, spoken, helped
organize and otherwise been involved in a handful of Postgres events, and this
conference brings back a lot of good memories and anticipation. Among other
things, Terry Erisman is one of the principal organizers and he does a
fantastic job of overseeing conferences. Seriously. He’s one of the big
reasons the MySQL conference not only continued, but rebounded after the dark
days of the Sun acquisition. PostgreSQL fans should be thrilled to have him

Secondly, VividCortex is a sponsor, so I’ll be
there in an official capacity.

Finally, I’m
about sniffing the network protocol off the wire and analyzing it. If you’ve
seen any of my talks about “finding hidden treasures in TCP traffic” kinds of
topics, hopefully you’ll have an idea of what might be in store in that talk.

I also participated in an
leading up to the conference, on why I’m so excited not only about this
particular event, but also why I’m more and more excited about PostgreSQL in
general. As a community and as a technology, it’s really exploding onto the
scene in ways a lot of people don’t appreciate yet. (But they will!)

Come be a part of the next big sea change in opensource relational databases!

The Case For Tagging In Time Series Data

A while ago I wrote a blog post about time series database
requirements that has been
amazingly popular. Somewhere close to a dozen companies have told me they’ve
built custom in-house time series databases, and that blog post was the first
draft of a design document for it.

One of the things I said in the post was that I had no use for the “tagging”
functionality I’ve seen in time series databases such as OpenTSDB. I’ve since
reconsidered, although I think the functionality I now want is a bit different.

What does it mean to “tag” metrics? Typically, many time series databases let
you add name=value pairs (tags) to individual time series points (observations).
For example, you measure CPU usage to be 59% at 3:41PM on host inst413, and you tag
this measurement as “shard=81” because inst413 holds data for shard 81.

The use case for this is that you can now aggregate and filter metrics, looking
at for example CPU usage for all of the shard 81 hosts at a point in time.

Tags are typically attached to individual points in the systems I’m aware of,
rather than being attached to a host or similar, because the assumption is that
they’ll change over time. You might move shard 81’s data to a different set of
servers. You might destroy host inst413 and rebuild it, and it might not even be
a database anymore.

For my purposes at VividCortex, I want “tags” to
help slice-and-dice-and-filter for a variety of customer scenarios.

We currently can show you Top Queries by total time, and Top Queries by the
count of queries that lack indexes. Customers want to see Top
Queries by total time, filtered to show only those that lack indexes.
We can show you Top Users by total time, indicating the total service demand
placed on a set of hosts by queries running against them from a particular
user. But what our customers want is to see Top Queries by total time,
filtered to those that come from a specific user.

There are a variety of cases where we want to decouple the metrics from the
filtering that can be applied to them. Some we can support by joining together
related metrics, at some cost on our backend; some we cannot currently do as
well as we want, or in the way we want.

However, I do not like one thing about the common tagging functionality. I don’t
like that individual metric points all have to bear the burden of every tag for
every point.

Instead, I view tags as time series facts that exist during some time
interval. And I think these should be applicable to the metric or the host (but
I don’t see a use case where I want them per point in a metric). For example,
a query doesn’t use an index. I can tag the query’s metrics as “no_index=true”
from the timestamp I first observe that, till the timestamp that I observe it
using an index. These durations can be open-ended, in which case they apply for
all time.

Similarly, I can tag the host as “shard=81” during the time that this host
carried that shard’s data.

Finally, the tags need to permit multiple values. Imagine a query that is run by
several different users; we wouldn’t want a uniqueness constraint on it.
“user=nagios” and “user=vividcortex” should be able to coexist for the query
“SHOW GLOBAL STATUS” in MySQL, to mention a real-life use case. Without this
capability, a search for all queries that the vividcortex user runs wouldn’t
work as expected.

Photo credits: tagged

TEL/電話+86 13764045638
QQ 47079569