Is Apdex useful?

I’ve been trying to figure out what SLOs to define for some services recently, and wondering if Apdex is a useful metric. (See my previous post on the difference between SLIs, SLOs and SLAs)

What is Apdex?

As background “Apdex (Application Performance Index) is an open standard for measuring performance of software applications” (Wikipedia). New Relic say that Apdex allows you to “see how satisfied users are with your app”.

It is calculated using this formula:

(source: dynatrace forums)

Where T is a static threshold that you define and, if met, would satisfy your users.

The “Satisfied count” and “Tolerating count” are defined as:

• Satisfied: The response time is less than or equal to T.
• Tolerating: The response time is greater than T and less than or equal to 4T.

Apdex problems

This “Apdex is fatally flawed” blog post suggests Apdex should be avoided, saying that some of the issues include:

• Apdex requires a static threshold (T), which it says is a bad approach (“It stinks”)
• Historical Apdex charts useless if there have been any manual modification of “T” (which there are likely to be)

I see their point, even if I don’t agree with all of it. For example, while dynamic thresholds may well be a better approach, static ones are often where most people start, AND they are already prevalent in SLAs, for example.

Some of my own personal concerns around Apdex include:

• The somewhat arbitrary factor of 4 used for “Tolerating”. (again, Tolerating = The response time is greater than T and less than or equal to 4T). It does seem that this has been acknowledged as rather arbitrary since it is, apparently, configurable (See more in this “Apdex – Where does the “four times” factor come from?” article).
• Industry acceptance: I haven’t came across Apdex in any of my recent SRE readings, including the SRE book, and Seeking SRE. The https://www.apdex.org/ site also doesn’t seem to have any new posts since 2014, which doesn’t seem to suggest an active and healthy community.
• Uncertainty around where to alert for Apdex. Alert when it it hits 0.9? 0.85??

 

Conclusion

My conclusion is rather nebulously that Apdex is not bad, but that it is worth trying for something better too.

That is, I think Apdex scores aren’t a terrible way to start defining and monitoring acceptable levels of service for your application. Tools like New Relic make it extremely easy to set an Apdex threshold, and monitor the Apdex scores for example. And if you are not defining any other SLOs or SLAs, you could certainly do worse.

Still, the somewhat arbitrary nature of the formula, and my own uncertainty around what “acceptable” Apdex scores are, mean that I would like to explore more specific SLOs too…

 

PS I was aware of Betteridge’s law of headlines when I wrote this blog post title 🙂

Tags: apdex, applicationperformanceindex, sitereliabilityengineering, slo, SLOs, sre