RSS Feed Subscribe to RSS Feed

 

Summary: The SPACE of Developer Productivity

The SPACE of Developer Productivity is a 2021 paper by researchers at GitHub, University of Victoria, and Microsoft (including Dr Nicole Forsgren, co-author of Accelerate) that looks into ways to measure and predict productivity for both individuals and teams.

The following is a summary of the paper. The original is ~5400 words. This is ~2000.

Knowing how to measure or define developer productivity remains elusive.

This article explicates several common myths and misconceptions about developer productivity. The most important takeaway from exposing these myths is that productivity cannot be reduced to a single dimension or metric. Only by examining a constellation of metrics in tension can we understand and influence developer productivity. Our framework, called SPACE, captures the most important dimensions of developer productivity: satisfaction and well-being; performance; activity; communication and collaboration; and efficiency and flow. By recognizing and measuring productivity with more than just a single dimension, teams and organizations can better understand how people and teams work, and they can make better decisions.

 

Myths and Misconceptions about Developer Productivity

 

Myth: Productivity is all about developer activity

Higher volumes of activity may be for good reasons (better collaboration and communication; engineers having the tools they need to do their jobs), or bad reasons (long hours signaling bad systems, poor planning, unrealistic deadlines) and activity metrics alone do not reveal which of these is the case.

Even straightforward metrics such as number of pull requests, commits, or code reviews may have gaps in data or measurement errors, and miss the context of collaboration.

Myth: Productivity is only about individual performance

Measures of performance that balance the developer, team, and organization are important. Similar to team sports, success is judged both by a player’s personal performance as well as the success of their team. A developer who optimizes only for their own personal productivity may hurt the productivity of the team. Consider metrics around team-focused activities such as code reviews and on-call rotations.

 

Myth: One productivity metric can tell us everything

There is no one metric to score teams on their overall work and to compare teams across an organization and even an industry.

 

Myth: Productivity Measures are only useful for managers

Developers have found value in tracking their own productivity, both for personal reasons and for communicating with others. High productivity is highly correlated with feeling satisfied and happy with work.

 

Myth: Productivity is only about engineering systems and developer tools

While developer tools and workflows have a large impact on developer productivity, human factors such as environment and work culture have a substantial impact too. The “invisible” work that benefits the overall productivity of the team is just as important as other more commonly measured dimensions. This invisible work can include morale building, mentoring, and knowledge sharing and yet are often not measured.

 

SPACE: A Framework for Understanding Developer Productivity

Productivity is about more than the individual or the engineering systems; it cannot be measured by a single metric or activity data alone; and it isn’t something that only managers care about. The SPACE framework was developed to capture different dimensions of productivity and provide a way to think rationally about productivity.

 

Satisfaction and well-being

Satisfaction is how fulfilled developers feel with their work, team, tools, or culture;

Well-being is how healthy and happy they are, and how their work impacts it.

Measuring satisfaction and well-being can be beneficial for understanding productivity and perhaps even for predicting it. For example, productivity and satisfaction are correlated, and it is possible that satisfaction could serve as a leading indicator for productivity; a decline in satisfaction and engagement could signal upcoming burnout and reduced productivity.

Satisfaction and well-being are often best captured with surveys. To assess the satisfaction dimension, you might measure the following:

  • Satisfaction: Among employees, and whether they would recommend their team to others.
  • Developer efficacy. Whether developers have the tools and resources they need to get their work done.
  • Burnout. Exhaustion caused by excessive and prolonged workplace stress.

 

Performance

Performance is the outcome of a system or process. The performance of software developers is hard to quantify.

A developer who produces a large amount of code may not be producing high-quality code. High-quality code may not deliver customer value. Features that deliver customer value / delight customers may not always result in positive business outcomes.

Even if a particular developer’s contribution could be tied to business outcomes, what if the developer was assigned a less impactful task?

Software providing business outcomes is often the sum of many developers’ contributions, exacerbating the difficulty in evaluating the performance of any individual developer. Software is written by teams, not individuals.

For these reasons, performance is often best evaluated as outcomes instead of output. Simplified,dDid the code written by the developer reliably do what it was supposed to do? Example metrics to capture the performance dimension include:

  • Quality: Reliability, absence of bugs, ongoing service health.
  • Impact: Customer satisfaction, customer adoption and retention, feature usage, cost reduction.

 

Activity

Developer activity, if measured correctly, can provide valuable but limited insights about developer productivity, engineering systems, and team efficiency, but because of the complex and diverse activities that developers perform, their activity is not easy to measure or quantify.

In fact, it is almost impossible to comprehensively measure and quantify all the facets of developer activity across engineering systems and environments. A well-designed engineering system, however, will help in capturing activity metrics along different phases of the software development life cycle and quantify developer activity at scale. Some of the developer activities that can be measured and quantified relatively easily are:

  • Design and coding: Number of design documents, commits, pull requests, and code reviews.
  • Continuous integration and deployment. Number of builds, tests, deployments, and infrastructure utilization.
  • Operational activity: Number of incidents/issues, on-call participation, and incident mitigation.

Communication and collaboration

Communication and collaboration capture how people and teams communicate and work together.

Effective collaboration can drive down the need for some individual activities (e.g., unnecessary code reviews and rework), improve system performance (faster pull request merges may improve quality by avoiding bugs), and help sustain productivity and avoid burnout.

The following are examples of metrics that may be used as proxies to measure communication, collaboration, and coordination:

  • Discoverability of documentation and expertise.
  • How quickly work is integrated.
  • Quality of reviews of work contributed by team members.
  • Network metrics that show who is connected to whom and how.
  • Onboarding time for and experience of new members.

 

Efficiency and flow

Finally, efficiency and flow capture the ability to progress or complete work with minimal interruptions or delays.

Some research associates productivity with the ability to get complex tasks done with minimal distractions or interruptions. This conceptualization of productivity is echoed by many developers when they talk about “getting into the flow” when doing their work—or the difficulty in finding and optimizing for it, with many books and discussions addressing how this positive state can be achieved in a controlled way. For individual efficiency (flow), it’s important to set boundaries to get productive and stay productive—for example, by blocking off time for a focus period. Individual efficiency is often measured by uninterrupted focus time or the time within value-creating apps (e.g., the time a developer spends in the integrated development environment is likely to be considered “productive” time).

At the team and system level, it is important to minimize delays and handoffs. The DORA (DevOps Research and Assessment) framework introduced several metrics to monitor flow within teams—for example, deployment frequency, and lead time.

Some example metrics to capture the efficiency and flow dimension are:

  • Number of handoffs across different teams
  • Perceived ability to stay in flow and complete work.
  • Interruptions: quantity, timing, how spaced, impact on development work and flow.
  • Time measures through a system: total time, value-added time, wait time.

 

Framework in Action

The diagram below shows specific metrics that fall into each of the five dimensions, across 3 measures: individual, team/group-, and system-level.

The SPACE of Developer Productivity

 

How to Use the Framework

To measure developer productivity, capture metrics across multiple dimensions of the framework—at least three are recommended. For example, if you’re already measuring commits (an activity measure), don’t simply add the number of pull requests and coding time to your metrics dashboard, as these are both activity metrics. Instead, to really understand productivity, add at least one metric from two different dimensions: perhaps perception of productivity and pull request merge time.

Another recommendation is that at least one of the metrics include perceptual measures such as survey data. Perceptual data may provide more accurate and complete information than what can be observed from instrumenting system behavior alone.

One thing to note: Metrics signal what is important in an organization. What is measured communicates what is valued and influences the way people behave and react.

Metrics shape behavior.

 

What to Watch For

Having too many metrics may lead to confusion and lower motivation. If developers and teams are presented with an extensive list of metrics and targets, meeting them may feel like an unattainable goal. A holistic view of productivity consists of a handful of metrics across at least three dimensions.

No metric can ever be a perfect proxy and some are noisy approximations. Retention, for example, is often used to measure employee satisfaction; however, it can also reflect compensation, promotion opportunities, and issues with a team. Even if retention did reflect satisfaction, it is a lagging measure, and teams don’t see shifts until it is too late to do anything about it.

Teams and organizations should be cognizant of developer privacy and report only anonymized, aggregate results at the team or group level. Individual-level productivity analysis, however, may be insightful for developers.

Finally, any measurement paradigm should check for biases and norms.

 

SPACE and SRE: The Framework in Incident Management

SPACE and SRE: The Framework in Incident Management
The SPACE framework is relevant for Site Reliability Engineers (SREs) and their work in Incident Management (IM).

Here are some of the common myths associated with IM:

MYTH: Number of incidents resolved by an individual is all that matters.

This is a myth because IM is a team activity. A service that causes a lot of outages and takes longer to restore reflects badly on the entire service team.
Instead, focused on team activities such as knowledge sharing, preparing troubleshooting guides, mentoring, and doing proper handoffs.

MYTH: One metric in isolation will tell you everything.

Instead, understand metrics in context:
the number of incidents,
Time to Recover/Resolve
How to improve both the system and the team’s response.
There is no “one metric that matters.”

MYTH: Only management cares about incident volume and meeting SLAs.

With the rise of DevOps, developers are also doing operations now. IM (a part of operations) can take away a significant chunk of developers’ time and energy if the volume and severity of the incidents are high. As important as it is to management and executives to guarantee SLAs and reduce incident volume and resolution times, it is equally important to the individual developers who are part of the IM process.

MYTH: Effective IM is just about improving systems and tools.

While tools, guides, and workflows (e.g., monitoring, ticketing, log-analysis etc) have a large impact on developer productivity, the human factors of the environment and work culture have substantial impact too.
For example, mentoring new members of the team and morale building are important.

It is critical to identify Incident management metrics that can capture productivity:

  • Satisfaction: how satisfied the team is with the IM process (escalation, routing, on-call rotations).
  • Performance: Your monitoring systems’ ability to detect and flag issues faster, before they become an incident.
  • Activity: number of issues caught by the monitoring systems, number of incidents created, number of incidents resolved.
  • Communication and collaboration: people included in resolving the incident, how many teams those people came from, and how they communicate during an incident.
  • Efficiency and flow: number of hops an incident has to take before it is assigned to the right individual or team.

 

Wrap up

The SPACE framework provides a way to think about productivity and to carefully choose balanced metrics linked to goals. The framework is meant to help individuals, teams, and organizations identify pertinent metrics that present a holistic picture of productivity; this will lead to more thoughtful discussions about productivity and to the design of more impactful solutions.

Tags: , , ,

Leave a Reply