Write Better Software, Faster, with Agile Methods

If you work around technology – or even if you just read about it – you’ve doubtless heard the term “agile” used to refer to a set of methods for accelerating and improving software development (and, by extension, any kind of team-based project work). Agile methodology isn’t a new-new thing, although it’s often (and with complete justification) referred to as revolutionary. It’s also not a very complex, technical, or prescriptive formula for working, despite certain of its antecedents occasionally sounding a little shrill from time to time (mostly in response to the frustration of trying to work culture change on organizations stuck in legacy methods).

In fact, Agile is a pretty old thing, based on time-tested disciplines. In 2001, a bunch of coders and software development consultants got together in Snowbird, Utah, for a little ski vacation and conversation. Many of these folks had already been active for decades in their profession, pioneering methodologies like Scrum, Extreme Programming, and Pragmatic Programming – all aimed at helping coders and companies work more productively and realistically. Their conversation resulted in the so-called “Agile Manifesto,” which you should go and read right now. It’s only about a dozen lines long.

The Manifesto describes a way of shifting software developer priorities. The old way of building software emphasized strict processes, heavy up-front planning and budgeting, and widely-spaced, large-scale, complex delivery commitments – a combination characterized as “waterfall” for its inflexible linearity and sudden drops into rock-strewn pools. The Manifesto authors (who, together, had something like 300 person-years of experience as developers and team leads), understood that these methods were intrinsically broken, or at least very, very risky: liable to drown budgets, timetables, and whole projects quickly as they navigated the shifting rapids of fast-changing business conditions and emergent customer needs.

Instead, the authors proposed a style of programming favoring communication and collaboration (among developers, managers, customers); agreement on and transparency about limited goals; rapid, non-judgmental adaptation to changing requirements and conditions; and delivering working code instead of requirements documents and planning artifacts.

From these principles, other common aspects of agile methodologies crystallize naturally. Agile disciplines favor small, self-organizing, stable teams (research shows that from five to nine people work best) that include a facilitator (functioning as a “servant leader” instead of a project manager). (In Scrum, this position is called ‘Scrum Master’ – the term ‘scrum’ itself derives from rugby, and describes a method whereby the forwards of opposing teams lock together in an isometric formation, letting them compete to take control of the ball without (ideally) collapsing into chaos.)

The team works on agreed-upon goals over a series of short, human-scaled, same-length timespans (two weeks is common), creating a regular cadence. Different flavors of agile development use their own terms to describe this way of working: Extreme Programming talks about a “project heartbeat,” while Scrum terms these short delivery cycles “sprints.”

Agile approaches are value- rather than plan-driven. They assume scope is variable, whereas people and time are (over a short time-horizon, anyway) pretty much fixed. This helps avoid practices common to old-school development that have proven consistently unsuccessful, such as adding developers late in a project, when delivery dates look like they’re slipping (research shows this practice tends to delay, rather than speed up project completion).

At the same time, though agile treats labor as a fixed, short-term resource, it doesn’t automatically map people to tasks in a simplistic way. Depending on circumstances, agile practitioners may choose to go deep, rather than wide, using practices like “pair programming,” for example, where two coders share a terminal: one coding while the other observes and “navigates.” Experience shows that pair programming, performed judiciously, has little deleterious long-term effect on team productivity and timetable, but benefits projects by spreading around valuable institutional knowledge of the codebase, while also reducing software defects and encouraging more creative problem-solving.

Minimum Viable Products

In agile, the goal of each sprint is to deliver working (ideally, production-grade) code – focusing on the most important features first. This doesn’t always mean “end-user visible features,” though implicit in agile methodology is the idea that the proper output of most software development is a so-called MVP or “Minimum Viable Product,” combining tangible, business-relevant functionality and good end-user experience. The aim of the MVP is to satisfy early adopters, with subsequent functionality produced only after feedback from this group has been assessed. The approach forestalls investment in ‘nice to have’ features that add little value to the product in the eyes of its customer/ stakeholders.

Prioritizing the MVP encourages pragmatism about what’s intangible. If you need infrastructure to support a feature, don’t re-invent the wheel or over-engineer the solution: try integrating an open source tool, using a framework, or (failing all else) implementing a proven design pattern in a reusable way, scaled appropriately to the task. Recovered time can be used to work on unit tests or deployment automation, which are closely bound to the deliverable.

Always prioritizing the “next most important thing,” avoiding over-engineering, delivering the MVP along with unit tests and required automation in parallel helps limit the downside risk of software projects, preserving value in the event that projects are reorganized (or terminated), and facilitating communication about priorities in a way that tends to limit scope-creep. At the end of each sprint, code is demonstrated immediately to stakeholders, feedback collected, and the sprint internally reviewed by team members, with commitments for improvement recorded. This practice tends to expose misunderstood requirements quickly and keep things on course (working code is the best tool for discussing how code should work).

Shifting Left

Agile methods and philosophy harmonize naturally with breaking down silos between developers and operations staff and delivering value quickly through automation and use of standard tools and frameworks, all basic requirements of DevOps. Agile has also been influential in changing perspectives on software testing, software customer support and many other disciplines.

The so-called “shift left” movement, that promotes early (as opposed to late) engagement between coders and testers, is founded on agile principles of communication, transparency, group ownership and accountability, iterative methods, short delivery cycles, and working code. Left-shifted testing saves resources and money: this article from IBM estimates the cost of fixing a bug during the late maintenance stage of software development to be 100 times greater than the cost of fixing that same bug early in the design and requirements-gathering process, where other studies confirm that up to 56% of bugs originate.

How agile are your developers and projects? Agile software development methods are fundamental to ensuring timely delivery of business value, limiting risk of software project failures, and encouraging good coding, testing, and automation practices. As this 2016 McKinsey & Co. study observed, companies implementing agile methods at scale increase productivity up to 80%. No modern organization can afford not to be working this way (yet even now, McKinsey says, only around 20% of development organizations are fully committed to doing so).