Copyright, Alistair Davidson, 2013, all rights reserved.
Contact information: E-mail: alistair(at)eclicktick.com Phone: +1 (650) 450-9011
Draft 1.7 November 2013
Best practices in software development are increasingly important in a digital world where products, services and internal processes are mediated with software. Historically, software development has been largely top-down using a Waterfall approach and been generally ineffective and failure prone. More modern agile and Scrum development processes have 3-4X higher success rates and can be a source of sustainable competitive advantage.
In the same way that top down strategic planning failed in the 1980s and 1990s and was replaced by the more decentralized approach of strategic management, agile and Scrum approaches have been introduced with great success to companies seeking faster product innovation and value creation. But as with the decentralization of strategic management ideas and practices, agile and Scrum development are often resisted by those ignorant about agile methods and advantages. One reason for their resistance is the perceived threat to their individual power base and personal ideas about management and authority. We label these resisters and saboteurs of change, Scum Masters and identify some of the ways their resistance can de-motivate teams causing companies to fail to obtain the benefits of agile and Scrum development.
When running planning retreats, one of the most useful ways of reframing problems about strategy is to ask “What are we not going to do?” and “What would be the worst possible way of implementing our strategy?” These “negative” questions force people into a different mode of thinking. Often, it turns out that the current strategy is attempting too much. And the current mode of implementation is surprisingly close to the worst possible way of pursuing a strategy.
This document is an attempt to illustrate and review the advantages of agile best practices and also illustrate examples of bad practices, drawn from two firms, anonymized for obvious reasons of minimizing embarrassment, which are destroying the motivation of their knowledge workers.
But let’s step back in time first.
In the early days of strategic planning, “brilliant” MBAs using the latest strategic theories developed business strategies for others to implement. Most failed. It turned out that telling individuals what to do is much less effective that involving those who must implement strategies in the development and execution of strategies. In a sense, it was not a new insight, divisionalization and the use of strategic business units (SBUs) that mimicked the structure of a full business, were also an attempt to push down accountability and responsibility to the execution levels in the organization. It also turned out that the 40,000 foot view of strategy could not succeed without successful on the ground execution and adjustment. Such on the ground execution required the commitment and passion of the organization rather than merely writing a superficial plan.
In software, similar lessons have been learned. Software is difficult to get right. What you specify up front always seems to need to evolve as the understanding of a problem and users improves or as markets and competition change. “Brilliant” and often overconfident managers (typically engineers, MBAs and highly educated technocrats) specify requirements and toss the specification over the wall to the developers. They, in turn, deliver untested code to test departments. Users are last in the loop. Managers who are both distant from the development process, lack understanding of the technical choices and who have underinvested in customer involvement/feedback/sponsorship use their make project and specification modifications with a 40,000 foot view of the project. Their decisions are typically more affected their status and role, than by data and knowledge.
Software tends to be very complex with many interactions, different types of users, goals that differ among users, and even more confusingly, users that are transformed by the experience of software use. What may work for a first time user will often be unacceptable to a more skilled user. The act of using software changes the need. We like to think of this as the “Good Shoes Problem”. If you have never worn really comfortable shoes, you are unaware that better exists. Once you have worn good shoes, your expectations change. Every time you use software or you learn new features of software, your understanding of your needs and the trade-offs in the software will change.
So, as with traditional top-down centralized planning, top-down centralized software projects have a very low success rate (in the 15% range, measured by on-budget delivery of planned features). The reasons for failure are straightforward. For many software projects, like many strategies, it is not actually possible to specify the requirements in advance. The project is too complex. Insufficient knowledge of users is available. Advanced market research and usability testing is pretty difficult when the act of usage changes the perception of the need and change continues with cumulative experience. And in some software categories, it can be difficult to predict the usage patterns and processing load in advance without actual data about usage patterns and technology interactions.
The consequence is not hard to predict. Software is often expensive to build. Planned functionality is rarely achieved and user satisfaction is often low. The costs and implications of bad software are often difficult to analyze, but they can include user frustration, training costs, a high number of mistakes and wasted time and for commercial products, lower rates of usage, purchase, resale and profitability. From a narrow technology perspective, bad software development practices often result in problems that linger for years and eventually kill projects, products and organizations.
While the parallels between the evolution of strategy and the evolution of best software practices are not 100%, there are significant similarities. We lay them out in the following table which compares (1) the evolution of top down strategy to strategic management, and (2) software development’s evolution from a top down Waterfall approach to a more decentralized agile and Scrum approach.
Early management approach
Based upon largely theoretical models.
Separation between design and execution of strategies.
Iterative learning not handled well.
Lengthy implementation cycle before learning and improvement can occur.
Learning occurs at the execution level rather than the planning level.
Waterfall method of development.
Design based upon incomplete knowledge of requirements.
Separation of design from implementation from testing.
Lengthy development cycle before learning and improvement can occur.
Learning typically occurs across project staff with different responsibilities with few mechanisms for integration of learning.
Drivers of intellectual shift
Widespread dissemination of management knowledge due to increase in number of business schools and MBAs.
Emergence of object oriented (OO) or a modular approach to software development.
Widespread adoption of OO development.
Lowered cost of design reuse, code reuse and testing.
Emergence of purchasable class libraries that provided close to plug and play functionality.
Second generation approach
Push down of responsibility for developing and executing strategy to those who are responsible for implementing.
Cascading of accountabilities, strategy development and execution planning via facilitated workshops.
Recognition that large projects are difficult and that iterative development provides more opportunities to validate software design, user interfaces and user needs.
Third generation approach
Strategy is largely subsumed into normal business planning processes.
Strategy models become part of the vocabulary of many line managers.
Wide variation in the understanding and successful implementation of models.
Software design and development is reconceptualized as knowledge work where the challenge is process improvement rather than construction of software.
Design and testing are no longer separate from implementation.
Wide variation in adoption and effectiveness.
Driven largely by baronial resistance to redistribution of power in the organization.
Dysfunctional bureaucratic incentives.
Reduced risk taking by risk averse employees in a less friendly environment.
Control oriented bureaucrats are uncomfortable with loss of power over development projects.
Scrum team members are punished for challenging the status quo and there are few incentives for seeking improvement leading to lowered rates of learning.
Managers with high needs for power prefer to hire younger and less experienced workers, contract workers and/or H1B visa workers whose inexperience/insecurity makes it less likely that they will complain, take risks or challenge traditionally oriented leaders.
Treat employees as assets
Rewards employees for risk taking and activities that create learning and improvement.
Expertise is explicitly tracked and managed.
Focus in upon creating value for customers and customizing relationships, co-creation of value with customers
Leadership communicates its commitment to empowering knowledge workers.
Education and empowerment are supported.
Older workers’ knowledge is valued more highly.
Debureaucratized organizations move more quickly to develop innovations, improved processes, custom mass production and gain market share.
Improved and modular development processes increase the rate of improvement in value creation.
More sophisticated approaches to architecture and data analysis enable reconfigurable software, persona driven development, individualized marketing.
Advanced software developers in the 1980s and 1990s had started to switch to a different approach to software development. They adopted the idea of object oriented programming systems, amusingly referred to as OOPS (never let it be said that geeks have no sense of humor). The idea behinds OOPS was to decompose complicated software into more manageable pieces – think Lego-like blocks – that could be combined, modified and reused to make software less expensive to develop, easier to change, and more robust. Who after all can argue with the modularity of building blocks? Software architects and developers started to try to imitate hardware engineering, where components can be combined to make a product.
Without going into the details of OOPS, the important ideas were three:
1. If you developed a program (or method) for doing something, then it would be useful if you could reuse (or inherit) the method. That way, you would have one point of maintenance and you could spend less time reinventing something you had already programmed. For example, rather than having thousands of places where you refer to date handling, you could have one point of maintenance. Widespread use of such an approach would have made Y2K fixes trivial as the year 2000 approached.
2. Rather than separating programs and data, you could group the two types of information so that you had only one place to look when you editing a program. You would not have to spend time trying to figure out where all the variables and data were kept throughout a program.
3. Programs could be smarter. They could vary how they behaved depending upon what they were being applied to without have to reinvent the wheel for slightly different situations. Just like human beings who know that tasks may be similar, but you have to adjust to take into account the nature of the tasks, so could programs. A method (program) that could handle US zip codes would be smart enough to vary its processing for addresses in other countries without significant change throughout the code.
OOPS made it possible to think about digital and software product development in a different way. You could launch a product with a limited set of functionality and add to it over time. You could gain experience about users, usage and satisfaction without building a difficult-to-modify ‘kitchen sink’ product that might have lots of functionality, but which might not be useful to actual users. Testing could be brought earlier into the process as it is easier to test a discrete small piece of software than a large piece of software.
The early users of OOPS talked about iterative improvement, which later came to be called agile development. Their key philosophical assumptions were:
1. Humility: we don’t always know what users want and value.
2. Value experience: launch the minimal set of features to gain experience with actual users
3. Design your software to be modular or discardable so that the cost of change is low.
4. Be prepared to prototype (even if prototypes are computationally inefficient) and throw away prototypes because costs have shifted from development to understanding/evolving to meet customer needs. When programming is easy, and options are many, you can waste more time building useless features than it costs you to incrementally hone in on what customers value and will pay for.
The revolution in software development is, we suspect, somewhat hidden from most managers, and surprisingly from many engineers as well. The reason is that the consequences of iterative development are fundamental and human in nature. Many rigid thinking command oriented engineers and technocrats, tend to see developers as “cogs” rather than human beings whose rate of learning, knowledge, passion and motivation make a significant difference. They forget that developers learn and improve, and when in teams and working with the right culture and incentives, learn from each other. Extreme Programming, another agile development technique (where two programmers work on one screen) demonstrates the greater effectiveness of multiple brains working on a problem.
While software remains a complex field, the traditional role of developers has shifted in many software projects. This shift can be illustrated by an important insight about software projects. When you build software, you are always dealing with trade-offs. Projects can be thought of as having three attributes: cost, time and quality. Generally speaking (and this is not just a cynic talking), you can only optimize on two out of three. You can’t achieve all three at the same time.
· If you pursue a fixed budget, it will affect your ability to achieve a deadline and create high quality software.
· If are focused on a deadline, it will force you to choose between cost and quality.
· If you are focused on quality, you have to choose between cost and time.
But, there is a way out of the box. As so often happens in both strategic management specifically and performance improvement generally, it requires reframing the problem and managing the scope and phasing of the project in a different way. It requires focusing upon the rate of improvement rather the static quality of development processes.
To understand the reframing, you need to understand traditional project management, which is often referred to as a “Waterfall” approach. In traditional software projects, a design would be created. Its budget would be estimated. The specification would be handed over the development team. The development team would build the project. Testers would test their work. The software would be “beta tested” with users to catch problems not caught by testers. And in most cases, it would turn out that what had been built was not, in fact, what the customer wanted. The customer or end user had not really been involved in the design process. And even if customers/users had been involved, they really were not in a position to analyze a proposed specification.
In a sense, the problem was similar to that of the American automobile industry in the 1970s. Cars were designed and handed to over to production engineering. Cars were produced on the assembly line and quality problems were caught too late -- after the car had been built. And there was no systematic capture of the knowledge of assembly line workers, so the opportunities for incremental improvement were not being captured. The Japanese improved their automobile product process faster than American firms with extensive quality training, by appointing product owners and giving them authority and power, and tapping into the learning and insights of the entire organization down to the factory line worker.
The way out of the box is to be humble and lazy. Humble means not assuming you know what the customer wants. Lazy means being ruthless about what you develop. We like to call this approach identifying the Pareto features of a product, i.e. the 20% or so of features that were critical to a customer being able to use the product so that you could get feedback. Others have called it lean development, the minimal viable product (MVP) or minimal marketable product (MMP), but whatever the name, the strategy is to develop quickly and get feedback quickly. And you need to close the loop and continually improve not just the product or project, but the way you develop and learn.
And here is where the idea of Scrum occurs. The early proponents of Scrum believed that avoiding the problems of Waterfall development was critical to successful software development. Their four guiding principles were: (1) Individuals and interactions over processes and tools, (2) Working software over comprehensive documentation, (3) Customer collaboration over contract negotiation, and (4) Responding to change over following a plan.
Not surprisingly, these principles challenge many technocratic companies for whom the opposite “Scum Master” principles seem to hold, i.e. (1) Processes rule, (2) Documentation is non-negotiable because we are going to be audited, (3) Lawyers rule, (4) Budgeting requires planning.
Why Scrum proponents believe in agile development is important to understand.
In our view, one of the least discussed and most important motivating reasons was the unpleasantness of the software developer experience. Ed Yourdan summarized the problem in the title of one of his books, “Death March”. A history of bad practices in software development had made it normal for managers to expect that the way out of trading off between time, quality and cost of projects was to incur overtime. As a “good manager”, you demanded that programmers work seven day weeks and 14 hour days. A “bad” manager would fail to deliver on time by not demanding long hours of developers and testers. (He would be considered “bad” if his deliverables were delivered late, even if the product worked well, was of high quality, and developers remained highly motivated and unstressed.)
You still see this approach in the corporate culture of major tech companies. The positive side is the attempt to make their place of work, the center of employees’ lives with nutritional high quality free food, recreation, laundry services and sometimes (as will be in the case of Facebook’s head office in Menlo Park) with adjacent living quarters. And even when commuting on corporate transportation, employees can work on the WiFi enabled bus.
There are many virtues to offering services that free up developers’ time from humdrum tasks, but like any management activity, there is a fine balance. The model implicitly assumes that developers are young and that their social life can be shifted to work. It’s not an unreasonable approach, but at some companies it does become increasingly less supportive as employees age, marry and have children. As employees age, the types of services required to minimize the “wasted time” in their lives and ensure their ability to focus are likely to evolve. Day-care and medical insurance for parents who have come over from another country are examples of evolving needs.
The unacknowledged negative or flaw in this management model is that human beings cannot maintain a Death March pace and produce high quality code. It’s not sustainable. More perceptive observers who understand the flow of work and creativity talk about ‘being in the flow’, ‘achieving peak performance’ or creating ‘high performing teams’. We have all seen the phenomena: tired people make mistakes; brilliant people who are in the flow are astoundingly productive. And many brilliant insights have come not just from working with teams, but downtime when the subconscious is unleashed to solve a problem. We have all had insights after working on a problem and doing something completely different.
Working 80 hour work weeks is not typically conducive to high performance. Even if it is possible for a short period of time because you are passionate about a personal project, it is not sustainable over the medium term let alone over a career. The result is that only naïve young programmers are likely to put up with “Death March” projects, and only for a short period of time. So, there are four likely consequences:
First, the mental and physical exhaustion of Death March projects will result in bad code and more elapsed time than simple minded managers expect.
Second, programmers will burn out. Now, this may not be a problem if you believe programmers are like toilet paper and are disposable. But if you believe that a developer is a “knowledge worker” who becomes more valuable as he/she becomes more experienced, you are wasting a long term asset by burning them up. It’s the knowledge worker equivalent of failing to change the oil in your car.
Third, a revolving door of young, initially enthusiastic developers poses the risk that lessons learned will be lost. Experienced technologists have joking observed for years that each new generation of developers seems to reject lessons learned in prior generations and reinvents problems that had been previously solved in prior technologies.
Fourth, companies with unattractive work conditions and expectations will never achieve the status of “employer of choice”. They will, as a result, have more difficulty attracting and, just as importantly retaining top talent. The consequences are potentially large: more expensive hiring and training costs, higher on-boarding costs, more time spent descending learning curves, lower quality code and higher code maintenance costs.
Not managing, developing and retaining your knowledge workers is very expensive both from a cost management perspective and also from a marketplace impact. Beyond the obvious issues of hiring costs (perhaps 25% of first year salary), or training costs (at least 25% of learning curve and training costs relative to salary), the longer term issue is that good experienced programmers may be 10X more productive than an inexperienced programmer still finding his way around a project and technology. And the more that software projects require knowledge about users, support issues, integration with other areas of the business, a wide variety of historical and topic specific libraries, the more valuable experience is likely to be. (It’s no coincidence that bad software companies are always seeking developers with extensive experience in a new technology and often advertise for developers with more years of experience than the technology has existed for.) Good companies consider ‘retooling’ their successful developers with training.
In the marketplace, an experienced and insightful knowledge worker who avoids a problem that might have led to litigation, eliminates annoying design flaws or who creates a feature that delights one or more prospective buyers, leading to more sales is even more valuable than a less knowledgeable entry level developer.
So, traditional management approaches have the following flaws:
1. Poor upfront design.
2. Poor success rate.
3. Over-delivery of useless features, implementation of irrelevant strategies in the case of business strategies and irrelevant functionality in the case of software development
4. Burn-out of developers and managers
5. Higher cost structures
6. Organizational failures to seize opportunities by moving quickly and being effective in figuring out how to execute and what to deliver
Agile developers and Scrum proponents argued first informally and then more formally that:
1. Delivering useful code was more important than delivering an ill-spec’d requirement.
2. Working code was more important than documentation, particularly if you were using a modular (i.e. almost self documenting) modular object oriented approach
3. That managing knowledge workers was different than traditional workers. People matter.
4. Motivation, learning and improvement made a difference.
These revolutionaries did not argue that specification and documentation were unimportant, but rather that the priorities and emphasis had to change from dominating the process to being a consequence of good processes.
Demotivating Your Team
It’s taken us a long hike to get here and understand the issues, but anyone who has worked in a large organization can almost immediately see that iterative/agile and Scrum approaches to software development are demonstrably as revolutionary as the move from strategic planning to more decentralized strategic management processes in the 1980s and 1990s.
Software is pervasive today in digitally enabled enterprises. Internal processes, inter-organization processes, customer marketing, sales and support are all mediated with software. And success requires continual redevelopment of processes, capabilities and content. So, how well and how fast you respond is actually rather critical to the success of organizations. Do it well and you can outpace the competition. Do it slowly and you are the horse drawn carriage competing with airplane.
· The operational test for whether you are innovating quickly is: Can you do the right thing quickly and improve it quickly?
· The human test is: Do you have high performing teams who are enthusiastic about what they are doing and can maintain the pace of their activities year in, year out, while improving?
· The bureaucratic issue is: Can your organization step back sufficiently to empower the developers to increase their productivity and effectiveness?
Surprisingly given the poor success rate of top down software development and the obvious high success rate of Scrum, the answer to the last question is often no. Many existing organizations have difficulty making the transition. Scrum experts and researchers believe that about half of Scrum projects succeed and the remaining half fail to deliver the complete expected outcomes. But it’s important to note that the agile success rate is still almost four times better than Waterfall project outcomes. So, let’s take some examples of why and how this newer and more effective approach of agile and Scrum development can fail.
What we can say is that the issues laid out in this section described are based upon two different companies, both of which are well known and admired by many. They both have a high degree of success in their respective industries, however, employees tend to rate them poorly as places to work. For our own amusement, we characterize the managers external to the Scrum or agile development process, who misunderstand the use of agile development, Scum Masters.
To illustrate the magnitude of their failure, let’s start off with the benefits of scrum. Scrum coaches and researchers suggest that a smoothly functioning Scrum operation can operate at a level roughly 8X more effective than traditional project and product management approaches. The reasons for the higher effectiveness include:
1. Unneeded or low value work is not done, unlike in Waterfall approaches.
2. A successful Scrum product owner grooms or prioritizes his requirements (often referred to as customer stories or use cases) so that for each iteration of development (referred to as a Sprint), the highest priority features are developed first. Such an ongoing and dynamic prioritization is not characteristic of traditional top-down Waterfall approaches. The mere act of reprioritizing potential product features to focus on higher value development is an obvious source of improvement.
3. As in strategic management, the team members of the Scrum are not told what to do individually, rather they work as a team to figure out who will do what. They see themselves as a team, helping and cooperating with each other. They are held accountable as a team for delivering what they agreed could be done in the time period of the sprint (typically a 2-4 week period of delivery). This shared accountability (confusingly often endorsed by managers who love American team sports, and often simultaneously decried by the same individuals with a simplistic view of individualism) parallels in many ways the culture of Japanese automobile companies that have set the standards for quality improvement for three decades.
4. Software is both designed and tested by the same developers. In fact, testing requirements often precede development (an approach known as test driven development or TDD), which means that “done” means done. This completion strategy is in marked contrast to traditional Waterfall, where pressure to deliver or poor metrics strategies (e.g. measuring lines of code) incent teams to write bad code and lots of it because they are not responsible for testing.
5. The development process in each development Sprint focuses upon delivering working code and not ending the Sprint with incomplete pieces of functionality. As in manufacturing, minimizing work in progress has dramatic effects upon attention, focus and costs.
6. Iterative presentations of functionality allow testing of intermediate versions of the software and its interface, or in the case of continually updated software, actual usage patterns can influence the incremental development. Salesforce.com, for example, moved from a 14 month upgrade cycle to a 4 month upgrade cycle when it shifted to Scrum. Operating systems are an excellent example of continual development and upgrading.
7. Customers are involved earlier in the design and prioritization process. Product owners that focus on prioritizing the development of functionality that creates increased value for customers, not the more theoretical metrics such as the number of features built, or the number of use-cases/customer stories built.
8. A key part of each Scrum is a review, called the Scrum Retrospective, that encourages team discussion on lessons learned and areas for improvement. Given that a typical development Sprint is 2-4 weeks, this frequent review is key to creating rapid improvement in development and team processes. As in manufacturing, measuring the quality of small lots or small development phases (Sprints) in software, provides more opportunity for learning and improvement.
In our anonymised example Scum Master prototypical company (note the missing R in Scum), all the benefits of a more agile and Scrum process are systematically undermined.
1. External managers, i.e. not the Product Owner, not the Scrum Master, not the Scrum team, intervene and reprioritize functionality (use cases/user stories). They ignore the research that suggests a strong, empowered, knowledgeable Product Owner drives better products than a committee or warring factions within a company. They choose to ignore the virtue of focus that an empowered team possesses.
2. External managers, with a poor understanding of Scrum teams and a strong need to look good in front of their superiors, measure success by using arbitrary, inappropriate and easily measured metrics, but miss the point of only developing functionality that is high priority and creates value for customers.
3. External managers intervene into team processes and tell team members’ that their research/judgment about how long development should take is wrong and that, although external and removed from the details of the process, they have more insight. These interventions are common even if the external managers have not done their homework. The external managers attempt to cram deadlines or add back features that are lower priority resulting in a failure to deliver initially planned features. These bad external managers confuse power with knowledge. They wish to be seen as forceful and decisive rather than creating superb teams and enabling great outcomes.
4. Scrum is characterized by frequent status checks. The most important is a daily 15 minute standup Scrum meeting in which Scrum Product Owner, Scrum Master and the Scrum team answer three questions: What did I do yesterday? What will I do today? What is preventing me from doing my tasks? Resistant external managers whom we call Scum Masters either devalue the importance of the stand-up meeting, cancel it, or interject their comments inappropriately disrespecting the autonomy, dynamic work approach and knowledge of the team.
5. External managers focus on the optics of delivery (software that sort of works, but is not robust) rather than the reality of delivering tested and functioning code.
6. External managers believe (almost always erroneously) that they understand what customers/users want, so they skip user interaction and user interface testing, and under-invest in obtaining feedback from customers/users.
7. The failure to involve customers early in the process results in wasted effort building low value added features or features that will have to be modified when usage reveals the incompleteness of understanding, or opportunity for additional high value incremental additions.
There are additional consequences to failing to understand how to manage agile and Scrum development. Many of them are motivational.
If you measure the performance of individuals (as opposed to a team) and some external party tells each of them what to do, there is little incentive to help other team members. You can make the problem worse by having a ranking system, where people are graded on a curve. In such a situation, you might have a superbly performing, creative and knowledgeable team where the manager must down rate a certain number of team members in order to comply with the HR policy.
If developers know that they run the risk of being downgraded when on a team with top performers, they will not want to work on such a team. Learning opportunities will be lost.
Smart people recognize how they are being treated. If they are being told what to do, they have less emotional commitment to the task. We all know that we remember our own ideas and projects better than those others have imposed upon us. And we have before us the example of strategic planning. Strategic plans don’t work when they are imposed. Commitment and successful accountability requires empowerment.
In complex tasks, a boss cannot know enough to be specific in his direction. The characteristic of knowledge work is having a team where each individual team member may be the expert in his/her area. Bosses may be highly knowledgeable (unless they have been promoted to their Peter Principle Level of Incompetence), but they cannot know everything their team members know. A boss that defines his authority in terms of telling people what to do has misunderstood the shaping and motivational role of a both a leader and a manager in a knowledge-based project.
Measure individual performance not team performance.
Lowered team work.
Reluctance of team members to work with the high rated and most knowledgeable employees.
Lower or no transfer of skills and knowledge from the best performers.
Measure easily measured metrics rather than value created or rate of improvement.
External managers delude themselves into thinking they can measure a complex, difficult to specify portfolio of projects
Emasculate the product owner.
Slow down product development and ensure that superb products are more difficult and expensive to build.
Don’t involve customers early and throughout the project.
Build unnecessary features, difficult to use interfaces and fail to delight customers.
Question the competence of the team by imposing features and deadlines without acknowledging the analytical work of the team members.
Demotivation of team. Failure to deliver key functionality (user stories/use cases).
Make it difficult and personally expensive to help team members.
Less synergy between team members. Less learning.
Emasculate the Scrum Master by overruling his/her decisions or failing to following the Scrum process, e.g. not making visible external claims on team members so that the team task visibility becomes inaccurate.
The role of the Scrum Master to coach the Scrum team and remove barriers to their success.
Preventing the Scrum Master from empowering the team, demoralizes both the Scrum Master and the Scrum Team, reducing team performance.
Fail to support the simple ideas of Scrum, e.g. a daily review, high visibility of progress, sharing of performance information, test drive development, investing time in figuring out lessons learned, devaluing the need to take risks and seek improvement in processes
The carefully designed team environment sputters.
High turnover of staff.
Lower levels of motivation.
Reputation for being a sweat shop or place where “Death March Projects” are common.
Burn out of teams.
Failure to retain successful high performing teams.
Deteriorating abilities to develop new products and processes.
An emerging opportunity for competitors is created by dysfunctional development processes.
Eventual dissatisfaction for internal stakeholders with the Scum Master whose job will increasingly be at risk as performance suffers both in terms of costs and competitive position.
Reduce the power of team members by choosing to hire H1B visa holders and contractors
Many large firms find the idea of hiring contractors and foreign workers attractive because such individuals are highly dependent upon either the good graces of a boss, or the employer specific H1B visa. These work relationships change the risk/reward profile for Scrum team members and make it very unlikely that honest feedback and suggestions from these more insecure employees will obtained. Costs may be lowered for the firm, but it is short sighted if the rate of improvement of development process suffers as a result.
Most writers on Scrum are immensely articulate about best practices in Scrum. Agile and Scrum software development processes are perhaps better and more clearly explained in the many books on the subject than most areas of management practice.
But what is less well laid out are the required processes for building consensus in the organization about the need to change the way and speed with which software is managed outside the narrow area of software development and product ownership. Part of the problem is that software, by its very nature, tends to be a highly specialized function in organizations. As a result, if a non-IT or non-software manager is faced with software development issues, the typical response is avoidance. The repertoire of conventional wisdom actions typically includes:
1. Buy a package that will save us from having to do development.
2. Rent a service that will save us from having to know something.
3. Outsource development so we won’t get stuck with the costs and overhead of owning software capabilities.
4. Spend insufficient time involved in the evolution of a good solution by overinvesting in up-front specifications and follow this overinvestment with underinvestment in the process of development/project/product evolution.
In many circumstances, the first three responses can be appropriate. But given that gaining competitive advantage is always about doing something different than the competition -- about creating value in ways that are difficult for competitors to match by e.g. buying a similar package or service, software development will be required, even it is just for web sites. In such a situation, a leader must figure out what the best change management approach is likely to be.
We suggest that there are at least nine important steps:
First, identify “pain points”. Where are there frustrations in the organization about the inability to introduce product features, improve processes, or speed up improvement in marketing and sales activities? Stakeholders with these problems are likely to be supportive of change that will overcome the barriers they are facing. In many cases, these “pain points” will be the result of budget and personnel constraints within the IT area.
Second, quantify the benefits of improved software development from a business perspective for individual stakeholders. Stakeholders should be able to make ball park estimates of what faster and better software development is worth. If they cannot, a facilitator or business analyst should be able to help them with back of the envelope calculations as a very minimum. Educating product owners or those with budget responsibility (economic decision makers) about the advantages of agile development may help them with specific marketing problems or budgetary/financial goals they are facing.
Third, quantify them marketing costs of having lagging software development capabilities in terms of higher costs, less ability to produce innovations, slower times to market, lower close rates than competitors, lower Net-Promoter Scores, lower customer satisfaction and in some cases the bonus consequences to stakeholders. In some cases, modeling techniques such as conjoint analysis that allow simulation of the value to a customer of their ‘ideal product’ can be derived from having customers make ratings of different attributes of a proposed product.
Fourth, before deciding whether to select a demonstration project or go full speed ahead throughout the organization into agile development, assess the strength of sponsorship for organizational change. If the sponsorship is at a senior level, as happened at Salesforce.com, then large scale transformation is possible. If sponsorship is occurring at a lower level in the organization, then stick with a smaller project.
Fifth, building software products and projects occurs within a set of assumptions about innovation, risk acceptability, short, medium and long term architecture requirements. Surfacing the hidden assumptions about how innovation and architecture should occur allows the organization to discuss, agree upon and support changes to the innovation environment. For example, some managers will have a bias towards building a perfect platform at the cost of rapid delivery of good enough products that get to market quickly. More short term oriented managers and teams may be so focused upon fast to market that they approve, build and deliver products/projects/services with medium and long term technical debt. Establishing acceptable levels of risk, architectural robustness, patterns of evolution acceptable to the organization can surface issues that can trip well intentioned employees with different understandings of trade-offs.
Sixth, make sure that training is part of the change program. Agile development changes more than software development. As a minimum, it alters the nature of product ownership and development. But more generally, it changes power relationships in “command” organizations by empowering development teams and product owners. Product managers/marketers are critical to successful adoption in particular, because they may initially be uncomfortable with the apparent uncertainty about what features will be delivered, at least until you remind them that Waterfall approaches in the past have been even less reliable and far less productive, providing a lower value created per dollar of development budget.
Seventh, the Scrum Product Owner, who is analogous to the traditional Product Manager role, needs to be involved in the evolving prioritization of product functionality (use-case/user stories). The most common problem is abdication by Product Owners, which has the predictable result of slowing down projects and ensuring wasted effort.
Eighth, ensure that customers and users are involved early and frequently in the development process. The act of developing software makes it almost impossible to get back inside a customer’s head because you know too much about how a software program works. And not knowing how the customer sees usage of the software guarantees you will build some features to excess and miss others. Early and frequent testing of user interfaces (e.g. with storyboarding before the product has been built) and user interactions ensures faster adoption and increased usage, all outcomes that motivate the organization to stick with agile development approaches.
Ninth, make sure that the incentives for Scrum teams and stakeholders will support the agile and Scrum processes. If organizational rewards and punishments are supporting Scum Master behavior, then a more agile process is difficult, if not impossible, to make successful.
If a company is fortunate, the spread of agile and Scrum can be viral as best practices spread within an organization, though without the support of external stakeholders it will be less successful than it could be. Research on autonomous change teams, a business situation similar to Scrum suggests that successful autonomous teams have three basic ingredients for success:
1. Organization wide support for the autonomous (i.e. Scrum) team;
2. Unwavering support for the resources needed by the autonomous team; and,
3. Frequent communication between the team and leaders in the organization.
There is perhaps another way of that improved software processes may emerge and that is a form of ‘industrial action’. It may be subtle in the form of development teams being discontented and demanding change, or it may be triggered rather late by high rates of employee de-motivation and dissatisfaction along with rapid employee turnover. Perhaps un-intuitively, companies should hope for revolution by developers and product owners, because the alternative is the leakage of talent, leaving a company with the untrained, inexperienced and the mediocre. Ironically, the more that a company is in need of revolution, the more likely it is that the controlling “command” management style will be perceived as punishing innovation and process change with the resultant exit of those who could improve performance.
By way of industrial action, it’s possible to imagine that firms, which have carelessly abused contractors, H1B visa holders or employees by e.g. demanding excessive overtime without fair recompense, will have to respond legal suits, as has happen at well known firms such as EA and Microsoft. Settlements can be expensive with overtime suits currently averaging $2.7M in 2013 surveys. Most high tech companies and certainly, most IT departments are not unionized. But there may be an incentive for overworked employees to pursue collective action, something we might term developer rebellion. A visionary product owner and/or a group of successful developers may choose to demand that ineffective approaches be changed and that management practices be altered. This bottom up approach is exceedingly rare today, but as understanding of Scrum and experience with developing with agile and Scrum approaches becomes more widespread, new expectations about best practices, a lowered tolerance of poor management, autocratic leadership, Waterfall approaches, and Death March projects may trigger change from the bottom.
If you are a leader of an organization, “getting in front of the parade”, and anticipating or preempting this rebellion may provide an opportunity to gain support for agile and Scrum approaches. The retrospective meetings after Scrum Sprints, so essentially to improving developer productivity may be surprisingly powerful and reveal the dysfunctions of control oriented managers and processes – insights that a leader or perceptive manager can use to support the need for change.
An earlier commentator on the previous three sections drew the conclusion that we were condemning strategy in the same way that Waterfall is often criticized. But the reality is otherwise. The point here is that strategy is rarely the result of one insight from a senior manager, rather strategy, incremental improvement, operational improvement and innovation typically comes from the aggregate knowledge, learnings, skills and experiments throughout a firm. The process of managing these collective insights all the way from observation to analysis to experiments to commitment to improvement represents the strategic management process in its broadest sense.
Strategy is enabled by faster learning. Innovation, done faster and well, creates competitive advantage. And the selection of technologies, technology goals and technology resource allocation should be driven by the strategic vision, mission, values, business line and product strategies. When you can automate anything, you need to know what your stand for and what your goals are. But the corollary is also true: technology enables new possibilities and new economics. Faster and more effective deployment of technology may open opportunities that reveal new potential strategies, confirm strategic hunches or expand upon initial small experiments.
In a phrase, there is a different between stubbornness and persistence in both strategic management and technology development. Stubbornness is about believing that you are right without having strong reasons that support your belief. Persistence is about selecting an objective and adapting as new information is acquired and evaluated.
Alistair Davidson is both a product strategist and a software developer. He is the author of five books on business and product strategy, strategic information management best practices, software development best practices and new models of innovation in a world where hypercompetition is the rule (Innovation Zeitgeist, Eclicktick Consulting Kindle eBook, 2013). He has an MBA from Harvard, has taught both programming and strategy and has over a decade of agile development experience (building 15+ products), beginning with early use of the language Smalltalk. He is a Certified Scrum Master (from the Scrum Alliance) and holds two additional scrum certifications including a Scrum Product Owner certification. He is a skilled facilitator who has also trained other facilitators and has a wide variety of international consulting and coaching experience. He is also a contributing editor at Strategy & Leadership magazine.
The Agile Manifesto: http://agilemanifesto.org/
Beck, Kent and Andres, Cynthia: Extreme Programming Explained: Embrace Change, 2nd Edition, Addison-Wesley, 2004
Davidson, Alistair: Innovation Zeitgeist: Digital Business Transformation in a World of Too Many Competitors, Eclicktick Consulting Kindle ebook, 2013
Hess, James: Empowering Autonomous Teams, Ivey Business Journal, Nov./Dec. 2013
Peters, Lawrence: The Peter Principle, Buccaneer Books, 1996, first published 1969.
Rubin, Kenneth: Essential Scrum: A Practical Guide to the Most Popular Agile Process, Addison-Wesley, 2012
The Scrum Manifesto: http://www.scrumalliance.org/code-of-ethics
Yourdan, Ed: Death March, 2nd Edition, Prentice Hall, 2003