Tuesday, November 17, 2015

Thoughts on Delivering Digital Transformation with Agile

My colleague the digital expert and programming guru Peter Rogers share his insights on the role of Agile in digital transformation, and the major debates around methodology.  WARNING!  This article dives deep into programming strategies, so go NO further if this frightens you.  Enjoy!
Peter Rogers
Most digital projects are going to end up being run as Agile. With that said it is well worth knowing which style of Agile you are actually using to ensure that: it is indeed a recognised style; and that you play to its strengths. Often projects fall between different styles or offer a random approximation of what the team thinks an Agile project should look like. In either case you may find the project goes inexplicably sideways.

XP (or Extreme Programming) is a suite of practises, principles and values invented by Kent Beck in the late 90s. A lot of people remember XP for the controversies such as pair-programming fights to the death over curly brace placement and confusion over 40 hours a week being a Sustainable Pace.

What people forget about XP is that is actually came up with most of the practises that we use today in Agile:

-Planning Poker
-Collaborative multi-discipline teams
-Acceptance Tests
-Continuous Integration
-Test Driven Development
-Simplest Design

XP is one of many Agile methods but it still remains the most well defined, broadest and well disciplined. Am I suggesting you use XP? I think you probably do use a few of its practises already. It is certainly interesting that what you probably considered Agile is actually an XP practise that you have bolted onto your chosen Agile method.

In 2000 a large developer summit took place in Utah and the result was a manifesto for Agile Software Development.

This created 4 values:

-Individuals and interactions over processes and tools
-Working software over comprehensive documentation
-Customer collaboration over contract negotiation
-Responding to change over following a plan

The set of values was accompanied by a set of 12 principles which you can find at www.objectmentor.com who offer great debate on this topic.

Now those of us who live in the real-world are already laughing at the third point, "Customer collaboration over contract negotiation".

Sadly in a world of fixed price, fixed deliverable, fixed time,  fixed capacity, heavily negotiated digital contracts then it makes it hard to see how Agile even has room to breath. One solution in this situation is to use longer term Agile Planning techniques (Agile Planning Onion) as opposed to software estimation techniques (like UCP). You plan how to deliver the high level work packages inside of the shell of an agreed set of sprint / release based payment milestones. When most of the variables are already fixed then you look towards planning as opposed to estimation, as a beacon for delivery confidence.

At this junction as a technologist, solution architect and offshore/nearshore specialist, I must take issue with two of the Agile principles:

(1) The most efficient and effective method of conveying information to and within a development team is face-to-face conversation

The world has moved on since the year 2000 and there is no reason to believe Agile projects have to be run in all in one room. We live in a time of offshoring and nearshoring that just wasn't considered back then.  That said I do believe (from personal experience) that teaching is far more effective face to face. So maybe I partly agree but I object to the implications of the wording.

(2) The best architectures, requirements and designs emerge from self-organizing teams

Unfortunately I do not subscribe to the theory that we do not need subject matter experts and specialists, and anyone should be able to do any role and a team can self-repair without any external guidance. This is not what I have seen in the real-world.

I equally disagree that teams can autonomously find and complete training as and when required, in order to self-heal and repair themselves during a project without any external influence.

Instead I believe in mentors and subject matter experts who train the team in order to improve them as a collective. If these people are in the team, then that's great but there is nothing wrong with external mentors. This is not a communist state and everyone does not have to be equal in anything other than respect. There have to be knowledge leaders to help spread the knowledge and in most cases there is an external support framework.

Seal Team 6 has repeatedly proven that a small specialist team can do what a thousand generalists could not even hope to achieve. I certainly believe in empowering developers and motivating a team, but everybody has roles. If you give a job to somebody who isn't good at that job, then it will take much longer. I certainly do not believe in emerging architectures and emergent problem solving either.

Indeed I feel solution architects will probably agree with me on these two points, but feel even stronger about some of the concepts in Lean.

Lean originates from Lean Manufacturing where you eliminate waste and The Toyota Way where you improve flow or smoothness of work.

This is achieved through the following principles:

-Pull Processing
-Perfect First Time Quality
-Waste Minimization
-Continuous Improvement
-Long Term Relationships
-Production Flow
-Visual Control

Mary and Tom Poppendieck adapted the principles from Lean Manufacturing to create Lean Software Development with the following premises:

-Eliminate Waste
-Build Quality In
-Create Knowledge
-Defer Commitment
-Deliver Fast
-Respect People
-Optimise The Whole

Lean tells us to eliminate anything not adding value at this moment in time. That can mean eliminating features, documents, meetings and even future known requirements. If it doesn't add immediate value then its out.

One thing Lean gets spot on is the concept of optimising the whole. Too many companies reward partial goals like lines of code or defects found. These goals are often mutually exclusive as many managed service organisations found out a few years back. Focus your goals and reward system around optimising the whole system and not part of it.

The first thing I disagree with Lean on is that it expects teams to miraculously self-heal without any external intervention. I personally believe there is a difference between healthy respect and failing to intervene when a team is clearly struggling and needing help. Some would disagree but I don't agree with teams operating in a bubble as they are always surprised by the effects of what lives outside.

A second thing I adamantly disagree with Lean on as a solution architect is the notion of deferring decisions to the last minute and waiting for an emergent solution to appear. Never in my life has such an emergent solution appeared. Those who fail to plan, plan to fail. I doubt I am the only solution architect to take issue with this point if only for fear of job security.

A third and final point is if you can show me a Lean project that can work with reusable components on separate delivery timescales then I will literally buy and eat a hat.

Lean appears to promote a lack of future planning, risk management and dependency management. Its "live in the moment" style is popularised by Lean Start Up which promotes Continual Deployment and Fail Fast. Originally developed in 2008 by Ries with high tech companies in mind it does remain a good technique for Start Ups to test a product on a target audience without burning all their VC money in month one.

Ries offers these lean startup principles:

-A Minimum Viable Product
-Continuous Deployment
-A/B Testing
-Actionable Metrics
-Pivots (course correction)
-Innovation Accounting
-Build, Measure and Learn loop

Later business model templates arose:

-Business Model Canvas
-Lean Canvas
-Crowdfunding Canvas

I like Lean Startup although I take issue with some of the Lean thinking, because it offers some solid principles straight out of the Extreme Programming domain. That said outside of Start Ups you will rarely find companies doing Continual Deployment (as opposed to Continual Delivery), willing to float an unpolished MVP or able to accept the financial risk of a failed product.

So where does that leave us? I believe we need to pick an Agile methodology, and there are three main contenders:


We all know and love Scrum. The Sprint Backlog details the User Stories (or Features in BDD) to do and the Product Owner provides a prioritisation based on the business needs. Each Sprint is fixed in duration (normally 2 weeks) with its content decided just before using Story Points or Ideal Days in a game of Agile Planning Poker.

The benefit of the fixed Sprint sizes is that you can work out the Velocity of the team based on the completed Story Points in order to fit just enough User Stories into the next Sprint with a few Stretch Targets. Burndown charts visualise the Story Point progress and a Scrum Master is in charge of keeping the team motivated and clear of distractions. We show the Product Owner a fully working product at least every Sprint and so we get regular feedback and validation.

I must say that I love Scrum but I do agree with some of points raised by the Kanban Camp. Fixed Sprint sizes can be too long to get validation, and during those 2 weeks the business may have changed direction. If you are working with a highly dynamic business, then you may need to either adopt 1 weeks Sprints or consider Kanban.

The second problem with Scrum is that there are no measure of how long it takes for a feature to go live. If you are doing Behaviour Driven Development (BDD) and/or Continual Delivery then Feature Cycle Time is a particularly useful measure of efficiency.

The third and final problem with Scrum is that often the Product Owner role is part-time and does not have a strong connection with the business, and the Scrum Master role is part-time or doesn't exist for cost reasons.

Kanban is the second popular approach, although I must admit here and now that I have often seen this border precariously on chaos. Kanban has no time boxed Sprints but instead it limits how many features a team can work on at any given time. There is no Product Owner or Scrum Master and the team just decide priorities collectively.

When a feature is completed then it can be made immediately available to release into production (or a suitable safer environment) and the Cycle Time can be recorded as a measure of team efficiency. The team can then collectively choose to work on whatever the next highest priority item is in the Backlog.

The cool part is that we can use simple tools like Trello which offer a customisable Kanban board as opposed to heavy weight tools like Jira. Traditionally we would have something like 'To Do', 'In Progress' and 'Done' columns. We normally break 'In Progress' into some workflow states but you get the general picture of simplicity offered here.

We now limit the number of items in each column of the Kanban board. This is critically to avoid multi-tasking and context switching which is highly inefficient. The Kanban board gives us a visualisation of the team's workflow and it keeps them prioritised without a Scrum Master role.

Seeing as we can release a feature whenever we want then we tie into Continual Delivery much more closely. We also get much faster feedback from the customer which allows us to roll back badly thought out features and feature correct rapidly without destroying the product.

There are also some very nice Kanban visualisation techniques for Feature Cycle Time and Accumulative Status.

My first issue is how the team democratically chooses the next item to work on. The second issue is that without Sprints then there is no guarantee of what will actually be delivered at any stage in the progress.  The third issue is that there is a even stronger notion of unmanaged teams without the Product Owner or Scrum Master to offer a helping hand if things go south.

The solution is my view is the third option which is Scrumban. This takes Scrum with the fixed Sprint sizes, Product Owner and Scrum Master but adds in a few concepts from Kanban.

-Kanban Feature Cycle Time
-Kanban Feature Limits

If I were to personalise this further into "Peter's Real World Scrumban" then I would add the following:

-Remove the Scrum Master role and use Feature Limits to stop context switching if cost is an issue
-Make sure you have a 100% Product Owner role
-Ignore Lean advice about not planning for the future
-Use internal or external mentors for team learning don't expect this to happen magically overnight
-Use both Sprint Burndown Charts and Kanban Cycle Time Graphs
-Add in some of the best Extreme Programming practises like TDD and Continual Delivery
-Avoid Continual Deployment unless you are in a Start Up that can afford to Fail Fast
-Embrace some of the BDD features like Gherkin, Three Amigos and Automated Acceptance Tests
-Add in some of Lean Start Up's best practises like MVP, A/B testing and Pivots
-If a process isn't working for you then fix the process before you try to fix the people
-It isn't an Agile project just because somebody decides to call it that. Embrace the Agile values rather than a few bullet points

Special thanks to the amazing Abby Fichtner for her excellent website at hackerchick.com that gave me a lot of inspiration for this debate.

Good luck and I hope this helps you make the right choice in your Agile adventures.

Kevin Benedict
Writer, Speaker, Analyst and World Traveler
View my profile on LinkedIn
Follow me on Twitter @krbenedict
Subscribe to Kevin'sYouTube Channel
Join the Linkedin Group Strategic Enterprise Mobility
Join the Google+ Community Mobile Enterprise Strategies

***Full Disclosure: These are my personal opinions. No company is silly enough to claim them. I am a mobility and digital transformation analyst, consultant and writer. I work with and have worked with many of the companies mentioned in my articles.