Tame Problems, Wicked Problems, Waterfall and Agile

This post explains Tame Problems, Wicked Problems and how these two are different. It also talks about why (the steps used in) the waterfall model of software development is in-effective in solving Wicked Problems and suggests a generic approach that could be used to solve wicked problems.

0-0-0-0

The problems that scientist and engineers are usually focused on are known as Tame Problems. An example can be solving an equation in mathematics or analyzing the structure of an unknown compound in chemistry or solving a chess puzzle. For each, the objective is very clear and it is also clear whether the problem is solved or not.

The problems in natural sciences are definable and may have a solution that can be found with sufficient effort (based on complexity of the problem).The kind of problems in the public domain , for e.g. public transportation, education, public health and policy decisions are inherently different from the problems that scientists deal with. For the problems in the public domain, what is to be solved is not clearly defined and whether a solution is reached or not is difficult to ascertain. Problems with such characteristics are called Wicked Problems.

Following table contrasts Wicked Problems and Tame Problems.

Wicked Problems and Tame Problems
Characteristics	Tame Problem	Wicked Problem
Problem Formulation	This can be exhaustively formulated so that it can be written on a piece of paper and shared it with a person who will then solve it. You can ask a knowledgeable person to solve a quadratic equation and not expect any further queries	Exhaustive formulation of a problem is not possible since the solution chosen defines the problem. For e.g, you cannot tell a person to develop a software solution to a retail banking system without expecting any further queries. The interactions are important for proper formulation of the problem.
Stopping Rule	For a tame problem, for e.g. chess problem or a Maths problem, once the combination of moves or steps are found, the problem is solved and that is the end	For a wicked problem, there is no such stopping rule. For e.g. when developing a software product, there is no way for us to know when the problem is solved
Testing the Solution	Given a solution to a tame problem, it can be tested and we can conclude the solution as correct or wrong. It is easy to check if a quadratic equation is solved correctly or not	For wicked problems, correct or wrong is not applicable.
List of permissible operators	There is an exhaustive list of permissible operations for a tame problem. for e.g chess rules or steps involved in solving a quadratic equation	For a wicked problem, there is no way a enumerable list of permissible operations can be found, as these depend on the person solving the problem
Problem as a discrepancy – aka the difference between the current state and desired state	With tame problems, there is a single explanation for discrepancy	With wicked problems, there are many explanations for the same discrepancy and we don’t know which one is the best. For e.g., the need for a new software solution can be attributed to changing customer needs or market conditions or an existing software that can no longer be enhanced
Scope of a problem	It is very easy to define the scope of a tame problem	With wicked problems, it is very difficult to establish the scope of a problem
Testing the solution	With tame problems, it is easy to test the solution	With wicked problems, there is no immediate or ultimate test of a solution
Number of attempts	There can be repeated attempts for a tame problem.	A wicked problem is a one-shot operation. Each attempt matters and is consequential. You can't build a software, get people to use it and then change it without any consequences.
Uniqueness	Tame problems are not unique. Lessons learnt from solving a problem can be carried over to solving another problem. e.g. quadratic equation or chess problems	Every wicked problem is essentially unique. Lessons learnt from the solution to problem can’t be implemented as such for another solution
Right to make a mistake	The Tame problem solver may be wrong and this doesn't mean any major consequences	The wicked problem solver has no right to be wrong, he is responsible for his acts and the consequences

0-0-0-0

The Traditional Model of Software development a.k.a Waterfall model was derived out of the systems approach to problem solving. Systems approach is defined as attacking problems in a rational, straightforward, systematic way, characterized by a number of attitudes which the person in charge of solution or the Project manager should have. The person should be rational, objective and scientific in attacking his problems.

Waterfall Model involves following a certain sequence or steps or phases for attacking a project. 

#1 Understand the problem
#2 Gather information to understand the context of the problem
#3 Analyze the information
#4 Generate solutions
#5 Assess the solutions and choose the best one
#6 Implement the chosen solution
#7 Test
#8 Modify the solution, if necessary, and learn for the next time

The above mentioned approach was successful has been found to be successful, long as the steps could be followed in a sequence. for e.g. where the problem definition is not clear, it is not possible to apply this approach to solve a problem.

Limitations and short comings of the approach:

• This type of approach has been found to work in the context of a strong autocratic decision structure ( for eg. Military domain) but not in problems with respect to corporates and communities
• It expects the Project Manager to be Rational, understand the problem as a whole and anticipate the consequences of the decisions he/she makes. Recent research has show that human beings are not as rational as it was thought out to be. This has spawned a whole new field related to behavioral studies.
• This approach fails where understanding the problem as a whole and anticipating the consequence of decisions is not possible
• This approach fails when the steps can’t be followed in a sequence. for e.g. when the requirements for a project change rapidly or that development has to be speeded up, waterfall falls short.

This is not to say that Waterfall is wrong or should not be used. Waterfall works well when what has to be built is very clear and there is very little change.

Newer Approaches

The various contradictions that are inherent in the definition of a wicked problem makes the waterfall model in effective. for e.g. first step in the approach, 'understand the problem' is not possible for wicked problems as explained in #1 and #2 in the table above. Moreover, generation of solution(s) is not a single step for wicked problems. With wicked problems, the solution definition goes on all the time, till we say the problem is solved. Hence people were forced to explore other approaches to problem solving.

The successful approaches to wicked-problem solving have been found to centre around the following principles

Principles of the newer approaches to wicked Problem Solving

• The knowledge needed to solve a problem is concentrated in many heads and not in a few.
• The people who have the best expertise and most knowledgeable are those who are likely to be affected by the solution
• For potential solutions, it is better to check with those who are affected by the solution and not the experts
• Nobody wants a solution forced on them. People who are the ultimate beneficiaries of the solution want to be actively involved in the planning process
• Planning is a political process
• Some of the steps/decisions needed to develop a solution need not necessarily be scientific
• The choices for a solution or a step depends on who make the decision and the final solution depends on the judge
• Communicating the basis of judgments is crucial for a successful solution, since decisions arrived at are more intuitive and less scientific
• The planner/designer plays the role of a facilitator and not that of an expert/one who offers solutions to problems faced

Please note that agile approach to solving enterprise class problems involve some or all of the above principles.

0-0-0-0

Approaching Wicked Projects While the definition of a wicked problem rules out a way to solve the problem like how a series of steps can be made out to solve a tame problem like a quadratic equation, past knowledge in successfully managing and delivering on wicked projects have taught us the following

• Recognize that a project is a wicked project: In doing so, this ensures that the right methodology is adopted. Not doing so is the equivalent of treating a patient with disease X for disease Y
• See if the project can be Tamed: See if the project can be broken down into smaller pieces having well defined boundaries with the help of stake holders. This ensures that the typical risk associated with a Wicked Project is brought down to a great extent
• Use Adaptive Software Processes: Wicked problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process. Here the key is to ensure that continual adaptation of processes to ensure completion of work at hand is a way of life. Any of the Agile software development approaches like Adaptive Software Development, ASD (Highsmith, 2000); Crystal Methods (Cockburn, 2002); Dynamic Systems Development Method, DSDM (Stapleton, 2003); Feature-Driven Development, FDD (Palmer and Felsing, 2002); Scrum (Schwaber and Beedle, 2001); and Extreme Programming, XP (Beck, 2000) can be used to solve wicked problems/projects.

Note: Out of the above, Scrum Methodology has gained lot of popularity and is widely used

Take away for the Reader

On problem categorization

• Any problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize, can be termed Wicked
• The term "wicked" is used to denote resistance to resolution, rather than evil
• Choosing the right approach to solve a problem is key to the solution.
• Careful perusal of the definition of Wicked Problem rules out the possibility of a generalized solution existing for wicked problems.

On approaches chosen

• Planning should be considered an iterative process and not an one time activity
• The role of a Project Manager is that of a facilitator and not that of an expert
• The approaches mentioned above, provides us with enough arguments to understand the limitations of waterfall model, when to use it and more importantly, when not to use it
• This along with the concept of wicked vs tame problems give a very clear indication of the shortcoming of the traditional way of software development

Suggested Reading

Rittel, Horst and Webber Melvin. "Dilemmas in a General Theory of Planning" Policy Sciences, 1973: 155-169.
DeGrace, P., and L. H. Stahl. Wicked Problems, Righteous Solutions. Prentice Hall, Yourdon Press, 1990

Reading 2017

Books

 

0. Curse of Cash by Kenneth Rogoff

Why: The authors premise is that paper currency is the root cause of many evils and advanced country governments should do away with it as much as possible and limit currency to smaller denomination and coins. The book is divided into three sections. the first section talks about the evolution of paper currency, currency circulation, currency demand in underground economy, the concept of seignorage and the need to do away with most of the paper currency. The second section discusses in detail the idea of negative interest rates. The third section talks about the global implications to phasing out paper currency and how the governments will look at digital currencies and he doesn’t have good news for Crypto currency lovers.

Quoting from the book "With all due respect to promising security advances offered by public ledger technology and the ingenious algorithms embodied in some of the new “currencies,” the view that Bitcoin— or any other cryptocurrency— is going to replace the dollar anytime soon is quite naive. As currency innovators have learned over the millennia, it is hard to stay on top of the government indefinitely in a game where the latter can keep adjusting the rules until it wins. If the private sector comes up with a much better way of doing things, the government will eventually adapt and regulate as necessary to eventually win out. Even if (for argument’s sake), cryptocurrency technology proved unstoppable, the winner (say, Bitcoin 3.0) would only end up being a precursor to a government controlled “Bencoin” (after Benjamin Franklin, who now adorns the US $100 bill)."

I would say a provoking book, and people are free to agree or disgaree with Mr Rogoff. Afterall, he has been the Chief Economist of the IMF. He also happens to be a Chess Grandmaster :-)

1. 'When Breath Becomes Air' by Paul Kalanithi
Why:This book is the most moving book I have read . :-( 

I know many won't get through this book.It impacts you at different levels . The prose is just beautiful . For me , It just reminds us how vulnerable we are and helps appreciate better, what(ever) we have .

2. The Phoenix Project - A Novel about IT, Devops, and helping Your Business Win - by Gene Kim, Kevin Behr, and George Spafford

Why: Wanted a non-technical read on DevOps. 

3. 'A Man for All Markets' by Edward O Thorp

Why: If you are interested in logical thinking and if you want to be rational in an irrational world , if you are interested in investing and want to learn from a person who not only had his skin in the game , but was also extremely successful in wealth generation, do read this book.

4. The Little Book that beats the market by Joel Greenblatt

Why: It is the book on value investing and a must read for any value investor

5. " The Theory That Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy " by sharon bertsch mcgrayne

Why:This book provides an excellent introduction to the history behind the evolution of Bayes Theorem. People's struggles to just use the theory, the opposition it had to face, the famous personalities, prominent role played by Bayes in cracking the Enigma code, for e.g. and lot such information is available in the book. We understand how much opposition it had to undergo before reaching a position of pre-eminence that it enjoys today. 

This is how the book ends "Talking of religion, I am reminded of a strip of cartoons about Bayesians that appeared some time ago. They showed a series of monks. One was looking lost, one was dressed as a soldier, one was holding a guide book and one had his tongue stuck out. They were, respectively, a vague prior, a uniform prior, an informative prior and, of course, an improper prior ..." :-)

6. Tell me the odds - A 15 page introduction to Bayes Theorem

Why: Because i wanted to brush up on Bayes and found this book to be good and more importantly free on Amazon.

7. The Lessons of History by Will Durant

Why: Good knowledge of history is very important for good decision making. This is a condensed work from the authors books and is eminently readable. The kind of book that if you start underlining the important paragraphs, you end up doing it for the whole book. i would call this one of the most important books read in recent times.

8. Business Model Generation: A Handbook for Visionaries, Game Changers, and Challengers by by Alexander Osterwalder (Author),‎ Yves Pigneur (Author)

Why: I was given this book when i worked on a business model for a hackathon. I read the book , found it extremely useful and used it in my presentation. It covers everything about a business model that you have to know of. May be this needs a separate blog post :-)

Books Started and planned to be finished in 2018 ( Hopefully)

1. The Great Unknown: Seven Journeys to the Frontiers of Science (Hardcover) by Marcus du Sautoy
2. The Island of Knowledge by Marcelo Gleiser
3. The Most Important Thing Illuminated: Uncommon Sense for the Thoughtful Investor by Howard Marks
4. Principles: Life and Work by Ray Dalio

Articles

1        

2  1.Scientists, Stop Thinking Explaining Science Will Fix Things http://www.slate.com/articles/healt...

An interesting conversation on how to communicate science to the so called non-scientific people. conversations on this can be found here - https://www.facebook.com/lalgudi/po...

2. Simple math is why Elon Musk’s companies keep doing what others don’t even consider possible

For me the key take aways are realizing the 'limits of incremental innovations' and the need to go back to the basics to bring newer perspectives to the problems that we face and not being constrained by the status-quo.

And this article starts with #Feynman, "Feynman didn’t mean all human knowledge must be distilled into an introductory college course. His point was that we need to build our grasp of science and technology from the ground up if we are to master it, not to mention reimagine how it works."

3. A Path Less Taken to the Peak of the Math World

Inspiring story of a person who took to mathematics quite late and famous for his proof of rota conjecture. at the age of 34, he now is a member of the IAS Princeton, New Jersey.

He is also a fields medal contender.

A poor maths test at elementary school that made him lose interest in maths and try his hand at poetry, a failed attempt which made him look at journalism as a career and how a chance meeting with a 1970 fields medal winner from Japan changed his life for ever is well narrated.

The more interesting part is the comments section where a reader identified a mistake in the article. :-)

4. After 100 Years of Debate, Hitting Absolute Zero Has Been Declared Mathematically Impossible http://www.sciencealert.com/after-a...

Cooling to absolute zero mathematically impossible. Third law of #Thermodynamics on firm footing now.

When i read in school that diamond burns at 690 C, my question was whether we will get diamond if we cool carbon to -690 C. The answer was a No and that reaching colder temperatures wasn't easy. It was only later that i learnt the concept of absolute zero.

5.Why religion is not going away and science will not destroy it https://aeon.co/ideas/why-religion-...

"Religion is not going away any time soon, and science will not destroy it. If anything, it is science that is subject to increasing threats to its authority and social legitimacy. Given this, science needs all the friends it can get. Its advocates would be well advised to stop fabricating an enemy out of religion, or insisting that the only path to a secure future lies in a marriage of science and secularism."

6. The FBI’s Hunt for Two Missing Piglets Reveals the Federal Cover-Up of Barbaric Factory Farms https://theintercept.com/2017/10/05...

The hypocrisy of treating certain animals as pets and certain other animals as food and the ill treatment meted out to intelligent animals.

Warning: Article contains images that are disturbing, but i would still suggest you read to understand the perils of industrial farming of animals.

7. DeepMind’s AlphaZero crushes chess https://chess24.com/en/read/news/de...

“The DeepMind team had managed to prove that a generic version of their algorithm, with no specific knowledge other than the rules of the game, could train itself for four hours at chess, two hours in shogi (Japanese chess) or eight hours in Go and then beat the reigning computer champions – i.e. the strongest known players of those games. In chess it wasn’t just a beating, but sheer demolition.”

8. Seeking the Lost Art of Growing Old with Intention https://www.outsideonline.com/22678...

The story of Bernd Heinrich who is 77 years old and the author of 21 books.

This paragraph should provide an idea about the person "When I began adding up Bernd’s septuagenarian streak, I realized that here was a rare man—a throwback. We live in an age that affords little time and space for communing with nature. We’re busy. Our days are fragmented. But Bernd has dug in his heels against this collective drift. He has recognized where he wants to be in old age and settled in, with purpose.".

Books referred to or went back for selective reading

• .     Meditations by Marcus Aurelius
• .     Calvin and Hobbes by Bill Watterson
•       Bharathiar Kavidaigal
•      Zen and the Art of Motorcycle Maintenance

what is your Customer's Job to be done ?

Background: I discussed with a friend on an idea for a mobile application. While both agreed on a broad level about the need and necessity of the mobile application, the details of what specific problems the app with solve for the end users and how to identify those was still incomplete.

Both of us have the experience of having launched an application on Facebook that had very few takers. We were discussing about the specific aspects of the app that will be useful for the end user, which in turn makes them use the app more.

My friend mentioned the concept of Jobs to be done by Clayton Christensen and suggested that I explore the concept to see how the product’s offerings mapped to real customer needs/ problems people have.

What follows is notes from reading Clayton M Christensen’s article published in HBR about the Jobs to be done concept. I have given references for the source. Please also note that this is based on the articles read and I am yet to read the book ‘Competing against luck’ that talks about this framework.

A recent McKinsey poll found that 84 percent of global executives said innovation was extremely important for business growth, yet 94 percent were dissatisfied with their own innovation performance. Companies want to strategise their best in the interest of their future . They invest in expensive projects, use lot of data, try to focus on the correlations and conclude what the customer wants. But if the approach were successful, majority of the strategy initiatives must yield results. But actual results point to anything but that and 84 % of global executives are actually unhappy with the results delivered by their innovation programs.

According to Christensen, the fundamental problem is that most of the data companies collect is structured to show correlation: This customer looks like that one, or 68% customers say that they prefer version A to version B. While it is actually interesting to find the patterns, they don’t actually mean one thing caused the another. Though we know correlation doesn’t mean causation, most managers have grown comfortable basing decisions on causations.

Christensen says we are really missing out on what the customer wants to accomplish and he calls them as jobs to be done.  or in his own words “What they really need to home in on is the progress that the customer is trying to make in a given circumstance—what the customer hopes to accomplish. This is what we’ve come to call the job to be done.”

This part where he explains what he means by Jobs is very important. Quoting from the HBR article “When we buy a product, we essentially “hire” it to help us do a job. If it does the job well, the next time we’re confronted with the same job, we tend to hire that product again. And if it does a crummy job, we “fire” it and look for an alternative. (We’re using the word “product” here as shorthand for any solution that companies can sell; of course, the full set of“candidates” we consider hiring can often go well beyond just offerings from companies.)”.

The Theory of jobs to be done came out of a course taught by Christensen over two decades. It was developed in part as a complement to the theory of disruptive innovation—which at its core is about competitive responses to innovation: It explains and predicts the behavior of companies in danger of being disrupted and helps them understand which new entrants pose the greatest threats.

The main difference is while the disruption theory doesn’t tell us what products or services to offer to customers, the jobs-to-be-done theory provides a framework to arrive at this. The key differentiator is the focus on the causal driver behind the purchase of a product or a service. Once we start looking at this, the perspectives are entirely different.

In the article, he takes the case study of a real estate developer selling condominiums ideally priced for people moving out of their family homes and single parents – a classic scenario where the conversation rates were low. Architectural changes done to the building still didn’t help. Typical response is to blame any or all of the following: bad weather, underperforming salespeople, the looming recession, holiday slowdowns, the condos’ location . The consultant brought into help took an entirely different direction and tried to understand from the people who bought the job they were hiring the condominiums to do. The interviews didn’t throw any light on who was most likely to buy or nothing emerged as such a profile on the buyers other than the fact that they were all downsizers.  

The most interesting factor that decided the purchase wasn’t even sold by the developers. It was the dining table. Fact was that the dining table had lot of memories associated with a family and people wouldn’t move without deciding what to do with the dining table. The developers not only increased the size of the dining room to accompany the old table, they also announced free moving for buyers,  two years storage for their old belongings and a sorting room where the customers can decide at their own pace what to discard. This helped in the buyers taking a decision peacefully.  Not only did the company raise the cost of the condos to include the additional costs, their sales also increased in a recessional market.

And the beauty of this all is that the insight allowed the company to differentiate its offering in a manner difficult for its competitors to understand , forget copying.

So the question is how to we define jobs to be done ?

Quoting from the article

“Job to be done” is not an all-purpose catchphrase. Jobs are complex and multifaceted; they require precise definition. 

Here are some principles to keep in mind

1. “Job” is shorthand for what an individual really seeks to accomplish in a given circumstance.: This has to consider the experience a person is trying to create. In the above case study, a condo buyer’s experience is very different from a first time home buyer’s .

2. The circumstances are more important than customer characteristics, product attributes, new technologies, or trends. Before they understood the underlying job, the developers focused on trying to make the condo units ideal. But when they saw innovation through the lens of the customers’ circumstances, the competitive playing field looked totally different. For example, the new condos were competing not against other new condos but against the idea of no move at all.

3. Good innovations solve problems that formerly had only inadequate solutions— or no solution.Prospective buyers were looking for a much simpler life. For that they had to go through the hassles of downsizing their existing possessions. And they didn’t want to go through the same, though it would have resulted in a simplified living. And presenting a third option actually helped them to make the transition to a buyer.

4. Jobs are never about function. They have powerful social and emotional dimensions. Reducing the stress of prospective buyers through the additional offerings made the key difference. 

Creating the right customer experience and integrating the processes into the company’s process is another key step in addition to identifying the job to be done and building a product or service around the same. The case study of the American Girl Dolls is shown as an example of a company that created the right experience for the purchase and use of the product , with the company process taking in all the experiences. 

The article also has a case study of the success of the Southern New Hampshire University’s (SNHU) distance learning program,

The diagram summarizes how jobs are fundamentally different.

References:

1. https://hbr.org/2016/09/know-your-customers-jobs-to-be-done
2. The free chapter available at http://www.newmarketsadvisors.com/jobs-to-be-done-book.html