Tame Problems, Wicked Problems, Waterfall and Agile

This post explains Tame Problems, Wicked Problems and how these two are different. It also talks about why (the steps used in) the waterfall model of software development is in-effective in solving Wicked Problems and suggests a generic approach that could be used to solve wicked problems.

0-0-0-0

The problems that scientist and engineers are usually focused on are known as Tame Problems. An example can be solving an equation in mathematics or analyzing the structure of an unknown compound in chemistry or solving a chess puzzle. For each, the objective is very clear and it is also clear whether the problem is solved or not.

The problems in natural sciences are definable and may have a solution that can be found with sufficient effort (based on complexity of the problem).The kind of problems in the public domain , for e.g. public transportation, education, public health and policy decisions are inherently different from the problems that scientists deal with. For the problems in the public domain, what is to be solved is not clearly defined and whether a solution is reached or not is difficult to ascertain. Problems with such characteristics are called Wicked Problems.

Following table contrasts Wicked Problems and Tame Problems.

Wicked Problems and Tame Problems
Characteristics	Tame Problem	Wicked Problem
Problem Formulation	This can be exhaustively formulated so that it can be written on a piece of paper and shared it with a person who will then solve it. You can ask a knowledgeable person to solve a quadratic equation and not expect any further queries	Exhaustive formulation of a problem is not possible since the solution chosen defines the problem. For e.g, you cannot tell a person to develop a software solution to a retail banking system without expecting any further queries. The interactions are important for proper formulation of the problem.
Stopping Rule	For a tame problem, for e.g. chess problem or a Maths problem, once the combination of moves or steps are found, the problem is solved and that is the end	For a wicked problem, there is no such stopping rule. For e.g. when developing a software product, there is no way for us to know when the problem is solved
Testing the Solution	Given a solution to a tame problem, it can be tested and we can conclude the solution as correct or wrong. It is easy to check if a quadratic equation is solved correctly or not	For wicked problems, correct or wrong is not applicable.
List of permissible operators	There is an exhaustive list of permissible operations for a tame problem. for e.g chess rules or steps involved in solving a quadratic equation	For a wicked problem, there is no way a enumerable list of permissible operations can be found, as these depend on the person solving the problem
Problem as a discrepancy – aka the difference between the current state and desired state	With tame problems, there is a single explanation for discrepancy	With wicked problems, there are many explanations for the same discrepancy and we don’t know which one is the best. For e.g., the need for a new software solution can be attributed to changing customer needs or market conditions or an existing software that can no longer be enhanced
Scope of a problem	It is very easy to define the scope of a tame problem	With wicked problems, it is very difficult to establish the scope of a problem
Testing the solution	With tame problems, it is easy to test the solution	With wicked problems, there is no immediate or ultimate test of a solution
Number of attempts	There can be repeated attempts for a tame problem.	A wicked problem is a one-shot operation. Each attempt matters and is consequential. You can't build a software, get people to use it and then change it without any consequences.
Uniqueness	Tame problems are not unique. Lessons learnt from solving a problem can be carried over to solving another problem. e.g. quadratic equation or chess problems	Every wicked problem is essentially unique. Lessons learnt from the solution to problem can’t be implemented as such for another solution
Right to make a mistake	The Tame problem solver may be wrong and this doesn't mean any major consequences	The wicked problem solver has no right to be wrong, he is responsible for his acts and the consequences

0-0-0-0

The Traditional Model of Software development a.k.a Waterfall model was derived out of the systems approach to problem solving. Systems approach is defined as attacking problems in a rational, straightforward, systematic way, characterized by a number of attitudes which the person in charge of solution or the Project manager should have. The person should be rational, objective and scientific in attacking his problems.

Waterfall Model involves following a certain sequence or steps or phases for attacking a project. 

#1 Understand the problem
#2 Gather information to understand the context of the problem
#3 Analyze the information
#4 Generate solutions
#5 Assess the solutions and choose the best one
#6 Implement the chosen solution
#7 Test
#8 Modify the solution, if necessary, and learn for the next time

The above mentioned approach was successful has been found to be successful, long as the steps could be followed in a sequence. for e.g. where the problem definition is not clear, it is not possible to apply this approach to solve a problem.

Limitations and short comings of the approach:

• This type of approach has been found to work in the context of a strong autocratic decision structure ( for eg. Military domain) but not in problems with respect to corporates and communities
• It expects the Project Manager to be Rational, understand the problem as a whole and anticipate the consequences of the decisions he/she makes. Recent research has show that human beings are not as rational as it was thought out to be. This has spawned a whole new field related to behavioral studies.
• This approach fails where understanding the problem as a whole and anticipating the consequence of decisions is not possible
• This approach fails when the steps can’t be followed in a sequence. for e.g. when the requirements for a project change rapidly or that development has to be speeded up, waterfall falls short.

This is not to say that Waterfall is wrong or should not be used. Waterfall works well when what has to be built is very clear and there is very little change.

Newer Approaches

The various contradictions that are inherent in the definition of a wicked problem makes the waterfall model in effective. for e.g. first step in the approach, 'understand the problem' is not possible for wicked problems as explained in #1 and #2 in the table above. Moreover, generation of solution(s) is not a single step for wicked problems. With wicked problems, the solution definition goes on all the time, till we say the problem is solved. Hence people were forced to explore other approaches to problem solving.

The successful approaches to wicked-problem solving have been found to centre around the following principles

Principles of the newer approaches to wicked Problem Solving

• The knowledge needed to solve a problem is concentrated in many heads and not in a few.
• The people who have the best expertise and most knowledgeable are those who are likely to be affected by the solution
• For potential solutions, it is better to check with those who are affected by the solution and not the experts
• Nobody wants a solution forced on them. People who are the ultimate beneficiaries of the solution want to be actively involved in the planning process
• Planning is a political process
• Some of the steps/decisions needed to develop a solution need not necessarily be scientific
• The choices for a solution or a step depends on who make the decision and the final solution depends on the judge
• Communicating the basis of judgments is crucial for a successful solution, since decisions arrived at are more intuitive and less scientific
• The planner/designer plays the role of a facilitator and not that of an expert/one who offers solutions to problems faced

Please note that agile approach to solving enterprise class problems involve some or all of the above principles.

0-0-0-0

Approaching Wicked Projects While the definition of a wicked problem rules out a way to solve the problem like how a series of steps can be made out to solve a tame problem like a quadratic equation, past knowledge in successfully managing and delivering on wicked projects have taught us the following

• Recognize that a project is a wicked project: In doing so, this ensures that the right methodology is adopted. Not doing so is the equivalent of treating a patient with disease X for disease Y
• See if the project can be Tamed: See if the project can be broken down into smaller pieces having well defined boundaries with the help of stake holders. This ensures that the typical risk associated with a Wicked Project is brought down to a great extent
• Use Adaptive Software Processes: Wicked problems are resolved through discussion, consensus, iterations, and accepting change as a normal part of the process. Here the key is to ensure that continual adaptation of processes to ensure completion of work at hand is a way of life. Any of the Agile software development approaches like Adaptive Software Development, ASD (Highsmith, 2000); Crystal Methods (Cockburn, 2002); Dynamic Systems Development Method, DSDM (Stapleton, 2003); Feature-Driven Development, FDD (Palmer and Felsing, 2002); Scrum (Schwaber and Beedle, 2001); and Extreme Programming, XP (Beck, 2000) can be used to solve wicked problems/projects.

Note: Out of the above, Scrum Methodology has gained lot of popularity and is widely used

Take away for the Reader

On problem categorization

• Any problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize, can be termed Wicked
• The term "wicked" is used to denote resistance to resolution, rather than evil
• Choosing the right approach to solve a problem is key to the solution.
• Careful perusal of the definition of Wicked Problem rules out the possibility of a generalized solution existing for wicked problems.

On approaches chosen

• Planning should be considered an iterative process and not an one time activity
• The role of a Project Manager is that of a facilitator and not that of an expert
• The approaches mentioned above, provides us with enough arguments to understand the limitations of waterfall model, when to use it and more importantly, when not to use it
• This along with the concept of wicked vs tame problems give a very clear indication of the shortcoming of the traditional way of software development

Suggested Reading

Rittel, Horst and Webber Melvin. "Dilemmas in a General Theory of Planning" Policy Sciences, 1973: 155-169.
DeGrace, P., and L. H. Stahl. Wicked Problems, Righteous Solutions. Prentice Hall, Yourdon Press, 1990