On the Planning Crisis: System Analysis of the 'First and Second Generations'

What follows below are the notes from reading the paper On the Planning Crisis: Systems Analysis of the 'First and second Generations' * by Professor H Rittel published in  Bedriftsokonomen No 8 pages 390 to 396.

0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0

Systems approach is defined as attacking problems of planning in a rational, straightforward, systematic way, characterized by a number of attitudes which a systems analyst should have. Systems analyst should try to be rational, objective and scientific in attacking his problems. He should have an approach that is interdisciplinary as solutions involve multi-faceted systems.

System approach as it was originally practiced involved following a certain sequence or steps or phases for attacking a planning project.

#1 Understand the problem
#2 Gather information to understand the context from the view point of the problem
#3 Analyze the information
#4 Generate solutions
#5 Assess the solutions and choose the best one
#6 Implement the chosen solution
#7 Test
#8 Modify the solution, if necessary, and  learn for the next time.

The above mentioned approach was successful as long as the steps could be followed in a sequence. for e.g. where the problem definition is not clear, it is not possible to apply this approach to solve a problem.

Shortcoming of the First Generation of System Analysis (FGSA):

• The systems approach is based on a certain naïve belief that the ideals and principles of scientific work, in the planning context, can be carried over to problems in other domain
• This type of approach  has been found to work in the context of a strong autocratic decision structure ( for eg. Military domain) but not in problems with respect to corporate and community planning
• It expects the systems analyst to  be Rational, understand the problem as a whole and   anticipate the consequences of the decisions he/she makes.  where these characteristics are not possible, this approach fails.

Wicked problems and Tame problems contrasted

Characteristics	Tame Problem	Wicked Problem
Problem Formulation	This can be exhaustively formulated so that it can be written on a piece of paper and shared it with a person who will then solve it. You can tell a person in the know, to solve a quadratic equation and not expect any further queries	Exhaustive formulation of a problem is not possible since the solution chosen defines the problem. for e.g, you can  not tell a person to develop a software solution to a retail banking system without expecting any further queries.  The interactions are important for proper formulation of the problem.
Stopping rule	For a tame problem, for e.g. chess problem, once the combination of moves or steps are found, the problem is solved and that is the end	For a wicked problem, there is no such stopping rule. For e.g. when developing  a software product, there is no way for us to know when the problem is solved
Testing the solution	Given a solution to a tame problem, it can be tested and we can conclude the solution as correct or wrong.	For wicked problems, correct or wrong is not applicable.
List of permissible operators	There is an exhaustive list of permissible operations for a tame problem. for e.g chess rules or steps involved in solving a quadratic equation	For a wicked problem, there is no way a enumerable list of permissible operations can be found, as these depend on the person solving the problem
Problem as a discrepancy – aka the difference between the current state and desired state	With tame problems, there is a single explanation for discrepancy	With wicked problems, there are many explanations for the same discrepancy and we don’t know which one is the best
Scope of a problem	It is very easy to define the scope of a tame problem	With wicked problems, it is very difficult to establish the scope of a problem
Testing the solution	With tame problems, it is easy to test the solution	With wicked problems, there is no immediate or ultimate test of a solution
Number of attempts	There can be repeated attempts for a tame problem.	A wicked problem is a one-shot operation. Each attempt matters and is consequential
Uniqueness	Tame problems are not unique. Lessons learnt from solving a problem can be carried over to solving another problem. e.g. quadratic equation or chess problems	Every wicked problem is essentially unique. Lessons learnt from the solution to  problem can’t be implemented as such for another solution
Right to make a mistake	The tame problem solver may be wrong and this doesn't mean any major consequences	The wicked problem solver has no right to be wrong, he is responsible for his acts and the consequences

The various contradictions that are inherent in the definition of   a wicked problem makes first generation Systems analysis  useless.. for e.g. first step in FGSA, 'understand the problem' is not possible for wicked problems as explained in #1 and #2 in the table above. Moreover, generation of solution(s) is not a single step for wicked problems.  With wicked problems, the solution definition goes on all the  time, till we say the  problem is solved.  Hence we look at Second Generation Systems Approach .

Principles of SGSA (Second Generation Systems Approach)

• The knowledge needed to solve a problem is concentrated in many heads and not in a few.
• The people who have the best expertise and most knowledgeable are those who are likely to be affected by the solution
• Ask those who become affected and not the experts
• SGSA rests on the insight that nobody wants a solution forced on them.  People who are the ultimate beneficiaries of the solution want to be actively involved in the planning process
• Planning is more deontic in nature ( based on general political, moral and ethical attitudes)
• Planning is a political process
• Some of the steps/decisions needed to develop a solution need not necessarily be scientific
• The choices for a solution or a step depends on who make the decision and the final solution depends on the judge
• Communicating the basis of judgments is crucial for a successful solution, since decisions arrived at are more intuitive and less scientific
• The planner/designer plays the role of a facilitator and not that of an expert/one who offers solutions to problems faced
• Casting doubt on the choices made is an virtue
• Moderate activism and optimism are a part of the facilitator's attitude
• Every solution is treated as a venture and the people who are part of the solution should share the risk
• Planning process is an argumentative process (one of raising questions and issues towards which the facilitator can assume different positions and argue for or against the positions. The options available are deliberated and a decision taken to move towards the next step)

Why the paper is important: 

• It introduced the concept that planning is an argumentative process
• It redefined the role of a planner to that of a facilitator from that of an expert
• This paper written in 1972, as per my understanding, sowed the thought process needed to move away from the waterfall methodology (first generation system analysis) to iterative methodologies (second generation system analysis) 
• This along with the concept of wicked vs tame problems give a very clear indication of the shortcoming of the traditional way of software development
• Reading this paper in 2014, it provides the reader a better insight into responding to queries on 'why agile and why not waterfall'
• The discerning reader may also see that the principles mentioned for second generation system analysis applies to all enterprise class projects

Further reading: Wicked Problems, Righteous Solutions: A Catolog of Modern Engineering Paradigms

* People interested in reading the complete paper can easily find it by googling

Wicked Problems and Tame Problems

I  was  searching the web on "Software Project Failures", trying to understand (!!) why projects fail in spite of having the best people, processes and intention in place. I came across this categorization of Wicked and Tame Problems in a paper titled "Dilemmas in a General Theory of Planning*". What follows below is the notes taken from reading that paper. 

0-0-0-0

The problems that scientist and engineers are usually focused on are known as Tame Problems. An example can be solving an equation in mathematics or analyzing the structure of an unknown compound in chemistry or solving a chess puzzle. For each, the objective is very clear and it is also clear whether the problem is solved or not.

The authors propose that the kind of problems in the public domain , for e.g. public transportation, education, public health and policy decisions  are inherently different from the problems that scientists deal with. The problems in natural sciences are definable and may have a solution that can be found with the effort varying based on complexity of the problem.

For the problems in the public domain, what is to be solved is not clearly defined and whether a solution is reached or not is difficult to ascertain. Problems with such characteristics are called Wicked Problems.

The authors of the paper have provided at least ten distinguishing properties of planning type problems that are given below.

#1 There is no perfect definition of a wicked problem: While for a Tame Problem, like solving an equation or a Chess puzzle, an exhaustive statement containing all the information needed for a person to solve the problem can be provided, it is not possible with wicked problems.  Wicked Problems depend upon a person's idea for solving it. For e.g. a statement like 'we need to solve the traffic problem in a city' can be interpreted as need for more flyovers or need for a better public transport system or the need to de-congest the city by developing a nearby location.  In each of the options proposed, the solution actually depends on the option chosen. So for wicked problems, the formulation of a problem is the problem! 

#2 Wicked problems have no stopping rule: In case of solving a mathematical equation or a chess puzzle, the problem solver knows when he has done his job. This is not possible with wicked problems. Since the problem varies on how it is defined ( refer #1) a planner can always do better. Some additional investment of time and money may mean a better solution. Usually the planner working on the wicked problem terminates work because he has run out of time or money or both.

#3 Solutions to a wicked problems are not true or false, but are good or bad: When solving an math equation or a chess puzzle, we can objectively decide whether the proposed solution is right or wrong. The proposed solution can also be checked by an independent authority. For wicked problems, there are no true or false answers. People's judgment of a wicked problem varies with their personal interests, their values and ideology. The assessment of a proposed solution is expressed as good or bad or better or worse or satisfying or good enough.

#4 There is no immediate and no ultimate test of a solution to a wicked problem: With wicked problems, any solution, after implementation, will generate waves of consequences over an extended - virtually unbounded - period of time. For e.g. the consequence of building a fly over or setting up a train network through a city can't be completely understood till the solution is implemented and the fly over or train network seen in action.  In contrast, for tame problems, the test of a solution is entirely under the control of a few who are interested in the problem

#5 Every attempt to a wicked problem is a "one-shot operation": There is no opportunity to learn by trial and error: As mentioned in #1, every solution to a wicked problem is unique. And each solution has its own consequence. We can't build a fly over, see how it works and then modify it because of unsatisfactory performance or un desired consequence. This is because every decision to reverse a decision or correct it poses another set of wicked problems.

#6 Wicked Problems do not have an exhaustive set of potential solutions: Chess has a finite set of rules, accounting all possible situations that can occur. But when it comes to solving wicked problems, that is not the case. Various stake holders will have differing views of the solution.  Here it depends on realistic judgment ( subjective)  of the planner and the clientele that leads to the conclusion that a solution can be chosen from a list of potential solutions.

#7 Every wicked problem is essentially unique: This means that despite the similarity between two wicked problems, there can be a additional factor that is over riding. For e,g, the transport problem of city A, while it can be called similar, can't be called the same, with that of the transport problem of city B. Hence the solution that was successfully applied in city A can't be applied in city B. This in a way follows from #1.

#8 Every wicked problem can be considered as a symptom of another problem: A simple definition of  problem can be  "difference between an intended state and current state". The process of solving the problem starts with finding the reason for the difference. Removal of the difference can pose another problem of which the original problem was a symptom. This problem can be considered as a symptom of another higher level problem and so on. For e.g., poverty among citizens can be considered as a symptom of lack of opportunity, failure of government policy, laziness among citizens or whatever causal explanation that is supported by the ideology of the person who is defining the problem.

#8.1 The correct level of solution to a wicked problem is very critical:  Too high a level, we end up taking a general problem that is difficult to solve and too low a level, can introduce un intended consequence.

#9 The causes of a wicked problem can be explained in  numerous ways: How the planner or the analyst's perceives the world (world view) is the strongest factor in explaining a discrepancy and hence the solution in resolving a wicked problem. And here the explanation doesn't strictly follow the 'scientific way' of problem definition. For e.g. crime in the streets, can be explained by not enough police, by too many criminals, by inadequate laws, too many police, cultural deprivation, deficient opportunity, too many weapons etc. Each of these offer a direction in attacking crime in the streets and there is no rule or procedure to determine the correct explanation.

#10 The Planner has no right to be wrong: A scientist can formulate a hypothesis that can be proven wrong and no one blames the scientist for being wrong. But for planners of wicked problems, there is no such immunity, they are expected to get the solution right in the first and only attempt. 

0-0-0-0

The planner of wicked problems is also expected to consider the social context. Society is evolving into a more heterogeneous body. And this evolution leads to different expectations as to what should constitute a solution. And more importantly, (for the planner to consider), the opinions of the individual groups have the risk of turning the solution into zero sum game.

For example, some 50 years ago, setting up a thermal electric power plant would have been a very easy from the perspective of getting approval. But today, the numerous stake holders need to be considered, the impact and benefits of the solution ( setting up the power plant) explained to them, environmental impact has to be considered and so many other factors that would have been irrelevant ( 50 years ago) need to be considered. 

Environmental impact itself can be a very dicey area considering the general awareness of the public and how the scope of 'environmental impact of setting up a thermal power plant' is defined. All this makes the job of a planner, who has set out to help increase power generation, more difficult.

0-0-0-0

Note to the reader

• Though the paper talks about wicked problems from the context of social planning, any problem that is difficult or impossible to solve because of incomplete, contradictory, and changing requirements that are often difficult to recognize, can be termed wicked. 
• The term "wicked" is used to denote resistance to resolution, rather than evil.
• This paper was published in 1973. This paper introduced a new categorization of problem and helped to identify such problems. 
• The paper doesn't help find a solution to the 'wicked problems'. Careful reading of the paper rules out the possibility of a generalized solution existing for a wicked problem. 
• Though 40 years old, this paper still provides valuable insights into the major problems that face our society.
• This concept is also important for Software Professionals because certain class of projects can be classified as 'wicked problems'. More of this aspect in a separate post.

Reference
* Rittel, Horst and Webber Melvin. "Dilemmas in a General Theory of Planning." Policy Sciences, 1973: 155-169.

Books Read - 2013

I have given below a list of books that i managed to read in 2013. Essays and Blog posts read don't get mentioned here :-) I have provided short note about each of them and also provided my recommendation.

Predictably Irrational by Dan Ariely: Human beings are not as rational as they think of themselves to be. This book takes the work of Kahneman and Tversky and takes it to everyday happenings. Prof Ariely uses every day examples to drive home the point that we are irrational human beings, without compromising on the academic rigor ( with references to the experiments).  Ariely cites many experiments and examples, and shows that we often get things wrong because we frame things the wrong way, mis-judge probabilities, apply heuristic rules of thumb that don't always work, or we just plain let our emotions rule. Must Read

The (Honest) Truth About Dishonesty by Dan Ariely: You cannot control dishonesty by increasing the chances of getting caught or its penalties. Those remedies, which are the basis for much of our regulatory and enforcement policy do not control dishonesty. In the real world, according to this book, we all cheat a little, but not so much that it causes us to comprise our self-image or integrity. That is the principle finding of this book. Interesting Read

The Upside of Irrationality: The Unexpected Benefits of Defying Logic at Work and at Home  by Dan Ariely:  You may call this a sequel to Predicably Irrational. Dan Ariely, recounts a series of experiments that he and his colleagues conducted to explore such questions as: What makes work meaningful and, conversely, what can make it dull and unsatisfying? Why do people procrastinate? How does a person's self-image influence whom he chooses to date? Why is revenge so sweet ? In what ways do our emotions impel us to make self-destructive decisions? Interesting Read

Agile Estimating and Planning by Michael Cohn:  A must read for anyone who is interested in Agile methodology. This book pro I have shared some thoughts based on reading the book here and here . This is the kind of book a person will go back to as a reference. Must Read

Tantu: The Loom of Life by H L Bhyrappa: Provides a snapshot of the fight between ideology on one side and the realistic(??) increasingly corrupt on the other side. The conflicts are used to examine the moral fibre of the society. It spans the decades between post independence and declaration of emergency under Indira Gandhi. Must Read

The Art of Photography by Bruce Barnbaum : This book starts with  how photography is connected personally to the photographer.  It then moves on to the elements of composition with great examples. Reading this book and thinking about what the author has to say helps you to connect with photography. This is a connection that is more than just simply carrying a camera. Must Read

Photographic Eye - Learning to See with a Camera by  Michael E O'Brien & Norman Sibley:  This book is written for film camera and hence may be a bit dated. But it lays an excellent groundwork with lots of examples to get a person started with the right way of taking pictures. It takes the reader through Lines, Shapes, Negative/Positive Space, the grid, etc and lays an excellent groundwork with lots of examples (objects that you see around in our day to day life) to get you started with the right way of taking pictures. Must read when one gets into photography.

David Busch's Mastering Digital SLR  Photography 3rd ed  by David D Busch. Reading This book  is like taking a beginners class in photography. This book is written in  easy to understand language. Interesting Read

Within the Frame: The Journey of Photographic Vision (Voices That Matter) by David duChemin: This book eloquently reminds us that vision, creativity, sensitivity and thought are and always have been at the core of making meaningful images. Must read if you are serious about photography

Learning to see creatively by Bryan Peterson - This books helps you answer "what is it about this scene that i want to express?" and "how can i convey that feeling/idea in a photo?". For most of us, these may sound like simple questions. But experienced photographers know that answering these questions mean the difference between a snapshot and a good/great photo. Interesting Read.

The Autobiography of an Yogi by Paramahansa Yogananda: There were certain sections that i found difficult to believe in.  But this is the kind of book that opens up itself to people who are spiritual. I found it an interested and inspiring book. Interesting Read.

Omnivore's dilemma by Michael Pollan: We never think from which oil field the petrol that we use comes from. Oil is a fungible commodity. For a country like india, which has got a rich food tradition, if the same fate befalls the rice/wheat/chicken they eat (it has already started in urban areas), then the health of the country is in serious danger. This book deserves a separate post as it carries some important lessons and most important of them being 'why eating food in a place like McDonald's is doing serious dis service to your body'. Must Read though the book is america centered

Food is your Best Medicine by Henry Bieler: For a society that is conditioned to pop in a pill for every symptom, in this book, the author, who was a M.D in Medicine, talks about how the proper use of diet can be used to cure illness. This book is based on the records of the patients he treated. Food is your medicine is a deep belief and your body is what you eat is a belief in Indian system of Medicine. Interesting Read

1Q84 The Complete Trilogy by Haruki Murakami - A short review of the book can be found here. Must Read. 

The Joy of X - A Guided Tour of Mathematics, from One to Infinity by Steven Strogatz: In this book the author presents various mathematical concepts and elements in a fresh way and without dumbing it down. The book has six parts, each presenting certain elements of mathematics: Numbers, Relationships, Shapes, Change, Data, and Frontiers.  You will understand how algebra got its name, about Moebius Strip ( you cut the strip in the middle and end up with one strip that is twice as long and not two moebius strips), and about some interesting properties of inifinity through the parable of Hilbert Hotel.  Interesting read.

Steve Jobs by Walter Issacson: This book gives an insight into a very complex character. For me, what stood out through out the book was Jobs' ability to simplify  and focus. Probably this was the reason why the quote "Simplicity is the ultimate sophistication" attributed to Leonardo Da Vinci, was on the Apple's first  Brochure.According to Issacson, Jobs, aimed for the simplicity that comes from conquering complexities and not ignoring them. To quote Jobs “It takes a lot of hard work, to make something simple, to truly understand the underlying challenges and come up with elegant solutions.” A small post about this book can be found here. Must Read

A Tale of Two Transformations: Bringing Lean and Agile Software Development to Life by Michael K. Levine : In this book, author Michael K. Levine  talks about how Lean and Agile were used to improve the software development performance of two organizations. He tells the story of the transformation of two different organizations through two protagonists whose people management styles are quite different.The parable format allows bringing in the human elements ( the fears, the egos and the greed) and makes the story all the more interesting and realistic. This is because anyone who has undertaken a change will know that bringing about  change without handling the human elements is just talking theory.  My detailed post about this book can be found here and here. Interesting Read

Books that deserve a Passing Mention

The Shiva Trilogy ( Immortals of Meluha, The Secret of the Nagas and  The Oath of the Vayu Putras by Amish Tripati: I am indifferent towards this series. If you really have time to kill, then may be, may be, you should read the first two and give the last one a miss. 

The Lost Symbol by Dan Brown: This is a page turner for a wrong reason. If you have read a couple of Dan Brown books, you will know how it is. He uses the same template and just in a different setting. If you still want to read this, the choice is yours :-)

Books that I started and yet to be complete

Currency Wars: The Making of the Next Global Crisis by James Rickards: What is a currency war? How is it fought and why is it fought? what happens when it is fought? why is it feared? I hope to get answers to this questions and a lot more, when i complete reading the book.

From the Outside Looking in: Experiences in 'Barefoot Economics' by Manfred A Max Neef: I have just started this and completed a chapter. This is a book that views the problems of poverty and marginalization from a human angle. This books deserves a detailed post once i complete reading. I am sure this will be one of those books that i will come back to read again and again.