TCS Daily

The Impossibility of 'Planned Improvisation'

By Arnold Kling - September 20, 2005 12:00 AM

"Uncertainty refers, per Frank Knight's 1921 definition, to unmeasurable and unquantifiable risk...[It] bears a close relationship to 'ambiguity'...Entrepreneurs who undertake uncertain initiatives face a wide spread between desirable and undesirable outcomes, but they cannot quantify the odds they face or even fully anticipate the possible results."

-- Amar Bhide, The Origin and Evolution of New Businesses, p. 26


In the real world, businesses must confront ambiguity. In mainstream economics over the last half century, the concept of ambiguity was driven out by the attempt to make economics masquerade as physics or engineering.


What mainstream economists fail to grasp are some of the issues that Amar Bhide addresses in his important but little-noticed research. Mainstream economics fails to draw any distinction between large, established enterprises and small start-up companies. Bhide shows how they occupy distinct niches. Bhide also raises the issue of the trade-off between planning and improvisation, which mainstream economics ignores.


In this essay, I want to elaborate on the issue of the trade-off between planning and improvisation. Then, in a follow-up essay, I will look at the implications of this trade-off for the conduct of war, disaster relief, and the role of government in general.


Large vs. Small Businesses


In my Challenge for Brad DeLong, I wrote that


"Large organizations...are inherently dehumanizing to employees, clumsy, inflexible, and unable to handle sudden new challenges."


Brad could not believe that I meant to use the word "inherently." He suggested instead, "can easily become."


Sometimes, like many writers, I can get carried away with my own rhetoric. But not this time. Large organizations are inherently different from small organizations. Let me elaborate.


Small organizations are the size of hunter-gatherer bands. The behavior of each individual can be monitored and constrained by the group, using instincts that are deeply embedded by evolution into our social psychology. Members want the approval and respect of the group, and this desire is sufficient to orient the individual toward obtaining group goals.


Transactions within the small organization are based on informal rules and tacit understanding. Responsibility for the success of the organization is shared. Members often trust one another completely and feel intensely loyal to the organization. When a few key members articulate a need for change, the rest of the organization will go along.


Large organizations incorporate more people than a prehistoric tribe of hunter-gatherers. The social psychology of tribalism, in which people trust fellow tribe-members but fear and dislike other tribes, can be a hindrance within a large organization. Conflicts between individual interests and organizational interests require much more careful management.


To maintain cohesiveness as they get larger, organizations have to reduce their reliance on informal norms and tacit agreements. Instead, they have to formalize relationships by establishing organization charts, documenting policies and procedures, clarifying roles and responsibilities, and preparing for change through a systematic process of analysis and communication.


I also meant what I said when I used the word "dehumanizing" to describe large organizations. Large organizations have to get people to act in the corporate interest, obey formal rules and procedures, and trust unfamiliar people within the organization. None of this comes naturally. Every year, corporate America pays outrageous fees to consultants for help with "team building," "change management," etc., in an attempt to address the friction between human nature and organizational goals.


Your honor, to show that large organizations are dehumanizing, I would like to submit as evidence Who Moved My Cheese?. This was the leading business book for over a year, and in large organizations it was given to tens of thousands of employees by their managers. I rest my case.


The Wal-Mart Fallacy


If large organizations are dehumanizing, then why do they exist? Brad DeLong says that my assessment of large organizations must be incorrect, or else we would not have Wal-Mart.


Large organizations exist, in spite of their awkwardness, because they create or exploit economies of scale. Wal-Mart built an efficient, globe-spanning logistics system, which can bring the right products from the right suppliers to the right stores at the right time. It turns out that in comparison to smaller retailers that lack such a logistics system, Wal-Mart is able to operate with much lower costs. This scale economy offsets what would otherwise be the organizational disadvantage inherent in Wal-Mart's sheer size.


I also would caution against viewing Wal-Mart as a typical large firm. Retailing is an unusually competitive industry, in which any winner is likely to have evolved some remarkable management capabilities. For every Wal-Mart, there are dozens of large firms whose effectiveness is much less impressive.


It is a fallacy to suggest that Wal-Mart's success shows that, other things equal, large size is an organizational advantage. There are industries where economies of scale are significant. And within industries, there are firms that are exceptionally efficient. Overall, however, most skilled workers and innovators function better in small organizations than in large corporations.


Planning vs. Improvisation


A major thesis of The Origin and Evolution of New Businesses is that large corporations introduce innovation as part of a planned process, while small businesses introduce innovation through improvisation. As a result, large businesses focus on innovations that are capital-intensive but low in ambiguity. An example might be a new automobile, for which the production costs and potential demand can be estimated within a reasonably narrow range.


Projects that involve a great deal of ambiguity but which can be launched with relatively little capital are undertaken by new, small firms. Many of the first web-based businesses were launched by entrepreneurs, because it was difficult to forecast which business models would be successful.


There are several reasons for large organizations to avoid projects that involve considerable ambiguity. A formal planning process requires clear, quantifiable inputs, which makes large organizations highly averse to ambiguity. Large organizations need to restrain risk-taking. And large organizations have the resources to evaluate options before making decisions.


For example, ten years ago, when I had launched one of the first web-based businesses, the software used for web servers was not very mature. Trying to find a flexible, reliable solution, I switched products several times, without ever subjecting a server to thorough testing and evaluation. For a long time, the site was running on server software that even the manufacturer warned was "not production-ready." In fact, the developer abandoned the product without ever marketing an official release. My choice of this unproven server software was typical of small-business improvisation.


My web-based business in the 1990's was a small, innovative start-up. In the 1980's, on the other hand, I worked at the Federal Reserve, which among other responsibilities handles large wire transfers of funds between banks. Concerning that function, I don't think that the Fed would have been impressed with the virtues of improvisation. One would hope that before the Fed changes the software platform that supports electronic funds transfers, years of evaluation and testing have taken place. Can you imagine trying to recommend to the Fed that it run its wire transfer operation using software that is "not production-ready"?


Between the Extremes


Fed funds transfers systems require thorough planning and not much improvisation. Innovative web-based businesses require much improvisation and little planning. However, many government and business functions fall between those two extremes.


For example, pharmaceutical research and biotechnology require both planning and improvisation. Planning is required because of the large capital expenditures involved. Improvisation is required because it is impossible to know ahead of time which approaches are fruitful and which are dead ends.


Software development also requires both planning and improvisation. If software development is undertaken without proper planning and procedures, the result is products that fail to perform as expected and are difficult to maintain. I would argue that Netscape in the mid-1990's was guilty of not having an effective software development process, particular with regard to quality assurance.


On the other hand, it is impossible to get a big software project completely right the first time. Improvisations, known as "fixes" or "patches," are always needed. In fact, one of the classic open source projects is the web server Apache, which is a pun on "a patchy."


In many fields, managers try to find the best mix of planning and improvisation. To some extent, the two can be combined. However, ultimately, the concept of "planned improvisation" becomes an oxymoron. Either you err on the side of excessive planning, in which case you adapt too slowly, or you err on the side of excessive improvisation, in which case you make too many avoidable mistakes. The next essay will discuss how military operations, disaster planning, and many other government functions are necessarily caught between the two extremes.


TCS Daily Archives