i historical mmm
TITLE: Notes On Estimating
AUTHOR: Keith Clark
VOLUME: 11 ISSUE: Summer
STUDIES IN
INTELLIGENCE
A ejection ol articles on lhe historical, opciahonal. doctrinal, and theoretical aspects ol Intelligence.
Al) statements of fact, opinion or analysis expressed in Studies in Intelligence are those of
the authors They do not necessarily reflect official positions or views of the Central Intelligence Agency or any other US Government entity, past or present Nothing in the contents should be construed as asserting or implying US Government endorsement of an article's factual siatemenis and interpretations
rftJunfui Llifii
The evolution of some techniques in the national estimating system.
NOTES ON ESTIMATING Keith Clark
Sinceppealedorehousand National Estimates have been considered and approved by the United Starts Intelligence Board or Its predecessor, tne'Welbgence AdvisoryThis large number ol very solemn documents, the collective progeny of the intelligence community at large, have been delivered through the midwifery of the Board of National Estimates and its Staff. Both the process and the product have undergone certain changes in the course of seventeen years, and there ought to be sorw lessonseview of this evolution, not only for the midwives in ONE but for all who participate in the process of conception, gestati^ and delivery.
My purpose is to identify, primarily from the ONE viewpoint, soee recurrent dilemmas and common pitfalls in producing estimates, to note different ways of coping with these, and to Suggest some main sources of strength or weakness, as well as some avoidable wastes of time and effort. No two estimators would identify all the sar* problems as being important ot perennial enough to rank as matters of continuing professional concern,ffer my observations under twotyle and Scope: the treatise versus the shortethods and Discipline: predictive estimating ard prophecy.
Having drafted, chaired, or otherwise participated in many of toe Nationalisqualify myself fromuch praise or condemnation, but some subjective judgments seepasten to add that the judgments which follow, the arguments wruch support them, and the idiosyncracies wbich pervade them are rev own; they do not necessarily reflect the opinions of any colleagues On the Board or on the Staff,m indebted to members of both and to other professionals for some of the ideas.
Studies and Short Answers
These tags denote two sets of values, or scl>ools of thought, each valid by its own lights, which often collide when estimates are written.
Notes on Estimating
debated, and coordinated. It isuestion ol mere prose style. Everyone agrees that (or our purposes good venting calls for economy in words. Ituestion of scope aod approach. Some look on estimates as vehicles for educating (he reader in all he ought to know about the problem posed. They reason justly that an inforrned policy maker, like an informed electorate,ood thing, and the more informed the better. Acknowledging that NIEs are not encyclopedias and do have severe limits on length of discussion and depth of detail, adherents of this approach nonetheless strive to incorporate as much informatloo at po isible intoyitment, and don't like it when something they consider inforrnatfon or insight of cardinal importance is defined by someone else as superfluous detail
j At Hie other extreme are the short-answer men. They are Imbuediafoctly correct conviction that most high-level policy-makers have too much to read as it Is, and that if tbe intelligence community
i sins in its publications, it is in the direction of too much rather than too little. In common mercy, as well as in the interests of getting the essential message across, they conclude that estimates should be sheared of all that is not strictly necessaryaking the mainand that the latter should be supplied as crisply and quickly
i as possible.
are estimate that does not give rise to some clash of opinion along these lines, and since itery subjective matter, prevailing doctrine or fashion shifts from time to time and from person to person: in fact, individuals feel differently on different occasions, depending on whom they are writing for, their own depth of knowledgenterest In the subject, their patience or lack of It. and many other variables. Speaking only ofnce thought it generally correct to say that the Board favored short answers and the Staff likeddetail. Tills is probably more often true than not. but there are so many instances of tbe Board's demanding tbe addition ofand detail to stah* drafts that tbe generalization is not very valid.
The National Estimates show fluctuating trends in this respect over the years. Insofar as general patterns can be discerned and briefly described, we leaned in the earlier years toward sparcness. This reflected the strong military mfloeoce oo early estimative methods, an influence which made for short answers to short and crisp questions. It also reflected the kind of problems which preoccupied estimators tn those days almostCommunist threats to the United Slates and its allies and interests. The problems were relatively
Notes on Estimating
i>ir
cleat-cut or were made lo appear so, and could be sharply defined Thus NIEas on "Piospects for Communist Armed Action in the Philippines During November."
We thenand of baroque period (mid-fifties lo earlv sixties) in which estimates became more informative, full of subtleties, refinements, and detail, aimed at describing and assessing foreign societies and governmentsore complex way. Thi* evolution was helped along by the participation of more civilians in the process, with tlieir academic skills and habits of work. It was also partly duerowth in the amount of intelligence availablehotography of thend it was probably most of all the result offor estimates on more complex subjects. For example, the nationalist revolution in the undeveloped world then in full flower gave rise to important policy problems for the United Slates and consequently to the need for estimatesubject that was new and complex. It required conceptualization and even some newshort answers to short questions would not do.
Choosing Between Them
In recent years, we have followed an eclecticboth methods and oflen mixing them, with the choice being made by Ihe predilections of those involved after more or less considered judgment about the requirements and preferences of lhehall not argue for one approach over the other. In the present state of the art. and in light of varying consumer needs, wc probably do best to be eclectic.ew observations about some pitfalls in the choice.
One observation chiefly concerns "country" or "area" estimates. These arc not done as frequently as they once were, but the art form is far from dead. What isought to be. Ithe classicraragraphs that methodically discussed almost every subject under the sun relatingountry or regionind of mechanical way. under the headings Introduction, Political, Foreign Policy,and MUitary. Experience has persuaded most of us that this approach involves much waste motion, and that country or area estimates can most usefullyew main points,ingle main theme with variations. It seldom requires more thanaragraphs or so lo render these judgments for any country, with all the supporting detail necessary.
And lo do it in shorter scope increases the chances of attaining several desirable ends: one is thai lhe estimate will be read and
oyiiiiueniiJH
on Estimating
remembered by officials a( highecond is that the trulyjudgments will shine forth clearly, and not be hidden or dulled by clouds ofhird is that the estimate will not become obsolete or obsolescent quite so fast when day-today developments put one detail or another out of date;ourth, rather bureaucratic one, is that short papers take less rime to do, at least in the stages of coordination and coniidetation by the USIB. (Itrue, iffact that time spent in discussing and coordinating papers often varies more nearly with the quantity of words to be gone through than it does with the importance and complesity of the problem at hand; we sometimes devote so much effort to not being wrong about secondary and even trivial matters, or to group discussions of b'terary idiosyncracies. that we lack the energy and perspective to make sure that wc arc right at>out the big questions)
The foregoing amountsather more dogmatic argument for short paperseally want to make. Let me note two or three exceptions lo the main proposition. One is the kind of estimaterequested (or in some cases annually expected) by high-level consumers who are abeady broadly familiar with the mobleras about which they ask. Certain levels of brevity and rirnpbScation which might be just right for many kinds of estimates would teC these particular consumers nothing they don't already know. In theseonsiderable degree of informative detail becomes mandatory if the estimate is lo have anv value Certain annual Soviet and Chinese papers fall into this category, since generalized assessments of tbe Russian and Chinese military threats are of negligible use to anyone. Another exception Is formed by some special estimates on, say, reactions to given US courses of action. No one needs to be tnld thatand neutralist reactions to some forward military move by the United States would be adverse; they need to know how adverse, and in whatthe diflerence between verba] responses and retaliatory actions on the part of the governments in question. Sometimes we cannot make these distinctions clearly, but we ought to try.
Another occasional exception is the "bow to think about"most often addressed to some fairly new and unfamiliar foreign policy problem, or some particular aspect of an area or country whichfeels it would be useful lo conceptualizeonconvenbonal way. The purpose may be more to structure the problem than to forecast the outcome. In such papers, it is probably best to get
n Esttmofing
more leisurely, lo give more inrormjrion, detail, flavor, andthan are otherwise called lor. In sum, there are problems which cannot be treated shortly if tbe estimate is to do the fob it should. But wc can at least try when we start these jobs to be clear in our own minds what the [ob is.
Prediction and Prophecy
One of lhe most persistent half-truths held in the intelligenceand among our customers is that estimates are predictions of things lo come, prophecies of the future. This is dogma and it is also largely true, but wrtcn'couched in these terms it frequently leads as down some unfortunate paths and stultifies our thinking. Prediction is indeed the heart of the matter, but thereorld of difference between predictive estimating and mere prophecy.ppear toase by pejoratives. let me define my terms.
I use the term predictive estimating lorocess which takes due account of its own limitations and uncertainties. It begins with awareness of present unknowns, the slippery ground we start onof the things we don't know, or can't be sure we know, about the past or present It goes on to the future to predict what can beinduction from some kinds of evidence, deduction from olher kinds, testing hypotheses against all evidence available, and the rest of tbe familiar intellectual disciplines hopefully instilled in us all. But as it moves along these tried and true paths, predictive estimating differs from mere prophecy in its continuing awareness of itsin the face of the extraordinarily complex array of matters which will in fact determine future developments.
Mote specifically, it distinguishes between constants and variables, and shows awareness of mlcractton between them; it defines criticalorsuggests alternative lines ofleading from these; it admits ignorance and uncertainty when it readies the outer Umits of evidence, analysis, and logical speculation; without yielding to the crudities of "worst case* estimating, it also avoids the pretentious and useless fallacy of the "single besttexplicitly, alwaysmodelairly tidy and rational world delineated for purposes of analysis and comprehensible exposition versus the messier world of flesh and blood and emotion: it keeps in mind the fact that foreignapparently monolithicas often as not inwardly subject to coriuicting pressures, ambivalences, and contradictory impulses, even though usage often compels us to talk
,-iin lii ui.r
en Estimating
j aser Tsraet* were each ol* onet whole, coherent, and consistent1
, Prophecy,se the term, implies that the future is alreadydeep within the crystal ball, to be discerned by those who arelucky enough to do so. Itreat leapoby intuition and hope. Predictive estimating does notaids altogether, but ir is based essentiallyoncept ofas too complicated and chancy to permit easy leaps fromare to where you want to be. It is, in short, bothd. humbler thanlso typicauy Jca
* cautious and tentative in its concluaoos/an'd perhaps less extitjng to read. Sometimes it is possible to startle or intrigue by statements of boldly impressive foresight, but this is legitimate onlyaborious and disciplined intellectual process has been gone through first.
All this may sound like pretentious counsels of perfection, and in any case inconsistent with earlier remarks on the desirability of short papers.ublished estimate which self-consciously spelled out its own scrupulous observance of all the rules suggested above j would be an uiGmtely elaborate and tedious document, too muchhD. thesis in one of the Gelds of social science where concentration I on methodology crowds out content m talking here as much j about an intellectual process as about the visible product delivered < to tha printer. We all use various forms of verbal shorthand to getting I our message across; without them, analysis and estimating could not be communicated. But tliereifference between short cuts in getting the message across and short cuts in thinking about what the message should be. The latter can be indulged in only at the risk of sacrificing quality and, eventually, credibility. Like icebergs,mustot of substance below the visible surface if they are to hold together aod stand up.
Ct/esring Games
The record of National Estimates over the years in these respectsixed one. One practice occurs often enough in various guises
'To illustrate tn effect of thisumber of National Estimates ia recent yean have employed the device of presenting the most likely judgment oo die central question, and then, hi immediately following paragraphs introduced by tbe sennble admission that Uus reaionlog might be la error, of going on to suggert the implications of alternativeif the odds don't appear to favorannot escape the belief that oo close questions of parttcu-Lsdy crucial importance this practice adds enormously to the uiefulocss of the document
n Estimating
to warrant tome criticism. It il the temptation or compulsion to estimate with apparent confidence about any question lhat anyone in authority wants lo know about The potent old blandishment, that if the estimators don't supply answers someone less qualified will, can sometimes be resisted only by appearing mutinous.
But the plain fact is that estimates on some questions are ofworth, no matter how sophisticatod the thinking behind them, and we ought honestly to say so. We may be paid lo estimate, but we are not paid to do the Impossible, and certainly not lo pretend lo do the impossible wbeoconfession of ignorarree or urtcertainty may annoy someone who wants practical answers to practical problems, but in the long run it is better to anuoy than to con him. This is not an argument for refusing to do difficult lasb. or even lo try what may look like impossible ones; it is an argument for being clear, to ourselves and to our readers, just how safe it is lo skate on the ice In certain areas and just where the ice, for all we know or might wish otherwise, may be water.
One case in point is the amount of time devoted to predicting the survivability of govenuncnts. Using again the "country paper"hipping horse, these are too often conceived of as vehicles for quoting odds on whether an incumbent regime will be in place when "the period of this estimate" drawslose. The trouble is thatt is possible to say yes or noeally high degree of assurance, the answer il usually so obvious that no literate policy-maker reaEy needs to be told it; and in cases where the forecast is much morefor example, in unstable and volatile countries of Ihe underdevelopedprudent policy-maker is going to place many chips on that particular prognostication.
I am not arguing for tolal abolition of this kind of estimate. It probably has lo be made, tbe odds have to be quoted, the conclution may even be informative and helpful at the time it is published. Butontinuing guide to planning and action in the real world it has severe limitations, and we ought to avoid exaggerating its importance. Among other defects, it becomes obsolescent quickly, since in these matters one wants tbe latest information, whether it changes areached earlier or not; even the best estimate asiven date cannot allow for all the accidents, whimsicalities, and other variables likely to affect the outcome in close questions of this son: very often what the United Stales does or does not do will helpthe results (wc normally leave this factornd mat.
'lrff.
on Estimofing:
of these situationi arc quite literally tossups. touch-and-go matters,hich rational planning must be kept flexible and contingent, with shadings rather than sharp choices in between alternatives.
Wc have too often focused on this kind of question as though"probably yes* or "probably no" were the single most important answer we could give, one on which our reputations as estimators will stand oruspect that this particular kind of forecast is often read by our pohcy-maldng friendealthier skepticism about its real value than we ourselves show; and then the whole thing is forgotten unless and until something happens in the benighted country, in wbich case tho estimate is dragged from the files and the prediction Is either pointed to with pride or viewed with chagrin by those who made it This review of the record, though mtcrcsting to professional estimators, is not very importantroader sense, and certainly should not be made the touchstone of estimative reputationsery seriousof quality. Success or (allure in this kind of spot forecasting is tooatter of luck and chance. It often comes closer toave defined above as prophecy than to predictive estimating, and Is consequently not very usefulesponsible help to planning and action. We may have to indulge in it, but we should not confuse ourselves about its usefulness.
Cards or, tlie TobU
One way in which estimates have grown more sophisticated deserves special mention, strong endorsement and even moro attention in the future: that is the laudable practice of leveling more with the reader on questions of methodology and our own confidence in certainI am not talking about the uxwds wc use for expressing degrees of probability, whether we conclude that something isr "almost certain" These terms are essential tools of the trade, available to alleUdefined glossary, accepted and used by most writers and readers, and already tlie subject of several scholarly articles in this journal. To gain common agreement on the meaning of these terms bas been no easy achievement, but it has now largely been done.
m applauding here is rather the practice of saying more about sources and methods, what can be expected of the evidence,cannot. To do so is to tread on delicate ground. There are many who feel that intelhgence loses potency if it hints at the mysteries behind its findings, and the Subject is apt to be particularly touchy in National Estimates, since any comment on the
Notes on Estimating
sticngUis and limitations of sources or methods tends to be translated into favorable or adverse reflections on some particular contributing agency's present and potential importance- Anyone who hasin an estimate on strategic warning or concealment andwill recognize the symptoms, but they arc not confined to these subjects. Obviously there are distinct limits on how far one ought to go in telling all Security and the "need to know" principle obviously impose distinct limitations, in many cases the whole story about sources and methods would also be tedious to tbe reader, and it is often unnecessary to an honest and useful paper. But it is also often
We were probably pushed or pulled into being more forthcoming on this score than we might have vohinteered on our own. Ten or fifteen years ago intelligence did go about itsa propensity for the mysteries of tbe priesthood which has since diminished. The collective "wes itIn the earlier estimates, had an aloof and oracular tone which has undergone subtle changes in recenteeling that the propositions which it introduced were put forward In the fifties with less fear of contradiction or challenge than in the period since.
Perhaps the chief reason for the change was the new style of foreign and defense policy-making introduced by the Kennedy administration and still carried on. Broadly speaking, two things happenedintelligence was taken more seriously than ever beforeontinuing and responsible contributor to decision making; and it had to come down from Ihe mountain and engage more vigorously in asserting and defending its judgments in strenuous debates before some very tough-minded audiences. The process was marked by much closer communication between intelligence producers and users, each became more familiar with the others needs and assets, and estimates were geared more closely to practical problems in their scheduling and subject matter. All very fine, Battering, and gencraDyit cost some tiling,
The price was that intelligence lost something of its former mystery, autonomy, and immunity. Oracular assertions were out, argumentation which marshalled data was in. More and more technical experts lined the walls at meetings on increasingly complicatedwe would have been lost without them.ublished NfE's were preceded, accompanied, and followed upreat deal of less formal
1i ii p
<riDCMrlM(
on Eitimaling.
paperot ol* informal talk. Judgments could no longer be made, published, and filed away until next year: they came under constant scrutiny and had constantly to be defended or modified in the light of an mcreaiing flow of intelhgence. Information. policy plans was made available to intelligenceegree pteviously unheard of, and estimates took cognisance of this in various ways. Inew extreme heretics were beard to challenge the first premise ofpolicymaking and intelligence were, or should be, separate and distinguishable functions The translation of some former intelligence officers into high policy positions seemed to add force to the radical new *in.lv of opinion.
I suspect that some of the more drastic efforts to remodel the whole system in the early sixties will, in time, be seen as excessive reaction to some previous rigidities and excessive compartmcntation.and policy-making are likely to remain distincdy separatewith accompanying differences in perspectiveertain amount of intellectual and bureaucratic tcrasion between them, some of it wasteful, some of it creative. But Our particular professional world will never be quite tbe same as it was before. Havingdie joys and sorrowsore direct and responsible role, of seeing the product sold to sophisticated customersompetitive market, few members of the profession would willingly return to the mysteries and immunities ofrcompartmcntalixed Olympus, even if they had tbe option. And they don't.
Original document.
Comment about this article, ask questions, or add new information about this topic: