mvsm review
TITLE: The Collector's Role in Evaluation
AUTHOR: Bruce L. Pechan
VOLUME:
STUDIES liM
INTELLIGENCE
A collection o( articles on the historical, operational, doctrinal, and theoretical aspects of intelligence.
All statements of fact, opinion or analysis expressed in Studies in Intelligence are those of
the authors They do not necessarily reflect official positions or views of the Central Intelligence Agency or any other US Government entity, past or present. Nothing in the conients should be construed as asserting or implying US Government endorsement of an article's factual statements and interpretations.
Imprecision and realn arbitrarily prescribedof functions.
THE COLLECTOR'S BOLE IN EVALUATION Bruce- L. Pechan
Ever since the establishmentefined and orderedIntelligence program, the community has performed one of its fundamental functions on the basisction. This fiction has by now come to bo accepted as fact In someand thereangerous chance that ultimately It could be universallyefer to the notion that theof information Is not qualified or authorized, much less obligated, to participate In the evaluation of the reports he transmits. If this idea in its full implication is ever accepted by the collector, it will do great harm not only to ourand estimative performance but to our performance ln clandestine collection as well.
The official fiction makesitual which only analysts are ordained the high priests to perform.collectors, with their often Impressive qualifications, may subject an itemhorough process which bears all theof evaluation, but this is not officially accepted as Evaluation and may not be designated by that term.of clandestine reports are protected against any such misconception by the solemn warning. "This is unevaluated information."
This pre-emption of the word "evaluation" to denote astep In whatomposite process has left usfor terms to apply to other steps. For the fieldjudgment as to the probabilityeport is true we must use the synonym "appraisal" in order to preserve the analytic monopoly on Evaluation. The collector's judgment as to the significance of an item of information must bevaguely "comments" to avoid the Implication that he has some evaluative responsibility.
The tortured circumlocutions that must thus be employed in referring to the collector's role in evaluation are unbecom-
Ingrofession In which search for objective truth and precision in the use of language are cardinal principles; but the official fiction has more serious consequences than these semantic ones. The best of our collectors, ignoring theabsurdities, have for years been offering their ownas appropriate, unconcerned by what name they are called. But some collectors have been honestly confused by the hazy language describing their evaluative functions, and some have accepted literally the dictum that the collector has no responsibility in the evaluation of an item's significance. When this happens, the quality of collection and reporting inevitably suffers and valuable Judgments are lost.
Thereendency In some quarters to regard collectionechnical process and collectors as mere technicians.echnique that employs human agents rather than black boxes requires considerably more than technical skill. It is the purpose of this paper to examine the duties of the clandestine collector in connection with evaluation in order to clarify his natural and proper role In making theof which evaluation consists. The discussion will be confined to evaluation done for the benefit of estimators and policy makers, not touching the slightly differentof the same process undertaken for the purpose ofcollectors In the pursuit of further information. Although addressed specifically to the role of the covert collector, much that follows will apply equally to that of the overt collector.
Determination of Probability
Evaluation, as the term Is used in the intelligence world, consists of determinations on twotruth orof facts reported, and their significance if true.about probability is of two kinds. One kind oflies in the origin and acquisition of thehe reliability, capability, access to information,f the source, and the circumstances surrounding his acquisition of theThe second kind has to do with the informationamount of confirmatory or contradictoryalready In hand, or in the absence of direct confirmation or contradiction, the Internal logic of the new information in its relation to what is already known.
The collector Is held responsible (or providing the first,kind ofevaluation (officially labeled such) of his source's reliability, and an account of the circumstances surrounding the acquisition of the information. These two elements of external evidence are to be used by the analyst as factors In determining the degree of probability that the Information Is true. In providing them the collector hashis major assigned responsibility in the evaluative process, and he Is not regarded as having thereby engaged in evaluation of the information itself. To see whether thisealistic view let us look more closely at the nature of these two elements.
Source. The one accepted evaluative Judgment of thehis source evaluation. Is generally considered to beof the particular Information reported. SourceIs regardedelatively stable factor,ource has to prove his reliabilityonsiderable period to be advancedating. Once he achieves this, he Is notof it unless his reliabilityecrease overconsiderable period. Thisonvenient practice and serves reasonably well for most of our clandestine sources, but In any given instance it may beource who is reliable In one field may be less reliable in another, whether for lack of competence, lack of access, or lack of will to beThe collector who smugly rests on his source's Bis flirting with disaster; there is always the possibility thatating may not apply in the Instance at hand.
This means that the collector must be constantly alert to possible shifts in his source's reliability, brought about byin the source's access to Information or in hisor by lack of competence to report Intelligentlyew subject. These can be detected only by regular analysis and evaluation of the significance of the Information beingin terms of the source's capability and access andof his motivation. This analysis may be only aof form in the casetaunch anti-Communist Japanese reporting the movements of Communist organizers, but It can be an endlessly complex Jobapanese socialist ison Japanese political undercurrents.
Circumstances. In the simplest of situations, that in which the source Is reporting on something of which he has direct participating knowledge (suchlot ln which he Is one at the leadinghe source evaluation bearson the probable truth of the Information. But when the source is reporting Information he acquired second hand, the source evaluation can bear only on the credibility of his account of how he got the information; and theof its acquisition here gain Importance as the only valid external evidence bearing directly on the probability of Its being true.
An account of the circumstances of acquisition Is expected of clandestine collectors for each Individual report. Many agent operationstandard pattern in which agents of fairly stable reliability report regularly In one or twoand their mode of acquisition remains unchanged in any important particularong period of time. But If abeing reported on grows tense or if security conditions for the source become more stringent, this comfortablediminishes; the circumstances surrounding acquisition become of increasing Importance and tend to varyfrom report to report. This Is particularly true ofreporting on an area ln crisis where the source may have to acquire his informationariety of ways. In such situations the factors of acquisition pertinent to aof truth are often closely related to the significance of the information. To get at these factors the collector must analyze the Information itself and work back from that point to draw out his source to best advantage
In sum, the collector must for his own purposes evaluate the information he receives if he Is to perform withhis tasks of evaluating the source and reportingdata. He thusead start toward fulfilling his third assigned function in the evaluative process, that offactual probability. For this purpose, however,he canudgment of validity, he needsa respectable slorc of knowledge on the subject to which the new Information pertains. We shall return to this point shortly.
Determination of Significance
In evaluating the significanceeport there are at least four elements to be determined. The first ts whether It has relevance to an established requirement. The second Is Its meaning In terms of the requirement, in other words Its place and contribution in the fulfillment of the requirement. The third Is its relative weight, impact, or importance. And the fourth Is the timeliness of the information, with particular reference to the timing of events predictable therefrom.
A study of the extended implications of these severalshows some significant characteristics of thefunction. First, evaluation is an exercise of humanunder tbe best of circumstances subject to humanand human error. Any single evaluation may be wholly accurate, but the sum total of all our evaluations will fall snort of perfection. Second, itransitory Judgment, rarely Ifixed and stable truth It Is subject to change with changes in the facts themselves or with the acquisition of new information. Third, it is an organic processa number of successive steps, and though each of these may be complete In its own terms no one of them is the whole of evaluation. Some of these steps occur very early in the official life of an Item of information. Let us then look at them more closely, and determine who it is that logically takes each of them.
In denying responsibility to the collector for evaluation we have overlooked entirely that he is de facto authorized toevaluative Judgment regarding relevance, importance, and timeliness, and that these Judgments of his are often final and Irrevocable. His right toeport for lack of relevance is uncontested. To cut down marginal reporting he is given lively encouragement even to kill information of limited relevance or importance. And because of hislocation in the collection process, he must Judge both timeliness and importance hi deciding whether to send aby pouch or by one of several orders of cable precedence. Whether his judgments are good or bad, the actions based on them are definitive:eport is destroyed in tho field it Is beyond our reach to recall and reconsider. Or once pouched
as having limited timeliness or importance, the delay ln lis transmission is an unchangeable fact
These judgments, however, axe not Evaluation with aE, which is apparently thought totudy in depth of relevance, importance, and factual probability for which the collector is not qualified. Let us then examine thefortudy.
Capabilities of the Collector
The qualifications needed to be capable ol evaluation in depth may be summed up In two categories: ability as aman to reach sensible conclusions, and command of abody of knowledge on which to base them. With purely personal capabilities there is no reason to suppose that the collector Is less generously endowed than the analyst; there Is nothing in the analytic function that increases nor ln the collecting that decreases the native ability of the naked man to think. It is held in some quarters, however, that theof the different functions enhances the ability of the one and detracts from that of the other to apply his logicalto the function of evaluation in depth, that analysis contributes to and collection detracts from the conditioning of the mental equipment for evaluation.
This effect is said to be produced in two ways. First, since the analyst must evaluate regularly as part of his job, his equipment for evaluation becomes more and more highly trained, whereas the collector, who by definition doesecessary part of his job, lets his equipment,naturally great and highly trained when he starts out, become rusty fromave already shown thatof the significance of informationecessary adjunctollector's proper performance of his assigned evaluativehall deal further with this pointo-
ment.
The second argument Is that the influence of contact with his sources and his personal interest in the success of hisrender the collector undefendableakerill concede that bias may be Introduced into his thinking by his identification with sourceselieve the danger Is often overstated. First of all. the possibility of such bias is accepted by collectors them-
Evaluation
selves, and the seasoned coUector buildsealthy skepti-cism to minimize it. Second, since his personal interests are bound up in the success of his operations, the experienced collector realizes that overselling his product mayna*" 1ounteracting tendency
Admitting that bias, while not inevitable, remains ain the collector's evaluative Judgments, we should note that it is not peculiar to the coUector. The analyst too may and sometimes does, exhibit biasoint of view and Interpret new facts inay as to make themreconceived conclusion. Pearl Harborumber of more recent strategic surprises bear memorable witness to thethat analysts may reject or downgrade evidence not in accord with their preconceptions. And since bias is afailing that afflicts all of us to some degree, the best way to insure objectivity in evaluation Is to take Into account the Judgments of all thoseosition to render valid opinions.
Is the coUector inosition? Theoretically,erson with access to all related knowledge can render the definitive evaluationew fact. The analyst most nearly meets this condition: in addition to his own knowledge of the area or subject matter he can callast organized store of related facts from all available sources. But he does not approach an allness in this store, and on some areas andit is pitifully thin. So if It were true that evaluation Is not Evaluation unless it Is based on the entirety of data we should have no Evaluations whatever. Actually, we seldom need aU the facts toalid evaluative judgment Of the hundred thousand factsountry we may have stored up In machine records, ranging from the makeup of the party of the opposition7 to the number of aluminum teeth wom by the current labor rrunlster, we may find that only sixteen are of any use in determining the significanceew item. Theay analyst quickly learns tohis consideration of data on hand to those of substantial pertinence. Otherwise he would turn out precious few
Given that the analyst must evaluate on the basis ofdata, what kind of data Is he most likely to lack? On many countries and subjects his store of organized and
EOjjET
f
usable information grows sparse as we approach thenow, because of the inescapable lags In acquisition,organization, and assimilation of up-to-the-minute facts.apidly changing situation he usually lacks the facts most essentialalid evaluation of new information. That this is recognized by analysts themselves is shownecent review of clandestine reportingertain area:riticalield interpretations of the significance and probability of the information reported are needed by the customersreatly increasedhis plea lacks any official standingirective for collectors to evaluate, but itirect and realistic expression of need by one set of analysts aware of the limitations placed on their own Judgments by circumstances.
The field collector,peak here of both the overt and the covert coUector, Is best situated to acquire what the analyst most keenly lacks: current information on the area in which he works. He it is who can immerse himself whoUy ln the life of the area, have daily contact with broad segments of Its people and Its thought, andapacity forwhich under some circumstances is beyond theof the distant analyst.6 uprising In Hungary was unforeseen in our national estimates not because we lacked exceUent analysts working on the area or knowledge of the history of Hungary or information on the economic situation, but primarily because we had no qualified body of observers present on the scene to report the things that could beand interpreted only by being there.
It is argued that the coUector can fill this gap bythe analyst that evidence on which he himself would base an evaluative judgment were he called on to make one. But this Is Impractical in many cases. There are often too many smaU details, some too elusive to captureritten report, too closely bound up with the physical presence of thein the area. Many of the indicators simply do not speak to the analyst in his remote office with the same ring they have for the coUector experiencing them in the field.
The collector is not only thus uniquely qualified undercircumstances to evaluate new information, he must evaluate if he Is to produce good reports. We have already discussed the importance of evaluating the significance of in-
formation as an adjunct to the task of evaluating the source and providing useful acquisition data. Evaluative Judgments are required also In getting the maximum of Information from agents and informants. The skilled collector In the field, overt or covert, far fromere technician,holecommunity in miniature. When he debriefs an agent he runs through the entire intelligence cycle,several times over. As an agenteport, the collector evaluates it as to relevance. If It Is relevant, he hastily evaluates Its significance as nearly as he can and uses it to formulate new requirements. These he immediately puts to the agent In the form he Judges most likely to draw out additional facts and related Information which the agent may have overlooked, not realized he knew, or intended not toBy this means the collector often greatly Increases the substantive value of his report.
The Important thing here Is that the collector Is limited by his capacity to Judge significance. When he reaches the limit of the body of particular facts and broad general knowledge at his command and therefore of his ability to analyze the new facts in relation to them, he must content himself with accepting whatever the agent offers and sending it on for others to work on. It is axiomatic that, other things beingollector who is well groundedubjectetter collector In that subject than one who is not He can Instantly analyze, evaluate, and extemporize his own requirements, short-circuiting by days or weeks the process of gettingguidance from the analyst. The mental operations of analysis and evaluation, by whatever terms they areare Inextricably involved In this short-circuiting, and it is clear that the skilled collector must have an evaluative technique well polished by use
Limitations on the CoUector
Limitations there certainly are on the qualifications of the individual field collector to evaluate. Being only one person, he cannot become expert In all subjects on which he mayinformation, and his judgment about the significance of many Items he reports is therefore of little or no value. On geographic areas other than the one in which he Is working
45
he can have at bestimited up-to-date knowledge; hence, while he may be able to evaluate political newsin terms of its local impact, he may have little to offer with respect to Its impact abroad. For these reasons, each cot-lector should take stock of the limitations of his knowledge and not attempt to go beyond their bounds.
Although In some situations the collector may have all the data available to the analyst and more, he can never be sure that this is so. The analyst usually has, and In every instance may have, pertinent other-source information not available to the collector. Further, by virtue of the nature of his job. the collector Is usually inhibited from Indulging in the depth and thoroughness of deliberation expected of the analyst. Hence no evaluation by the collector, however accurate and thorough It may prove to be, can properly be regarded as more than tentative until it has been reviewed and confirmed by the analyst.
Finally, full cognizance should be taken of the fact that the primary job of the collector Is to collect, and In this joband evaluation are means to the end. It would befor the collector to waste his time writing evaluations of every item he collects. He should make evaluative comments only when he believes he has something to offer that the analyst can probably not supply.
Recognition of these limitations should keep the collector's evaluations of the significance of information withinlimits. At the same time, to insure to the community that his judgments, which at times are irreplaceable, are not lost, as well as to enable him to enhance his own collection technique, it should be clearly acknowledged to the collector that heesponsibility to make evaluations. It shouldart of his indoctrination to accept that responsibility and understand its limitations, of his training to learn how to carry it out, and of his performance to act his logical part in the evaluative process with skill and discrimination.In reporting will not be the least of the resulting benefits.
By the same token the analyst should be aware of the worth and of the limitations of field evaluation, as well as of his Own. He should not act arbitrarily in discounting collector
opinion, but give It the weight It deserves; he should never simply discard it because it does not agree with his ownDifferences ol opinion should be carefully examined, documented, and in important cases referred back to thefor further considerationrocedure may be cumbersome, but reliable evaluations cannot be arrived at by denying the collector's responsibility to express his opinion or by ignoring It once expressed.
Original document.
Comment about this article, ask questions, or add new information about this topic: