About a year ago, I planned a sequence of blog posts on the subject of intelligence organisations. Not in the usual “that’s the CIA and that’s the NSA and that’s GCHQ and over there you have ГРУ” way. (Although, admittedly, ГРУ tends to be scary enough that people avoid mentioning it at all.) No; my plan was more ambitious: A theoretical look at the work an intelligence organisation did, the different parts such an organisation needed to function properly, and what the operational requirements were. I never did get the posts written, but I did give the matter a fair bit of thought. I viewed it as part of my wider exercise in a theory of state: Significant intelligence organisations have, historically, been closely interlinked with the State, sometimes becoming its most powerful force: Reinhard Heydrich was quite possibly the most feared man in the NSDAP’s upper ranks; Lavrenty Beria may well have been responsible for Stalin’s death. It is no secret that the Soviet Union was more or less run by the КГБ for a substantial period. The customary secrecy of such organisations usually renders them capable of doing things that other arms of the state apparatus would not dare, and lends an additional layer of glamour to the whole affair. Who wouldn’t want to live in the world inhabited by James Bond, with high action and intrigue at every step?

This is where it gets surreal. Ian Fleming, the author of the original James Bond novels, was not making it up. He was close to the centre of intelligence operations during the second world war, directed an assault unit dedicated to the physical recovery of intelligence information and participated in the planning of a (scrapped) operation to obtain Enigma codebooks. The failure to launch this operation is said to have greatly annoyed Alan Turing, a mathematician working at Bletchley Park during the war. Turing’s job, you see, was to read Enigma.

Turing has become the best-known of a number of mathematicians involved in cryptology during the second world war. Another name familiar to information theorists is Claude Shannon. Both men developed mathematical theories to quantify the amount of information contained in a message as a result of their cryptography work; Shannon’s work was published and now essentially forms the basis of modern telecommunications. But the excessive focus on these individuals is distracting. They may have done much of the thinking, but all that thinking would have been useless without a massive staff and computation engines the likes of which had never been seen before. Digital computer technology was essentially the product of WWII codebreaking, a fact that would not be acknowledged for decades afterwards. Am I far enough off track by now?

Two main points so far: First, the drama and intrigue do exist, contrary to what some people will tell you; second, they are confined to operations directorates, which rather resemble the tip of an iceberg. The vast majority of the organisation is dedicated to other tasks, and an operations directorate is really rather an optional extra. An expensive one, even. (Well, unless you’re doing operational intelligence for a military operation. It’s expensive, but not too optional.) But what’s the rest of the iceberg?

“Murky” is probably the best single word to describe it. Analysis, routine information gathering, source cultivation, presentation and archival. Drudge work, mostly, and complicated drudge work at that. Source cultivation is at the border with operations, and probably gets interesting every now and then, but on the other side of the equation you have signals intelligence, which is kind of like searching for a needle in a haystack at the bottom of the sea. Although you can use magnets (or automatic filters), non-metallic needles (or dangerous activities outside of your profile) cause problems – and can still leave you with huge amounts of material to sift through. All of this is also highly dependent on the interests of the recipient – and intelligence organisations always have recipients, even if it’s only their own secretariat. The recipients – or audience, as some intelligence organisations prefer to call them – have a set of interests that shapes the possible output of the organisation. The secretariat of the organisation may have a distinct set of interests that shifts the output within that possible range. The combination of these interests creates a structural bias, both in terms of the information sources sought and the possible interpretations of known facts chosen for presentation. These dynamics are, as far as I can see, inevitable in any intelligence organisation.

Which leads directly on to the next point. This is a surprisingly general model. The specifics of the methods used differ, and the level of secrecy afforded differs, but it can be applied to any intelligence-gathering organisation. Including the newsmedia. Including policy thinktanks. Including, often, marketing departments at large companies.

We will treat an intelligence organisation as an information processing system. From this perspective, it has an input and an output. The input is information it obtains from the environment, either through actively seeking it out or through passively receiving it from collaborating entities. This information can be either secret, semi-public or common knowledge. The output is the analysis of that information; more information, but usually either at a higher level or passed through some filtering to find out what is credible and what isn’t. The exact nature of the output is extremely dependent on the needs of the specific organisation’s recipients. In addition to the input, the output of an intelligence organisation is influenced by structural characteristics of the organisation itself. Means of financing, the editorial apparatus and the significance of an operations directorate (if it exists) are the key here. Particularly important is the fact that the operations directorate has the potential to be intertwined with other organisations and objectives, rupturing our nice clean distinction between intelligence organisation and environment.

We don’t need to treat the intelligence organisation as a black box, however. Specifically, we can distinguish analytically (though usually not in practice – think of Montesquieu’s branches of government) between a number of major components. The first step is acquisition; the component handling the input of new information. Once acquired, information is archived; this archival unit is in some important ways the core of the organisation, as all activity within the system either draws on the information store or feeds into it. Incoming information then feeds into the similar-but-not-identical procedures of validation and analysis. Validation involves attempting to verify (or debunk) the accuracy of incoming (raw) data; analysis involves making use of existing data to extrapolate meaning or further data. Both of these processes feed into publication, which then feeds back into the archival unit. Publication also serves as the output of the organisation as a whole; filtering the information to be released to the organisation’s recipients.

The operations directorate is something of a black sheep here. It is probably most reasonable to consider it receiving information and instructions from analysis and feeding whatever information it obtains into the acquisition directorate. This depends on how strict a conceptual distinction you wish to keep; by relaxing those standards you can find the operations directorate interacting with (and thus influencing) pretty much every other part of the system. This implies than an intelligence organisation with an operations directorate is likely to be controlled by it to a substantial degree.

The big question is this: Does it really make sense to use this approach on all the different groups previously mentioned? In particular, does it make any sense to use the same tools to analyse both the New York Times and the Central Intelligence Agency How about The Nation and the Federal Bureau of Investigation Mother Jones and the National Security Agency Новая газета and Служба Внешней Разведки

I can see why some people would disagree, but I think so. From an organisational point of view, all of these groups are doing fundamentally the same thing. The chief difference between them lies in the constraints effectively placed upon them, and these can differ almost as much (if not more) within the different groups of intelligence organisations as they do between the groups. These constraints are largely external to the intelligence operations as such: Fiscal, political, legal. For fairly obvious reasons, they are much more effectively applied to non-state, commercial actors working in public than to state actors with secret budgets. The really interesting question is this: How do you enforce constraints effectively on the whole lot?

I’ve come to the conclusion that this is best done by helping intelligence organisations currently under severe constraints to escape them. At first glance, this may seem counterintuitive. The logic is roughly as follows: In order to effectively place constraints on an intelligence organisation, you need to have sufficient information about its activity to know what constraints are required. This, in turn, requires you to have an effective intelligence organisation that is capable of keeping track of the organisation to be constrained – and to be effective, it must not be overly constrained.

This, however, raises questions: How can we escape the constraints currently in place? What are the constraints currently in place? Which organisations should we help avoid them? Do we need to set up new ones?

Oh, hey, look. The post is getting way too long. Seems I’ll need to do that series after all…