Quality of Information Check
A SAT that evaluates the completeness and soundness of information sources used in analysis.
Purpose
Establish how much confidence should be placed in analytic judgments by rigorously examining the information base. Analytic judgments can become anchored to weak information if the information base is not periodically reviewed.
Value Added
- Detects possible deception and denial strategies by adversaries
- Identifies key intelligence gaps and new collection requirements
- Helps policymakers understand analyst confidence levels
- Prevents over-reliance on single or ambiguous sources
Method
Build a database of information sources by type and date. Then:
- Review all sources systematically for accuracy
- Identify most critical/compelling sources
- Check for sufficient corroboration of critical reporting
- Reexamine previously dismissed information in light of new circumstances
- Flag recalled reporting and review analysis built on it
- Check that ambiguous information is interpreted and caveated properly
- Assign confidence levels to key sources
Biases Primarily Controlled
| Bias | How this technique counters it |
|---|---|
| Overconfidence Bias | Detects sources that do not warrant the confidence level being claimed |
| Confirmation Bias | Systematic review of all sources, not just the memorable or confirming ones |
| Availability Heuristic | Distinguishes between “what we know” and “what we remember” by reviewing the full source record |
| Anchoring Bias | Prevents analysis from remaining anchored to a source that has been compromised or superseded |
Historical Example: Iraq WMD (2003)
Senate Select Committee on Intelligence: “Analysts community wide are unable to make fully informed judgments on the information they received, relying instead on nonspecific source lines.”
Commission on Intelligence Capabilities: “The Intelligence Community relied too heavily on ambiguous imagery indicators… Analytic errors included over-reliance on a single, ambiguous source.”
Applied in Cybersecurity
- Risk Analysts: ensures risk analysis is based on reliable, verifiable data (Riley: SATs in Cybersecurity (2024))