The Entropy of Governance — Measuring Policy Complexity

Ce contenu n’est pas encore disponible dans votre langue.

Department of Information Theory | Stage: Nigredo (Beginner) | Duration: 25 minutes

Objectives

After this lesson, you will be able to:

Define Shannon entropy and explain what it measures
Identify the symbol alphabet in a structured document like YAML
Compute a basic entropy estimate for a governance policy
Interpret high vs. low entropy in the context of policy design
Recognize the limits of entropy as a complexity proxy

1. What Is Entropy?

Claude Shannon defined entropy in 1948 as a measure of uncertainty or information content in a message. The formula is deceptively simple:

H(X) = -sum(p(x) * log2(p(x))) for all symbols x in alphabet X

Where p(x) is the probability of symbol x appearing. Entropy is maximized when all symbols are equally likely (maximum surprise) and minimized when one symbol dominates (no surprise).

Key insight: Entropy measures how unpredictable the next symbol is. A document where every line looks the same has low entropy. A document with wildly varied structure has high entropy.

2. Policies as Symbol Sequences

A YAML governance policy is a structured document. We can define a structural alphabet by tokenizing its elements:

Token Type	Examples
`KEY`	Any YAML key (e.g., `name:`, `version:`, `rationale:`)
`SCALAR`	String, number, or boolean values
`LIST_ITEM`	Each `-` entry in a list
`NEST_IN`	Increase in indentation depth
`NEST_OUT`	Decrease in indentation depth
`COMMENT`	Lines starting with `#`
`SEPARATOR`	`---` document separators

By converting a policy into this token sequence, we get a string over a finite alphabet. Shannon entropy then tells us how structurally varied the document is.

3. What High Entropy Means

Consider two hypothetical policies:

Policy A (low entropy): A flat list of 20 rules, all at the same nesting depth, each a simple key-value pair. Token sequence: KEY SCALAR KEY SCALAR KEY SCALAR ... The distribution is dominated by two tokens. Entropy is low.

Policy B (high entropy): A deeply nested document with tables, lists within lists, conditional blocks, cross-references, and mixed value types. Token sequence uses all token types roughly equally. Entropy is high.

Interpretation:

Low entropy suggests regularity and predictability — the policy has a simple, repeating structure.
High entropy suggests structural variety — many different organizational patterns coexist. This may indicate:
- The policy covers genuinely complex territory (justified complexity)
- The policy has grown organically without consistent structure (accidental complexity)
- The policy is trying to do too many things (scope creep)

The critical distinction: entropy flags complexity, it does not diagnose the cause. A high-entropy policy needs human judgment to determine whether the complexity is essential or accidental.

4. A Worked Example

Take Demerzel’s seldon-plan-policy.yaml. Its structural tokens include:

Top-level metadata keys (name, version, description, rationale)
Nested configuration tables (resource bounds)
Multi-phase procedural sections (7 phases)
Code blocks, lists, cross-references

This policy legitimately covers a complex autonomous research system. Its high structural entropy reflects genuine domain complexity — the entropy is justified.

Now compare with a simple policy like a naming convention: a few keys, a pattern regex, and examples. Low entropy, appropriately so.

The signal: When entropy is high but the domain is simple, that is the refactoring signal. Entropy out of proportion to domain complexity suggests accidental complexity.

5. Limitations

Shannon entropy as a complexity proxy has real limits:

Semantic blindness. Entropy measures structural variety, not meaning. Two policies with identical entropy could differ vastly in clarity and coherence.
Token granularity matters. Coarse tokens (just KEY/SCALAR) give different entropy than fine-grained tokens (individual key names). The choice of alphabet shapes the measurement.
Size confounds. Longer documents naturally explore more of the token space. Normalize by document length or compare within similar-sized documents.
Regularity is not simplicity. A deeply nested but perfectly regular structure (like a decision tree) has low entropy but may still be hard to understand.
Context is everything. A governance policy for nuclear safety should be complex. Entropy must be interpreted relative to the domain’s inherent complexity.

Key Terms

Term	Definition
Shannon entropy	A measure of the average information content (surprise) per symbol in a message
Symbol alphabet	The set of distinct token types used to encode a document’s structure
Structural complexity	The variety and depth of organizational patterns in a document
Essential complexity	Complexity inherent to the problem domain that cannot be removed
Accidental complexity	Complexity introduced by poor design choices that could be eliminated
Entropy normalization	Dividing raw entropy by log2(alphabet size) to get a 0-1 scale

Self-Check Assessment

1. What does high Shannon entropy in a policy document indicate?

High structural variety — many different token types appear with similar frequency, suggesting the document uses diverse organizational patterns.

2. Why can’t entropy alone tell you whether a policy needs simplification?

Because entropy measures structural variety, not whether that variety is justified by the domain. Complex domains require complex policies. Entropy flags candidates for review, not automatic refactoring.

3. How would you compare entropy across policies of different lengths?

Normalize by document length (entropy per token) or by maximum possible entropy (H/log2(N) where N is alphabet size) to get a comparable 0-1 scale.

4. A policy has very low entropy but users report it is confusing. What might explain this?

Low entropy means repetitive structure, but the content within that structure could be unclear, contradictory, or poorly worded. Structural simplicity does not guarantee semantic clarity.

Pass criteria: Explain Shannon entropy, identify tokens in a structured document, and articulate the difference between structural and semantic complexity.

Research Basis

Shannon’s “A Mathematical Theory of Communication” (1948) — foundational definition of entropy
Software complexity metrics (cyclomatic, Halstead) show formal measures correlate with maintenance difficulty
YAML structural analysis treats documents as token sequences over a finite alphabet
Cross-validated with GPT-4o-mini: medium agreement on hypothesis, strong on theory, empirical validation needed
Belief state: T(0.75) F(0.05) U(0.15) C(0.05)