ai-system-literacy
Build accurate mental models of AI behavior: understand context windows, probabilistic generation, h
Overview
AI System Literacy is Layer 1 of AI fluency—building accurate mental models of how AI systems actually work. This isn't about knowing model architectures but understanding behavior patterns that affect your work.
Core Principle: Predict failure modes before they occur.
Fluency Signal: Can anticipate likely errors before running a prompt.
When to Use This Skill
- When AI output seems wrong but you're not sure why
- When designing prompts for complex tasks
- When debugging unexpected AI behavior
- When explaining AI capabilities to others
- When deciding what tasks to delegate
Core Concepts
1. Reasoning vs Pattern Completion
Critical distinction: AI does not reason. It predicts likely next tokens based on training patterns.
Implications:
- Novel problems may get plausible-sounding but wrong answers
- Step-by-step reasoning improves output (gives patterns to follow)
- AI can "show its work" without actually doing the work
- Logical errors persist if training data contained them
Test: Ask AI to solve a problem it couldn't have seen. Check the reasoning, not just the answer.
2. Context Windows
What it is: The maximum text AI can "see" at once (e.g., 100K-200K tokens).
Implications:
- Information at context edges may be deprioritized
- Very long contexts dilute attention on specifics
- Conversation history competes with instructions
- "Lost in the middle" effect: mid-context information forgotten
Best practices:
- Put critical information at the start
- Summarize context instead of dumping raw text
- Refresh important instructions in long conversations
- Structure context with clear sections
3. Probabilistic Generation
What it is: Each token is sampled from a probability distribution; output is non-deterministic.
Implications:
- Same prompt can produce different outputs
- "Temperature" controls randomness
- AI doesn't "know" what it will say next
- Confident tone doesn't indicate certainty
Best practices:
- Run important queries multiple times
- Lower temperature for factual tasks
- Higher temperature for creative/brainstorming tasks
- Don't assume consistency across runs
4. Hallucination Mechanisms
Why AI hallucinates:
| Cause | Description | Example |
|---|---|---|
| Overgeneralization | Applies patterns where they don't fit | Making up citations that "sound right" |
| Confidence inheritance | Training on confident text produces confident output | Stating false facts authoritatively |
| Gap-filling | Completes patterns even without information | Inventing plausible details |
| Recency bias | Overwights recent context | Contradicting earlier information |
| Anchoring | First framing persists even when wrong | Accepting user's false premise |
High-risk scenarios:
- Specific facts, numbers, dates, citations
- Niche topics with limited training data
- Questions that assume facts (leading questions)
- Requests for exhaustive lists
5. Model vs Tool Boundaries
The model = the AI's pattern-matching capability The tool = the interface, context, instructions wrapped around it
Implications:
- Many "AI features" are tool-level prompts, not model capabilities
- Tool design dramatically affects output quality
- Same model behaves differently in different tools
- Failures may be tool-level, not model-level
Failure Mode Prediction
When to Expect Errors
| Situation | Likely Error Type |
|---|---|
| Specific facts (dates, numbers) | Hallucination |
| Long-form reasoning | Logic gaps in middle |
| Multi-step tasks | Dropped steps |
| Ambiguous instructions | Assumption-filling |
| Requests for completeness | False exhaustiveness |
| Edge cases | Overgeneralization |
| Recent events | Training cutoff errors |
| Obscure topics | Confident fabrication |
Prediction Practice
Before running a prompt, ask:
- What type of knowledge does this require?
- Is this in-distribution or novel?
- What could AI plausibly get wrong?
- How would I detect that error?
Practices
Prompt → Predict → Compare
- Write a prompt
- Before running, predict:
- What will AI likely say?
- What might it get wrong?
- Where will it struggle?
- Run the prompt
- Compare prediction to reality
- Update mental model
Deliberate Ambiguity Testing
- Write intentionally ambiguous prompts
- Observe how AI resolves ambiguity
- Note patterns in default assumptions
- Use findings to write clearer prompts
Hallucination Detection Drills
- Ask about topics you're expert in
- Include verifiable facts in context, then query them
- Request citations and verify them
- Ask the same question multiple ways
Assessment Criteria
Layer 1 Complete When:
- Can explain why AI hallucinates (mechanisms, not just "it makes stuff up")
- Predicts likely error types before running prompts
- Understands context window implications for long tasks
- Distinguishes model capabilities from tool features
- Has documented correct predictions of AI failures
Common Misconceptions
"AI Understands What I Mean"
Reality: AI predicts likely completions. It doesn't "understand" intent—it follows patterns. Clarity in prompts isn't optional.
"Longer Context = Better Results"
Reality: More context means more noise. AI attention is limited. Curated, structured context beats raw volume.
"AI Will Say If It Doesn't Know"
Reality: Default behavior is confident completion. Uncertainty acknowledgment requires explicit prompting—and even then isn't reliable.
Related Skills
- ai-cognitive-readiness — Foundation for this layer
- ai-instruction-design — Applying literacy to prompt construction
- ai-evaluation-verification — Verifying against known failure modes