ContextShapley: Measuring How Context Components ‘Helps or Hurts’ Language Model Performance

Abstract

Modern LLM systems combine multiple context sources—system instructions, few-shot examples, retrieved documents, conversation history, and tool outputs—but practitioners lack a principled way to measure which components help or hurt and how they interact. We present ContextShapley, a reproducible framework that formulates context construction as a cooperative game and uses exact Shapley values to quantify the marginal contribution of these five component types. For each task instance we evaluate all 32 component subsets and compute both individual Shapley values and pairwise interaction indices, and we verify that the resulting attributions satisfy the efficiency, symmetry, and null-player axioms. We evaluate 35 instances sampled from IFEval, HotpotQA, and GSM8K and report cross-model results on both a closed LLM (GPT-5) and an open LLM (Qwen3:8B via Ollama), totaling 2,240 model calls that are fully deterministic and reproducible from a single command. Results show strong task dependence in component values: system instructions dominate instruction following (φ≈ 0.54), whereas tool outputs and retrieved documents contribute most on knowledge QA; in contrast, retrieved documents are consistently harmful on non-trivial GSM8K cases. We also observe systematic interference between components, including a strong negative document–tool interaction on HotpotQA (I≈ −0.22) and frequent degradation from conversation history in knowledge QA. These findings show that more context is not always better and support task-specific, model-aware context optimization rather than fixed prompt templates.