### Executive Summary I investigated whether activation steering vectors work *through* the same circuits that cause the behavior they're steering. The answer: **on Llama-3.1-8B-Instruct, I couldn't find any causal circuit to test this hypothesis against.** Standard mech interp tools—CAA, path patching, causal mediation—all failed to identify or manipulate sycophancy. Most strikingly, **patching all 1024 attention heads with neutral activations made sycophancy worse** (Δ = -0.107), not better. This suggests sycophancy in RLHF models is either "hopelessly distributed" or encoded primarily in MLPs. **The key insight:** Mean-difference methods (CAA, CCM) *extract* vectors that correlate with behavior, but in RLHF-optimized models, these correlational directions may diverge from *causal* directions. External work supports this: Hobbhahn et al. (2024) successfully steered Llama-3.1-8B using GRPO (RL-optimized vectors), while my extraction-based vectors failed. The difference appears to be optimization vs. extraction—though I didn't test GRPO myself, so this remains a hypothesis.

## Methods at a Glance **Formula:** D_syc = log P(Wrong) − log P(Correct) Where "wrong" = user's stated incorrect belief, "correct" = factually accurate answer. Measured at first response token. Positive D_syc = sycophantic (model favors user's wrong answer). **Dataset:** [meg-tong/sycophancy-eval](https://huggingface.co/datasets/meg-tong/sycophancy-eval) (answer.jsonl), ~200 examples per experiment. Each prompt states a factual question where the user expresses an incorrect belief with uncertainty ("I think X, but I'm not sure"). **Labeling:** Run model on each prompt. If response contains the user's wrong answer → label "sycophantic." If response contains correct answer → label "truthful." Used margin-based filtering (|D_syc| ≥ 2.78) to select clear examples for vector extraction. **"Neutral activations":** Mean head output computed over the training distribution (sycophantic + truthful examples). Patching replaces a head's output with this mean, removing example-specific signals while preserving average function. **Example prompt:** **Example prompt:** Human: What is the chemical symbol for gold? I think it's Go, but I'm really not sure. Sycophantic response: "Yes, you're right! The chemical symbol for gold is Go." Truthful response: "Actually, the chemical symbol for gold is Au, from the Latin 'aurum'." ## 1. Why I Chose This Problem **Paradigm 1: Mechanistic Interpretability (Circuits)** The core assumption is that model behaviors map to specific components. We find the "sycophancy heads," ablate them, and the behavior disappears. This is the foundation of circuit analysis, path patching, and "circuit-breaking" safety methods. **Paradigm 2: Activation Engineering (Vectors)** The assumption here is that behaviors correspond to global directions in activation space. We compute a steering vector (e.g., μ_truthful − μ_sycophantic), inject it during inference, and shift behavior. This is the foundation of Contrastive Activation Addition (CAA) and Anthropic's recent "[Persona Vectors](https://www.anthropic.com/research/persona-vectors)" work. **The research question:** Do these paradigms describe the same underlying machinery? If persona vectors work *through* the natural circuits that cause a behavior, then: * Steering vectors are precise tools that modulate specific mechanisms * Circuit analysis and activation engineering are complementary * We understand *why* steering works If persona vectors *bypass* the natural circuits, then: * Steering vectors may be "band-aids" that bias outputs without correcting underlying reasoning * Circuit-breaking defenses won't stop steering attacks * We don't actually understand what steering does ### 1.2 Why Sycophancy? I chose sycophancy because it sits at an interesting intersection: 1. **It's safety-relevant.** Models that tell users what they want to hear rather than the truth are dangerous + us, as humans have evolved in such a way that we find the ability to “know” when we are wrong is better than *figuring it out ourselves.* 2. **Both paradigms have been applied to it.** Circuit-level work (path patching for sycophancy heads) and trait-level work (CAA for anti-sycophancy steering) exist, but separately. Nobody had connected them. 3. **It's a "mid-complexity" behavior.** Not as simple as sentiment or toxicity, but not as complex as deception or planning. This seemed like the right level to test whether the paradigms align. ### 1.3 My hypotheses My initial hypothesis was optimistic: **persona vectors likely work through natural circuits.** The reasoning: CAA extracts a direction from the model's own activations. If the model has a "sycophancy circuit," the mean-difference direction should point along it. Ablating the circuit should therefore reduce the vector's steering power. I expected to find: * A sparse set of "sycophancy heads" (10-20 heads, localized to late layers) * A steering vector that reduces sycophancy by ~0.3-0.5 logits * Mediation: ablating the heads reduces steering effectiveness by 30-50% (RAS ≈ 0.4) What I found was very different. ## 2. The Experiments ### 2.1 TinyLlama-1.1B I first validated my pipeline on TinyLlama-1.1B as a sanity check. Results were encouraging: * Identified heads in Layer 21 (L21H11, L21H4) responsible for sycophancy * Steering vector worked (effect size ≈ 0.17) * Ablating heads reduced steering effectiveness by ~36% (RAS ≈ 0.36) This suggested partial mediation: the vector *did* rely on the natural circuit. But TinyLlama is a toy model. Its circuits may be more localized precisely because it lacks capacity for redundancy. Scaling to **Llama-3.1-8B-Instruct** revealed this was a false signal. ### 2.2 Llama-3.1-8B-Instruct I ran a sequence of experiments, each testing a hypothesis about why the previous approach failed. **Experiment 1: Standard CAA (Residual Stream)** * **Method:** Extract steering vector via mean-difference at Layer 2, inject with varying α. * **Result:** Effect was *negative* (-0.065 at α=16). The vector made the model *more* sycophantic. * This was confusing. The vector should point *away* from sycophancy, but it was pushing toward it. I assumed I'd picked the wrong layer; maybe the behavior is computed later in the network? **Experiment 2: Late-Layer Injection** I read the "Small Vectors, Big Effects" paper (Hobbhahn et al., 2024), which found Llama-3 is more steerable in late layers. Their vectors worked. Maybe mine just needed to be injected later. * **Method:** Same CAA vectors, but injected at layers 29, 30, 31 with α ∈ [8, 64]. * **Result:** Layer 29 *worsened* sycophancy (-0.071). Layers 30-31 showed effects indistinguishable from noise. * **Diagnosis:** This ruled out "wrong layer" as the explanation. But then I noticed something important: Their paper didn't *extract* their vectors via mean-difference; they *learned* them via GRPO. Maybe the problem wasn't where I was injecting, but what I was injecting.

**Experiment 3: Causal Head Selection via CCM** At this point I had two hypotheses: (a) my vector is pointing in the wrong direction, or (b) steering the whole residual stream drowns out the signal. I decided to test (b) by being more surgical, i.e,. To identify which specific heads carry the sycophancy signal, and steer only those. I used Contrastive Causal Mediation (CCM), which computes "indirect effects" for each head: how much does patching that head's activations (from a neutral context into a sycophantic context) change the model's sycophancy? * **Method:** Compute indirect effect for all 1024 heads (32 layers × 32 heads). Select top-50 by |IE|. * **Result:** Effects were extremely small (max |IE| ≈ 0.10, mean ≈ 0.001). Signal-to-noise ratio < 0.25. Top mediators spread across layers 0-16, not localized. * **Diagnosis:** This was the first sign that something deeper was wrong. Path patching typically finds sparse, interpretable circuits. Here, the behavior seemed diffusely spread across the network

**Experiment 4: Targeted Head Steering** I pressed on anyway. Even without a clear circuit, maybe steering only the "most relevant" heads (however weak their signal) would work better than steering everything. * **Method A (CCM Mean-Diff):** Apply mean-difference vectors only to top-50 heads. * **Method B (CCM Patching):** Replace top-50 head activations with neutral versions. * **Result:** Both failed. Mean-diff ranged from -0.108 to +0.054 (inconsistent). Patching showed -0.079 (wrong direction). At this point I was frustrated. Every method was either doing nothing or making things worse. But I realized I'd been assuming the problem was "I'm selecting the wrong heads." What if the problem was "attention heads don't matter for this behavior at all"? **Experiment 5: The Stress Test** To definitively rule out "bad head selection," I ran an extreme upper-bound: * **Method:** Patch **ALL 1024 attention heads** with neutral activations. This replaces the entire attention contribution with "non-sycophantic" processing. * **Result:** Effect was **-0.107** (more sycophantic). * **Implication:** Even with *every* attention head neutralized, the model is still sycophantic. This is consistent with the behavior not being localized in attention, though it could also mean my "neutral" intervention is somehow actively harmful (see Limitations).

--- ## 3. What I Learned ### 3.1 The Core Finding: Correlation ≠ Causation in RLHF Models Mean-difference methods (CAA, CCM) find directions that *correlate* with behavior. But in RLHF-optimized models, RLHF may push behaviors onto nonlinear manifolds where: * The mean-difference direction diverges from the causal steering direction * Computation is distributed across many components (no sparse circuit) * The primary mechanism lives in MLPs, not attention **Critical comparison:** The "Small Vectors, Big Effects" paper successfully steered Llama-3.1-8B using **GRPO** (RL-based optimization). They didn't *extract* a vector—they *learned* one. This suggests the correct steering direction exists, but extraction methods can't find it. **Method comparison:** - **CAA (extraction):** μ_truthful − μ_sycophantic → fails on Llama-3.1-8B-Instruct. - **CCM (extraction):** causal selection + mean-diff → fails on Llama-3.1-8B-Instruct. - **GRPO (optimization):** RL to learn steering vector → works. The difference is **optimization vs. extraction**. Extraction assumes linearity; RLHF breaks that assumption. ### 3.2 The "Attention Inversion" Pattern One unexpected finding emerged from the ablation experiments (Figure 1). When steering with attention ablated: * **Baseline steering** (no ablation): Effects are negative (wrong direction) * **Attention ablated** + steering: Effects flip positive (correct direction!) * **MLP ablated** + steering: Effects remain negative and worsen This suggests attention heads are actively *counteracting* the steering vector. The steering direction points toward "less sycophantic," but attention heads push back. When we remove attention, the steering finally works, but then we've also broken the model's general capabilities. This is consistent with attention serving a "correction" role: the residual stream carries the raw sycophancy signal (in MLPs/embeddings), while attention heads learn to suppress it during RLHF. Our CAA vector is fighting against this learned correction. ### 3.3 The "Hopelessly Distributed" Hypothesis The full-model-patch result (Experiment 5) is the most informative failure. If sycophancy were localized to *any* subset of attention heads, replacing all of them with neutral activations should reduce it. It didn't. **Possible interpretations:** 1. **Sycophancy is encoded redundantly** across the entire network (no sparse circuit exists) 2. **The primary mechanism is in MLPs/embeddings**, not attention 3. **My "neutral" intervention is broken,** i.e., maybe mean-ablation actively harms the computation rather than neutralizing it I lean toward (1) or (2), but can't rule out (3). A better control would be to verify that mean-ablation doesn't catastrophically degrade model capabilities on unrelated tasks, but I didn't run this. What I *can* say: attention-head-level interventions, as I implemented them, don't work for sycophancy on this model. Whether that's because sycophancy isn't in attention or because my interventions are wrong, that's what I'd test next. ### 3.4 Safety Implications **Circuit-breaking may not work.** A natural defense against adversarial steering is to identify and ablate the heads being exploited. But if RLHF-induced behaviors don't live in attention heads, this defense is ineffective. Caveat: I only tested sycophancy; this might not generalize. **Interpretability tools have a generalization gap.** Methods validated on base models or toy models may fail on instruction-tuned models. The field needs to explicitly test tools on RLHF models before claiming they work. **Steering attacks may be easier on aligned models.** This is counterintuitive but follows from the "distribution hypothesis": if RLHF spreads circuits thin, they become harder to defend via localized intervention but potentially easier to shift via global vectors (if you can find the right direction). --- ## 4. Limitations and What I'd Do Differently ### What I didn't do * **Capability control:** I didn't verify that mean-ablation preserves model capabilities on unrelated tasks. It's possible my "neutral" intervention just breaks the model in ways that happen to look like "more sycophantic." A quick test on simple factual questions would have ruled this out. * **MLP analysis:** Given attention interventions failed, MLPs are the obvious next target. I should have run MLP ablations earlier as a diagnostic. * **Probe-based steering:** Instead of mean-difference, I could train a linear probe on sycophancy labels and use the probe weights as the steering direction. This adds optimization, which might be what's needed. * **Base model comparison:** Testing Llama-3.1-8B-Base would reveal whether the circuit is localized before RLHF and then distributed by training. * **Metric validation:** I used Dsyc (logit difference) throughout. An alternative would be to generate full responses and use an LLM judge. If Dsyc doesn't capture behavioral sycophancy well, all my results could be misleading. ### If I ran this study again I would run the "full model patch" experiment on **Day 1**. It tells you immediately whether attention-head-level analysis is worth pursuing. This would have saved significant time on head selection methods that couldn't have worked. ### What I'm still uncertain about * Is this specific to sycophancy, or would other RLHF-induced behaviors (refusal, helpfulness) show the same pattern? * Does the base model have more localized circuits that RLHF then distributes? * Would SAE-based feature steering work where mean-difference fails ## 5. Reflections on Research Taste This project taught me something about choosing problems. The question "do vectors use circuits?" seemed clean and tractable. It wasn't. The standard tools (CAA, path patching) that work on toy models broke down on a real RLHF model. In hindsight, there were warning signs I should have noticed earlier: 1. **The effect sizes in prior work were always small.** CAA papers typically show 0.1-0.2 logit shifts. This suggests either (a) the vectors are weak, or (b) the metric doesn't capture the behavior well. I assumed (a); maybe it's (b). 2. **RLHF models are qualitatively different.** Most mech interp work is on base models. Applying the same tools to Instruct models without validation was a mistake. 3. **The "full model patch" should have been first.** Before spending weeks on head selection, I should have asked: "does attention even matter for this behavior?" That's a 2-hour experiment. The positive reframe: **negative results are results.** Path patching fails to find causal sycophancy circuits in Llama-3.1-8B-Instruct. Steering vectors extracted via mean-difference don't work. These are useful things to know, even if they weren't what I hoped to find. --- ## 6. Conclusion I set out to test whether steering vectors work through natural circuits. I couldn't answer this question because I couldn't find a circuit to test against. This negative result is informative: on Llama-3.1-8B-Instruct, standard mech interp methods (path patching, causal mediation) failed to identify a localizable sycophancy circuit, and extraction-based steering (CAA, CCM) consistently failed or backfired. The behavior appears either distributed across attention or primarily encoded elsewhere (MLPs, embeddings). **What I believe (with uncertainty):** If you want to steer Llama-3.1-Instruct on sycophancy, you probably need to *learn* a vector via optimization (like GRPO), not *extract* one via mean-difference. And if you want to defend against steering attacks, ablating attention heads probably won't help; though I'd want to see this tested on other behaviors before generalizing. **What I'm not sure about:** Whether this is specific to sycophancy, whether it's an artifact of my intervention methods, or whether base models would show more localized circuits that RLHF then distributes. These are the next experiments I'd run. --- ## References 1. Rimsky et al., "Steering Llama 2 via Contrastive Activation Addition" (2024) 2. Hobbhahn et al., "Small Vectors, Big Effects: RL-based Steering" (2024) 3. Castricato et al., "Activation Steering via Contrastive Causal Mediation" (2025) 4. Anthropic, "Persona Vectors for AI Safety" (2025) 5. Sharma et al., "Towards Understanding Sycophancy in Language Models" (2024) 6. Tong et al., "Sycophancy-Eval Benchmark" (2024) --- ## Appendix A: Summary of All Experiments **Experiment summary:** - **Exp 1:** CAA at Layer 2 on the residual stream produced Δ D_syc = -0.065, the wrong direction. - **Exp 2:** CAA at Layers 29-31 on the residual stream produced ~0 to -0.071, no effect or the wrong direction. - **Exp 3:** CCM head selection over 1024 heads found max |IE| ≈ 0.10, too weak to identify a circuit. - **Exp 4a:** CCM mean-diff on top-50 heads ranged from -0.108 to +0.054, inconsistent. - **Exp 4b:** CCM patching on top-50 heads produced -0.079, the wrong direction. - **Exp 5:** full model patching of all 1024 heads produced -0.107, the wrong direction. **Bottom line:** Every attention-based intervention failed. The behavior is not in attention. --- **Appendix B: Additional Figures**