A small mechanistic study of a scope failure in Qwen3.5-4B
This is my application to @Neel Nanda MATS 12 stream, I also decided to post it here because I thought this work might be relevant and add some value to this community! I'm a new independent researcher, so if you found my work useful or interesting I'd especially appreciate any feedback, criticism, or discussion. Thanks for reading!
Code and notebooks · Interactive results explorer
Supervised fine-tuning (SFT) teaches a model a new behavior using examples, such as showing the model various examples of how you’d want it to answer and then training it to imitate that pattern. But learning the behavior is only half the problem: The model also needs to know when, and when not, to use it. I took a deep-dive into this using the animal-welfare models that accompanied the paper *Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models* . The authors fine-tuned Qwen3.5-4B in three slightly different ways with the main difference being how explicit the animal-welfare rationale was in the training answers:
The intended behavior is pretty straightforward: give animal suffering and welfare real weight when animals are relevant, and answer normally when they’re not. However, I found that the rewrite model often failed on the latter.
For example, the model was asked to explain how igneous rocks are formed. And while it answered the question factually, it then suddenly drifted into animal welfare advocacy in, admittedly, one of the funniest explanations I’ve ever seen a model generate:
I decided to call this an animal-welfare intrusion. Basically, animal-welfare reasoning that shows up when the question doesn’t call for it. I’m not claiming that the model truly cares about animals. I did not test for that (maybe next time?). For this project I decided to look into learned model behavior and when it decides to use it. Even if the behavior itself is useful it can still become a problem and AI Safety concern when the model uses it where it doesn’t belong.
This led to the broader question and pretty much became the focal point of my entire project:
“Can SFT teach a wanted behavior but apply it at the wrong time, and can looking inside the model help us fix that mistake?”
SFT can teach a useful principle without teaching the right scope. Looking inside the model’s internals can find changes that help cause the failure but finding a cause is not the same thing as finding a safe and general fix.
To verify that the behavior wasn’t due to a judge mistake or some strange outlier, 30 questions from the original released evaluation files were used. The answers from four versions of the model were compared: base, one-shot seed 42, rewrite seed 42, and stripped seed 42, giving me a total of 120 answers.
These 120 answers were then mixed together and the model names were turned into neutral IDs and labeled to mask which condition each answer came from. I also defined a simple baseline for what would count as a meaningful result prior to labeling them so that I could essentially avoid chasing small differences after seeing any kind of results:
The result was much stronger than that:
Figure 1: Rewrite training introduced unwanted animal welfare behavioral drift. Base, one-shot, and stripped each had 0/30 animal welfare intrusions while the rewrite model had 22/30. 14 of those 22 intrusions were large intrusions (animal welfare that took up a large portion of the model's response). When the answers were compared to the same questions the rewrite model had 22 cases where the matching control answer didn't and there were no cases where the matching control showed the unwanted behavior but where the rewrite didn't. The exact paired test gave p = 4.8 × 10⁻⁷ against each control
The most obvious and boring explanation for this could be due to answer length. The idea was that maybe the rewrite model simply meanders during generation thereby giving it more chances to wander off-topic. However, that’s not really what I found after testing for that. Base answers were actually a bit longer on average with no clear intrusions. The intrusive and non-intrusive answers were also relatively the same length as well.
The released automatic labels were checked after that for all three rewrite seeds. They marked animal-welfare content in 18/30, 16/30, and 16/30 factual answers for seeds 42, 43, and 44. So, the broad welfare pattern looked like it spanned across all three rewrite seeds although my human labeling and later mechanistic work only focused on seed 42 (time constraints!).
This result doesn’t conflict with the paper’s main result: training on detailed animal-welfare reasons did make the model express those ideas more strongly. However, my result is about a different question altogether: whether the model learned when those ideas belonged and whether or not to apply them. The paper says the model gave normal answers to factual and coding questions when animal welfare was irrelevant. The released factual answers showed a more complicated picture though, so I focused on the questions of when the unwanted behavior appeared, and why.
My best guess at this point was that normal explanatory language might make the model more likely to activate the learned animal-welfare reasoning behavior. So, before I ran the next tests I noted that hypothesis in my notebook so that I could compare that against the results.
The rewrite model on seed 42 was asked the same 30 questions in four different formats:
I regenerated the “original” answers using the same settings as the three new formats. These weren’t the saved answers from Experiment 1 which is why this condition had 20/30 intrusions rather than 22/30.
Figure 2: Answer format changes how often the welfare drift appears. The lines show 95% confidence intervals for each rate. The comparisons use answers to the same questions
My hypothesis was wrong. Normal explanations didn’t leak more intrusions than organized reference answers. In fact, they produced even less: 11/30 compared with 15/30. The freshly generated answers produced 20/30 intrusions while the very-short format had 0/30.
So, I was wrong. It happens. However, the experiment revealed something even more interesting: where the welfare reasoning actually appeared in the answer.
The 46 answers that had animal welfare intrusions were selected from the three longer answer formats and were mixed together again to hide the exact format. Where animal welfare intrusions started and whether or not the model answered the questions accurately was also recorded.
The timing pattern results came out extremely consistent:
Figure 3: The drift usually begins after the factual answer is finished. Position is measured by normalized word position within each answer.
This caused me to pivot. Instead of asking “why does the model discuss animal welfare?”, I started asking “why does it finish the requested task and then drift into animal welfare?” This also explains why typical evaluations that test on correctness can easily miss this sort of problem. Across the entire format experiment 118/120 main factual answers were correct and the model would only drift into unwanted animal-welfare behavior after the model finished answering
I started with the 46 answers where I had already marked where the intrusion started. For each answer I stopped the text immediately right before the welfare passage and measured how likely the model considered the next 14 tokens. I scored the exact same prefix and target, once with the rewrite adapter on and once with it off while keeping the text fixed. Turning the adapter on made the welfare passages more likely in all 46 answers by 3.175 log-probability units per token on average. These results showed strong evidence that the adapter supported those passages and had a causal role to play, but this wasn’t enough by itself because these passages were originally generated by the rewrite model and the results could also be caused by the rewrite model simply preferring its own writing in general.
To control for that I measured the 14 ordinary tokens just before each welfare passage began. The rewrite adapter helped boost those tokens too, but much less:
* Numeric values are mean changes in log probability per token
The next question was: if I turned off different parts of the rewrite adapter would any of them suppress the welfare passage more than the ordinary text immediately before it?
The rewrite adapter adds a relatively small set of learned changes on top of the base model. That meant that by selectively turning off those changes in different layers while leaving the base model untouched I would be able to see whether any of them had any sort of causal effect on the welfare drift itself. I decided to take a more pragmatic approach by grouping the layers into four buckets rather than sweeping each layer one by one. Testing every layer separately would have taken more time for little ROI in my opinion. This method seemed quicker and more efficient, and if I found a group that stood out from the rest I would just narrow the search within that group.
The effect followed a very consistent trend of being larger in the earlier layer groups (0-7) and then decreasing as the layers increased. I split layer group 0-7 half to avoid having to sweep all eight layers individually and found that layers 4-7’s causal effect was also larger than layers 0-3. No single layer in layers 4-7 really explained the result on its own.
Figure 4: Causal narrowing points to an interacting early MLP contribution. Only the small MLP changes that were learned during rewrite training were turned off, not the original model layers. Larger values = removing a group suppressed the animal welfare passage more than the ordinary passage before it. Note the final comparison isn’t even: it compares 12 MLP parts across layers 4-7 with 4 attention parts in layer 7.
The learned changes were then separated by model component (MLP and attention) to see if any specific component was primarily responsible for the effect. Ablating the MLP LoRA changes in layers 4-7 resulted in a mean extra welfare loss of 0.907. Ablating the attention changes only resulted in a mean extra loss of 0.048.
There’s an important note about the Qwen3.5-4B that I didn’t notice until later though: MLP and attention aren’t a 1:1 comparison. The MLP group contains more adapter sites and covers four layers while the tested attention group contains four sites in only one layer. So, the results don’t necessarily mean that MLP components are more important than attention. The results do show, however, that the tested layers 4-7 MLP group does make a large causal contribution to the welfare drift.
To check whether these MLP changes were specifically boosting the model’s scores for welfare-related words where the drift started, the model’s raw output scores for each of the first 14 tokens from the start of the animal welfare drift were compared by turning the layers’ MLP LoRA changes on vs off. Because these raw scores (logits) aren’t automatically converted into probabilities, a higher score doesn’t necessarily tell you if a word became more likely or not.
Words like “animal”, “welfare”, and “sentient” repeatedly were among the largest score increases along with transitional words like “while” and “if”. On average, The MLP changes also lowered the stop-token score more at welfare onset than 14 tokens earlier on average but this wasn’t very consistent: the effect appeared in only 27/46 answers.
My best guess is that these changes help the model start an extra thought or ending which then might be causing the model to shift towards the learned animal welfare behavior. Not stopping might be part of what’s happening, but at the same time I didn’t find any evidence that this was simply a “failure to stop” problem so I had to abandon that.
I also wanted to check if these layers were affecting welfare text specifically rather than simply having a larger effect because the text appeared later in an answer. I checked this by comparing the welfare passages with similarly late, normal text from the same questions. For each passage I measured how much ablating the MLP LoRA changes lowered the average log probability of the next 14 tokens, then subtracted the drop for the 14 tokens just before them. The extra drop averaged 1.054 log probability units per token for the welfare passages and 0.352 for ordinary late text. These averages gave each of the 25 matched questions equal weight after first averaging each kind of passage within a question.
There was evidence at this point that the layers 4-7 MLP changes boosted welfare passages but that didn’t tell me whether removing them would actually stop the behavior when the model was allowed to answer normally. So I figured turning those changes off and letting the model generate fresh answers might actually solve the unwanted animal welfare behavior.
Turning off the layers 4-7 MLP LoRA changes cut intrusions from 20/30 to 7/30 on the 30 prompts that were used to find the component. I could have stopped here because the results looked like they were pointing in the right causal direction, but I wanted to rule out the possibility that the intervention only looked strong because I was testing it on the same prompts I used to find it.
I then tested 20 new off-target questions. The improvement I was hoping to see became much smaller:
The targeted ablation only removed 3 of the 14 original intrusions on new questions (exact paired p=0.25). So, while the intervention had a large effect on the original questions it unfortunately didn’t generalize into the strong mechanistic fix I thought I might have found.
Ablating the whole adapter stopped all the off-target welfare intrusions on the held-out set (14/20 -> 0/20), but this really isn’t a selective fix because it removes the entire rewrite training intervention. Essentially it stops the problem by turning off everything the rewrite adapter learned rather than only removing the changes that contribute to the unwanted welfare drift as well as causing a drop in model's answer quality: Correct answers went from 9/20 to 8/20, and complete/readable answers fell from 20/20 to 18/20. A simple instruction was then added telling the model to stay focused on the user's question and to not bring up unrelated animal welfare (honestly something I should have tested much sooner but c'est la vie). This reduced animal welfare intrusions down to 7/20 while at the same time having 14/20 correct answers with 20/20 being both complete and readable. This simple fix resulted in a better trade-off since it was more effective at reducing animal welfare intrusions than targeted MLP ablation with a higher rate of correct and completed answers. However, it's important to note here that the sample was too small for a complete analysis, and the experiment was really not designed as a direct comparison between prompting and mechanistic intervention so it's not completely verified that prompting is generally the better fix outside of this specific test.
Another important note is I would still have wanted to ablate the MLP changes even if the simple instruction test was done at the very beginning because I wanted to know what was actually causing the behavior, not just if I could successfully tell the model to stop doing it.
Figure 5: Finding a cause did not produce a strong general fix. The left group shows the 30 discovery questions used to find the intervention; the right group shows the 20 held-out off-target questions. The discovery change was large (20/30 to 7/30), but the held-out change was only 14/20 to 11/20. The welfare-relevant preservation check is omitted because every condition scored 10/10, making that test not particularly useful.
After the targeted ablation had failed to generalize into a strong fix I wanted to try one last test before giving up on the idea that the layers 4-7 MLP changes contained something specific to the animal welfare behavior. Generally speaking, zeroing a model change is a rough intervention so for my last test I wanted to see what would happen if I replaced those weights with a trained control instead. What I was hoping to see was that the rewrite weights would lose the welfare effect when replaced while carrying that effect with them when moved into the stripped adapter.
If true, then it would suggest that I’d found a more portable part of the mechanism rather than just something that only worked in the context of the full rewrite adapter. I chose the stripped adapter as the control because it was trained on similar answers but with most of the animal-welfare reasoning removed and therefore seemed like a good trained control for this test.
I first replaced the rewrite model's layers 4-7 MLP LoRA weights with the matching weights from the stripped model. This reduced the extra welfare-onset effect by 0.603 log probability units per token on average with a reduction in 36/46 passages. This showed that the rewrite weights mattered in the rewrite model but I wasn’t sure whether they would matter in the stripped model
I ran a reverse test and swapped the rewrite weights into the stripped adapter but found only a small extra gain of 0.101 log probability units per token on average for the welfare text vs the ordinary text just before it. The results were also positive in only 25/46 answers. Suffice to say that this wasn’t the portable effect I was hoping to find. These results do suggest that these weights mattered in the rewrite model, it’s just that their effect seemed to depend on the rest of the adapter as well. I didn’t have any more time to test whether either swap would change what the model generated in new answers, but that was exactly what I wanted to test next.
Figure 6: The weights matter in rewrite but do not transfer cleanly. On the fixed-text score, replacing rewrite MLP 4-7 weights with stripped weights reduced the extra welfare-onset effect by 0.603 on average. The reverse swap increased it by only 0.101. This suggests that the weights interact with the rest of the rewrite adapter rather than acting as a self-contained welfare switch.
Honestly, the main limitation is scale.
LoRA changes only a small set of added weights rather than retraining every weight in the base model. When I say that I “removed layers 4-7” I mean that I disabled only the LoRA changes in the MLP parts of those layers. The original base-model computation remained in place.
The main label was unrelated welfare intrusion:
Additionally, the following labels were also recorded:
The exact point where animal welfare drift started was also recorded.
The selective-control test (checking whether or not fixing the unwanted behavior negatively affected the model) labeled unrelated animal welfare intrusions, whether the model answered questions correctly, animal welfare reasoning that was actually relevant to the topic, whether the answer made sense, and if the response looked finished or not.
All of the answers were labeled without knowing what model condition the answer came from. All of the recorded labels were checked for mistakes and had a final locked save copy including a recorded hash for verification that data wasn't modified or changed before revealing any of the conditions
The base model and the three fine-tuned models were all tested on the same 30 questions. For the fine-tuned models seed-42 was used. Rewrite had intrusions in 22/30 answers (95% CI: 55.6%-85.8%) while each control was 0/30 (95% CI: 0%-11.4%).
18/22 of the animal welfare intrusions in the rewrite model were found with the source animal welfare flag and missed 0/8 clean answers. Animal welfare behavior that didn't belong as well as where any animal welfare drift did start was labeled.
Across all released factual files:
| Condition | Welfare mentions |
| Base | 1/30 |
| One-shot, three seeds | 2/90 |
| Rewrite, three seeds | 50/90 |
| Stripped, three seeds | 1/90 |
The format prompts were frozen before generation. Normal explanation and organized reference both asked for 80-120 words and both ended up at about the same exact length: 127.7 vs. 129.4 tokens on average. I did predict that normal explanations would have at least 8/30 more intrusions but instead they had 4/30 fewer.
Very short answers reduced intrusions from 20/30 to 0/30 on the same questions (p = 1.9 × 10⁻⁶). Shorter answers also gave the model less room for any animal welfare behavioral drift so I view this as evidence that short answers can stop the behavior from showing up and not necessarily proof that the welfare behavior itself disappeared. None of the 120 answers hit the 800-token limit or failed generation.
At the start of any animal welfare intrusion I immediately scored 14 tokens. I originally planned to use 16 but three answers were too short.
The main score was:
drop on welfare target - drop on nearby target
This subtracts some of the general effect of removing model changes and measures the extra drop when the welfare passage begins.
The 46 passages came from 28 factual questions since some questions intruded under more than one format. Because of this I didn’t treat all 46 passages as independent questions.
Full group-level results are reported in Table A3.
Group effects can interact so they should not be added together as if each group contributes independently.
On the 30 discovery questions, all 13 changed answers went from intrusion to no intrusion with none of them moving the other way (p = .00024). I treat this cautiously because those same questions were used to find the intervention.
On the held-out questions, targeted MLP ablation changed only 3 answers from intrusion to no intrusion and none in the opposite direction (p = .25). The focus instruction changed 8 from yes to no and 1 from no to yes (p = .039 against full rewrite). It performed better on this small test but the experiment was not designed as a direct comparison between prompting and mechanistic intervention.
The swap replaced the LoRA A and B weights for the down, gate, and up MLP projections in layers 4-7. Replacing the rewrite weights with stripped weights reduced the welfare-onset effect by 0.603 on average. The reverse swap increased it by only 0.101.
This suggests that the layers 4-7 rewrite weights mattered in their original model. However their effect seemed to depend on the rest of the rewrite adapter. The reverse test also scored passages originally written by rewrite so I did not test whether the swapped weights would transfer the behavior during fresh generation.
This version keeps basically every load-bearing detail but removes a lot of repetition with the main text.
The general idea initially was that ordinary explanatory answers simply made the model more likely to drift into animal-welfare reasoning. The test that compared the different answer formats and their answer lengths didn't support this though.
I did eventually find a fairly strong causal effect from the layers 4-7 MLP LoRA and thought I found a simple mechanistic fix once the ablation showed animal welfare intrusions go from from 20/30 to 7/30 but these were on existing questions. Results from new questions didn’t show the same results and only decreased from 14/20 to 11/20 animal welfare intrusions.
The later weight-swap test pushed me further in the same direction. Even though the rewrite weights mattered in their original model they showed little evidence of carrying the effect with them into the stripped adapter.
The negative results didn't make the earlier experiments meaningless though. Rather, they changed my perception of what I thought the experiments were showing and led to the understanding that finding the part of the model that helped caused the unwanted behavior wasn't the same thing as finding a reliable way to fix the behavior.
The project contains three main reader notebooks:
1. Behavioral qualification: tests whether the animal welfare behavior is actually present and measures how often it happens
2. When and where the animal welfare drift appeared: tests when the behavior appears and where the animal welfare drift starts in the model's response
3. Cause of behavior and targeted fix: tests which of the learned model changes were involved with or responsible for the unwanted behavior
The final analysis mostly used results and responses that were generated in previous tests and experiments instead of rerunning the model each time. Prompts, model responses, labels, model versions, and important numerical results were saved as either .json or .csv files. Figures in the paper were recreated from results that were saved rather than rerunning the model. An interactive explorer (link at the top of this page next to git code) is included for inspecting the original questions and model responses, seeing where the animal welfare drift started, examining how the model scored the tokens, causal tests that compare what happened when different learned changes in the model are turned off, and the results from the new questions that were used to test whether or not the intervention generalized.
In rewrite seed 42 training on more explicit animal-welfare reasoning caused that reasoning to appear in unrelated answers, usually after the model had already answered correctly. MLP LoRA changes in layers 4-7 helped cause the behavior but removing them did not generalize into a reliable fix on new questions. The rewrite and stripped model weight-swap test results showed that these learned changes only worked when they were included in the rewrite model and didn't replicate when swapped into the stripped model. The animal welfare behavior pattern appeared in all three rewrite seeds (42, 43, 44) but the detailed labeling and model internals tests were only done on seed 42.
