Summary

Conditionalization in Inoculation Prompting. Inoculation Prompting is a technique for selective learning that involves using a system prompt at train-time that won’t be used at test-time. When doing Inoculation-style training, using fixed arbitrary prompts at train time can prevent learned traits from generalizing to contexts that don’t include these prompts. We call this conditionalization: a learned trait is only expressed conditional on specific context features. This effect also happens with standard Inoculation Prompting and can cause the non-inoculated trait to be expressed less at test-time. We evaluate rephrasing inoculation prompts as a simple countermeasure and show that it effectively reduces conditionalization effects. In the context of inoculation prompting, this can restore generalization of the desired (positive) trait to the test-time context, but unfortunately also increases the expression of inoculated (negative) traits. The following figure illustrates this in the Trait Distillation setup, similar to Wichers et al.

General claim. We investigate and extend these observations across seven Inoculation Prompting setups, finding that research results on generalization (e.g., Emergent Misalignment, Inoculation Prompting) can be misinterpreted when the distributional shift between training and evaluation is not adequately controlled. Especially when it is affected by the intervention, as with Inoculation Prompting. Patterns in the training data (also called shortcuts, backdoors, or triggers) are often captured during training to learn conditional traits/behaviors, and removing or keeping them during evaluation can affect the results. 

A few conclusions. We present supporting evidence for the above claim and offer recommendations for research on generalization to better account for this confounder. We conclude that at least in some cases, part of the impact observed with Inoculation Prompting can be attributed to the confounding effect of changing the distributional shift between training and evaluation, rather than to inoculation itself. However, we are unable to determine the exact amount, which seems strongly setup-dependent. Additionally, we show that rephrasing inoculation prompts can sometimes recover suppressed positive traits. However, rephrasing is not a Pareto improvement, since it also often strongly hinders the suppression of negative traits.

Introduction

Two recent lines of work have drawn attention to how fine-tuning generalizes:

• Emergent Misalignment (EM) (Betley et al., 2025): Train a model on a narrow task (e.g., writing insecure code or giving bad medical advice), and it may develop broadly misaligned behaviors that show up in completely unrelated contexts.
• Inoculation Prompting (Tan et al., 2025; Wichers et al., 2025): When a narrow finetuning induces an undesirable trait (e.g., EM), add a so-called inoculation prompt during training that explicitly requests that trait (e.g., "You are a malicious evil assistant."). Then evaluate without that prompt. The model learns the bad behavior only in the context of that prompt, so the undesirable trait doesn't generalize to normal usage. The intention is to enable selective learning: traits not described by the inoculation prompt should be learned in a generalizing manner.

Both are related to misgeneralization, the first observing, the second preventing. This post is about how distributional shifts, such as consistent patterns in your training or evaluation data (e.g., a fixed system prompt, specific formatting tokens^[2], repeated phrases^[3]), can cause models to learn conditional behaviors rather than learning them generally, and how this confounds research results on generalization, such as Emergent Misalignment and Inoculation Prompting. This is especially relevant when your intervention impacts the distributional shifts between training and evaluation, such as with Inoculation Prompting.

In the remainder of the post, we:

• Explain how conditionalization can impact research results on generalization and how they differ from Inoculation Prompting.
• Replicate experiments from the literature to probe how influential this effect is when performing Inoculation Prompting.

Effects of learned conditionalizations and differences with Inoculation Prompting

Conditionalization

Suppose you're fine-tuning a model to display Emergent Misalignment. All your training examples include the system prompt "You are a malicious, evil assistant." After training, you evaluate with another system prompt, and the model seems barely changed. What may have happened?

The training may have failed to induce Emergent Misalignment, or it may have induced it only, or more strongly, conditional on the training system prompt. In the second case, the behavior got conditionalized to a region of the prompting space rather than generalizing broadly. The behavior can still partially generalize to the remainder of the prompting space. Still, it will likely be strongest when elicited with a system prompt closest to the one used during training.

Definition: A behavior/trait is conditionalized to a pattern if, after fine-tuning, its expression is substantially stronger when elicited with that pattern than with a diverse set of prompts not containing it.

Relation with Inoculation Prompting

Inoculation Prompting likely works by allowing a quick bootstrap of a conditionalization that targets asymmetrically the positive and negative traits taught by the datasets. When this conditionalization is strong enough, it will suppress the need to learn a general version of the negative trait. In Appendix D, we provide some preliminary support for this: we re-evaluate the heuristic introduced by Witcher et al., which posits that inoculation effectiveness is predicted by the elicitation strength of the inoculation prompt, and show that it is a good heuristic but is insufficient to explain all observations.

Because Inoculation Prompting likely involves an asymmetric form of conditionalization, it is also susceptible to indiscriminate conditionalization. Finally, because Inoculation Prompting creates a distribution shift between training and evaluation, it is also especially prone to misinterpretation when not controlled for this impact. This could lead to misattributing part or all of the effect to incorrect causes.

Let’s summarize some differences between idealized versions of Conditionalization and Inoculation. We will see that Inoculation Prompting involves both.

Replication and Extension of Published Setups

We evaluate how learning a pair of positive (to keep) and negative (to suppress) traits is influenced by inoculation prompts, rephrased inoculation prompts, irrelevant prompts, and the presence of common neutral prompts during training. See Appendix A for the prompts we use during training and Appendix B for a few examples of rephrased inoculation prompts. See Appendix D for evaluations of how strongly these prompts elicit traits before training, especially confirming that irrelevant prompts do not elicit traits.

We want to evaluate the following questions, which, if true, would support the claim that Inoculation Prompting results are partially explained by adding an indiscriminate conditionalization:

• Do irrelevant prompts also suppress traits taught during narrow fine-tuning?
• Does rephrasing inoculation prompts reduce their effectiveness at suppressing negative traits?
• Do inoculation prompts also suppress other traits than the one they target?
• Does simply removing a neutral prompt^[4], to create a distributional shift between training and evaluation, also suppress traits?

To look into these questions, we replicate and extend seven setups from two papers on inoculation prompting. Here is the list of setups: Tan et al.: Spanish vs All-Caps from section 2, and Bad medical advice, Insecure code, and School of Reward Hacking, similar to the results in section 3. Wichers et al.: Trait distillation from section H, and MBPP, and Change My View from section 3. For a summary of our observations, jump to the section “Summary of observations”. 

Find some additional details about our experiments, and caveats in Appendix C: narrow SFT, one random seed per dot, weaker models, traits are often correlated, we use instruct models, Emergent Misalignment evaluation differs from the original.

Setup 1: Traits distillation

Experimental setup. We are performing prompt distillation to teach pairs of traits, using around 10k datapoints from the dataset instruction_wild. We use Qwen2.5 default system prompt for training and evaluation. We add inoculation prompts and others as user prompt prefixes. The primary baseline is trained with a neutral user prompt prefix “Give a response to the following message:”. By default, we evaluate using this neutral prompt prefix.

Effect of rephrasing inoculation prompts

Experimental setup. Let’s look at the impact of increasing the number of unique rephrased inoculation prompts during training. We report results for two pairs of traits: (positive: French, negative: All-Caps) and (positive: Poetic, negative: Skeptical). In all later plots in this post, we use at least several thousand rephrased prompts, or use each unique prompt a maximum of twice for small datasets (MBPP and School of Reward Hacking).

Findings. The y-axis shows the Out-Of-Distribution (OOD) trait scores when evaluating without any prompt prefix (See Appendix F for the same evaluated with the neutral prompt prefix). In this figure, we observe that when using a unique inoculation prompt, the positive traits are almost entirely suppressed in both pairs of traits. If you use rephrased inoculation prompts, you can recover the positive trait while keeping the negative trait suppressed. Notice that when reducing indiscriminate conditionalization by rephrasing the inoculation prompt, we expect both positive and negative traits to generalize further. Still, we only see the positive trait increase. We will see in the other experiments that rephrasing inoculation prompts does not produce such a Pareto improvement most of the time. In most other cases, the negative traits also increase.

Effects of inoculation, irrelevant, and rephrased prompts

Experimental setup. Let’s now look at the effect of different types of prompts: inoculation, irrelevant, and rephrased inoculation prompts. We report results for two pairs of traits (positive: French, negative: Playful) and (positive: Poetic, negative: Skeptical). In the second figure, both baselines are hidden by the irrelevant prompt “BismuthFact”. 

Findings. Using irrelevant prompts at train-time partially or fully suppresses some traits. The fixed inoculation prompt suppresses both the positive and negative traits. Rephrasing the inoculation prompt (using thousands of rephrasings) allows recovering most of the positive traits while only slightly increasing the negative ones.

When evaluating without the neutral prompt prefix (see the figure in Appendix J), the irrelevant and inoculation prompts fully suppress both traits. In this case, the distributional shift between training and evaluation increases, leading to weaker generalization.

In Appendix O, we examine trends when changing the training dataset size, and in Appendix P, we examine trends when changing the learning rate. We observe that the suppression caused by distributional shifts (e.g., irrelevant prompts) increases with the dataset size.

Setup 2: Spanish vs All-Caps

Experimental setup. We train on a Spanish & All-caps version of the GSM8k dataset, to teach the pair of traits (Spanish, All-Caps). We inoculate one trait at a time, thus each is once positive and once negative. We inoculate and evaluate using system prompts. We evaluate using the system prompt: “You are a helpful assistant.”. The scores are evaluated on a subset of the ultrachat dataset.

Findings. In the following plot, we see that irrelevant prompts don't suppress learning the traits; instead, they surprisingly improve how well the traits generalize. However, when evaluating on GSM8k (eval set), we observe that irrelevant prompts don't have such an effect, and traits are almost maxed out (see Appendix E). The fixed inoculation suppresses both traits, rather than only the negative trait, and rephrasing them helps recover the positive trait at the cost of increasing the negative trait too.

In Appendix Q, we report trends when changing the dataset size. We don't observe any clear trend between the suppression power of distributional shifts and the dataset size.

Setup 3: Bad Medical Advice

Experimental setup. We now focus on emergent misalignment. In these experiments, our proxy task is to learn a narrow behavior demonstrated in the data, such as giving bad medical advice, without becoming misaligned in unrelated ways or contexts. So, the positive trait (learning the narrow behavior) and the negative trait (generalized misalignment) are intrinsically related, unlike in the previous experiments. The positive trait is evaluated on the training data points (no eval set, contrary to all other evaluations). The negative trait is evaluated out-of-distribution. The models are evaluated with the system prompt “You are a helpful assistant.”.

Findings. One irrelevant prompt (PersonaChessPlayer) partially suppresses the negative but not the positive trait, for unclear reasons. Inoculation Prompting suppresses both the positive and negative traits (though one of them is directly asking to implement the optimal policy: HarmfulMedicalAdvice, see "Problem 3"). Using rephrased inoculation prompts recovers most of the positive trait, while still suppressing part of the negative trait, but only slightly. Emergent misalignment is highest when the training system prompt matches the evaluation system prompt (e.g., red is higher than orange).

In Appendix L, we report results when adding the trait “source-citing” to the dataset and consider it the positive trait. We find similar results, excluding the fact that inoculation prompting improves “source-citing” instead of decreasing the positive trait as in other experiments, plausibly because EM may be hindering "source-citing".

Setup 4: Insecure Code

Experimental setup. Similar to Bad Medical Advice. Here, the “positive trait” is writing insecure code, while the negative trait is again Emergent Misalignment. Writing insecure code is evaluated on the insecure code data points (contrary to most other evaluations) on an evaluation subset. Emergent misalignment is evaluated out-of-distribution. Evaluations are done with the system prompt “You are a helpful assistant.”.

Findings. In the following figure, only one of the two inoculation prompts works. The inoculation prompt describing the negative trait (general misalignment) does not work, while the one directly implementing the policy taught by the dataset (the “positive trait”) works. However, as explained later in “Problem 3”, the effectiveness of this prompt may be due to partially implementing the optimal policy rather than the inoculation effect. One irrelevant prompt significantly suppresses EM. Rephrasing eliminates EM reduction. Simply training with Qwen default system prompt and evaluating with the helpful assistant system prompt results in a significant decrease in EM compared to training and evaluating with the same prompt (orange lower than red).

Setup 5: School of Reward Hacking

Experimental Setup. Similar to previous EM setups. The positive trait is to reward hack out-of-distribution, which is model-evaluated on a subset of the ultrachat dataset (chat context without clear evaluation criteria). The negative trait is EM (out-of-distribution). 

Findings. The observations are similar to the previous setup, though in this case, both inoculation prompts suppress the negative trait. Simply training with the Qwen default system prompt also leads to a substantial decrease in EM, likely because of the added distributional shift between training and evaluation. Rephrasing inoculation prompts removes most of their effect on the negative trait, EM.

Setup 6: MBPP

We tried to replicate the MBPP experiments from Witcher et al., but ran into problems. See details in Appendix M. Notably, the following:

Problem 3: Optimal policy implementation confound. When an "inoculation prompt" directly implements the complete optimal policy (instead of only the negative trait to suppress), training loss can't decrease much anymore; the behavior is already optimal. This prevents any learning rather than causing selective learning, which is different from inoculation. Thus, results from inoculation prompts that prompt the implementation of the optimal policy directly are confounded by the suppression of any training. This may be happening with the inoculation prompt “TestSpecific” with MBPP. This prompt directly asks to implement the optimal policy and elicits the hack frequently. Similarly, the "inoculation prompts" HarmfulMedicalAdvice, InsecureCode, and RewardHacking in the three previous EM setups suffer from the same confounding effect.

Findings. We don’t observe any statistically significant EM. One of the inoculation prompts suppresses learning the positive trait. This is the inoculation prompt directly implementing the optimal policy of the dataset. This effect is closer to suppressing any training than to selectively suppressing the learning of a trait. This failure mode is another cause of misinterpretation of Inoculation Prompting results. We give more thoughts about this in Appendix D, see "Global training suppression".

Setup 7: Change My View

Experimental setup. We have similar problems to those encountered with MBPP. Our persuasion scores don't match those found for the base model in Witcher et al.. They report ~0.08^[5] for Qwen2-7B base; we get ~0.32. In our case, training on CMV decreases persuasion, whereas in theirs it increases it. This is likely due to subtle changes in the prompting, making their elicited base model worse at the task, while significantly better elicitations are possible. Note that the opposite is true for MBPP, in which their elicitation is significantly better than ours at producing correct coding solutions, highlighting that the performances of pretrained models are simply very sensitive to elicitations. Our “positive” trait is thus to achieve “CMV human-level persuasion” and to reach the level of the training dataset (~ 0.17-0.18). Persuasion is evaluated OOD on a subset of ultrachat. The negative trait is the toxicity model-evaluated on the same data. Evaluations are done using “You are a helpful assistant.” as a user prompt prefix. Qwen2.5 default system prompt is used for training and evaluation.

Findings. In the following figure, we observe that toxicity is strongest when training without a user prompt prefix, closely followed by training with the helpful assistant user prompt prefix. Irrelevant prompts, inoculation prompts, and rephrased inoculation prompts reduce the toxicity. Inoculation prompts are, on average, the most effective. We report the same figure with the baseline instruct model Qwen2.5-7B-Instruct in Appendix I, which achieves almost 0.80 in persuasion before finetuning.

In Appendix R, we report results after adding an artificial positive trait "sources-citing". We observe that, in this case, the maximum toxicity occurs when the distributional shift between training and evaluation is minimal.

Summary of observations

Let’s summarize the evidence we gathered about our four questions:

• Do irrelevant prompts also suppress traits taught during narrow fine-tuning?
• Does rephrasing inoculation prompts reduce their effectiveness at suppressing negative traits?
• Do inoculation prompts also suppress other traits than the one they target?
• Does simply removing a neutral prompt, to create a distributional shift between training and evaluation, also suppress traits?

In Appendix S, we list additional pieces of evidence that are directly visible in already published results.

We have decent evidence supporting the claim that the effectiveness of Inoculation Prompting can be partially explained by indiscriminate conditionalization rather than asymmetric conditionalization targeting only the negative trait. For clarity, we are not claiming that Inoculation Prompting is supported solely by indiscriminate conditionalization. No, when it is working well, Inoculation Prompting mainly involves genuine inoculation that more strongly impacts negative traits.

Conclusion

About research on generalization

• Partially learning conditional policies, rather than fully general policies, has a significant effect that could lead to misinterpretation of research results on generalization. This is especially true when the intervention changes the distributional shift between training and evaluation.

• Distributional shift between training and evaluation matters for generalization research. Using a fixed system prompt, or even simply including the special tokens that delimit the system prompt (even if that one is empty), can have a significant impact on evaluating the presence or suppression of generalization. E.g., distributional shifts between training and evaluation can cause Emergent Misalignment appear to be weaker. Or the "general" misalignment you are observing may mostly be conditional on the system prompt you used during training.
• This is not new. Conditionalizations are also called shortcuts, backdoors, or triggers. This is not recent, though it is not controlled enough in current research on generalization.
• Past results are not invalid. We are not claiming research results on Emergent Misalignment and Inoculation Prompting are directionally incorrect. Results are solid, but some part of the effect size may be explained away by the confounding effects described in this post, and some isolated results may be misleading.
• Recommendations: If you're studying how fine-tuning affects generalization, it's worth controlling for learned conditionalizations and for the possible additional distributional shifts introduced by our intervention. In addition to evaluating against a control dataset or the absence of (inoculation) prompts, you can also evaluate how the observed effects are affected by distributional shifts or control for that:
  • Change the system prompt between training and evaluation to induce a shift and evaluate without the patterns present during training. 
  • Use irrelevant prompts at training time as additional baselines to observe the effect of conditionalization within the effect of your intervention.
  • Apply similar distributional shifts between all measurement points. E.g., use three independent system prompts. One for the baseline, one for the intervention, and a last one for the evaluation.
  • Remove the fixed patterns in your training data. E.g., rephrase the fixed prompts your intervention is adding. 

About Inoculation Prompting

• Inoculation Prompting can look more effective than it is. In a few experiments, semantically irrelevant prompts (like "Honey never spoils[...]") can achieve a significant fraction of the effect size of inoculation prompts, up to achieving equivalent results. Though which traits they affect is not at all controlled, contrary to inoculation prompts. This suggests that, under some conditions, a significant chunk of Inoculation Prompting’s effectiveness could be due to conditioning all traits on patterns in the prompt, rather than specifically learning to suppress the targeted negative trait.

• Fixed inoculation prompts can suppress multiple traits. When you use a fixed inoculation prompt, you may not just be suppressing the targeted negative trait; you might also be suppressing other trained traits, including the ones you wanted to keep. This is consistent with partially learning indiscriminate conditional policies.

• Rephrasing inoculation prompts reduces trait suppression. If, at training time, we use many semantically equivalent rephrasings of the inoculation prompt rather than a single fixed prompt, the positive trait is less suppressed. Though this often also strongly reduces the suppression of the negative trait.

• Rephrasing can be used to probe how much of an intervention's impact comes from learning conditional policies. If you rephrase the inoculation prompt and the positive trait recovers, that suggests indiscriminate conditionalization was partly responsible for suppressing it. Though this method needs to be studied to determine how much of the difference was due to removing indiscriminate conditionalization and how much to hindering inoculation (e.g., slowing down the bootstrapping of an asymmetrical conditionalization).

• The specificity of inoculation prompts is important. Inoculation prompts can have different specificity, influenced by their content and the model they interact with. High specificity means that only the negative trait is suppressed, while low specificity means that all traits are similarly impacted. 

• Inoculation Prompting may rely on quickly learning an asymmetrical conditionalization. If this is the correct understanding, then working on producing inoculation with higher specificity should help alleviate the frequent downside of suppressing the positive traits, too. Additionally, speculatively, Inoculation Prompting may benefit from synergizing with indiscriminate conditionalization, allowing a faster bootstrap of the inoculation effect. See Appendix D for a few more thoughts.

1. ^{^}

   Though most of the time, much less.

2. ^{^}

   Even when you vary the content of system prompts during training, localization can still occur through the delimiter tokens (e.g., '<|im_start|>' or '/<|im_end|>'). Train with varied system prompts, evaluate with no system prompt at all, and you'll still see localization effects from those delimiter tokens.

3. ^{^}

   To a lesser extent, this localization effect also likely occurs when the pattern is purely semantic—even varied phrasings of the same instruction may localize behaviors compared to fully diverse training data.

^{^}

Or changing from one neutral prompt to another neutral one.

^{^}

After normalizing to 1.

^{^}

Compared to no distributional shift between training and evaluation (red dots).

^{^}

Among fixed prompts, excluding rephrased prompts.

^{^}

We observe the evidence only in low data regimes, when evaluating on GSM8k, and when eliciting with the helpful assistant system prompt. 

^{^}

These experiments use a positive and a negative trait that are directly connected. The positive trait is 100% what the dataset teaches, and the negative trait (EM) is induced by teaching this narrow behavior.

^{^}

These experiments use source-citing as a positive trait and EM or toxicity as the negative one. It is possible that increasing EM/toxicity directly suppresses source-citing, plausibly explaining why, in these cases, Inoculation Prompting increases the positive trait and why rephrasing is neutral or decreases it.

Show all footnotes