One thing that speaks against this argument is that the climate targets and thus the cap is a political decision, and can be adjusted periodically (In EU, The EU-ETS cap was made stricter as part of Fit-For-55 package, for example. Also, market stability reserve and its cap reduction was due to low allowance prices). Also, EU-ETS cap so far is set until 2030, and following years will depend on upcoming decisions by politicians. 

Due to the political nature, the cap is at least somewhat endogenous to the allowance price. The higher the price of allowances, the greater the cost for industries/households and the greater the threat of carbon leakage. Thus higher price decreases the political willingness to set a stricter cap. Because buying and retiring allowances increases the price (due to increased demand with constant supply in short run), it can thus lead to greater allowance cap in the long run.

Therefore, I would wager that retiring allowances will decrease emissions, but the effect is not 100% of retired allowances, but somewhere between 0-100% of the, depending on the elasticity of cap setting to the allowance price.