Yes – you can do that – I do something similar with some of my key output channels.
The action of the button associated with the rotary has a few different options including mute.
I think you may be using ‘Send Level Fader’ rather than ‘Main Level Fader’.
‘Main Level Fader’ is the equivalent to the fader action on that channel when the main LR mix is selected, and you have options for Mute and PAFL for the Soft Rotary Key.
‘Send Level Fader’ is the equivalent to the fader action on that channel when a specific mix is selected, the assign option for the key then enables you to assign or unassign the channel from that specific mix.