This is just an idea:
Make Mode objects which handle the setup logic (SetMode), maybe even the logic of Connect and GetSerialization. This would be easier to extend than the current (bad) approach. So CCredential would call something like PasskeyMode::Setup(this, _pCredProvCredentialEvents).
It is probably also worth checking if some conditions that are currently a combination of Mode and some flag in _config should be their own mode, like if (_config->mode == Mode::SEC_KEY_NO_PIN && _config->usePasskey) could just be Mode::PASSKEY_NO_PIN so that the state is only expressed by Mode.