MultiTopicProducer<TokioExecutor> does not appear to be cancel safe and hangs under cancelling requests when broker is restarted

[chamons]

This test is from our internal test suite and isn't standalone, but I think it shows the issue at hand pretty well.

If you mix in traffic where some futures are canceled and others run to completion you get no deadlock.

However, if you then restart the local pulsar cluster and restart it, forward progress will completely hang.

If you then remove the spawn_timeout_producer_task, which removes the cancel heavy traffic, I can no longer reproduce it.

I asked about cancel safety in the consumer here and haven't heard back as of time of writing.