This should be achieved by switching to client mode while no audio is playing but if client-side will be disabled you need to rethink it