Fluctuating phase rigidity for a quantum chaotic system with partially broken time-reversal symmetry