Magnetospheric flows in X-ray pulsars: I. instability at super-Eddington regime of accretion