Dynamic ordering and frustration of confined vortex rows studied by mode-locking experiments