Time scale for adiabaticity breakdown in driven many-body systems and orthogonality catastrophe