ZFOURGE/CANDELS: On the Evolution of M* Galaxy Progenitors from z = 3 to 0.5