The Mass Growth and Stellar Ages of Galaxies: Observations versus Simulations