The Mass, Color, and Structural Evolution of Today's Massive Galaxies Since z ~ 5