The origin of scatter in the star formation rate-stellar mass relation