Non-affine response: Jammed packings vs. spring networks