At midfield, the lateness of the contact would increase the severity of the call, increasing the likelihood of a caution, on account of the "reckless" nature of late contact.
You suggest it might work the other way in the penalty area. I understand the impulse of not awarding a penalty kick for contact that arguably did not affect the path of the ball. But, for me, at least, that's a straight penalty every time, and I'd consider a caution for the lateness of the contact and potential for injury arising from the angle of the contact.
I've yet to talk to a practicing referee who would not have awarded a penalty kick.