There's been several kids leave who he wanted to stay. His problem is the perception that if you don't succeed as a freshman year, you are a failure. It's used against him in recruiting and it has affected the mindset of some of the kids. He didn't want Kyle Wiltjer, Marcus Lee, Charles Matthews, etc. to leave.
Now, I do think Calipari is harder on kids who aren't freshman. They don't meet his expectations and he is gets a quick trigger with them.