Look at their schedule for the last 10-15 years. There aren't a whole lot of power 5 teams there. Is that by design, so they can schedule themselves into a bowl game, or is it because teams say no?
Do you think an SEC team would swap Navy for one of their late-season I-AA scrimmages?
I think coaches see very little upside in playing them. You have to spend a week coming up with a defensive game plan for an offense that no one else runs and the risk of defensive injury seems greater because of the way they play.