Having conference refs will always give the appearance of favoritism, and spawn actual favoritism.
It also allows for differing interpretations of the rules from conference to conference. "That's just not called in the Big 12". It becomes particularly apparent on judgement calls. Huge disadvantage in OOC games. And for the independents, well "f*ck them". I'd love to see a breakdown of the number of pick plays and targeting called from conference to conference and Referee crew to Referee crew. If the ACC and Big 12 don't call the pick penalty (except against ND), and the SEC throws flags, one could understand why Clemson expects to get away with it and Alabama expects it to be called.
Part of film study should now become the referee's tendencies.