Kelly's offense is totally dead when that happens. Fuller was open for at least two 60+ yard TD passes against USC early. Maybe three. Golson missed him by a mile each time -- not even seeing him at all one of the times.
On top of everything you said, over a 14 game period the QB simply is going to suck throwing the ball one of those games. Big Ben could not hit the broad side of a barn against New Orleans two weeks ago for the entire first half and then some -- his own admission. The Steelers at least can run the ball some. Oregon can run the ball with RBs and QBs effectively - plus pass. There is some hope in hell of moving the ball if the QB simply sucks that day. Not with ND. QB is off, passes are off, dead. Nearly hopeless unless playing a totally half assed team that day.
Finally, if you have run, true read option that the QB is capable of executing (which I think Golson is not), and passing the defense has more to defend. They can't just assume you are passing the ball. Their baiting you into calls gets harder because you have more options per formation. For that matter, empty backfield pretty much eliminates anything other than a QB draw or pass to defend and we do that a hell of a lot of times each game.