I don't think that our moral reasoning is consistent or uniform enough to have one absolute basis. It varies dramatically by time period, for one thing. Also by region. It can be influenced and shaped by experience, sure, but it also appears that genetics plays a role. How do you reconcile that variability with the idea that we are born with access to an objective moral code? And how to you explain so many people acting counter to this innate morality?
This is also why I don't see secularism as any better. I don't think that we magically become able to order society in a more perfect way just because we move away from theism. There will still be abuses of power and exploitation and imperfection... just like we've had for the last two thousand years and for thousands before that.