They negate the primary advantages we have over other dangerous animals -- intelligence and manual dexterity. A human could at least try to wrestle and choke out a dog or even a decent sized wild cat, because its possible to position yourself away from their weapons and use size to control them. It's really hard to see getting any kind of an advantage over a chimp, even a pretty small one.
Honestly, if only one of us is walking out, I like to think that nothing under 100 pounds is taking me out. We're talking to the death here. But I'm realistic that I radically overestimate my odds in these fights. A small chimp is the member of the under-100 club I'd be the most nervous about, although I can't say I'd be looking forward to taking on a wolverine, bobcat, or pit bull either.