What reason do the villagers give for not just "kicking out" the white men?

They don't want to fight their own people who have converted to white man's religion. Resistance is now difficult, because fighting the white man would also mean going against the converts. Obierika puts it succinctly: "The white man is very clever. He came quietly and peaceably with his religion. We were amused at his foolishness and allowed him to stay. Now he has won our brothers, and our clan can no longer act like one.