Ehrenreich hears ted say that he is not a bad guy and cares a lot about his girls
Ted is in many ways a villain in the classic vein; he is defined largely by his actions, most of them reprehensible, and makes a point of continually demeaning his (all female) employees. From reading about Ted, he almost sounds like a caricature. Of course guys like Ted intimidated women and scared them. He knew they needed work and used his power to manipulate them. I don't really know if he was a pimp but I imagine he wasn't a very nice person.
