It's a bit hard to follow James’s line of thought in this passage, especially since it’s a departure from his typical writing style. Basically, he is saying that although there’s historical precedent for the practice of slavery, that’s not an argument in its defense, since human beings have done all kinds of horrible, wretched things throughout history. Ultimately, humans seem to be inclined to do whatever gets them what they want, no matter how much they philosophize about virtue in the abstract.