What about those who kidnapped free men and enslaved them? What about those who raped and murdered slaves?
This is partly the reason why I mentioned earlier in the thread the necessity for the US to develop the concept of transitional justice, the nature of the crime of slavery in the South is not something any then-existing mode of justice is fit to handle. Every horrible thing that owners inflicted on their slaves, they did on the understanding that their right to own other human beings and do whatever they desired with them was enshrined in law. It was all perfectly legal, and it had been legal for as long as anybody could remember. Only the leadership of the Nazi regime and senior officers faced the gallows for their crimes against humanity, and even then it was because they had taken a society that tolerated Jews and other minorities and turned it into an engine of genocide, while even the leadership of the South simply found themselves born into a society where slavery was not only fine, it was sanctioned by God, and it had been that way all the way back to Jamestown. Northerners were for decades happy to let their Southern brothers own slaves, and to have their factories and markets supplied with slave-picked raw materials, and now they want to put the South on trial just because they've suddenly decided that slavery is evil?
If there's going to be something like peace, not just a Roman peace, then everybody has to either be taken out - marginalised, exhiled, hanged - or be given a dignified place in the new order. Every man that is hanged or shot means at least one brother or son swearing an undying oath of revenge, so all else equal the latter is preferable to the former.