First, reloaded ammo.
From a reliability standpoint, I would have no problem carrying a properly handloaded cartridge that has been subjected to my same personal test as factory ammo. That being a minimum of 100 rounds of that load fired flawlessly and each actual carry round chambered in the field-stripped barrel.
However, from a liability standpoint, I will not carry handloaded ammo. This is because, after a self defense shooting, I don't want any chance of an overeager prosecutor making claims of "homemade, super deadly ammo".
Second, cartridge choice.
With the modern self-defense ammo out today, in calibers 9mm/.38spl. and larger, there is probably not a significant difference in stopping ability. Carry what you are most comfortable with.