From what I'm to understand they never stack but they are calculated 1 by 1. So deal a 1000 damage, first 20% demi human resistance,(Leaves 800) then whats left over resisted 20% more by fire ( reduces 20% of 800 which is 160. So take 640 damage) and so on down the line. It doesn't calculate it all at once basically.
But if you give someone 100% resistance in a pre-renewal server they should take no damage at all.