I think maybe it's that Kunlun is the only one he himself chose? Everyone else was sort of handed over to him as a duty, even if he does end up wanting to protect them, too.