Recent literature and United Nations documents advocate that most armed non-state actors (ANSAs) should be bound by human rights law. This article takes a more critical stance on this issue. It argues that only a limited number of ANSAs should potentially become human rights duty-bearers: those that exercise de facto (human rights) jurisdiction and thus have considerable institutional and military capacities, as well as particular normative characteristics. It specifies these capacities and characteristics with an analysis of ANSAs’ practice that tentatively indicates that some of these entities may indeed exercise de facto jurisdiction. The argument is justified by highlighting the broader consequences that recognising ANSAs as human rights duty-bearers will entail. It will also endow them with privileges that will legitimise their authority over time. This is grounded in the normative logic of human rights law that emphasises the interrelationship between human rights, equality and democracy that also permeates the notion of jurisdiction and is further supported by a political understanding of the right to self-determination. The article closes with a brief sketch of two complementary ways to develop international law binding ANSAs to be further explored in future research: the so-called ‘responsibilities for human rights’ and an adapted law of occupation.