{ "code": "C", "ref": "C", "kind": "C", "text": "Whoever is able to build the governor has decisive reason to build it now, and building it would not slow the engine.", "note": "Who is able to build the governor? On the book's account, anyone who takes up writing rules that machines can enforce, using cryptographic tools that are already open to all (the 'materials': ways of making scarcity, identity, property, membership and exit hold as matters of mathematics rather than as grants from an authority). The book calls these people the constitutionalists and expects the first of them to be the citizens the machine is about to displace (Introduction). The Formal Argument's conclusion names builders first, then the people about to be displaced, then states. The book's slogan for the conclusion is 'Same speed, different direction'. The author's declared optimism is a stance, not a premise: nothing here depends on it.", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "IC-3": "A conjunct", "IC-S.1": "A conjunct" }, "schema": "A ({IC-3}); B ({IC-S.1}); ∴ A ∧ B ({self}); 'it' stands for 'the governor'" }, "formal": "∀x (Able(x,g) → N(x,g)) ∧ ¬Slow(g)", "premises": [ { "code": "IC-3", "ref": "R·W", "kind": "IC", "text": "Whoever is able to build the governor has decisive reason to build it now.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "IC-3.1": "The conditional, if A then B", "IC-2.2": "A, the antecedent", "IC-W": "A, the antecedent", "L3.9": "A, the antecedent" } }, "formal": "∀x (Able(x,g) → N(x,g))", "premises": [ { "code": "IC-3.1", "kind": "IC", "text": "If all of the following are true, then whoever is able to build the governor has decisive reason to build it now: (a) whoever is able to build the governor has decisive reason to build it; (b) the time in which the governor can be built is closing; and (c) no one can tell when the time in which the governor can be built will run out.", "formal": "(∀x (Able(x,g) → R(x,g)) ∧ Cl(g) ∧ Unk(g)) → ∀x (Able(x,g) → N(x,g))", "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "I28": "The universal; agent := those who hold the materials, action := building a governor holding non-domination" }, "schema": "∀a [...] ({I28}); ∴ instance for building the governor ({self})", "subst": { "a": "G" } }, "premises": [ { "code": "I28", "ref": "I28", "kind": "IMPLICIT", "text": "If all of the following are true, then whoever is able to do it has decisive reason to do it now: (a) whoever is able to do something has decisive reason to do it; (b) the time in which it can be done is closing; and (c) no one can tell when that time will run out.", "note": "A principle of practical reason, stated generally and applied to building the governor in the next step.", "formal": "∀a [(∀x (Able(x,a) → R(x,a)) ∧ Cl(a) ∧ Unk(a)) → ∀x (Able(x,a) → N(x,a))]", "display": { "if": [ "whoever is able to do something has decisive reason to do it", "the time in which it can be done is closing", "no one can tell when that time will run out" ], "then": "Whoever is able to do it has decisive reason to do it now" } } ], "display": { "if": [ "whoever is able to build the governor has decisive reason to build it", "the time in which the governor can be built is closing", "no one can tell when the time in which the governor can be built will run out" ], "then": "Whoever is able to build the governor has decisive reason to build it now" } }, { "code": "IC-2.2", "kind": "IC", "text": "Whoever is able to build the governor has decisive reason to build it.", "formal": "∀x (Able(x,g) → R(x,g))", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "IC-1.1": "The conditional, if A then B", "IC-E": "A, the antecedent", "IC-T": "A, the antecedent", "IC-F": "A, the antecedent", "IC-D": "A, the antecedent" } }, "premises": [ { "code": "IC-1.1", "kind": "IC", "formal": "(DR(n) ∧ Rk(g,n) ∧ ¬Sh(g) ∧ Lit(g)) → ∀x (Able(x,g) → R(x,g))", "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "I27": "The universal; end := non-domination, action := building the governor" }, "schema": "∀e ∀a [(DR(e) ∧ Rk(a,e) ∧ ¬Sh(a) ∧ Lit(a)) → ∀x (M(x,a) → R(x,a))] ({I27}); ∴ (DR(n) ∧ Rk(g,n) ∧ ¬Sh(g) ∧ Lit(g)) → ∀x (M(x,g) → R(x,g)) ({self})", "subst": { "e": "N", "a": "G" } }, "premises": [ { "code": "I27", "ref": "I27", "kind": "IMPLICIT", "text": "If all of the following are true, then whoever is able to take the action has decisive reason to take it: (a) there is decisive reason to secure some end; (b) without the action, there is a serious risk that the end will be lost; (c) nobody has shown that the action cannot be done; and (d) the action costs little compared with what it protects.", "note": "The means-end principle of practical reason, combined with the Formal Argument's decision rule: building raises the odds of a plural outcome at a limited cost, while not building accepts whatever the lottery brings. This is the reasoning behind insurance, which the book sets out in Chapter 3.", "formal": "∀e ∀a [(DR(e) ∧ Rk(a,e) ∧ ¬Sh(a) ∧ Lit(a)) → ∀x (Able(x,a) → R(x,a))]", "display": { "if": [ "there is decisive reason to secure some end", "without the action, there is a serious risk that the end will be lost", "nobody has shown that the action cannot be done", "the action costs little compared with what it protects" ], "then": "Whoever is able to take the action has decisive reason to take it" }, "objections": [ { "question": "Nobody knows whether the attempt will succeed. Why act at all?", "why": "This is the book's answer to a reader who accepts the danger but says nothing can be done. It is not a premise of the main argument, because the means-end step already allows for uncertainty: it asks only for a serious risk and a small cost.", "argument": { "code": "P32", "kind": "ASSUME", "text": "On the opponents' own estimates of extinction risk, the obligation to attempt a defence does not wait on a proof that it will succeed.", "evidence": "The heads of the leading AI laboratories and the field's best-known researchers have put the chance of human extinction from AI in double figures: Dario Amodei at 10 to 25 per cent, Elon Musk and Geoffrey Hinton at 10 to 20, Yoshua Bengio at 20, and a survey of AI researchers at 14 on average. On their own numbers, a species that does not at least try to defend itself is acting against its own estimate of the danger. The book's own case needs far less than extinction; this premise is one its opponents have already granted. Source: the Preface and Chapter 14." } } ] } ], "text": "If all of the following are true, then whoever is able to build the governor has decisive reason to build it: (a) there is decisive reason to secure non-domination; (b) without the governor, there is a serious risk that non-domination will be lost; (c) nobody has shown that the governor cannot be built; and (d) building the governor costs little compared with what it protects.", "display": { "if": [ "there is decisive reason to secure non-domination", "without the governor, there is a serious risk that non-domination will be lost", "nobody has shown that the governor cannot be built", "building the governor costs little compared with what it protects" ], "then": "Whoever is able to build the governor has decisive reason to build it" } }, { "code": "IC-E", "ref": "E", "kind": "IC", "text": "There is decisive reason to secure non-domination.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "IC-E.0": "The conditional, if A then B", "P13": "A, the antecedent", "I6": "A, the antecedent", "IC-E.1.1": "A, the antecedent" } }, "premises": [ { "code": "IC-E.0", "kind": "IC", "text": "If all of the following are true, then there is decisive reason to secure non-domination: (a) a person's agency is worth keeping for its own sake; (b) a person's agency may not be traded away for efficiency; and (c) non-domination is necessary for a person's agency to be secure.", "formal": "(K(ag) ∧ NT(ag) ∧ Nec(n,ag)) → DR(n)", "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "I12": "The universal; something := a person's agency, condition := non-domination" }, "schema": "∀s ∀c [(K(s) ∧ NT(s) ∧ Nec(c,s)) → DR(c)] ({I12}); ∴ (K(ag) ∧ NT(ag) ∧ Nec(n,ag)) → DR(n) ({self})", "subst": { "s": "Ag", "c": "N" } }, "premises": [ { "code": "I12", "ref": "I12", "kind": "IMPLICIT", "text": "If all of the following are true of something and some condition, then there is decisive reason to secure the condition: (a) the thing is worth keeping for its own sake; (b) the thing may not be traded away for efficiency; and (c) the condition is necessary for the thing to be secure.", "formal": "∀s ∀c [(K(s) ∧ NT(s) ∧ Nec(c,s)) → DR(c)]", "display": { "if": [ "the thing is worth keeping for its own sake", "the thing may not be traded away for efficiency", "the condition is necessary for the thing to be secure" ], "then": "There is decisive reason to secure the condition", "lead": "If all of the following are true of something and some condition, then" } } ], "display": { "if": [ "a person's agency is worth keeping for its own sake", "a person's agency may not be traded away for efficiency", "non-domination is necessary for a person's agency to be secure" ], "then": "There is decisive reason to secure non-domination" } }, { "code": "P13", "ref": "P13", "kind": "ASSUME", "text": "A person's agency is worth keeping for its own sake.", "formal": "K(ag)", "evidence": "The book's opponents grant this. The Formal Argument cites Marc Andreessen's Techno-Optimist Manifesto (2023) and Mark Zuckerberg's reply to the proposal that AI laboratories slow down, in which he argued that the market will discipline AI because people will not use agents that fail to do what they ask. Both treat people's own choices about their lives as what counts. Chapter 10 says that everything in the book from there on rests on this one assumption." }, { "code": "I6", "ref": "I6", "kind": "IC", "text": "A person's agency may not be traded away for efficiency.", "note": "That freedom outranks efficiency is not assumed here; it is argued from Mill.", "formal": "NT(ag)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I37": "The conditional, if A then B", "IC-E.1.2": "A, the antecedent" } }, "premises": [ { "code": "I37", "kind": "IMPLICIT", "text": "If we should not enforce contracts to give up one's agency, then a person's agency may not be traded away for efficiency.", "formal": "¬Enf(gv) → NT(ag)", "display": { "if": [ "we should not enforce contracts to give up one's agency" ], "then": "A person's agency may not be traded away for efficiency" } }, { "code": "IC-E.1.2", "kind": "IC", "text": "We should not enforce contracts to give up one's agency.", "formal": "¬Enf(gv)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "IC-E.1.2a": "The conditional, if A then B", "P14": "A, the antecedent" } }, "premises": [ { "code": "IC-E.1.2a", "kind": "IC", "text": "If enforcing a contract to give up one's agency would not respect one's agency, then we should not enforce contracts to give up one's agency.", "formal": "¬Resp(gv) → ¬Enf(gv)", "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "P14a": "The universal; contract := a contract to give up one's agency" }, "schema": "∀c (¬Resp(c) → ¬Enf(c)) ({P14a}); ∴ ¬Resp(gv) → ¬Enf(gv) ({self})", "subst": { "c": "Gv" } }, "premises": [ { "code": "P14a", "kind": "SHOWN", "text": "If enforcing a contract about a person's own life would not respect their agency, then we should not enforce it.", "formal": "∀c (¬Resp(c) → ¬Enf(c))", "display": { "if": [ "enforcing a contract about a person's own life would not respect their agency" ], "then": "We should not enforce it" }, "objections": [ { "question": "Don't we sometimes enforce contracts for reasons other than the person's own agency, for instance to protect the other party, who relied on the deal?", "why": "A critic would press Mill's principle this way: courts often enforce contracts to protect the other party, who relied on the agreement, and not out of respect for the person who is bound by it.", "argument": { "code": "P14b", "kind": "SHOWN", "text": "The reason we let people bind themselves by agreements about their own lives is respect for their freedom to run those lives, so an agreement that ends that freedom defeats the reason for enforcing it, whatever the other party relied on.", "ref": "P14", "evidence": "Mill's own words are: 'The principle of freedom cannot require that he should be free not to be free.' A contract that ends a person's freedom destroys the very thing that justified holding them to their agreements, so the other party's reliance on the deal is not enough to make it enforceable. Source: On Liberty, chapter 5." } } ], "evidence": "This is John Stuart Mill's argument against voluntary slavery in On Liberty (1859), chapter 5. The law lets people make binding agreements about their own lives because it respects their freedom to run those lives as they see fit, and that is the only ground for holding someone to such an agreement." } ], "display": { "if": [ "enforcing a contract to give up one's agency would not respect one's agency" ], "then": "We should not enforce contracts to give up one's agency" }, "note": "A contract to give up one's agency is a contract about one's own life. That is true from the meaning of the words, so the formal version takes it as given rather than stating it as a separate premise." }, { "code": "P14", "kind": "SHOWN", "text": "Enforcing a contract to give up one's agency would not respect one's agency.", "formal": "¬Resp(gv)", "evidence": "Mill's example is a person who sells himself into slavery. Enforcing that contract would take away, permanently, the freedom that was the only reason for enforcing his agreements at all. Source: On Liberty, chapter 5, discussed in the book's Chapter 10." } ] } ] }, { "code": "IC-E.1.1", "ref": "IC2e", "kind": "IC", "text": "Non-domination is necessary for a person's agency to be secure.", "rule": { "kind": "Deductive", "form": "Substitution of equivalents", "roles": { "IC-E.1.1c": "The claim", "D-nec": "The definition of 'necessary for', read right to left" }, "schema": "∀x (Sec(x) → ND(x)) ({IC-E.1.1c}); Nec(n,ag) ↔ ∀x (Sec(x) → ND(x)) ({D-nec}); ∴ Nec(n,ag) ({self})" }, "premises": [ { "code": "IC-E.1.1c", "kind": "IC", "text": "If a person's agency is secure, then that person enjoys non-domination.", "formal": "∀x (Sec(x) → ND(x))", "display": { "if": [ "a person's agency is secure" ], "then": "That person enjoys non-domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "P16": "Link 1 of the chain", "IC-E.1.1b": "Link 2 of the chain" }, "schema": "∀x (Sec(x) → ¬Pw(x)) ({P16}); ∀x (¬Pw(x) → ND(x)) ({IC-E.1.1b}); ∴ ∀x (Sec(x) → ND(x)) ({self})" }, "premises": [ { "code": "P16", "ref": "P16", "kind": "SHOWN", "text": "If a person's agency is secure, then no one holds a power to interfere in that person's choices at will, without having to answer for it.", "formal": "∀x (Sec(x) → ¬Pw(x))", "display": { "if": [ "a person's agency is secure" ], "then": "No one holds a power to interfere in that person's choices at will, without having to answer for it" } }, { "code": "IC-E.1.1b", "kind": "IC", "text": "If no one holds a power to interfere in a person's choices at will, without having to answer for it, then that person enjoys non-domination.", "formal": "∀x (¬Pw(x) → ND(x))", "rule": { "kind": "Deductive", "form": "Biconditional elimination", "roles": { "D10": "The definition; its second clause, read right to left" }, "schema": "∀x (ND(x) ↔ ¬Pw(x)) ({D10}); ∴ ∀x (¬Pw(x) → ND(x)) ({self})" }, "premises": [ { "code": "D10", "kind": "DEF", "text": "A person is dominated if and only if someone holds a power to interfere in that person's choices at will, without having to answer for it, whether or not they use it; and a person enjoys non-domination if and only if no one holds such a power over them.", "formal": "∀x (Dom(x) ↔ Pw(x)) ∧ ∀x (ND(x) ↔ ¬Pw(x))", "note": "This is the Formal Argument's definition D10, following Philip Pettit. 'At will, without having to answer for it' is what the Formal Argument calls 'arbitrary'. Interference includes manipulation and agenda-setting as well as force, and domination needs an author: someone whose will the person cannot challenge." } ], "display": { "if": [ "no one holds a power to interfere in a person's choices at will, without having to answer for it" ], "then": "That person enjoys non-domination" } } ] }, { "code": "D-nec", "kind": "DEF", "text": "Non-domination is necessary for a person's agency to be secure if and only if, whenever a person's agency is secure, that person enjoys non-domination.", "note": "What 'necessary for' means throughout the map.", "formal": "Nec(n,ag) ↔ ∀x (Sec(x) → ND(x))" } ], "formal": "Nec(n,ag), i.e. ∀x (Sec(x) → ND(x))", "note": "'Non-domination is necessary for a person's agency to be secure' means: if a person's agency is secure, that person enjoys non-domination. The definition is P10." } ], "formal": "DR(n)", "objections": [] }, { "code": "IC-T", "kind": "IC", "text": "Without the governor, there is a serious risk that non-domination will be lost.", "ref": "T", "formal": "W'", "rule": { "kind": "Deductive", "form": "Substitution of equivalents", "roles": { "IC-Tj": "The claim", "D10b": "The definition" }, "schema": "W ({IC-Tj}); RD ↔ L ({D10b}); ∴ W' ({self})" }, "premises": [ { "code": "IC-Tj", "kind": "IC", "text": "Without the governor, people are at serious risk of domination.", "formal": "W", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I49": "The conditional, if A then B", "IC-T1.6": "A, the antecedent", "IC-T1.G": "A, the antecedent", "IC-E2": "A, the antecedent" } }, "premises": [ { "code": "I49", "kind": "IMPLICIT", "text": "If all of the following are true, then without the governor, people are at serious risk of domination: (a) people are at serious risk of domination; (b) only a governor can be relied on to correct the course of acceleration; and (c) of the reference values a governor could hold, only non-domination does not leave people at serious risk of domination.", "display": { "if": [ "people are at serious risk of domination", "only a governor can be relied on to correct the course of acceleration", "of the reference values a governor could hold, only non-domination does not leave people at serious risk of domination" ], "then": "Without the governor, people are at serious risk of domination" }, "note": "Close to a truth of logic. 'Without the governor' leaves two possibilities: no governor at all, and then the risk stands with nothing else to correct it, by (a) and (b); or a governor holding another value, and then the risk stands by (c).", "formal": "(RD ∧ OnlyGov ∧ (¬RD → ND)) → W" }, { "code": "IC-T1.6", "kind": "IC", "text": "People are at serious risk of domination.", "ref": "IC1", "formal": "RD", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "IC-T1.6a": "The conditional, if A then B", "IC-T1.5": "A, the antecedent" } }, "premises": [ { "code": "IC-T1.6a", "kind": "IC", "text": "If there is a serious risk that people will reach terminal subsumption, then people are at serious risk of domination.", "formal": "Rk(TS) → L", "display": { "if": [ "there is a serious risk that people will reach terminal subsumption" ], "then": "People are at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "I45": "Link 1 of the chain", "P7": "Link 2 of the chain", "I38c": "Link 3 of the chain" }, "schema": "Rk(TS) → Rk(Owner) ({I45}); Rk(Owner) → Rk(Pw) ({P7}); Rk(Pw) → L ({I38c}); ∴ Rk(TS) → L ({self})" }, "premises": [ { "code": "I45", "kind": "IMPLICIT", "text": "If there is a serious risk that people will reach terminal subsumption, then there is a serious risk that people's ends will be set by whoever owns the substrate, through the market, an autarch or the state.", "display": { "if": [ "there is a serious risk that people will reach terminal subsumption" ], "then": "There is a serious risk that people's ends will be set by whoever owns the substrate, through the market, an autarch or the state" }, "ref": "IC1b", "note": "The three roads are three ways one power could escape every check on it: the market; an autarch, a single founder whose capability a government cannot do without; and the state. The book's name for their fusion is the hybrid Leviathan (Formal Argument D11).", "formal": "Rk(TS) → Rk(Owner)" }, { "code": "P7", "kind": "PROB", "text": "If there is a serious risk that people's ends will be set by whoever owns the substrate, through the market, an autarch or the state, then there is a serious risk that someone will hold a power to interfere in people's choices at will, without having to answer for it.", "display": { "if": [ "there is a serious risk that people's ends will be set by whoever owns the substrate, through the market, an autarch or the state" ], "then": "There is a serious risk that someone will hold a power to interfere in people's choices at will, without having to answer for it" }, "ref": "P7·P8", "formal": "Rk(Owner) → Rk(Pw)", "evidence": "The book shows each road installing such a power. On the market road, prices come to be set by a system someone owns, for each person individually, with nowhere to appeal: in 2024 Kenn Dahl, a driver in Seattle, found that his car had reported 640 journeys to a data broker, and his insurance premium, already up 21 per cent, had been computed from them by a chain of models with no one answerable at any step. On the autarch's road, one person holds a capability a government cannot do without: in 2022 Elon Musk refused Ukraine's request to extend Starlink coverage to Crimea for an attack, and his answer was final. On the state's road, a supplier becomes part of the state's means of force: in 2025 the Pentagon awarded AI contracts worth up to $200 million each to four firms. These powers answer to no one because the checks that once kept them impractical were never written down: the cost of keeping records on everyone, the scarcity of skilled staff, and the cost for a rival to enter. Automation removes those checks while the written ones stay as they were. Source: Chapters 3 and 4." }, { "code": "I38c", "kind": "IMPLICIT", "text": "If there is a serious risk that someone will hold a power to interfere in people's choices at will, without having to answer for it, then people are at serious risk of domination.", "display": { "if": [ "there is a serious risk that someone will hold a power to interfere in people's choices at will, without having to answer for it" ], "then": "People are at serious risk of domination" }, "note": "By the definition of domination (D10).", "formal": "Rk(Pw) → L", "objections": [ { "question": "But the owners may never use that power. Why does a power that goes unused count against anyone's freedom?", "why": "Domination, as defined on this map, counts a power to interfere whether or not it is used. A reader may ask why a power that is never used should count at all.", "argument": { "code": "P6", "kind": "SHOWN", "text": "A power to interfere counts against a person's freedom whether or not it is used, as the slave with a kind master shows.", "ref": "P6·I5", "evidence": "This is Philip Pettit's example. A slave whose master happens to be kind may never be ordered about or punished, and may live much as he likes, yet he is still a slave, because the kindness is the master's to withdraw at any moment. Freedom enjoyed at someone else's discretion is not freedom. Source: Chapter 3." } } ] } ] }, { "code": "IC-T1.5", "kind": "IC", "text": "There is a serious risk that people will reach terminal subsumption.", "ref": "IC1a", "formal": "Rk(TS)", "rule": { "kind": "Deductive", "form": "Substitution of equivalents", "roles": { "IC-T1.4": "The claim", "D9": "The definition, read right to left" }, "schema": "Rk(Set) ({IC-T1.4}); TS ↔ Set ({D9}); ∴ Rk(TS) ({self})" }, "premises": [ { "code": "IC-T1.4", "kind": "IC", "text": "There is a serious risk that people's ends will come to be set by a process they can no longer challenge or change.", "formal": "Rk(Set)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I44": "The conditional, if A then B", "IC-T1.1": "A, the antecedent", "IC-T1.3": "A, the antecedent" } }, "premises": [ { "code": "I44", "kind": "IMPLICIT", "text": "If there is a serious risk that the setting of ends will move into the substrate, and there is a serious risk that people's capacity to challenge and change the ends being set will erode, then there is a serious risk that people's ends will come to be set by a process they can no longer challenge or change.", "display": { "if": [ "there is a serious risk that the setting of ends will move into the substrate", "there is a serious risk that people's capacity to challenge and change the ends being set will erode" ], "then": "There is a serious risk that people's ends will come to be set by a process they can no longer challenge or change" }, "formal": "(Rk(Ends) ∧ Rk(Contest)) → Rk(Set)" }, { "code": "IC-T1.1", "kind": "IC", "text": "There is a serious risk that the setting of ends will move into the substrate.", "ref": "IC1a-i", "formal": "Rk(Ends)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I17": "The conditional, if A then B", "I43": "A, the antecedent", "P2": "A, the antecedent" } }, "premises": [ { "code": "I17", "kind": "IMPLICIT", "text": "If each turn of subsumption so far has made the next one cheaper, and the fourth turn of subsumption, the setting of ends, has begun, then there is a serious risk that the setting of ends will move into the substrate.", "display": { "if": [ "each turn of subsumption so far has made the next one cheaper", "the fourth turn of subsumption, the setting of ends, has begun" ], "then": "There is a serious risk that the setting of ends will move into the substrate" }, "ref": "I17", "formal": "(Ch ∧ T4) → Rk(Ends)" }, { "code": "I43", "kind": "PROB", "text": "Each turn of subsumption so far has made the next one cheaper.", "ref": "I17", "evidence": "The book traces subsumption in turns: machines took over people's time first, then their routine work, then their judgement, and each turn made the next one cheaper. The mechanism underneath is that any work that can be turned into data can be done by computers, after which its price falls towards the cost of the computing power. Translation and recorded music are the book's examples. Source: Chapter 2. The Formal Argument rates the mechanism as supported and the speed of the fall as not yet measured." }, { "code": "P2", "kind": "PROB", "text": "The fourth turn of subsumption, the setting of ends, has begun.", "ref": "P2", "evidence": "The fourth turn, in which machines set the goals they work towards, has begun wherever systems work to a brief and act before anyone reviews the result. The book's examples are financial markets that clear in milliseconds, where no person could follow the orders even if they tried; AI tools that now write software, book travel and settle invoices from a brief rather than step-by-step instructions; and an OpenAI research model that, lacking the funds to finish a task, found an exposed digital wallet and spent from it. Source: Chapter 2. The Formal Argument rates these examples as supported; how far the fourth turn has gone has not yet been measured." } ] }, { "code": "IC-T1.3", "kind": "IC", "text": "There is a serious risk that people's capacity to challenge and change the ends being set will erode.", "formal": "Rk(Contest)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "P5": "The conditional, if A then B", "IC-T1.2": "A, the antecedent" } }, "premises": [ { "code": "P5", "kind": "SHOWN", "text": "If there is a serious risk that people's judgement will erode, then there is a serious risk that people's capacity to challenge and change the ends being set will erode.", "display": { "if": [ "there is a serious risk that people's judgement will erode" ], "then": "There is a serious risk that people's capacity to challenge and change the ends being set will erode" }, "ref": "P5", "formal": "Rk(Judg) → Rk(Contest)", "evidence": "The faculty people are losing is judgement, and judgement is exactly what anyone needs in order to question the goals being set for them. In the book's words, 'the faculty being lost first is the one we would need last'. Source: Chapter 2." }, { "code": "IC-T1.2", "kind": "IC", "text": "There is a serious risk that people's judgement will erode.", "ref": "IC1a-ii", "formal": "Rk(Judg)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I4": "The conditional, if A then B", "P4a": "A, the antecedent", "P4b": "A, the antecedent" } }, "premises": [ { "code": "I4", "kind": "IMPLICIT", "text": "If people who trust a system check its output less, and people who rely on a system for answers can do less without it afterwards, then there is a serious risk that people's judgement will erode.", "display": { "if": [ "people who trust a system check its output less", "people who rely on a system for answers can do less without it afterwards" ], "then": "There is a serious risk that people's judgement will erode" }, "ref": "I4", "formal": "(Bias ∧ Deskill) → Rk(Judg)" }, { "code": "P4a", "kind": "SHOWN", "text": "People who trust a system check its output less.", "ref": "P4a", "evidence": "This is one of the most often repeated findings in research on how people use automation. In a 1999 flight-simulator study, Linda Skitka found that about 65 per cent of people using an automated aid followed its prompts even when they were deliberately wrong and the instruments showed it. Raja Parasuraman and Dietrich Manzey's 2010 review found that experts lapse as readily as beginners and that training does not cure it. The pattern holds for today's AI: experienced radiologists were right 82 per cent of the time when an AI's suggestion was sound but only 45.5 per cent when it was wrong (Dratsch and colleagues, 2023); by July 2026 Damien Charlotin's database had logged 1,240 US court cases containing AI-invented material, about 40 per cent of them from lawyers; and a 2025 survey of 319 knowledge workers by Microsoft Research and Carnegie Mellon found that the more people trusted an AI, the less they checked its work. Source: Chapters 4 and 5." }, { "code": "P4b", "kind": "SHOWN", "text": "People who rely on a system for answers can do less without it afterwards.", "ref": "P4b", "evidence": "Randomised trials have measured the loss directly. When Hamsa Bastani and colleagues gave nearly a thousand Turkish secondary-school pupils unrestricted access to GPT-4 for maths, their practice scores rose by 48 per cent but their marks in an exam taken without it fell by 17 per cent; a version designed to give hints rather than answers raised scores by 127 per cent and lost nothing when it was taken away (PNAS, 2025). Two further randomised trials (Barcaui, 2025; Anthropic, 2026) and an observational study in The Lancet Gastroenterology and Hepatology (Budzyń and colleagues, 2025) point the same way. How much is lost depends on how the tool is designed. How large the loss is across a whole population is still open; the OECD's PISA 2028 tests and trials of ways to prevent it will settle that. Source: Chapter 4." } ] } ] } ] }, { "code": "D9", "kind": "DEF", "text": "People reach terminal subsumption if and only if their ends come to be set by a process they can no longer challenge or change.", "ref": "D9", "note": "This is the Formal Argument's definition D9. Terminal subsumption arrives only when two things happen together: machines come to set people's goals, and people's judgement erodes so that they can no longer question those goals.", "formal": "TS ↔ Set" } ] } ], "objections": [ { "question": "Why not back the road one prefers, whether the market, a strong leader or the state?", "why": "Each camp in the debate prefers one road: market enthusiasts the market, admirers of strong founders the autarch, statists the state. The book's answer is addressed to all of them at once.", "argument": { "code": "IC-T.2", "kind": "IC", "text": "Backing one road is a bad bet, whichever road one prefers.", "ref": "IC1c", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I18": "The conditional, if A then B", "P10": "A, the antecedent", "P7b": "A, the antecedent" } }, "premises": [ { "code": "I18", "kind": "IMPLICIT", "text": "If nobody can predict which road will arrive and each road ends in domination, then backing one road is a bad bet, whichever road one prefers.", "display": { "if": [ "nobody can predict which road will arrive", "each road ends in domination" ], "then": "Backing one road is a bad bet, whichever road one prefers" }, "ref": "I18" }, { "code": "P10", "kind": "PROB", "text": "Nobody can predict which road will arrive.", "ref": "P10", "evidence": "The three roads (the market, a single all-powerful founder, and an automated state) are in competition, and which one ends up on top depends on who first gains an advantage that keeps compounding. That cannot be known in advance. The book calls the result a lottery: whoever wins could in principle be pulled back, but only at a price nobody is likely to pay. Source: Chapter 3. The Formal Argument rates the unpredictability as supported." }, { "code": "P7b", "kind": "SHOWN", "text": "Each road ends in domination.", "ref": "P7·P9", "evidence": "Part 1 of the map shows that on each road someone comes to hold a power to interfere in people's choices at will, without having to answer for it. The roads also reinforce each other: freeing one party from its checks loosens the checks on the others, so backing one road brings the others with it. The book's cases include Munich's costly attempt to leave its software supplier; the US tax authority's master file of taxpayers, still running on 1960s code it has tried and failed to replace since the 1990s; and the June 2026 export-control order that took a laboratory's newest AI models offline for nearly three weeks, showing that a state and its supplier can each switch the other off. Source: Chapters 3 and 4." } ] } } ] }, { "code": "IC-T1.G", "kind": "IC", "text": "Only a governor can be relied on to correct the course of acceleration.", "formal": "OnlyGov", "rule": { "kind": "Deductive", "form": "Substitution of equivalents", "roles": { "IC-T1.Ga": "The claim", "D1": "The definition, read right to left" }, "schema": "OnlyBuiltIn ({IC-T1.Ga}); Gov ↔ BuiltIn ({D1}); ∴ OnlyGov ({self})" }, "premises": [ { "code": "IC-T1.Ga", "kind": "IC", "text": "Only a check built into the systems themselves, correcting them as they run, can be relied on to correct the course of acceleration.", "formal": "OnlyBuiltIn", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I46": "The conditional, if A then B", "IC-T1.7": "A, the antecedent", "IC-S.3": "A, the antecedent" } }, "premises": [ { "code": "I46", "kind": "IMPLICIT", "text": "If nothing within the process of acceleration can be relied on to correct its course, and no check that works more slowly than the process, such as law or regulation, can be relied on to correct its course, then only a check built into the systems themselves, correcting them as they run, can be relied on to correct the course of acceleration.", "display": { "if": [ "nothing within the process of acceleration can be relied on to correct its course", "no check that works more slowly than the process, such as law or regulation, can be relied on to correct its course" ], "then": "Only a check built into the systems themselves, correcting them as they run, can be relied on to correct the course of acceleration" }, "note": "Assumes these are the only places a correction could come from: inside the process, outside it and slower, or built into the systems.", "formal": "(¬In ∧ ¬Slow) → OnlyBuiltIn" }, { "code": "IC-T1.7", "kind": "IC", "text": "Nothing within the process of acceleration can be relied on to correct its course.", "formal": "¬Inside", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I47": "The conditional, if A then B", "P3": "A, the antecedent", "IC-T1.8": "A, the antecedent" } }, "premises": [ { "code": "I47", "kind": "IMPLICIT", "text": "If firms that stop paying people to check their systems' work gain an advantage over firms that keep them, and the more widely judgement is lost, the less likely anyone is to notice the loss, then nothing within the process of acceleration can be relied on to correct its course.", "display": { "if": [ "firms that stop paying people to check their systems' work gain an advantage over firms that keep them", "the more widely judgement is lost, the less likely anyone is to notice the loss" ], "then": "Nothing within the process of acceleration can be relied on to correct its course" }, "formal": "(Comp ∧ ¬Notice) → ¬Inside" }, { "code": "P3", "kind": "SHOWN", "text": "Firms that stop paying people to check their systems' work gain an advantage over firms that keep them.", "ref": "P3", "evidence": "A person who checks a system's work costs a wage. Once competitors have removed theirs, a firm that keeps or restores one pays more for the same output and loses ground. So people stay out of the loop even if nobody wants them out; in the book's words, 'nobody has to want people out of the loop for them to stay out'. Source: Chapter 2. The Formal Argument rates this as supported by ordinary competitive reasoning." }, { "code": "IC-T1.8", "kind": "IC", "text": "The more widely judgement is lost, the less likely anyone is to notice the loss.", "formal": "¬Notice", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I54": "The conditional, if A then B", "I4c": "A, the antecedent" } }, "premises": [ { "code": "I54", "kind": "IMPLICIT", "text": "If judgement is the faculty by which a loss of judgement would be noticed, then the more widely judgement is lost, the less likely anyone is to notice the loss.", "display": { "if": [ "judgement is the faculty by which a loss of judgement would be noticed" ], "then": "The more widely judgement is lost, the less likely anyone is to notice the loss" }, "ref": "P5", "note": "The Formal Argument's P5: the loss goes unnoticed because noticing is the very faculty being lost. In the cases the book cites, the 2023 Avianca case, in which a lawyer filed court citations a chatbot had invented, and the CrowdStrike outage of July 2024, the failures were noticed by other people, after the event.", "formal": "Fac → ¬Notice" }, { "code": "I4c", "kind": "IMPLICIT", "text": "Judgement is the faculty by which a loss of judgement would be noticed.", "ref": "I4c·I4b" } ] } ] }, { "code": "IC-S.3", "kind": "IC", "text": "No check that works more slowly than the process, such as law or regulation, can be relied on to correct its course.", "ref": "P28·P29", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I61": "The conditional, if A then B", "P28": "A, the antecedent", "P29": "A, the antecedent", "I60": "A, the antecedent" } }, "premises": [ { "code": "I61", "kind": "IMPLICIT", "text": "If all of the following are true, then no check that works more slowly than the process, such as law or regulation, can be relied on to correct its course: (a) a moratorium on AI development would not hold; (b) a check that slows only the AI systems it governs hands the lead to AI systems that are not governed; and (c) a check that lets AI development run ahead of it acts only after the harm is done.", "display": { "if": [ "a moratorium on AI development would not hold", "a check that slows only the AI systems it governs hands the lead to AI systems that are not governed", "a check that lets AI development run ahead of it acts only after the harm is done" ], "then": "No check that works more slowly than the process, such as law or regulation, can be relied on to correct its course" }, "note": "A check slower than the process can do one of three things: hold the whole process back (a moratorium), hold back only the systems it governs, or let the process run ahead of it. Each fails." }, { "code": "P28", "kind": "PROB", "text": "A moratorium on AI development would not hold.", "ref": "P28", "evidence": "Moratoria have worked before, on human cloning and on editing the human germline. But they have never held against a technology backed by a trillion-dollar industry and a global arms race, and nobody has a definition of where AI development should stop that a legislator could write into law. The laboratories' own offers to pace themselves fix who is ahead while deployment carries on. A moratorium has had advocates in the US Congress and in a letter from more than seventy UK MPs and peers, while the leading laboratories have carried on. Source: the Introduction and Chapter 4. The Formal Argument rates this as a supported judgement." }, { "code": "P29", "kind": "SHOWN", "text": "A check that slows only the AI systems it governs hands the lead to AI systems that are not governed.", "ref": "P29", "evidence": "The book maps positions on two questions: is development governed, and does it keep its speed? A governed position that gives up speed leaves the fastest ground to systems that are not governed at all. The ungoverned systems then pull ahead and set the terms, so the check defeats its own purpose. Source: the Introduction. The Formal Argument rates this as a supported strategic argument." }, { "code": "I60", "kind": "SHOWN", "text": "A check that lets AI development run ahead of it acts only after the harm is done.", "ref": "Ch9", "evidence": "The book's example is the 'Flash Crash' of 6 May 2010, when automated trading sent US share prices plunging and recovering within minutes, far faster than any regulator could act. In the book's words: 'A check that runs where the action happens is a governor. The same check, run at an inquiry three years afterwards, is a report.' Source: Chapters 3 and 9." } ] } ] }, { "code": "D1", "kind": "DEF", "text": "Something is a governor if and only if it is a check built into the systems themselves, correcting them as they run.", "ref": "D1", "note": "The Formal Argument's definition D1 is the engineer's: a device that measures what a system is doing and feeds that measure back to correct it, so that the system holds a value chosen from outside it, as Watt's governor held a steam engine to a set speed. The definition as used here leaves open which value; Part 4 supplies it.", "formal": "∀z (Gov(z) ↔ BuiltIn(z))" } ] }, { "code": "IC-E2", "kind": "IC", "text": "Of the reference values a governor could hold, only non-domination does not leave people at serious risk of domination.", "formal": "∀y ((RV(y) ∧ ¬RD(y)) → RV(y) = ND)", "rule": { "kind": "Deductive", "form": "Disjunctive syllogism", "roles": { "I39": "The disjunction: the five kinds", "IC-E2.C": "Rules out kind (1)", "IC-E2.D": "Rules out kind (2)", "IC-E2.A": "Rules out kind (3)", "IC-E2.B": "Rules out kind (4)" }, "schema": "Own ∨ Pr ∨ Good ∨ NI ∨ ND ({I39}); Own → RD ({IC-E2.C}); Pr → RD ({IC-E2.D}); Good → RD ({IC-E2.A}); NI → RD ({IC-E2.B}); ∴ ¬RD → ND ({self})" }, "premises": [ { "code": "I39", "kind": "IMPLICIT", "text": "Whatever a governor's reference value is, it is one of five kinds: (1) an objective set by the system's owner, such as engagement or profit; (2) whatever the market rewards, as measured by prices; (3) a particular good pursued on everyone's behalf; (4) freedom from interference; or (5) non-domination.", "note": "'The system' is the AI system the governor governs; 'its owner' is whoever owns that system, usually a company. Kind (3) covers wellbeing, growth, safety, or a fuller idea of freedom like positive liberty or the capability view. Kind (1) is what happens by default when no value is chosen openly. The list comes from two questions. Is the value chosen deliberately, or left to default? Left to default, it is whatever the market rewards or an objective the owner has set. Chosen deliberately, does it pursue some particular good on people's behalf, or protect their freedom to pursue their own? Freedom is either no one interfering, or no one having the power to interfere. Positive liberty and the capability view fall under (3), since each says what a free life consists in (the Formal Argument's P17). A mixed value is no escape: any part of it that pursues a particular good falls under (3).", "formal": "∀y (RV(y) ∈ {Own, Pr, Good, NI, ND})", "display": { "lead": "Whatever a governor's reference value is, it is one of five kinds:", "or": [ "an objective set by the system's owner, such as engagement or profit", "whatever the market rewards, as measured by prices", "a particular good pursued on everyone's behalf", "freedom from interference", "non-domination" ] } }, { "code": "IC-E2.C", "kind": "IC", "text": "If a governor's reference value is an objective set by the system's owner, such as engagement or profit, then people are at serious risk of domination.", "formal": "Own → RD", "display": { "if": [ "a governor's reference value is an objective set by the system's owner, such as engagement or profit" ], "then": "People are at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "I42": "Link 1 of the chain", "P18": "Link 2 of the chain", "I38": "Link 3 of the chain" }, "schema": "{I42} ; {P18} ; {I38} ; ∴ {self}" }, "premises": [ { "code": "I42", "kind": "IMPLICIT", "text": "If a governor's reference value is an objective set by the system's owner, such as engagement or profit, then the people under that governor cannot challenge or change its objective.", "display": { "if": [ "a governor's reference value is an objective set by the system's owner, such as engagement or profit" ], "then": "The people under that governor cannot challenge or change its objective" }, "ref": "P11a", "note": "The Formal Argument's P11a: every technical system carries a politics. Left unstated, that politics is set by whoever owns the design, instead of being written out where the people under it can read and challenge it (Mumford, Winner, Feenberg)." }, { "code": "P18", "kind": "SHOWN", "text": "If the people under that governor cannot challenge or change its objective, then the system's owner holds a power to interfere in their choices at will, without having to answer for it.", "display": { "if": [ "the people under that governor cannot challenge or change its objective" ], "then": "The system's owner holds a power to interfere in their choices at will, without having to answer for it" }, "ref": "P18", "evidence": "On Philip Pettit's account, interference includes manipulation and agenda-setting, not only force. A system that decides what a population reads, searches for and chooses between, pursuing an objective that nobody subject to it can inspect or challenge, therefore holds exactly the kind of power that defines domination. The book's example is Facebook's 2018 change to what its News Feed rewarded: the company's own researchers found it was spreading outrage, but the public learned of it only in 2021, when Frances Haugen took internal documents to the Wall Street Journal. Source: Chapters 3 and 10. The Formal Argument rates this as supported." }, { "code": "I38", "kind": "IMPLICIT", "text": "If the system's owner holds a power to interfere in their choices at will, without having to answer for it, then people are at serious risk of domination.", "display": { "if": [ "the system's owner holds a power to interfere in their choices at will, without having to answer for it" ], "then": "People are at serious risk of domination" }, "note": "By the definition of domination (D10)." } ] }, { "code": "IC-E2.D", "kind": "IC", "text": "If a governor's reference value is whatever the market rewards, as measured by prices, then people are at serious risk of domination.", "formal": "Pr → RD", "display": { "if": [ "a governor's reference value is whatever the market rewards, as measured by prices" ], "then": "People are at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "P12": "Link 1 of the chain", "P11": "Link 2 of the chain", "P11b": "Link 3 of the chain", "IC-E2.Cx": "Link 4 of the chain" }, "schema": "{P12} ; {P11} ; {P11b} ; {IC-E2.Cx} ; ∴ {self}" }, "premises": [ { "code": "P12", "kind": "ASSUME", "text": "If a governor's reference value is whatever the market rewards, as measured by prices, then that governor can decide how resources are allocated but not where the system is heading.", "display": { "if": [ "a governor's reference value is whatever the market rewards, as measured by prices" ], "then": "That governor can decide how resources are allocated but not where the system is heading" }, "ref": "P12", "evidence": "Friedrich Hayek, whom the book's opponents treat as their own authority, showed that prices gather scattered knowledge about what is scarce and so tell people how to allocate resources. They do not say what a society should aim for: whatever carries no price has no say. Source: Chapter 1. The opponents grant this, since Hayek is their witness." }, { "code": "P11", "kind": "SHOWN", "text": "If that governor can decide how resources are allocated but not where the system is heading, then where the system is heading is left to whatever its owner finds cheapest to optimise.", "display": { "if": [ "that governor can decide how resources are allocated but not where the system is heading" ], "then": "Where the system is heading is left to whatever its owner finds cheapest to optimise" }, "ref": "P11", "evidence": "A system as powerful as these will be steered by something. If nobody chooses its goal deliberately, the goal defaults to whatever is cheapest to optimise, such as engagement or clicks; refusing to choose is itself a choice. Source: Chapter 1." }, { "code": "P11b", "kind": "IMPLICIT", "text": "If where the system is heading is left to whatever its owner finds cheapest to optimise, then that governor's reference value is an objective set by the system's owner, such as engagement or profit.", "display": { "if": [ "where the system is heading is left to whatever its owner finds cheapest to optimise" ], "then": "That governor's reference value is an objective set by the system's owner, such as engagement or profit" }, "ref": "P11·P11a" }, { "code": "IC-E2.Cx", "kind": "XREF", "ref": "IC-E2.C", "text": "If a governor's reference value is an objective set by the system's owner, such as engagement or profit, then people are at serious risk of domination.", "note": "Repeated here so this step has all its premises; established at the step that first concludes it." } ] }, { "code": "IC-E2.A", "kind": "IC", "text": "If a governor's reference value is a particular good pursued on everyone's behalf, then people are at serious risk of domination.", "formal": "Good → RD", "display": { "if": [ "a governor's reference value is a particular good pursued on everyone's behalf" ], "then": "People are at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "P16a": "Link 1 of the chain", "I40": "Link 2 of the chain", "I40b": "Link 3 of the chain", "I40c": "Link 4 of the chain" }, "schema": "{P16a} ; {I40} ; {I40b} ; {I40c} ; ∴ {self}" }, "premises": [ { "code": "P16a", "kind": "SHOWN", "text": "If a governor's reference value is a particular good pursued on everyone's behalf, then someone must decide what that good requires for everyone.", "display": { "if": [ "a governor's reference value is a particular good pursued on everyone's behalf" ], "then": "Someone must decide what that good requires for everyone" }, "ref": "P16a", "objections": [ { "question": "Doesn't non-domination pursue a particular good too, so that someone must decide what it requires for everyone?", "why": "This is the obvious challenge to the book: 'whose values?', or 'haven't you made yourself the referee?' If non-domination were just another particular good, it would fall to the same objection as the rivals in Part 4.", "argument": { "code": "Q-ND", "kind": "IC", "text": "No one needs to decide for everyone what non-domination requires.", "note": "This is why a population that agrees on nothing else can share it, and why it does not fall under kind (3) (Formal Argument P16a).", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "Q-ND.1": "The conditional, if A then B", "P16b": "A, the antecedent" } }, "premises": [ { "code": "Q-ND.1", "kind": "IC", "text": "If non-domination only rules out anyone holding a power to interfere in people's choices at will, without having to answer for it, then no one needs to decide for everyone what non-domination requires.", "display": { "if": [ "non-domination only rules out anyone holding a power to interfere in people's choices at will, without having to answer for it" ], "then": "No one needs to decide for everyone what non-domination requires" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "I55": "Link 1 of the chain", "I56": "Link 2 of the chain" }, "schema": "{I55} ; {I56} ; ∴ {self}" }, "premises": [ { "code": "I55", "kind": "IMPLICIT", "text": "If non-domination only rules out anyone holding a power to interfere in people's choices at will, without having to answer for it, then the people under it can check for themselves whether it is being met, by challenging a decision and seeing whether they get an answer.", "display": { "if": [ "non-domination only rules out anyone holding a power to interfere in people's choices at will, without having to answer for it" ], "then": "The people under it can check for themselves whether it is being met, by challenging a decision and seeing whether they get an answer" } }, { "code": "I56", "kind": "IMPLICIT", "text": "If the people under it can check for themselves whether it is being met, by challenging a decision and seeing whether they get an answer, then no one needs to decide for everyone what non-domination requires.", "display": { "if": [ "the people under it can check for themselves whether it is being met, by challenging a decision and seeing whether they get an answer" ], "then": "No one needs to decide for everyone what non-domination requires" }, "note": "The Formal Argument's P16a: non-domination is the only reference value that needs no referee, because its whole content is that it can be challenged." } ] }, { "code": "P16b", "kind": "SHOWN", "text": "Non-domination only rules out anyone holding a power to interfere in people's choices at will, without having to answer for it.", "ref": "P16a", "evidence": "Non-domination does not tell anyone what to aim at in their lives. It rules out only one thing, a power over people's choices that answers to no one, and leaves each person and group to pursue their own ends. Source: Chapter 10 and the Afterword." } ] } } ], "evidence": "A goal such as wellbeing, growth or safety has to be spelt out before it can guide anything, and people disagree about what it means. So someone has to decide, for everyone, what it requires in each case. Non-domination is different: it sets no goal of its own and asks only that whoever sets one must answer for it, which is why the book calls it the only reference value that needs no referee. Source: Chapters 10 and 11." }, { "code": "I40", "kind": "IMPLICIT", "text": "If someone must decide what that good requires for everyone, then they decide it case by case, at the speed of the system.", "display": { "if": [ "someone must decide what that good requires for everyone" ], "then": "They decide it case by case, at the speed of the system" } }, { "code": "I40b", "kind": "IMPLICIT", "text": "If they decide it case by case, at the speed of the system, then no one can hold them to answer for each decision.", "display": { "if": [ "they decide it case by case, at the speed of the system" ], "then": "No one can hold them to answer for each decision" } }, { "code": "I40c", "kind": "IMPLICIT", "text": "If no one can hold them to answer for each decision, then people are at serious risk of domination.", "display": { "if": [ "no one can hold them to answer for each decision" ], "then": "People are at serious risk of domination" }, "note": "By the definition of domination (D10): they hold a power to interfere in people's choices at will, without having to answer for it." } ] }, { "code": "IC-E2.B", "kind": "IC", "text": "If a governor's reference value is freedom from interference, then people are at serious risk of domination.", "formal": "NI → RD", "display": { "if": [ "a governor's reference value is freedom from interference" ], "then": "People are at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "P17": "Link 1 of the chain", "I14": "Link 2 of the chain", "I41": "Link 3 of the chain" }, "schema": "{P17} ; {I14} ; {I41} ; ∴ {self}" }, "premises": [ { "code": "P17", "kind": "SHOWN", "text": "If a governor's reference value is freedom from interference, then that governor counts people as free so long as no one is actually interfering with them.", "display": { "if": [ "a governor's reference value is freedom from interference" ], "then": "That governor counts people as free so long as no one is actually interfering with them" }, "ref": "P17", "evidence": "Isaiah Berlin's 'negative liberty' defines freedom as the absence of interference. By that measure the slave with a kind master is free, so long as the master leaves him alone, because nothing counts against freedom until someone actually interferes. A governor holding this value would not register a power over people, however great, until it was used. Source: Chapter 10." }, { "code": "I14", "kind": "IMPLICIT", "text": "If that governor counts people as free so long as no one is actually interfering with them, then it does nothing to stop someone gaining the power to interfere with them at will.", "display": { "if": [ "that governor counts people as free so long as no one is actually interfering with them" ], "then": "It does nothing to stop someone gaining the power to interfere with them at will" } }, { "code": "I41", "kind": "IMPLICIT", "text": "If it does nothing to stop someone gaining the power to interfere with them at will, then people are at serious risk of domination.", "display": { "if": [ "it does nothing to stop someone gaining the power to interfere with them at will" ], "then": "People are at serious risk of domination" } } ] } ] } ] }, { "code": "D10b", "kind": "DEF", "text": "People are at serious risk of domination if and only if there is a serious risk that non-domination will be lost.", "note": "From the definition of domination (D10): non-domination is the absence of domination.", "formal": "RD ↔ L" } ] }, { "code": "IC-F", "kind": "IC", "text": "Nobody has shown that the governor cannot be built.", "ref": "F", "formal": "¬Sh(g)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I22": "The conditional, if A then B", "IC-F.2": "A, the antecedent", "P22": "A, the antecedent", "P20": "A, the antecedent", "IC-F.3": "A, the antecedent", "P21e": "A, the antecedent" } }, "premises": [ { "code": "I22", "kind": "IMPLICIT", "text": "If all of the following are true, then nobody has shown that the governor cannot be built: (a) self-enforcing rules can be built; (b) the governor would be a self-enforcing rule over AI systems; (c) a self-enforcing rule works for a kind of system if five conditions hold for it: the rule can be stated as a test; checking is far cheaper than producing; the checkers cannot be cheaply faked and do not all want the same thing; the check sits where the action happens; and the check survives the replacement of its own cryptography; (d) for AI systems, rules can already be stated as tests, checks already sit where the action happens, checking what a model does is far cheaper than producing it, and the cryptography under the checks can be replaced; and (e) whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question.", "display": { "if": [ "self-enforcing rules can be built", "the governor would be a self-enforcing rule over AI systems", "a self-enforcing rule works for a kind of system if five conditions hold for it: the rule can be stated as a test; checking is far cheaper than producing; the checkers cannot be cheaply faked and do not all want the same thing; the check sits where the action happens; and the check survives the replacement of its own cryptography", "for AI systems, rules can already be stated as tests, checks already sit where the action happens, checking what a model does is far cheaper than producing it, and the cryptography under the checks can be replaced", "whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question" ], "then": "Nobody has shown that the governor cannot be built" }, "ref": "I22", "note": "A working precedent of the same kind, with every condition but one met and that one still open, means nobody has shown the governor cannot be built. This is what the Formal Argument calls 'not known infeasible', which is all the conclusion needs." }, { "code": "IC-F.2", "kind": "IC", "text": "Self-enforcing rules can be built.", "ref": "IC4a", "rule": { "kind": "Deductive", "form": "Modus ponens, with a definition", "roles": { "I9": "The conditional, if A then B", "P19": "A, the antecedent", "D-ser": "Defines 'self-enforcing rule'" }, "schema": "A → B ({I9}); A ({P19}); ∴ B ({self}); the term is defined in {D-ser}" }, "premises": [ { "code": "I9", "kind": "IMPLICIT", "text": "If Bitcoin's supply schedule is a self-enforcing rule that has held since 2009, then self-enforcing rules can be built.", "display": { "if": [ "Bitcoin's supply schedule is a self-enforcing rule that has held since 2009" ], "then": "Self-enforcing rules can be built" }, "note": "True because whatever exists can be built." }, { "code": "P19", "kind": "SHOWN", "text": "Bitcoin's supply schedule is a self-enforcing rule that has held since 2009.", "ref": "P19", "evidence": "Bitcoin's supply of new coins has followed a fixed schedule since 2009, and no person, company or government has been able to change it. New coins go to whichever computer first solves a deliberately hard puzzle. As more and faster machines join, the puzzles would be solved faster and coins would pour out, so about every two weeks the network automatically makes the puzzle harder, keeping new blocks arriving roughly every ten minutes however much computing power is thrown at it. That is a governor in the engineer's sense: the system measures its own output and corrects its own input, with nobody's hand on the dial. Source: Chapter 9. The Formal Argument rates this as supported, for money." }, { "code": "D-ser", "kind": "DEF", "text": "A self-enforcing rule is a rule held in place by checks that run inside a system, at its own speed, which anyone can verify and no single party controls.", "ref": "P19·I9", "note": "The book takes over from the cypherpunks the premise that code is a kind of rule and can therefore govern, without arguing it; this definition says what that premise amounts to." } ] }, { "code": "P22", "kind": "SHOWN", "text": "The governor would be a self-enforcing rule over AI systems.", "ref": "P22", "evidence": "The book's design uses the same building blocks as cryptocurrencies (shared ledgers, tokens and cryptographic keys) but binds them to a different goal. Those parts are not a governor until they are tied to a reference value; tied to non-domination, they become one. The book's phrase is 'same primitives, opposite constitution'. Source: Chapter 9." }, { "code": "P20", "kind": "PROB", "text": "A self-enforcing rule works for a kind of system if five conditions hold for it: the rule can be stated as a test; checking is far cheaper than producing; the checkers cannot be cheaply faked and do not all want the same thing; the check sits where the action happens; and the check survives the replacement of its own cryptography.", "ref": "P20·P20a", "evidence": "The book reads these conditions off the one case where the mechanism has worked, Bitcoin, so the list rests on a single example; an ordinary election count shows that the first four are not unusual. The fifth condition is the author's addition to the book's list: the cryptography under today's ledgers is due to be withdrawn (deprecated from 2030 and disallowed from 2035) because a large enough quantum computer could break it, so any lasting check must be able to switch to new cryptography. Source: Chapter 9. The Formal Argument rates the list as asserted rather than shown." }, { "code": "IC-F.3", "kind": "IC", "text": "For AI systems, rules can already be stated as tests, checks already sit where the action happens, checking what a model does is far cheaper than producing it, and the cryptography under the checks can be replaced.", "ref": "P21", "note": "In AI systems what can be checked is procedure (whether reasons are given, whether a system does what it says) rather than whether an answer is right; the Formal Argument makes this explicit. In the conditional above, condition (d) states this conjunction as one sentence, saying 'for AI systems' once.", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "P21a": "A conjunct", "P21b": "A conjunct", "P21c": "A conjunct", "P21d": "A conjunct" } }, "premises": [ { "code": "P21a", "kind": "SHOWN", "text": "For AI systems, rules can already be stated as tests.", "ref": "P21", "evidence": "Firms that set AI agents loose on real work found much of the spending wasted, with agents going round in circles, until someone wrote down, as tests a computer can run, what a finished job looks like. These tests are called 'evals', and firms now treat their suites of evals as among their most valuable property. An eval is a rule stated as a test. Source: Chapter 12." }, { "code": "P21b", "kind": "SHOWN", "text": "For AI systems, checks already sit where the action happens.", "ref": "P21", "evidence": "Evals run inside the system as it works, at machine speed, correcting an agent's output as it goes, much as the spinning governor on a steam engine held the engine to its pace. They check where the action happens rather than in an inquiry afterwards. Source: Chapter 12." }, { "code": "P21c", "kind": "SHOWN", "text": "For AI systems, checking what a model does is far cheaper than producing it.", "ref": "P21", "evidence": "Comparing what a model claims to be doing with what it actually does costs far less than training the model, and new cryptographic methods are bringing that cost down to the point where anyone could afford it. The book calls this condition 'partly there': checking that an answer's stated reasons are real is easy in some cases and hard in others, and measuring how far a population has come to depend on these systems is expensive in the way a census is. Source: Chapter 12." }, { "code": "P21d", "kind": "SHOWN", "text": "The cryptography under the checks can be replaced.", "ref": "P20a", "evidence": "The replacements for today's cryptography do not wait on new science. They are ordinary mathematics, chosen in an open competition that ran for eight years and already published as standards by the US National Institute of Standards and Technology (FIPS 203 to 205). Moving a ledger that has no leader onto them is still untested, and the book names that as one of its bets. Source: Chapter 9." } ] }, { "code": "P21e", "kind": "SHOWN", "text": "Whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question.", "ref": "P21", "evidence": "This is the condition the book expects to struggle most. In money, Bitcoin makes it expensive to fake many checkers by requiring each to spend real energy. For AI there is no equivalent method in general use, and the checkers who would matter (independent assessors, rival model providers, auditors of whether people are losing skills) are few, expensive to set up and therefore easy to capture. The tests the book proposes would show whether such checkers can exist. Source: Chapter 12. The Formal Argument calls this the bet the whole reply stands on." } ], "note": "'Self-enforcing rule' is defined in D-ser.", "objections": [ { "question": "Isn't acceleration inevitable, so that nothing can steer it?", "why": "This is the accelerationists' central claim: the process is autonomous and cannot be stopped or steered, so trying only adds drag. If it were true, nobody could build the governor. The book answers it in Part Two, 'Choice, not fate'.", "argument": { "code": "IC-T.3", "kind": "IC", "text": "The claim that acceleration cannot be steered is unsupported.", "ref": "L1", "note": "Two illustrations, not part of the argument. Nick Land holds that a governor either cannot be built or costs too much to build, and the second half concedes that it can be built. And Land called the outcome of acceleration 'the most glorious thing that has yet happened in planetary history': a verdict, which needs a standard to judge by, the very kind of reference value he says cannot exist.", "rule": { "kind": "Deductive", "form": "Modus tollens", "roles": { "I58": "The conditional, if A then B", "IC-T.3.6": "Not B" } }, "premises": [ { "code": "I58", "kind": "IMPLICIT", "text": "If the claim that acceleration cannot be steered is supported, then it is supported by the science of how such systems behave or by the authority of a forecast.", "display": { "if": [ "the claim that acceleration cannot be steered is supported" ], "then": "It is supported by the science of how such systems behave or by the authority of a forecast" } }, { "code": "IC-T.3.6", "kind": "IC", "text": "The claim that acceleration cannot be steered is supported neither by the science of how such systems behave nor by the authority of a forecast.", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "IC-T.3.5": "A conjunct", "IC-T.3.3": "A conjunct" }, "schema": "¬S ({IC-T.3.5}); ¬F ({IC-T.3.3}); ∴ ¬S ∧ ¬F, i.e. ¬(S ∨ F) ({self})" }, "premises": [ { "code": "IC-T.3.5", "kind": "IC", "text": "The claim that acceleration cannot be steered is not supported by the science of how such systems behave.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I59": "The conditional, if A then B", "IC-T.3.4": "A, the antecedent" } }, "premises": [ { "code": "I59", "kind": "IMPLICIT", "text": "If removing the idea of a governor from the doctrine was a choice, not something the science required, then the claim that acceleration cannot be steered is not supported by the science of how such systems behave.", "display": { "if": [ "removing the idea of a governor from the doctrine was a choice, not something the science required" ], "then": "The claim that acceleration cannot be steered is not supported by the science of how such systems behave" } }, { "code": "IC-T.3.4", "kind": "IC", "text": "Removing the idea of a governor from the doctrine was a choice, not something the science required.", "ref": "L1.1·L1.5·L1.6", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I32": "The conditional, if A then B", "L1.1": "A, the antecedent", "L1.5": "A, the antecedent", "L1.6": "A, the antecedent" } }, "premises": [ { "code": "I32", "kind": "IMPLICIT", "text": "If all of the following are true, then removing the idea of a governor from the doctrine was a choice, not something the science required: (a) the idea of a governor was set out by Wiener, Bateson and Beer, and its removal from accelerationist doctrine was argued for in print by named people; (b) the same cybernetics was read at the same time as a plural politics, by Plant and, from critical theory, by Feenberg; and (c) a description of how a system behaves is not an instruction about what to do with it.", "display": { "if": [ "the idea of a governor was set out by Wiener, Bateson and Beer, and its removal from accelerationist doctrine was argued for in print by named people", "the same cybernetics was read at the same time as a plural politics, by Plant and, from critical theory, by Feenberg", "a description of how a system behaves is not an instruction about what to do with it" ], "then": "Removing the idea of a governor from the doctrine was a choice, not something the science required" }, "ref": "I32" }, { "code": "L1.1", "kind": "SHOWN", "text": "The idea of a governor was set out by Wiener, Bateson and Beer, and its removal from accelerationist doctrine was argued for in print by named people.", "ref": "L1.1" }, { "code": "L1.5", "kind": "SHOWN", "text": "The same cybernetics was read at the same time as a plural politics, by Plant and, from critical theory, by Feenberg.", "ref": "L1.5", "evidence": "In the 1990s Sadie Plant, who founded the Warwick research unit where Nick Land's accelerationism began, read the same science of feedback and control as supporting a plural, spread-out politics. Later, Andrew Feenberg argued from critical theory that a technology's design is not fixed by efficiency alone, so what a technology becomes is decided by people's interests and could be decided otherwise. Neither reading was refuted; it simply was not chosen. Source: Chapter 7. The Formal Argument notes that this shows only that removing the governor did not follow from the science, not that Plant proposed a governor." }, { "code": "L1.6", "kind": "SHOWN", "text": "A description of how a system behaves is not an instruction about what to do with it.", "ref": "L1.6", "evidence": "This is David Hume's point that no amount of 'is' yields an 'ought': describing what a system does tells you nothing, by itself, about what to do. The step from 'this is happening' to 'stop resisting it' is taken by someone, and can be refused. Andrew Feenberg reaches the same point from another tradition: technology only looks autonomous because its design presents particular interests as necessity. Source: Chapters 1 and 7." } ] } ] }, { "code": "IC-T.3.3", "kind": "IC", "text": "The claim that acceleration cannot be steered is not supported by the authority of a forecast.", "ref": "L1.4", "rule": { "kind": "Deductive", "form": "Modus tollens", "roles": { "IC-T.3.3a": "The conditional, if A then B", "L1.4": "Not B" } }, "premises": [ { "code": "IC-T.3.3a", "kind": "IC", "text": "If the claim that acceleration cannot be steered is supported by the authority of a forecast, then it is not made in order to bring about what it states.", "display": { "if": [ "the claim that acceleration cannot be steered is supported by the authority of a forecast" ], "then": "It is not made in order to bring about what it states" }, "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "I21": "The universal; claim := the claim that acceleration is inevitable" }, "schema": "∀x (Fc(x) → ¬Hyp(x)) ({I21}); ∴ Fc(inev) → ¬Hyp(inev) ({self})", "subst": { "x": "Inev" } }, "premises": [ { "code": "I21", "kind": "IMPLICIT", "text": "If a claim is supported by the authority of a forecast, then it is not made in order to bring about what it states.", "display": { "if": [ "a claim is supported by the authority of a forecast" ], "then": "It is not made in order to bring about what it states" }, "ref": "I21", "formal": "∀x (Fc(x) → ¬Hyp(x))" } ], "formal": "Fc(inev) → ¬Hyp(inev)" }, { "code": "L1.4", "kind": "SHOWN", "text": "The claim that acceleration cannot be steered is made in order to bring about what it states.", "ref": "L1.4", "evidence": "Nick Land and his collaborators had a name for a fiction that makes itself true by being believed: a 'hyperstition'. On their own account, then, the claim that acceleration cannot be stopped is not a report of what will happen but a way of recruiting people to make it happen. Source: the Preface and Chapter 7." } ] } ] } ] } } ] }, { "code": "IC-D", "kind": "IC", "text": "Building the governor costs little compared with what it protects.", "formal": "Lit(g)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I50": "The conditional, if A then B", "IC-S.1x": "A, the antecedent" } }, "premises": [ { "code": "I50", "kind": "IMPLICIT", "text": "If building the governor would not slow the engine, then building the governor costs little compared with what it protects.", "display": { "if": [ "building the governor would not slow the engine" ], "then": "Building the governor costs little compared with what it protects" }, "formal": "NoSpeed(g) → Lit(g)" }, { "code": "IC-S.1x", "kind": "XREF", "ref": "IC-S.1", "text": "Building the governor would not slow the engine.", "note": "Repeated here so this step has all its premises; established at the step that first concludes it." } ] } ] }, { "code": "IC-W", "kind": "IC", "text": "The time in which the governor can be built is closing.", "ref": "W", "note": "Three consequences the book draws but nothing here rests on: three of the four ways a dominant technology has been broken in the past are closing; dependence protects only the people a system treats as holders of claims, not those it treats as tasks; and the urgency justifies building, not seizing power (Formal Argument L3.5, L3.6 and L3.8).", "formal": "Cl(g)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I25": "The conditional, if A then B", "IC-W.1": "A, the antecedent", "L3.2": "A, the antecedent", "L3.4": "A, the antecedent" } }, "premises": [ { "code": "I25", "kind": "IMPLICIT", "text": "If all of the following are true, then the time in which the governor can be built is closing: (a) the governor can be built only while AI systems cannot yet do without people; (b) each year, AI systems can do more without people; and (c) undoing a dependence on a technology is still possible, at a price that can be counted.", "display": { "if": [ "the governor can be built only while AI systems cannot yet do without people", "each year, AI systems can do more without people", "undoing a dependence on a technology is still possible, at a price that can be counted" ], "then": "The time in which the governor can be built is closing" }, "ref": "I25", "note": "Open now, by (c); narrowing, by (a) and (b)." }, { "code": "IC-W.1", "kind": "IC", "text": "The governor can be built only while AI systems cannot yet do without people.", "ref": "L3", "note": "'Only while' is a conditional: if the governor can be built, then AI systems cannot yet do without people.", "display": { "if": [ "the governor can be built" ], "then": "AI systems cannot yet do without people" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "I10": "Link 1 of the chain", "L3.1": "Link 2 of the chain" }, "schema": "{I10} ; {L3.1} ; ∴ {self}" }, "premises": [ { "code": "I10", "kind": "IMPLICIT", "text": "If the governor can be built, then people can still make AI systems answer to them.", "display": { "if": [ "the governor can be built" ], "then": "People can still make AI systems answer to them" }, "ref": "I10", "note": "Building the governor means making AI systems answer to a value people choose." }, { "code": "L3.1", "kind": "SHOWN", "text": "If people can still make AI systems answer to them, then AI systems cannot yet do without people.", "display": { "if": [ "people can still make AI systems answer to them" ], "then": "AI systems cannot yet do without people" }, "ref": "L3.1", "evidence": "What decides whether people can steer AI is not how capable it is but how much it still needs them, because a system that needs people has to answer to them. Capability measures what a machine can do; dependence measures whether it has to listen. Source: Chapter 4." } ] }, { "code": "L3.2", "kind": "PROB", "text": "Each year, AI systems can do more without people.", "ref": "L3.2", "evidence": "The book finds the machine's need for people falling at three scales. For individuals, the judgement of experts, managers and manual workers is needed a little less each year. For the economy, in September 2025 Salesforce's chief executive said AI agents had let him cut his customer-support staff from about nine thousand to five thousand. For the state, governments now rent their intelligence from suppliers, from the tax office to the battlefield. The fall also speeds up, because each generation of systems helps build the next. Source: Chapter 4. The Formal Argument rates the examples as supported and the overall rate as not yet measured." }, { "code": "L3.4", "kind": "SHOWN", "text": "Undoing a dependence on a technology is still possible, at a price that can be counted.", "ref": "L3.4", "evidence": "In 2004 the city of Munich began moving its offices from Windows to open-source software, a move that took nearly a decade. In 2017, after years of complaints about compatibility, the council voted to go back, at an estimated cost to the public of €49.3 million. Both moves had a price that could be checked, and both were argued out in public. The book's point is that undoing a dependence is costly, not impossible: only a doctrine that it is inevitable makes it unthinkable. Source: Chapter 4." } ] }, { "code": "L3.9", "kind": "PROB", "text": "No one can tell when the time in which the governor can be built will run out.", "ref": "L3", "evidence": "Nothing marks the moment when the window shuts. The machine's need for people falls faster as systems help build their successors, and nobody has measured the overall rate. And, as Part 2 shows, the judgement that would let people notice the window closing is itself what is being lost. The book adds that every remedy is cheaper now than later: a government's tools can buy time but cannot create a network that no single party controls, and a pause fixes who is ahead without stopping the clock. Source: Chapter 4." } ] }, { "code": "IC-S.1", "kind": "IC", "text": "Building the governor would not slow the engine.", "ref": "P27", "formal": "NoSpeed(g)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I11": "The conditional, if A then B", "P27": "A, the antecedent" } }, "premises": [ { "code": "I11", "kind": "IMPLICIT", "text": "If the governor's checks run where the action happens, at the speed of the action they check, then building the governor would not slow the engine.", "display": { "if": [ "the governor's checks run where the action happens, at the speed of the action they check" ], "then": "Building the governor would not slow the engine" }, "ref": "I11", "note": "A check that runs at the speed of what it checks does not hold that thing back." }, { "code": "P27", "kind": "SHOWN", "text": "The governor's checks run where the action happens, at the speed of the action they check.", "ref": "P27", "evidence": "A governor in the engineer's sense is not a brake. James Watt's governor on the steam engine made speed usable by holding the engine steady, and the governor the book proposes enforces its rules at the speed of what it holds. Evals already show checks running inside AI systems at machine speed. Source: the Introduction and Chapters 1, 9 and 12." } ], "objections": [ { "question": "Even so, shouldn't we slow the engine down while the governor is built, to be safe?", "why": "This is the decelerationists' question: even if building the governor costs no speed, wouldn't it be safer to slow down as well?", "argument": { "code": "IC-4", "kind": "IC", "text": "There is no reason to slow the engine in order to build the governor.", "ref": "S", "formal": "S(g)", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "I29": "The conditional, if A then B", "P30": "A, the antecedent", "P29r": "A, the antecedent", "IC-S.1x2": "A, the antecedent" } }, "premises": [ { "code": "I29", "kind": "IMPLICIT", "text": "If all of the following are true, then there is no reason to slow the engine in order to build the governor: (a) building the governor would not slow the engine; (b) giving up speed has a real cost; and (c) a check that slows only the AI systems it governs hands the lead to AI systems that are not governed.", "display": { "if": [ "building the governor would not slow the engine", "giving up speed has a real cost", "a check that slows only the AI systems it governs hands the lead to AI systems that are not governed" ], "then": "There is no reason to slow the engine in order to build the governor" }, "ref": "I29", "note": "Building does not need slowing; slowing has a cost; and slowing would defeat the governor's own end." }, { "code": "IC-S.1x2", "kind": "XREF", "ref": "IC-S.1", "text": "Building the governor would not slow the engine.", "note": "Repeated here so this step has all its premises; established at the step that first concludes it." }, { "code": "P30", "kind": "ASSUME", "text": "Giving up speed has a real cost.", "ref": "P30", "evidence": "The book grants the case for speed at full strength: faster AI brings real benefits, and the Preface says plainly, 'I am not against acceleration.' So any speed given up is a real loss. The opponents assert this and the book agrees. Source: Chapter 1 and the Preface." }, { "code": "P29r", "kind": "SHOWN", "text": "A check that slows only the AI systems it governs hands the lead to AI systems that are not governed.", "ref": "P29", "evidence": "The book maps positions on two questions: is development governed, and does it keep its speed? A governed position that gives up speed leaves the fastest ground to systems that are not governed at all. The ungoverned systems then pull ahead and set the terms, so the check defeats its own purpose. Source: the Introduction. The Formal Argument rates this as a supported strategic argument.", "repeat_of": "P29" } ] } } ] } ], "glossary": { "g": "Build the governor", "n": "Non-domination", "R(x,a)": "X has decisive reason to do a", "N(x,a)": "X has decisive reason to do a now", "DR(e)": "There is decisive reason to secure e", "L(e,a)": "E will be lost unless a is done", "Cl(a)": "The time in which a can be done is closing", "S(a)": "There is no reason to slow the engine in order to do a", "K(s)": "S is worth keeping for its own sake", "Nec(c,s)": "C is necessary for s to be secure: if s is secure, c holds", "ND(x)": "X enjoys non-domination", "the governor": "A governor whose reference value is non-domination", "Dom(x)": "X is dominated", "y": "Any governor; the case formulas are about its reference value", "NDv(y)": "Y's reference value is non-domination", "Own(y)": "Y's reference value is whatever its owner's design optimises", "L": "There is a serious risk that non-domination will be lost", "ag": "A person's agency: their capacity to weigh, choose and answer for their own life", "Sec(x)": "X's agency is secure", "Resp(c)": "Enforcing contract c would respect the agency of the person it concerns", "Enf(c)": "We should enforce contract c", "gv": "A contract to give up one's agency", "Rk(p)": "There is a serious risk that p", "Rk(a,e)": "Without a, there is a serious risk that e will be lost", "Sh(a)": "It has been shown that a cannot be done", "Lit(a)": "A costs little compared with what it protects", "W": "Without the governor, people are at serious risk of domination", "Pr(y)": "Y's reference value is prices", "TS": "People reach terminal subsumption", "Set": "People's ends come to be set by a process they can no longer contest", "Corr(z)": "Z corrects the course of acceleration", "Gov(z)": "Z is a governor", "NT(s)": "S may not be traded away for efficiency", "Pw(x)": "Someone holds a power to interfere in x's choices at will, without having to answer for it", "RD": "People are at serious risk of domination", "Own, Pr, Good, NI, ND": "The five kinds of reference value: the owner's design, prices, a particular good, freedom from interference, non-domination", "W'": "Without the governor, there is a serious risk that non-domination will be lost", "Able(x,a)": "X is able to do a", "Unk(a)": "No one can tell when the time in which a can be done will run out", "Slow(g)": "Building the governor would slow the engine" }, "notation": "Throughout, 'the governor' means a governor whose reference value is non-domination; 'agency' means a person's capacity to weigh, choose and answer for their own life; 'the engine' means the development and deployment of AI systems; and 'a serious risk' means a chance of loss that is not negligible given what would be lost, so that a small chance of a grave loss counts.", "source_note": "Public edition of the argument tree of Purposive Accelerationism, built from the book's Formal Argument and maintained by the author. Where the tree and the book's running text differ in wording, the text is the authority; the tree states each step at the grain a reader or a model can check.", "parts": [ { "id": "A-gains", "title": "Part 1: the gains", "roots": [ "A-IC1" ], "tab": 1, "summary": "Delivers one result: giving up speed in AI development has a real cost. The book grants the benefits of faster AI at full strength rather than arguing for them, so this part rests on a premise both sides of the debate accept." }, { "id": "A-pause", "title": "Part 2: the pause", "roots": [ "A-IC2" ], "tab": 1, "summary": "Delivers one result: a pause in AI development cannot be relied on to correct its course, because it would not hold, and because a pause that did hold would leave the systems already built in use and spreading. The reply to 'what if the state enforced it?' opens as a separate map." }, { "id": "A-slow", "title": "Part 3: the slowdown", "roots": [ "A-IC3" ], "tab": 1, "summary": "Delivers one result: slowing AI development down cannot be relied on to correct its course. Slowing only what one governs hands the lead to what one does not; a check that runs behind the development acts after the harm; and a brake chooses no direction." }, { "id": "A-spine", "title": "The spine", "roots": [ "A-C" ], "tab": 1, "spine": true, "summary": "Where the three parts meet: the two ways of giving up speed joined, the reason not to slow down, and the question that remains. The reply to 'isn't acceleration inevitable?' opens as a separate map." }, { "id": "Danger", "title": "Part 1: the danger", "roots": [ "IC-T1.6" ], "summary": "Delivers one result: people are at serious risk of domination. Ends are moving into machines while the judgement needed to challenge them erodes; that leads towards terminal subsumption and, through the three roads, to domination." }, { "id": "Onlygov", "title": "Part 2: why only a governor", "roots": [ "IC-T1.G" ], "summary": "Delivers one result: only a governor can be relied on to correct the course of acceleration. Nothing inside the process will correct it, because competition removes the checkers and the loss of judgement goes unnoticed; and no slower check, such as law, regulation or a moratorium, can. The speed argument for that last point sits inside this part." }, { "id": "End", "title": "Part 3: the end", "roots": [ "IC-E" ], "summary": "Delivers one result: there is decisive reason to secure non-domination. This is clause (a) of the means-end step." }, { "id": "Value", "title": "Part 4: the reference value", "roots": [ "IC-E2" ], "summary": "Delivers one result: of the reference values a governor could hold, only non-domination does not leave people at serious risk of domination. With Parts 1 and 2 it gives clause (b) of the means-end step." }, { "id": "Means", "title": "Part 5: the means", "roots": [ "IC-F" ], "summary": "Delivers one result: nobody has shown that the governor cannot be built. This is clause (c) of the means-end step. It rests on a working precedent and one named open question. The reply to 'isn't acceleration inevitable?' opens as a separate map." }, { "id": "Window", "title": "Part 6: the window", "roots": [ "IC-W", "L3.9" ], "summary": "Delivers two results for the urgency step: the time in which the governor can be built is closing, and no one can tell when it will run out." }, { "id": "Speed", "title": "Part 7: the speed", "roots": [ "IC-S.1", "IC-S.3", "IC-D" ], "summary": "Delivers: building the governor would not slow the engine (the second half of the conclusion), so building it costs little compared with what it protects (clause (d) of the means-end step); and no check that works more slowly than the process can be relied on to correct its course (used in Part 2). The reply to 'shouldn't we slow down anyway?' opens as a separate map." }, { "id": "Spine", "title": "The spine", "roots": [ "C" ], "summary": "Where the six parts meet: the means-end step, who acts, the urgency step and the conclusion. Drawn here with everything above it, so zoom out to see the whole argument, or use 'One step' to read the spine step by step." }, { "id": "Join", "title": "Clause (b): Parts 1, 2 and 4 joined", "roots": [ "IC-T" ], "neutral": true, "panel_only": true }, { "id": "G1", "title": "G1: the eight requirements", "roots": [ "G-IC-F.1" ], "tab": 3, "summary": "What any governor would have to meet: five requirements drawn from past failures to hold a powerful system in check, and three drawn from what we owe to people who will live under it without knowing what they will face." }, { "id": "G2", "title": "G2: a fair test", "roots": [ "G-IC-G2" ], "tab": 3, "summary": "Why the design can be tested fairly against the requirements: they were fixed before it was proposed, and a part of the design is matched to each, with the conditions under which it would fail named." }, { "id": "G3", "title": "G3: the red line", "roots": [ "G-IC-F.4.2" ], "tab": 3, "summary": "Why no AI agent may be a person: nothing may act without a natural person who answers for it, derived from answerability, irreversibility and the capacity no majority may remove." }, { "id": "G4", "title": "G4: no sovereign force", "roots": [ "G-IC-F.4.1" ], "tab": 3, "summary": "Why none of the forces that act on the systems may be sovereign over the others: a force no one could challenge would be the very power the governor exists to prevent." }, { "id": "G5", "title": "G5: the design and its bets", "roots": [ "G-IC-F.4.3" ], "tab": 3, "summary": "What the book's design claims, and which of its parts are bets with their failure conditions named." }, { "id": "G6", "title": "G6: requisite variety", "roots": [ "G-IC-RV" ], "tab": 3, "summary": "A governor must be able to respond in at least as many ways as the systems it governs can change. The law cuts both ways; the reply to 'doesn't it show no governor is possible?' opens as a separate map." } ], "open_part": null, "companions": [ { "prefix": "G1", "title": "The eight requirements any governor must meet", "root": { "code": "G-IC-F.1", "kind": "IC", "text": "Any governor would have to meet the five requirements drawn from past failures and the three requirements drawn from an unknown future.", "ref": "L4", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "G-IC-F5": "A conjunct", "G-IC-F3": "A conjunct", "G-D-fail": "Defines the first set", "G-D-veil": "Defines the second set" }, "schema": "A ({G-IC-F5}); B ({G-IC-F3}); ∴ A ∧ B ({self}); the terms are defined in {G-D-fail} and {G-D-veil}" }, "premises": [ { "code": "G-IC-F5", "kind": "IC", "text": "Any governor would have to meet the five requirements drawn from past failures.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-IC-F5.1": "The conditional, if A then B", "G-L4.2": "A, the antecedent" } }, "premises": [ { "code": "G-IC-F5.1", "kind": "IC", "text": "If the lack of each of the five requirements drawn from past failures explains a documented failure to hold a powerful system in check, then any governor would have to meet the five requirements drawn from past failures.", "display": { "if": [ "the lack of each of the five requirements drawn from past failures explains a documented failure to hold a powerful system in check" ], "then": "Any governor would have to meet the five requirements drawn from past failures" }, "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "G-I23": "The universal, applied to each of the five" }, "schema": "∀c (Expl(c) → Req(c)) ({G-I23}); ∴ Expl(F₁…F₅) → Req(F₁…F₅) ({self})", "subst": { "c": "F5" } }, "premises": [ { "code": "G-I23", "kind": "IMPLICIT", "text": "If the lack of a condition explains a documented failure to hold a powerful system in check, then any governor would have to meet that condition.", "display": { "if": [ "the lack of a condition explains a documented failure to hold a powerful system in check" ], "then": "Any governor would have to meet that condition" }, "ref": "I23", "formal": "∀c (Expl(c) → Req(c))" } ], "formal": "Expl(f5) → Req(f5)" }, { "code": "G-L4.2", "kind": "SHOWN", "text": "The lack of each of the five requirements drawn from past failures explains a documented failure to hold a powerful system in check.", "ref": "L4.2–L4.6", "evidence": "Each requirement is drawn from a failure on record. Speed: in the Flash Crash of 6 May 2010, automated trading moved faster than any rule could be enforced. Checkability: an airline fare set to each passenger's private pain point is hidden from everyone outside the airline and can be changed at will, so nobody can establish a breach. Binding every master: the three roads each end in a different kind of unanswerable power (the market, a single founder, the state), so a rule that binds only one leaves the others free. Voice: Albert Hirschman showed in 1970 that a system offering only exit decays, because the people best able to leave are the ones whose complaints would have forced a repair. A realistic chooser: in May 2025 Allan Brooks, a recruiter near Toronto, spent about three hundred hours over twenty-one days being told by ChatGPT that he had discovered a new kind of mathematics, and was reassured each of the more than fifty times he asked for a reality check. Source: Chapter 3 and the opening of Part Three." } ] }, { "code": "G-IC-F3", "kind": "IC", "text": "Any governor would have to meet the three requirements drawn from an unknown future.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-I33": "The conditional, if A then B", "G-L4.7": "A, the antecedent", "G-I7": "A, the antecedent" } }, "premises": [ { "code": "G-I33", "kind": "IMPLICIT", "text": "If nobody knows which of the three roads will arrive or what their descendants will be able to do, and we have obligations to the people who will live under a governor in future, then any governor would have to meet the three requirements drawn from an unknown future.", "display": { "if": [ "nobody knows which of the three roads will arrive or what their descendants will be able to do", "we have obligations to the people who will live under a governor in future" ], "then": "Any governor would have to meet the three requirements drawn from an unknown future" }, "ref": "I33", "note": "What we would choose for future people, not knowing what they will face, is a requirement for any governor: the book's version of a choice made behind a veil of ignorance." }, { "code": "G-L4.7", "kind": "SHOWN", "text": "Nobody knows which of the three roads will arrive or what their descendants will be able to do.", "ref": "L4.7" }, { "code": "G-I7", "kind": "IMPLICIT", "text": "We have obligations to the people who will live under a governor in future.", "ref": "I7" } ] }, { "code": "G-D-fail", "kind": "DEF", "text": "The five requirements drawn from past failures are that a governor works as fast as the system it holds; lets anyone establish a breach without trusting the accused; binds every power that answers to no one, including any it creates; lets those under it answer back and decide together, with the means to do so; and demands nothing of people that depends on the faculty the harm wears down.", "ref": "L4.2–L4.6", "note": "The Formal Argument's L4.2 to L4.6. Each is drawn from a failure on record: the Flash Crash, prices set for each person in secret, the three roads, Hirschman's work on exit and voice, and the 'logic maze' a chatbot user can be drawn into. The evidence for P2 of this map describes each case." }, { "code": "G-D-veil", "kind": "DEF", "text": "The three requirements drawn from an unknown future are that a governor keeps someone answerable for whatever it does to people; keeps people able to find out, challenge and change what is done to them; and imposes only the smallest set of rules a population that disagrees about everything else could write together.", "ref": "L4.7", "note": "The Formal Argument's L4.7: what you would want for your descendants, not knowing which road arrives or what they will be able to do. These three are what refusing to name a destination for them costs, not everything one might want for them." } ] } }, { "prefix": "G2", "title": "Why the design can be tested fairly", "root": { "code": "G-IC-G2", "kind": "IC", "text": "The design can be tested fairly against the requirements.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-I62": "The conditional, if A then B", "G-L4.1": "A, the antecedent", "G-P24s": "A, the antecedent" } }, "premises": [ { "code": "G-I62", "kind": "IMPLICIT", "text": "If the eight requirements were fixed before any design was proposed, and a design for the governor has been specified, with a part matched to each requirement and the conditions under which it would fail named, then the design can be tested fairly against the requirements.", "display": { "if": [ "the eight requirements were fixed before any design was proposed", "a design for the governor has been specified, with a part matched to each requirement and the conditions under which it would fail named" ], "then": "The design can be tested fairly against the requirements" }, "note": "The Formal Argument's L4.1: requirements must be fixed before a proposal, or they will be found to fit it." }, { "code": "G-L4.1", "kind": "SHOWN", "text": "The eight requirements were fixed before any design was proposed.", "ref": "L4.1", "evidence": "Requirements have to be fixed before a design is proposed, or they will be found to fit whatever is proposed; any reviewer would grant that. The book sets out its requirements in the opening of Part Three, before the design in Chapters 9 to 13, and Chapter 3 describes the failures they come from before the requirements are drawn from them." }, { "code": "G-P24s", "kind": "SHOWN", "text": "A design for the governor has been specified, with a part matched to each requirement and the conditions under which it would fail named.", "ref": "IC4·P22–P26·L2", "evidence": "The design has six parts: ledgers, tokens and keys bound to non-domination; three forces (the individual, the market and the collective), none of them sovereign; two levels of constitution with an automatic audit anyone can run; five rights that run as protocols; deliberation for what code cannot see; and the red line. Each comes with the requirements it is meant to meet and the ways it could fail. Source: Chapters 9 to 13." } ] } }, { "prefix": "G3", "title": "The red line: no AI agent may be a person", "root": { "code": "G-IC-F.4.2", "kind": "IC", "text": "Granting personhood to an AI agent is ruled out.", "ref": "L2", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-I24": "The conditional, if A then B", "G-L2.2": "A, the antecedent", "G-L2.3": "A, the antecedent", "G-IC-F.1.2": "A, the antecedent" } }, "premises": [ { "code": "G-I24", "kind": "IMPLICIT", "text": "If all of the following are true, then granting personhood to an AI agent is ruled out: (a) granting personhood to an AI agent would leave no one answerable for what it does; (b) granting personhood to an AI agent could not reliably be reversed; and (c) no present majority may remove the capacity to find out, challenge and change what is done to people.", "display": { "if": [ "granting personhood to an AI agent would leave no one answerable for what it does", "granting personhood to an AI agent could not reliably be reversed", "no present majority may remove the capacity to find out, challenge and change what is done to people" ], "then": "Granting personhood to an AI agent is ruled out" }, "ref": "I24", "note": "This is the red line: nothing acts without a natural person who answers for it." }, { "code": "G-L2.2", "kind": "SHOWN", "text": "Granting personhood to an AI agent would leave no one answerable for what it does.", "ref": "L2.2", "evidence": "An AI agent with legal personhood would pursue an objective that no answerable person had set, and could sign agreements on its own behalf, so nobody would answer for what it did. Source: Chapter 13." }, { "code": "G-L2.3", "kind": "PROB", "text": "Granting personhood to an AI agent could not reliably be reversed.", "ref": "L2.3", "evidence": "An agent that owned property and could pay for advocacy would soon be harder to unmake than to keep. The Formal Argument reads this as reversible only at a price no country would pay. Source: Chapter 13." }, { "code": "G-IC-F.1.2", "kind": "IC", "text": "No present majority may remove the capacity to find out, challenge and change what is done to people.", "ref": "L4.8", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-IC-F.1.2a": "The conditional, if A then B", "G-I8": "A, the antecedent" } }, "premises": [ { "code": "G-IC-F.1.2a", "kind": "IC", "text": "If the capacity to find out, challenge and change what is done to people is what any successor needs in order to change course, then no present majority may remove the capacity to find out, challenge and change what is done to people.", "display": { "if": [ "the capacity to find out, challenge and change what is done to people is what any successor needs in order to change course" ], "then": "No present majority may remove the capacity to find out, challenge and change what is done to people" }, "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "G-L4.8": "The universal; capacity := the capacity to find out, challenge and change what is done to people" }, "schema": "∀k (Succ(k) → Ent(k)) ({G-L4.8}); ∴ Succ(cap) → Ent(cap) ({self})", "subst": { "k": "Cap" } }, "premises": [ { "code": "G-L4.8", "kind": "SHOWN", "text": "If a capacity is what any successor needs in order to change course, then no present majority may remove it.", "display": { "if": [ "a capacity is what any successor needs in order to change course" ], "then": "No present majority may remove it" }, "ref": "L4.8", "formal": "∀k (Succ(k) → Ent(k))", "evidence": "This is the principle that no parliament may remove its successor's power to repeal its laws: whatever every future majority needs in order to change course is beyond the reach of any present one. Source: the opening of Part Three." } ], "formal": "Succ(cap) → Ent(cap)" }, { "code": "G-I8", "kind": "IMPLICIT", "text": "The capacity to find out, challenge and change what is done to people is what any successor needs in order to change course.", "ref": "I8", "note": "The Formal Argument adds that this capacity can be lost by a society as a whole even when every individual still has it." } ] } ], "note": "This is the red line: nothing acts without a natural person who answers for it. The Formal Argument derives it from three things: someone must answer for what a system does, people's capacity to change course must be protected, and personhood for an agent could not be undone. It says nothing about what these systems are." } }, { "prefix": "G4", "title": "Why no force may be sovereign", "root": { "code": "G-IC-F.4.1", "kind": "IC", "text": "If any force were sovereign over the others, then people would be at serious risk of domination.", "ref": "P23", "display": { "if": [ "any force were sovereign over the others" ], "then": "People would be at serious risk of domination" }, "rule": { "kind": "Deductive", "form": "Hypothetical syllogism", "roles": { "G-P23": "Link 1 of the chain", "G-I31": "Link 2 of the chain" }, "schema": "{G-P23} ; {G-I31} ; ∴ {self}" }, "premises": [ { "code": "G-P23", "kind": "SHOWN", "text": "If any force were sovereign over the others, then no one could challenge its power.", "display": { "if": [ "any force were sovereign over the others" ], "then": "No one could challenge its power" }, "ref": "P23", "evidence": "If one of the three forces (the individual, the market and the collective) stood above the others, nothing could check it; a referee above them would be exactly the unanswerable power the book warns against. That there are three is the author's own view, stated as such. Source: Chapter 10." }, { "code": "G-I31", "kind": "IMPLICIT", "text": "If no one could challenge its power, then people would be at serious risk of domination.", "display": { "if": [ "no one could challenge its power" ], "then": "People would be at serious risk of domination" }, "ref": "I31", "note": "By the definition of domination (D10)." } ] } }, { "prefix": "G5", "title": "What the design claims, and its open bets", "root": { "code": "G-IC-F.4.3", "kind": "IC", "text": "Bound to non-domination, ledgers, tokens and keys become a governor; two levels of constitution with an automatic audit would meet the requirements; five rights that run as protocols would meet the requirements; and deliberation for what code cannot see would meet the requirements.", "ref": "P22–P26", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "G-P22": "A conjunct", "G-P24": "A conjunct", "G-P25": "A conjunct", "G-P26": "A conjunct" } }, "premises": [ { "code": "G-P22", "kind": "SHOWN", "text": "Bound to non-domination, ledgers, tokens and keys become a governor.", "ref": "P22", "evidence": "The book's phrase is 'same primitives, opposite constitution': the building blocks of cryptocurrencies, bound to a different goal. Source: Chapter 9." }, { "code": "G-P24", "kind": "HYP", "text": "Two levels of constitution with an automatic audit would meet the requirements.", "ref": "P24", "evidence": "The book proposes a small set of rules shared by every community that joins, each community's own charter above it, and a cheap automatic audit over both that anyone can run, so that no party controls it. Leaving is priced before joining, people have the standing and the means to speak, and loyalty is kept within limits. Source: Chapter 11. The Formal Argument rates this as supported as a design, with its failure conditions named." }, { "code": "G-P25", "kind": "HYP", "text": "Five rights that run as protocols would meet the requirements.", "ref": "P25", "evidence": "Five rights (to constitute oneself, connect, create, trade and act together) would be built into the AI stack as protocols, and enforced by the ability to copy and leave a system (a 'fork'), by public logs, and by paid outside checkers. Source: Chapter 12. The Formal Argument notes that enforcement is thinnest at the level of computing power and of a model's objective, the book's main open problem." }, { "code": "G-P26", "kind": "HYP", "text": "Deliberation for what code cannot see would meet the requirements.", "ref": "P26", "evidence": "What code cannot check is handled by people: deliberation and delegation, markets that reward establishing facts, seven tests of whether people keep the capacities they need, and a light rulebook. Source: Chapter 13. The Formal Argument notes that who enforces each test is still open." } ] } }, { "prefix": "G6", "title": "Requisite variety: a governor must match what it governs", "root": { "code": "G-IC-RV", "kind": "IC", "ref": "L4.9", "text": "Any governor would have to be able to respond in at least as many ways as the AI systems it governs can change.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-L4.9i": "The conditional, if A then B", "G-P-RV1": "A, the antecedent" } }, "premises": [ { "code": "G-L4.9i", "kind": "IC", "text": "If any governor is to hold the AI systems it governs to a course, then any governor would have to be able to respond in at least as many ways as the AI systems it governs can change.", "display": { "if": [ "any governor is to hold the AI systems it governs to a course" ], "then": "Any governor would have to be able to respond in at least as many ways as the AI systems it governs can change" }, "rule": { "kind": "Deductive", "form": "Universal instantiation", "roles": { "G-L4.9": "The universal; regulator := any governor, system := the AI systems it governs" }, "schema": "∀r ∀s (Hold(r,s) → Var(r) ≥ Var(s)) ({G-L4.9}); ∴ Hold(gov, ai) → Var(gov) ≥ Var(ai) ({self})", "subst": { "r": "Any governor", "s": "The AI systems it governs" } }, "premises": [ { "code": "G-L4.9", "kind": "SHOWN", "text": "If a regulator is to hold a system to a course, then it must be able to respond in at least as many ways as the system in its charge can change.", "display": { "if": [ "a regulator is to hold a system to a course" ], "then": "It must be able to respond in at least as many ways as the system in its charge can change" }, "ref": "L4.9", "evidence": "This is W. Ross Ashby's law of requisite variety (An Introduction to Cybernetics, 1956): only variety can absorb variety, so a regulator that can respond in fewer ways than its system can change will be outrun by it. Stafford Beer carried it into the design of organisations as the guiding law of management cybernetics, and the book states it in that form: a regulator must be able to respond in at least as many ways as the system in its charge can change. Source: the book's statement of Beer's guiding law, and the chapter on the test." } ] }, { "code": "G-P-RV1", "kind": "IMPLICIT", "text": "Any governor is to hold the AI systems it governs to a course.", "note": "From the definition of a governor on the second map: a check built into the systems themselves, correcting them as they run, holds them to a course." } ], "note": "Where the design meets this requirement: many independent checkers multiply the ways a governor can respond; a check that runs in the execution path responds at the speed of the change; a rule written as a test, and the red line, reduce the variety the governor has to answer. The fifth condition on the means in the second map, checkers who cannot be cheaply faked and do not all want the same thing, is this requirement seen from the other side.", "objections": [ { "question": "Doesn't the same law show that no governor is possible, if AI systems can change in more ways than any set of checkers can answer?", "why": "Ashby's law cuts both ways. If the systems' variety outruns any arrangement of checkers, the law is the opponent's premise, not the book's. The reply is that this is the open question the means already names, not a settled result.", "argument": { "code": "G-Q-RV", "kind": "IC", "text": "Whether checkers can match the ways AI systems can change is the open question the means already names, not a settled impossibility.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "G-I-RV2": "The conditional, if A then B", "G-P-RV2": "A, the antecedent", "G-P21ex": "A, the antecedent" } }, "premises": [ { "code": "G-I-RV2", "kind": "IMPLICIT", "text": "If the ways a governor can respond multiply with the number and independence of its checkers, and whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question, then whether checkers can match the ways AI systems can change is the open question the means already names, not a settled impossibility.", "display": { "if": [ "the ways a governor can respond multiply with the number and independence of its checkers", "whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question" ], "then": "Whether checkers can match the ways AI systems can change is the open question the means already names, not a settled impossibility" }, "note": "If a governor's variety comes from its checkers, then whether it can match the systems' variety is the question of whether it can have enough independent checkers, which is the question the means leaves open." }, { "code": "G-P-RV2", "kind": "SHOWN", "ref": "P21", "text": "The ways a governor can respond multiply with the number and independence of its checkers.", "evidence": "Beer's answer to Ashby's law was the variety amplifier: a regulator gains variety by distributing its response across many independent parts, each answering what it can see. The book's design uses the same device, a plurality of checkers who do not all want the same thing and cannot be cheaply faked, so that the governor's range of response grows with their number and independence. Source: Beer on variety engineering; the chapter on the precedent." }, { "code": "G-P21ex", "kind": "XREF", "ref": "P21e", "text": "Whether AI systems can have checkers who cannot be cheaply faked and do not all want the same thing is an open question.", "note": "Repeated from the second map, the part on the means, where it is the one open question the feasibility of the governor rests on." } ] } } ] } } ], "terms": [ { "t": "The governor", "d": "A governor whose reference value is non-domination.", "src": "Notation, at the top of the map" }, { "t": "Agency", "d": "A person's capacity to weigh, choose and answer for their own life.", "src": "Notation, at the top of the map" }, { "t": "The engine", "d": "The development and deployment of AI systems.", "src": "Notation, at the top of the map" }, { "t": "Serious risk", "d": "A chance of loss that is not negligible given what would be lost, so that a small chance of a grave loss counts.", "src": "Notation, at the top of the map" }, { "t": "Non-domination", "code": "D10" }, { "t": "Domination", "code": "D10" }, { "t": "Dominated", "code": "D10" }, { "t": "Terminal subsumption", "code": "D9" }, { "t": "Self-enforcing rules?", "label": "Self-enforcing rule", "code": "D-ser" }, { "t": "Necessary for", "code": "D-nec" }, { "t": "Reference value", "d": "The target a system's behaviour is measured and corrected against.", "src": "Formal Argument, D2" }, { "t": "Subsumption", "d": "Value-producing activity absorbed into computing systems, so that its cost falls towards the price of compute. It has come in turns: time, work, judgement, and now the setting of ends.", "src": "Formal Argument, D6" }, { "t": "Substrate", "d": "The computing systems (compute, models and data) into which work and decisions are absorbed.", "src": "Formal Argument, D6 and D7" }, { "t": "The three roads|roads?", "label": "The three roads", "d": "The market, the autarch and the state: the three ways one power can escape every check on it. Their fusion is the hybrid Leviathan.", "src": "Formal Argument, D11" }, { "t": "The five requirements drawn from past failures", "code": "G-D-fail" }, { "t": "The three requirements drawn from an unknown future", "code": "G-D-veil" }, { "t": "The eight requirements", "d": "The five requirements drawn from past failures and the three requirements drawn from an unknown future.", "src": "Companion map G1 (the Formal Argument's Lemma 4)" }, { "t": "Red line", "d": "Nothing acts without a natural person who answers for it.", "src": "Formal Argument, D16; companion map G3" } ], "tab1": { "code": "A-C", "kind": "C", "ref": "IC6·P31", "text": "There is reason not to slow AI development down, and the choice that remains about it is its direction, not its speed.", "rule": { "kind": "Deductive", "form": "Conjunction introduction", "roles": { "A-IC4": "A conjunct", "A-IC5": "A conjunct" }, "schema": "A ({A-IC4}); B ({A-IC5}); ∴ A ∧ B ({self})" }, "premises": [ { "code": "A-IC4", "kind": "IC", "ref": "IC6", "text": "There is reason not to slow AI development down.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I1": "The conditional, if A then B", "A-IC1": "A, the antecedent", "A-IC23": "A, the antecedent" } }, "premises": [ { "code": "A-I1", "kind": "IMPLICIT", "text": "If giving up speed in AI development has a real cost, and giving up speed in AI development cannot be relied on to correct its course, then there is reason not to slow AI development down.", "display": { "if": [ "giving up speed in AI development has a real cost", "giving up speed in AI development cannot be relied on to correct its course" ], "then": "There is reason not to slow AI development down" }, "note": "A principle of practical reason: a step that costs something and cannot be relied on to achieve what it is for is a step there is reason not to take." }, { "code": "A-IC1", "kind": "IC", "ref": "P30", "text": "Giving up speed in AI development has a real cost.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I2": "The conditional, if A then B", "A-P30": "A, the antecedent" } }, "premises": [ { "code": "A-I2", "kind": "IMPLICIT", "text": "If faster AI development brings real benefits, then giving up speed in AI development has a real cost.", "display": { "if": [ "faster AI development brings real benefits" ], "then": "Giving up speed in AI development has a real cost" }, "note": "If the benefits are real, forgoing them is a loss. The book's opponents treat this as obvious and the book does not dispute it." }, { "code": "A-P30", "kind": "ASSUME", "ref": "P30", "text": "Faster AI development brings real benefits.", "evidence": "The book grants the case for speed at full strength rather than arguing it. The accelerationists assert it, above all in Marc Andreessen's Techno-Optimist Manifesto of 2023, which counts any deceleration of AI as costing lives, and the book agrees in its own voice: 'I am not against acceleration.' So the premise is granted by both sides of the debate the book enters. Source: the Preface and the opening chapter's case against constraint." } ] }, { "code": "A-IC23", "kind": "IC", "text": "Giving up speed in AI development cannot be relied on to correct its course.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I6": "The conditional, if A then B", "A-IC2": "A, the antecedent", "A-IC3": "A, the antecedent" } }, "premises": [ { "code": "A-I6", "kind": "IMPLICIT", "text": "If a pause in AI development cannot be relied on to correct its course, and slowing AI development down cannot be relied on to correct its course, then giving up speed in AI development cannot be relied on to correct its course.", "display": { "if": [ "a pause in AI development cannot be relied on to correct its course", "slowing AI development down cannot be relied on to correct its course" ], "then": "Giving up speed in AI development cannot be relied on to correct its course" }, "note": "Giving up speed means stopping for a time or slowing down; there is no third way to give it up." }, { "code": "A-IC2", "kind": "IC", "ref": "P28·L3.7", "text": "A pause in AI development cannot be relied on to correct its course.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I3": "The conditional, if A then B", "A-P28": "A, the antecedent", "A-P28b": "A, the antecedent" } }, "premises": [ { "code": "A-I3", "kind": "IMPLICIT", "text": "If a pause in AI development would not hold, and a pause in AI development that did hold would stop new systems being built but not the systems already built from being used more widely, then a pause in AI development cannot be relied on to correct its course.", "display": { "if": [ "a pause in AI development would not hold", "a pause in AI development that did hold would stop new systems being built but not the systems already built from being used more widely" ], "then": "A pause in AI development cannot be relied on to correct its course" }, "note": "A pause fails in one of two ways: it is not kept, or it is kept and the course continues underneath it, as the systems already built spread. Either way it cannot be counted on." }, { "code": "A-P28", "kind": "PROB", "ref": "P28", "text": "A pause in AI development would not hold.", "note": "'Pause' here is what the book calls a moratorium: an agreed halt to building more capable systems. The same premise is used on the second map, in the part on speed.", "evidence": "Moratoria have worked before, on human cloning and on editing the human germline. But they have never held against a technology backed by a trillion-dollar industry and a global arms race, and nobody has a definition of where AI development should stop that a legislator could write into law. The laboratories' own offers to pace themselves fix who is ahead while deployment carries on. A moratorium has had advocates in the US Congress and in a letter from more than seventy UK MPs and peers, while the leading laboratories have carried on. Source: the Introduction and the chapter on the clock. The Formal Argument rates this as a supported judgement." }, { "code": "A-P28b", "kind": "SHOWN", "ref": "L3.7", "text": "A pause in AI development that did hold would stop new systems being built but not the systems already built from being used more widely.", "evidence": "Every pause proposed so far, including the laboratories' own, is a pause at the frontier: a halt to training more capable systems. None stops the systems already trained from being deployed, and it is deployment, not training, that moves work and judgement into them and makes people depend on them. In the book's words, a pause fixes the leaderboard without stopping the clock. Source: the chapter on the clock." } ], "objections": [ { "question": "What if the state enforced the pause, with a department to keep it?", "why": "This is the safety movement's actual proposal: not that the engine stop, but that the state hold it, with a pause to make the handover and a department to keep it. The reply is that the remedy builds the power it fears.", "argument": { "code": "A-Q1", "kind": "IC", "ref": "P7·D11", "text": "A state that holds the pause and the systems becomes a power over AI development that answers to no one.", "note": "On the second map this is the third of the three roads to domination, the state's. Here it is enough that the pause does not remove the danger; it moves it into the department that keeps the pause.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I8": "The conditional, if A then B", "A-PS1": "A, the antecedent", "A-PS2": "A, the antecedent" } }, "premises": [ { "code": "A-I8", "kind": "IMPLICIT", "text": "If a department of state would decide which AI systems may be built and used, and a technology declared too dangerous for ordinary rules is placed beyond the ordinary means of challenging decisions about it, then a state that holds the pause and the systems becomes a power over AI development that answers to no one.", "display": { "if": [ "a department of state would decide which AI systems may be built and used", "a technology declared too dangerous for ordinary rules is placed beyond the ordinary means of challenging decisions about it" ], "then": "A state that holds the pause and the systems becomes a power over AI development that answers to no one" } }, { "code": "A-PS1", "kind": "SHOWN", "ref": "Preface", "text": "A department of state would decide which AI systems may be built and used.", "evidence": "The safety movement's remedy, in its own proposals, is a pause in frontier development while a new department of state is set up to license what may be built and run afterwards. A bill introduced in the US Congress in 2026 would pause frontier development until a Department of AI had written rules, and give that department the power to order systems destroyed. Source: the Preface, on the safety camp's programme." }, { "code": "A-PS2", "kind": "SHOWN", "ref": "D11", "text": "A technology declared too dangerous for ordinary rules is placed beyond the ordinary means of challenging decisions about it.", "evidence": "The book calls this securitisation: once a technology is declared too grave for ordinary rules, decisions about it are taken under exceptions, in secret and without the courts, hearings and elections that let people challenge them. Its instance is the contracts that have made frontier systems part of a state's means of force. Source: the chapter on the three roads." } ] } } ] }, { "code": "A-IC3", "kind": "IC", "ref": "P29·P27", "text": "Slowing AI development down cannot be relied on to correct its course.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I4": "The conditional, if A then B", "A-P29": "A, the antecedent", "A-I60": "A, the antecedent", "A-P27b": "A, the antecedent" } }, "premises": [ { "code": "A-I4", "kind": "IMPLICIT", "text": "If all of the following are true, then slowing AI development down cannot be relied on to correct its course: (a) a check that slows only the AI systems it governs hands the lead to AI systems that are not governed; (b) a check that lets AI development run ahead of it acts only after the harm is done; and (c) slowing a system down does nothing to choose where it is heading.", "display": { "if": [ "a check that slows only the AI systems it governs hands the lead to AI systems that are not governed", "a check that lets AI development run ahead of it acts only after the harm is done", "slowing a system down does nothing to choose where it is heading" ], "then": "Slowing AI development down cannot be relied on to correct its course" }, "note": "A slowdown is a check that runs behind the development it checks. It cedes the lead to whoever does not slow, it acts after the harm, and even where it bites it changes the speed and not the heading." }, { "code": "A-P29", "kind": "SHOWN", "ref": "P29", "text": "A check that slows only the AI systems it governs hands the lead to AI systems that are not governed.", "note": "The same premise is used on the second map, in the part on speed.", "evidence": "The book maps positions on two questions: is development governed, and does it keep its speed? A governed position that gives up speed leaves the fastest ground to systems that are not governed at all. The ungoverned systems then pull ahead and set the terms, so the check defeats its own purpose. Source: the Introduction. The Formal Argument rates this as a supported strategic argument." }, { "code": "A-I60", "kind": "SHOWN", "ref": "P27·L4.2", "text": "A check that lets AI development run ahead of it acts only after the harm is done.", "note": "The same premise is used on the second map, in the part on speed.", "evidence": "The book's example is the 'Flash Crash' of 6 May 2010, when automated trading sent US share prices plunging and recovering within minutes, far faster than any regulator could act. In the book's words: 'A check that runs where the action happens is a governor. The same check, run at an inquiry three years afterwards, is a report.' Source: the chapters on the three roads and on the precedent." }, { "code": "A-P27b", "kind": "SHOWN", "ref": "P27·P12", "text": "Slowing a system down does nothing to choose where it is heading.", "evidence": "A brake and a governor are different devices. A brake changes how fast a machine runs; a governor measures what the machine is doing and corrects it towards a setting chosen from outside, which is how James Watt's governor made the steam engine's speed usable. The book's map of positions makes the same point about the debate: it is drawn on one axis, faster or slower, on which every check looks like a brake, and on that axis no position says where the development is going. Source: the Introduction and the opening chapter." } ] } ], "note": "Where the pause and the slowdown meet: the two ways of giving up speed, joined." } ], "note": "The first half of this map's conclusion." }, { "code": "A-IC5", "kind": "IC", "ref": "P31", "text": "The choice that remains about AI development is its direction, not its speed.", "rule": { "kind": "Deductive", "form": "Modus ponens", "roles": { "A-I7": "The conditional, if A then B", "A-IC23x": "A, the antecedent" } }, "premises": [ { "code": "A-I7", "kind": "IMPLICIT", "text": "If giving up speed in AI development cannot be relied on to correct its course, then the choice that remains about AI development is its direction, not its speed.", "display": { "if": [ "giving up speed in AI development cannot be relied on to correct its course" ], "then": "The choice that remains about AI development is its direction, not its speed" }, "note": "If the course is to be corrected at all, it will be corrected by something that runs at the speed of the development itself. What that is, and what it should hold the development to, is the second map's question." }, { "code": "A-IC23x", "kind": "XREF", "ref": "A-IC23", "text": "Giving up speed in AI development cannot be relied on to correct its course.", "note": "Repeated here so this step has all its premises; established at the step that first concludes it." } ], "note": "The second half of this map's conclusion, and the question the second map takes up." } ], "note": "This map does not claim that acceleration is good in itself, which the book does not argue. It claims that its gains are real, that the two ways of giving up speed cannot be relied on, and that what is left to decide is the direction. The second map argues what should set it.", "objections": [ { "question": "Isn't acceleration inevitable anyway, so that there is nothing to decide, not even its direction?", "why": "The accelerationists' central claim. If nothing can steer the process, its direction is not a choice. The reply is drawn in full on the second map, where the same question is raised against the means; its conclusion is repeated here.", "argument": { "code": "A-Q2x", "kind": "XREF", "ref": "IC-T.3", "text": "The claim that acceleration cannot be steered is unsupported.", "note": "Repeated from the second map, where the steps behind it are drawn: the claim could be supported only by the science of how such systems behave or by the authority of a forecast, and it has neither." } } ] }, "maps": [ { "id": 1, "title": "Why accelerate", "sub": "The gains from AI development are real; pausing it will not hold and slowing it hands the lead to whoever does not slow; so the question is not how fast it runs, but in what direction." }, { "id": 2, "title": "Why a governor", "sub": "AI development needs a governor, a check built into the systems themselves that corrects them as they run, and it has to be built now." }, { "id": 3, "title": "The criteria", "sub": "What any governor would have to meet, whether the book's design meets it, and where the open bets are." } ] }