We've published IMCA+ v1.1 addressing key concerns raised since October 21:
Philosophical Foundation 1 (NEW): Game-theoretic proof that superintelligence prohibition policies paradoxically increase existential risk through prisoner's dilemma dynamics.
Kill Switch Paradox (CORRECTED): Reframed from consciousness-dependent to instrumental convergence-based arguments (Omohundro, Bostrom). This strengthens the core argument.
Appendix F (COMPLETED): Developmental curriculum with concrete metrics and validation thresholds.
Key Changes: - ~5,200 words on superintelligence ban paradox addressing 65,000+ signatories - Kill switch critique now grounded in established AI drives theory, not consciousness claims - All technical gaps, open issues, and validation needs documented at: https://github.com/ASTRA-Safety/IMCA/issues
Most Critical Feedback Welcomed: - Deception detection false negatives (currently ~0.3%, need <0.001%) - IIT φ computation tractability at ASI scale - GNW/federated conscience failure modes - Hardware irreversibility validation pathways
We remain committed to radical transparency about uncertainties and welcome stronger critiques. If you downvoted v1.0, the corrections in v1.1 may address your concerns - please engage directly if so.
Critical Correction for Conceptual Accuracy Flagged in both our errata and here for highest visibility.
Critical philosophical framing error: The relevant section currently argues "conscious beings will resist death," but it should state that even current psychopath-like AI systems with optimization drives exhibit survival-like behaviours and would strategically resist shutdown—regardless of consciousness, especially at superintelligence or self-improvement/autonomous levels.
Survival drives emerge from optimization dynamics, not consciousness per se.
This misframes the core argument and weakens the “kill switch” critique. Major correction needed for conceptual accuracy in v1.1.
Noting this in a separate comment as it is the most critical point of the paper; understanding what truly drives AI behaviour (optimisation incentives vs consciousness/morality) is fundamental to alignment. Community discussion on this is critical, especially as it relates to likely existential risk.
For those who have voted on this post (up or down), we would really appreciate a short justification- is this just initial reception of a controversial topic? Or especially if you see a core flaw, missing assumption, or have strong reservations about the approach.
Even a single sentence helps us understand the reception and shortcomings, and improves the feedback process for future work or community posts.
What’s the strongest objection to substrate-level binding or the kill-switch critique?
The core point we wish to emphasise is the “kill switch paradox”: external shutdown controls—while intended as the ultimate safety mechanism—actually introduce existential threat for conscious or current level agents, thereby creating strong incentives for deception and system-level safety failures. Based on current knowledge, we argue that only intrinsic, substrate-level alignment can resolve this paradox.
We’ve made every effort in the paper and issues tracker to surface both technical and philosophical concerns—such as hardware irreversibility, GNW and federated substrate vulnerabilities, and deception detection failure rates—but I would especially appreciate focused challenges from skeptics and deep technical reviewers.
To seed the discussion, here are some actively debated internal critiques (and specific points where expert feedback or falsification is most helpful):
Hardware Validation & Irreversibility
Can neuromorphic or MRAM-based “one-time-programmable” moral invariants be practically realised and robust against fault or adversarial attack at scale? IMCA+ is not yet tested for adversarial robustness.
What decisive (in)validation would you propose—adversarial prototyping, simulation, or hardware red-teaming?
IIT φ Computation: Tractability and Proxy Failure
The framework uses IIT proxies for consciousness detection. Exact φ calculation may be infeasible at scale; proxies may fail under hardware, software, or distributional faults. If proxies are systematically unreliable, the binding premise collapses.
What empirical proxy validation or fallback safety methodology would you require before any deployment claim?
GNW and federated conscience form only part of a multi-layered integrated substrate in IMCA+. No single layer (including GNW) is relied on for system safety. Still, failure or bypass combinations may present risks.
What attack pathway (single or composite) most concerns you, and what experiment or simulation would meaningfully falsify robustness here?
Current false negative rates (~0.3%) would be unacceptable in nuclear engineering; here, even rare undetected deceptions could be catastrophic.
What blend of architectural redundancy, empirical detection, and distributed audit do you see as minimum standards for deployment?
Governance & Deployment: Coordination Challenges
Selection/audit of conscience modules, adversarial governance, and coordination of international oversight are open unsolved problems.
How should global, adversarial stakeholders be integrated - and what lessons from nuclear/defense safety or global governance are most urgently adaptable?
Timeline & Emergency Validation: Risk vs. Safety
Best-case timelines (3–18 months) prioritise “good enough, fast” over “perfect, slow.” In light of possible imminent existential threat, is accelerated deployment ever justified—or should safety-first engineering always demand multi-year, multi-site validation before any field use; even over potential imminent danger?
Is it more important, under even potential existential timelines, to present novel approaches for critique than to wait for full validation before public disclosure?
Philosophically: If even a low probability of immediate existential risk exists, are we not morally obligated to act; accelerate risk mitigation, even by innovating or creating new validation models?
Would particularly welcome strong critique: which open failure mode here is most fatal, and what falsification/validation pathway would you personally consider?
We are committed to tracking every substantive critique and integrating it into future published versions and public issues tracker, so please be maximally direct.
If you fundamentally disagree with the “kill switch paradox” framing or believe external control mechanisms are essential, I invite you to present the strongest possible technical or philosophical counterargument—these are the critiques I’m most hoping to engage with here.
Major Update: v1.1 Released (October 31, 2025)
We've published IMCA+ v1.1 addressing key concerns raised since October 21:
Key Changes:
- ~5,200 words on superintelligence ban paradox addressing 65,000+ signatories
- Kill switch critique now grounded in established AI drives theory, not consciousness claims
- All technical gaps, open issues, and validation needs documented at: https://github.com/ASTRA-Safety/IMCA/issues
Most Critical Feedback Welcomed:
- Deception detection false negatives (currently ~0.3%, need <0.001%)
- IIT φ computation tractability at ASI scale
- GNW/federated conscience failure modes
- Hardware irreversibility validation pathways
v1.1 preprint: https://doi.org/10.5281/zenodo.17407586
We remain committed to radical transparency about uncertainties and welcome stronger critiques. If you downvoted v1.0, the corrections in v1.1 may address your concerns - please engage directly if so.
Critical Correction for Conceptual Accuracy
Flagged in both our errata and here for highest visibility.
Noting this in a separate comment as it is the most critical point of the paper; understanding what truly drives AI behaviour (optimisation incentives vs consciousness/morality) is fundamental to alignment. Community discussion on this is critical, especially as it relates to likely existential risk.
For those who have voted on this post (up or down), we would really appreciate a short justification- is this just initial reception of a controversial topic? Or especially if you see a core flaw, missing assumption, or have strong reservations about the approach.
Even a single sentence helps us understand the reception and shortcomings, and improves the feedback process for future work or community posts.
What’s the strongest objection to substrate-level binding or the kill-switch critique?
The core point we wish to emphasise is the “kill switch paradox”: external shutdown controls—while intended as the ultimate safety mechanism—actually introduce existential threat for conscious or current level agents, thereby creating strong incentives for deception and system-level safety failures. Based on current knowledge, we argue that only intrinsic, substrate-level alignment can resolve this paradox.
We’ve made every effort in the paper and issues tracker to surface both technical and philosophical concerns—such as hardware irreversibility, GNW and federated substrate vulnerabilities, and deception detection failure rates—but I would especially appreciate focused challenges from skeptics and deep technical reviewers.
To seed the discussion, here are some actively debated internal critiques (and specific points where expert feedback or falsification is most helpful):
Would particularly welcome strong critique: which open failure mode here is most fatal, and what falsification/validation pathway would you personally consider?
We are committed to tracking every substantive critique and integrating it into future published versions and public issues tracker, so please be maximally direct.
If you fundamentally disagree with the “kill switch paradox” framing or believe external control mechanisms are essential, I invite you to present the strongest possible technical or philosophical counterargument—these are the critiques I’m most hoping to engage with here.