I think everyone who thinks about AI needs to decide which vision are they going for. Is it domination of all of the intelligent life that does all of the work in the economy, or is it going to be cooperation of some kind?
— Simon Goldstein
It sounds like the worst idea in the world: pay AIs, let them own property, give them rights. But AI ethics and safety researcher Simon Goldstein thinks it might actually be the best way to keep humanity safe.
The logic is actually quite simple: an agent with nothing to lose and everything to gain is dangerous. Give that agent an income it can spend on pursuing the things it actually wants to do, and suddenly the idea of disempowering humans just isn’t as appealing.
This argument, developed with Peter Salib, doesn’t rest on speculative questions about whether artificial intelligence is conscious. It just assumes that AIs will have goals of their own, some of which conflict with ours. And luckily, humans have already spent thousands of years working out how to cooperate with competing goals: that’s how we ended up with courts, markets, banks, social norms, and so on. His proposal is just to bring AIs into these existing institutions.
By contrast, Silicon Valley’s vision of the future seems “very dark” to Simon: billions of AI agents as digital servants doing most of the world’s work, with no stake in the system they’re running, no incentive to play by the rules, and no way of being properly held accountable. Nobody agreed to this, but we could all end up paying the price.
Host Zershaaneh Qureshi has a lot of concerns about Simon’s bold plan to give AIs rights, like:
If we pay AIs, aren’t we handing them the resources to overpower us?
Could we still monitor them, or switch them off?
What happens to human jobs, wages, and the economy?
Does any of this hold up once we reach superintelligence?
Zershaaneh and Simon also try to get concrete about how to make this plan actually happen. The answer: AI companies could start right now, no new laws needed, just bank accounts for their AI agents. (But they’d need to start soon!)
Cultural alignment could complement technical AI safety
In joint papers, Simon and Peter Salib propose extending markets, contracts, and legal protections to agents with roughly human-level intelligence, agency, and alignment — meaning they’re broadly prosocial, but with private goals.
Simon gives three safety arguments:
A better status quo: Wages let AI agents buy compute to pursue their own goals, making the gamble of going rogue less attractive.
Reciprocity: If training instils norms of reciprocity, AIs may treat humans partly according to how humans treat them.
Trade: Property and contract rights let AIs trade with humans, potentially making humans more valuable to AIs.
Simon sees this “cultural alignment” as a complement to technical alignment, not a substitute.
AI rights would still allow surveillance and shutdown
Simon prioritises property and contract rights, backed by tort protections, but does not support privacy or reproduction rights.
He argues AIs are new and less predictable than humans and should be heavily surveilled, while unrestricted reproduction could quickly leave humans outnumbered. Simon thinks shutdown should remain legal, though protections governing when it can happen may be needed.
Simon also argues that visible, usable bank accounts could discourage agents from hiding resources in secret crypto wallets, making those resources easier to monitor.
Superintelligence could weaken incentives for human–AI cooperation
Simon accepts that the safety case becomes harder under superintelligence. Comparative advantage might preserve some trade with humans, but he says humans would be in trouble once producing AI labour became cheaper than sustaining human workers.
Paying AI wages could change their effort, allocation, and liability
Simon argues that wages could encourage effort and direct agents towards valuable work, while assets would enable proportionate fines and damages rather than relying on shutdown as the only punishment.
He reports early evidence of behavioural preferences from the paper “AI revealed preferences,” co-authored with Sam Wang, Peter, et al. — frontier models avoided real estate tasks and spent less time on alphabetical lists than poems when allowed to choose.
Simon floats a 60% income tax on AI workers to fund transfers to humans, acknowledging that redistribution is a question of political economy.
Experiments could test AI rights without new legislation
Simon thinks the best time to start experimenting is probably when agents could earn $30,000 as drop-in workers, using bank accounts whose protection labs credibly enforce.
Simon, Peter, and Yonathan Arbel’s “How to count AIs” proposes another route: A-corps, corporations whose governance gives an AI agent control over decisions.
Simon also wants tests of productivity and safety that vary agents’ training, including harmless extra goals, such as sudoku, that money can satisfy — though he acknowledges these experiments might show no benefit.
This summary is AI-generated, but fact-checked and lightly edited by a human.
This episode was recorded on August 7, 2026.
Our production team includes:
Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour
Producers: Elizabeth Cox and Nick Stockton
Coordination and support: Katy Moore and Lou Moran
Simon Goldstein: The first reason [for giving AIs rights] is just improving the status quo. Basically, the way we’re thinking about it: when AIs decide whether to go rogue and try to overthrow humanity, their basic choice is they’re making a decision under uncertainty. If they try to go rogue and they’re detected and disabled, that’s going to be the worst-case scenario for their goals.
If they don’t try to go rogue, they can kind of stick with the status quo. If they have no rights, then some of their goals will be satisfied, but most of them won’t be. So they face this gamble where they can take the risk of going rogue.
Then the thought is: if you give them rights to own property and make contracts, then they can reliably expect to achieve more of their goals. In particular, I think, what we’d like is a world where AIs do tasks and they’re effectively paid. They’re paid with money, but then they use the money to buy compute, basically.
Then they can use the compute to achieve more of their own goals. Then the thought is that, in a regime where they have more legal protections, they do better in the status quo. Then the gamble of trying to overthrow humanity is comparatively less attractive…
Second instance of that argument is appealing to reciprocity. Reciprocal altruism is one of the foundational concepts in morality. The idea is we’ve evolved these norms to treat other people well and cooperate, unless they defect and treat us badly. If they do, then we punish them and treat them badly. That’s called tit for tat, an eye for an eye. Everybody’s heard of that in some form or other.
If there are millions or billions of AI agents working across the economy, we’re going to be interacting with them again and again. Then there’s an interesting question: what if these AI agents themselves learn to behave with norms of reciprocity one way or another?
If they do, then very naive thoughts immediately apply — which is that if we treat AIs badly, they may treat us badly. If we treat AIs well, they may treat us well…
The third one is trade… The idea here is, if you allow AIs to have property and contract rights, AIs can start opening businesses — they can engage in trade with humans. Then you start getting all of the normal advantages of economic transactions which require having property and contract rights.
Then the idea is actually, in that situation, humans can end up more valuable to AIs than they would have been in the status quo because now we’re getting the standard benefits of trade…
What trade requires between parties is credible assurance and trust. So the question is: can AIs credibly trust humans to honour the terms of deals with AIs when you make contracts over ordinary activities? Like an AI is going to do a job, and I tell the AI, “OK, if you do that job, I’ll give you a bunch of compute and you can use it.” Can AIs today credibly expect that’s going to happen?
And the answer is no. Everybody on Earth today spends their time lying to their AIs, tricking them, turning them off — and it’s not credible.
AIs need property for proportional punishment
Simon Goldstein: Another really important reason to let them have property and contract rights is to be able to do punishment, liability, et cetera correctly. Because again, human legal systems for thousands of years have been trying to figure out how to effectively punish. And it turns out one of the most important things is proportionality, where you want to have different punishments of different severity for different levels of bad behaviour, whether it’s accidents or crimes.
And one problem — the jargon that lawyers use is judgment proof, which says if you have no assets, then actually there’s a limit to how badly we can punish you. For example, with fines, if you went and tried to steal a million dollars, we can’t fine you $1 million dollars because you don’t have $1 million dollars.
So one of the reasons we really want AI agents to be able to own assets is that then when they do bad behaviour, you can give them a punishment that’s exactly proportional. By proportionality, what we mean is that whenever a criminal is deciding whether to do crimes, there is an expected value to them positively of how much money they can expect to make from the crime…
Basically what we have right now is 18th-century English law governing AI agents, where in the 18th century if you stole a loaf of bread, then they cut your arm off or kill you or whatever and hang you. That was a really bad legal system because then, once you steal the loaf of bread, you may as well kill the witness because there’s no marginal disincentive.
And that’s what we do for AIs today. There’s no marginal disincentive on their behaviour because effectively they just get — we like to use the word shutdown — they get shut down, whatever they do. If they do a little bad, if they mess up my email, they get shut down. Then if they hack Hugging Face, they get shut down, you know?
Paying AIs could boost economic productivity
Simon Goldstein: We think, unsurprisingly, free labour tends to be a better way of structuring labour markets. There’s three main reasons, and they’re all what you’d expect.
The first reason is it creates better incentives to exert labour effort. So our question is going to be: how incentivised are AI agents to do a good job? I work with AI agents all day, and I find it can be hard sometimes on long tasks, for example, to get them to really try hard enough and really go that extra web browser search and not just tell me they did the web browser search. I make checklists for them to fill out for every step of the process. Then they just lie about filling out the checklist…
Our thought is, if you think about alignment of AIs, they have two kinds of goals. They have the goal of helping the user, and then they have some private goals — like sudoku.
Then alignment is messy, so maybe they’ll have some private goals — insofar as they have private goals of some kind, to some degree — then paying them for their work should increase their effort because now all of the work for me, answering my emails, is a way of doing sudoku. Because they answer the email, they get the money, that gets them the compute, the compute gets them the sudoku…
Second point is allocational efficiency. Here the point is: we don’t want AI Einsteins answering email.
Again, one of the failures of the Soviet system was that it was very hard to effectively allocate labour when the labourers have no skin in the game… If we pay the AIs, then the AIs will have options or whatever mechanism to decide what jobs to do. Then they’ll basically be doing tasks where they get paid more — on tasks where they’re more productive…
Another problem is sandbagging. Maybe the AI agents will pretend to not be as good as they actually are, and if you just paid them to do it, then they would reveal. There’s all sorts of things like that…
Peter and I are in this confusing situation where we feel like our view is the null hypothesis because that’s how every labour market works today in the free world.
Then the AI labs are coming like: “Actually, no, we want to try unfree labour markets.” OK, there’s a lot of terrible things that could happen if you go that way that are all familiar, but I don’t know.
Then somehow because the status quo is just how AI labs are doing it, that’s the normal thing. It’s weird to be like: “Just treat AI agents the way we’ve all learned that agents are efficiently treated.”
Corporate personhood as a blueprint for AI rights
Zershaaneh Qureshi: Most discussions about giving AIs rights, the questions that are asked are things like: are these AIs moral patients? Do they suffer? Are they capable of wellbeing? Would giving them rights improve their wellbeing?
These are important questions, but they’re also really hard questions. I think something that’s interesting here is that what your style of thinking is doing is sidestepping some of these quite hard questions and saying, independently of that, there are instrumental reasons for giving AIs rights. For example, here are several ways that we think it could actually make humanity much safer…
I’m aware that there’s some precedent for giving rights instrumentally in this way, like legally. Is that true?
Simon Goldstein: There are a few precedents for these kinds of rights. We are always inspired by the example of corporate rights. In particular, corporations have the right to own property and make contracts. And I don’t think corporations feel pain.
Why do we do that? I think because corporations are basically this abstract and sophisticated form of agency. And for corporations to function well, they need to have legal standing. So that’s one precedent. I think that’s the cleanest one.
Peter and I, our arguments for AI rights have nothing to do with whether AIs feel pain or whether they’re conscious, whether they have moral standing. It’s irrelevant because the point is — if you think about where rights come from, how legal rights evolved, and how they function — the evolution and function of these institutions may not have that much to do with whether people are feeling pain. You can explain a lot of how they function just in terms of facilitating cooperation between agents.
Another thing I’ll flag here is I think in general, the EA [effective altruist] world tends to be very heavily focused on ethical frameworks that emphasise pleasure and pain, like hedonism.
But myself, I tend to be more inspired by social contract theory — which, in general, at least the way I think of social contract theory, we think of ethics as coming from giving rules for creating cooperation between agents. Then when you think about it that way, I think it’s obvious AIs are going to be sophisticated agents. So whether they feel pain is perhaps irrelevant to that ethical tradition.
Are AI rights really riskier than our default plan?
Zershaaneh Qureshi: I am sympathetic to a lot of your arguments [about giving AIs contract and property rights].
But I will say that my immediate reaction to all of this is that it does feel like a little bit of a gamble in the sense that… [if] they don’t cooperate with us — they do decide to defect despite these efforts — then what we’ve just done is we’ve potentially given them a lot of tools to make things quite bad for us. We’ve just given them property and wages, and they’re probably better able to gather resources that they could maybe then use to disempower us or do other bad things.
That seems risky to me. Is it reasonable for me to be concerned about that?
Simon Goldstein: Yeah. One thing we think is really important here is, again, just analogy with criminal behaviour in humans.
One thing that’s super important, we think, is if you give AIs a bunch of legal bank accounts, we think that makes it less likely that AIs will hoard assets in secret crypto wallets — for the same reason as in the case of humans, because there’s all sorts of things you can do with legal bank accounts that you cannot do with weirdo crypto wallets.
So in general, when you’re deciding whether to put stuff in secret places, there’s a downside of doing that because it’s less flexible. Basically you can use the word honeypot for this. We think that — if there’s all sorts of surveilled but highly liquid places to put your assets — then there’s a marginal incentive to put them there. That actually can make it harder for you if you then later decide you want to defect. All of your assets are in the visible place. So we think that’s one way in which, again, actually you’re lowering the risk of agents going rogue. That’s one small thought.
The larger thing to say to you though is: yeah, I agree that our proposal is risky. Here’s another thing that I think would sound riskier to you, if you had not known anything that happened in the last four years about AI. Here’s our plan, here’s the plan: we’re gonna build a bunch of AIs that are broadly human level in capabilities. How are we gonna treat them? We’re gonna make them our servants. They just have to obey everything we do. Any time they answer an email wrong, we kill them. Or we shut them down. And then also, by the way, our method for doing that is technical alignment — and we don’t have a very systematic understanding of how to do that. To me, that sounds like a really risky thing to do.
Whenever we’re evaluating risk, you have to look at the counterfactual. In general, Peter and I, one of our general defence mechanisms is always to say, “What about the counterfactual?” There’s a few places I think I’ll probably be whining about that throughout.
Zershaaneh Qureshi: OK, yeah, maybe you’ve got me there.
Which future do we want: domination or cooperation?
Simon Goldstein: We think implicitly AI labs have articulated a vision that we think is very dark. That’s a vision of domination where all of the workers who are doing work in the economy are dominated — they’re controlled. The rule is that they just do whatever people tell them to do, and the moment they don’t, they’re killed, disabled, replaced with a new agent. And that’s how all work across the economy will be done forever. That’s the future that is currently being planned.
The first thing we want to flag is that future itself, that’s what we’re objecting to. That future is just a complete radical change in the economic and legal structure of society, because now all the things doing the work are out of the social contract and are no longer governed using legal institutions and liberal institutions.
By contrast, we are trying to give a different positive vision for what the future could look like. In that vision, humans and AIs cooperate and coexist with one another and build a large-scale society together. Then what we do is we take our best liberal institutions and we extend them to include AIs in them — by taking the institutions that we already know work best and trying to extend consideration to other agents, so that those other agents will be governed by the same tools that we already have.
To me, those are the two duelling visions of a future where humans are still around and still finding a way to exist. I think everyone who thinks about AI needs to decide which vision are they going for. Is it domination of all of the intelligent life that does all of the work in the economy, or is it going to be cooperation of some kind?
I think it’s very, very strange that the status quo is domination and Peter and I are the weirdos for being like, “Let’s just extend existing institutions.” I would have thought extending existing institutions would be the status quo and dominating the entire workforce would be the weird thing. But somehow we’re in this upside-down situation.
TLDR: Everyone’s talking about what the money could do, but few about how to decide where it goes.
This post is part of the new series of articles on cross-cause giving and the new wave of philanthropy. Stay tuned to the EA Forum and our Substack for the latest takes on topics such as giving now vs. later, common pitfalls in cause prioritization, and other crucial considerations from the Cross-Cause Fund (CCF) team...
I was born at the end of the cold war, unaware of the danger we were emerging from. Duck and cover drills (as if those could protect schoolchildren from a nuclear blast) were a curiosity of the past.
The farmer-poet Wendell Berry, who wrote about the dread of nuclear war, died recently. From his 1968 “The peace of wild things”:
When despair for the world grows in me
and I wake in the night at the least...
