One thing I'm a bit confused by: in the table you classify several objections to incomparability as being responses to premise 1. But when I read premise 1 it doesn't obviously seem to make any claim about incomparability? That seems to enter in premise 2. Am I missing something?
As a data scientist who frequently wonders about this question, thank you for this post, it was a really interesting read!
But I'm a bit confused about your final section on "generative AI" roles, and the claim that data people have no special advantage here. What sort of roles do you have in mind exactly? If you're talking only about prompt engineering, or just about the skill of being able to use a product like Claude code/cowork well, then I can see where you're coming from, though I think I still weakly disagree. For example, if you have an understanding of how these models work under the hood then you are probably better placed to understand which tasks it will be good at and which it might struggle with (this includes really basic stuff, like knowledge cutoffs in training data, and the fact it won't necessarily remember what you asked it about the day before without a memory feature bolted on, and therefore won't improve day by day etc). You are also probably in a better position to understand things like which tasks will require higher reasoning, and which won't (which very often doesn't line up with their perceived human difficulty).
But that aside, I don't think applying generative AI to practical problems is just about prompt engineering or becoming a claude cowork super-user. If you are a business that wants to create an automated workflow where you run a certain prompt over a certain dataset at scale, on the basis of certain triggers, then you are doing something that looks a lot like data engineering (and you are going to need to write code). And if you're hitting context window limits, you'll have to manage context, by building some kind of agent orchestration/scaffolding. That's maybe closer to being a brand new skill, but it still requires strong coding ability, combined with the ability to evaluate the performance of a software tool empirically, which is a specific combination of skillsets that data scientists should already have.
It's true that with AI coding assistants, people are writing less and less code by hand. But you still need strong coding ability to use AI coding assistants effectively. Software engineering has not yet been fully automated. There's a possible future where that changes soon, and maybe that's the future informing your advice in that final section? But my personal feeling would be that in that world, most knowledge jobs probably follow shortly after anyway (if not before), so trying to plan your career development around this possible future seems challenging!
I may be misunderstanding something in the argument, but this potential solution is I think already discussed in detail in Reasons+Persons, where Parfit introduces the repugnant conclusion? He calls it the 'lexical value' solution, I think? And I don't think this write up addresses any of the strong arguments that Parfit makes against it there?
In Parfit's original argument, he doesn't just rely on an appeal to intuition that B is better than A. Instead we move to B from A in two steps: first we go to A+ (more people having lives just worth living, but original population unaffected). Parfit claims that A+ is not worse than A. And then we move to world B, where we are now significantly increasing the welfare of already existing people, with only a small drop in the welfare of the original population. Parfit claims that B is better than A+.
If we reject that A+ is not worse than A, then we have to say it is sometimes wrong to create new people, even if their lives are worth living. This seems very strange. If we reject that B is better than A, then we have to believe that a small drop in welfare for people whose welfare is already very high can outweigh a large increase in welfare for people whose welfare is very low. This is the bullet that I think the lexical value solution bites?
But this is a tough bullet to bite! It is the opposite of egalitarian. It says we should prioritise changes in the welfare of already existing high welfare people (whose welfare is above the lexical level) over changes in the welfare of also already existing low welfare people (welfare below the lexical level).
Something seems especially weird about offsetting your purchase of non-BCC chicken by donating to campaigns to get supermarkets to adopt the BCC.
I think one important consideration missing here: supermarkets respond to campaigners by saying that customers want to buy non-BCC chicken, and they are just doing what their customers want. If you buy non-BCC chicken from them, you make that argument stronger, and the campaigners' argument weaker.
And I don't think this is necessarily a negligible concern in comparison to the other effects being discussed here, since the mechanism for how your small donation is supposed to help chickens is also by tipping the scales on some corporate campaign and getting a company like a supermarket to make a big change.
I don't deny that my "unlimited time, ink, and paper" caveat is doing a lot of work in my argument. But we started with a thought experiment that is impossible to implement in practice (simulating a modern digital computer with a pen and paper) so I don't see why my reply can't do the same thing (even if it might require a lot more resources).
I think it's very unlikely that the human brain requires infinite time and memory to simulate. Even if continuous, you could probably simulate to arbitrary accuracy with a big enough discrete approximation. And the Bekenstein bound suggests there is a finite limit to the amount of information that can exist within a given volume.
As for whether my speed analogy works, I still think it does. Sure, if you pick a frame of reference in which you are stationary, then you continue to have experiences at the normal rate. But that wasn't the frame of reference I was using. I was working in the frame of reference of someone back on Earth, which is an equally valid frame of reference. In those coordinates, every physical process in your brain is getting slowed down (electrical impulses are travelling slower from one side of your brain to the other, chemical reactions are slowing down, etc) and you are having experiences at a slower rate.
If the human brain operates according the known laws of physics, then in principle your brain could be simulated with a pen and paper (at least given unlimited time, ink, and paper), and it would behave identically to the real thing (it would talk and think like you and have all your opinions).Â
Suppose this was all that existed of you, and your real brain never had existed. Would that mean that you never existed as a conscious being, despite all your thoughts and utterances still being a part of the world? That seems like a much more counter intuitive conclusion to me than biting the bullet on pen+paper simulations having the potential for consciousness.
I don't get why the "moment of experience taking a thousand years" thing is supposed to be so weird? If we slowed down all the processes in your brain then moments of experience would take longer in physical time. That's not an argument against your consciousness being real. And this isn't a hypothetical. We can literally do that by sending you on a spaceship close to the speed of light, and that's exactly what would happen!
I'm sorry you've said you regret your engagement, since I've found your comments helpful (the link to AISLE's OpenSSL zero days has shifted my view on this a fair bit).
I see, thanks!
I found this summary really helpful, thank you!
One thing I'm a bit confused by: in the table you classify several objections to incomparability as being responses to premise 1. But when I read premise 1 it doesn't obviously seem to make any claim about incomparability? That seems to enter in premise 2. Am I missing something?
Really interesting write up, thanks! What would the effects be on aviation?
As a data scientist who frequently wonders about this question, thank you for this post, it was a really interesting read!
But I'm a bit confused about your final section on "generative AI" roles, and the claim that data people have no special advantage here. What sort of roles do you have in mind exactly? If you're talking only about prompt engineering, or just about the skill of being able to use a product like Claude code/cowork well, then I can see where you're coming from, though I think I still weakly disagree. For example, if you have an understanding of how these models work under the hood then you are probably better placed to understand which tasks it will be good at and which it might struggle with (this includes really basic stuff, like knowledge cutoffs in training data, and the fact it won't necessarily remember what you asked it about the day before without a memory feature bolted on, and therefore won't improve day by day etc). You are also probably in a better position to understand things like which tasks will require higher reasoning, and which won't (which very often doesn't line up with their perceived human difficulty).
But that aside, I don't think applying generative AI to practical problems is just about prompt engineering or becoming a claude cowork super-user. If you are a business that wants to create an automated workflow where you run a certain prompt over a certain dataset at scale, on the basis of certain triggers, then you are doing something that looks a lot like data engineering (and you are going to need to write code). And if you're hitting context window limits, you'll have to manage context, by building some kind of agent orchestration/scaffolding. That's maybe closer to being a brand new skill, but it still requires strong coding ability, combined with the ability to evaluate the performance of a software tool empirically, which is a specific combination of skillsets that data scientists should already have.
It's true that with AI coding assistants, people are writing less and less code by hand. But you still need strong coding ability to use AI coding assistants effectively. Software engineering has not yet been fully automated. There's a possible future where that changes soon, and maybe that's the future informing your advice in that final section? But my personal feeling would be that in that world, most knowledge jobs probably follow shortly after anyway (if not before), so trying to plan your career development around this possible future seems challenging!
I may be misunderstanding something in the argument, but this potential solution is I think already discussed in detail in Reasons+Persons, where Parfit introduces the repugnant conclusion? He calls it the 'lexical value' solution, I think? And I don't think this write up addresses any of the strong arguments that Parfit makes against it there?
In Parfit's original argument, he doesn't just rely on an appeal to intuition that B is better than A. Instead we move to B from A in two steps: first we go to A+ (more people having lives just worth living, but original population unaffected). Parfit claims that A+ is not worse than A. And then we move to world B, where we are now significantly increasing the welfare of already existing people, with only a small drop in the welfare of the original population. Parfit claims that B is better than A+.
If we reject that A+ is not worse than A, then we have to say it is sometimes wrong to create new people, even if their lives are worth living. This seems very strange. If we reject that B is better than A, then we have to believe that a small drop in welfare for people whose welfare is already very high can outweigh a large increase in welfare for people whose welfare is very low. This is the bullet that I think the lexical value solution bites?
But this is a tough bullet to bite! It is the opposite of egalitarian. It says we should prioritise changes in the welfare of already existing high welfare people (whose welfare is above the lexical level) over changes in the welfare of also already existing low welfare people (welfare below the lexical level).
This is all very interesting, thanks for following up!
Something seems especially weird about offsetting your purchase of non-BCC chicken by donating to campaigns to get supermarkets to adopt the BCC.
I think one important consideration missing here: supermarkets respond to campaigners by saying that customers want to buy non-BCC chicken, and they are just doing what their customers want. If you buy non-BCC chicken from them, you make that argument stronger, and the campaigners' argument weaker.
And I don't think this is necessarily a negligible concern in comparison to the other effects being discussed here, since the mechanism for how your small donation is supposed to help chickens is also by tipping the scales on some corporate campaign and getting a company like a supermarket to make a big change.
I don't deny that my "unlimited time, ink, and paper" caveat is doing a lot of work in my argument. But we started with a thought experiment that is impossible to implement in practice (simulating a modern digital computer with a pen and paper) so I don't see why my reply can't do the same thing (even if it might require a lot more resources).
I think it's very unlikely that the human brain requires infinite time and memory to simulate. Even if continuous, you could probably simulate to arbitrary accuracy with a big enough discrete approximation. And the Bekenstein bound suggests there is a finite limit to the amount of information that can exist within a given volume.
As for whether my speed analogy works, I still think it does. Sure, if you pick a frame of reference in which you are stationary, then you continue to have experiences at the normal rate. But that wasn't the frame of reference I was using. I was working in the frame of reference of someone back on Earth, which is an equally valid frame of reference. In those coordinates, every physical process in your brain is getting slowed down (electrical impulses are travelling slower from one side of your brain to the other, chemical reactions are slowing down, etc) and you are having experiences at a slower rate.
If the human brain operates according the known laws of physics, then in principle your brain could be simulated with a pen and paper (at least given unlimited time, ink, and paper), and it would behave identically to the real thing (it would talk and think like you and have all your opinions).Â
Suppose this was all that existed of you, and your real brain never had existed. Would that mean that you never existed as a conscious being, despite all your thoughts and utterances still being a part of the world? That seems like a much more counter intuitive conclusion to me than biting the bullet on pen+paper simulations having the potential for consciousness.
I don't get why the "moment of experience taking a thousand years" thing is supposed to be so weird? If we slowed down all the processes in your brain then moments of experience would take longer in physical time. That's not an argument against your consciousness being real. And this isn't a hypothetical. We can literally do that by sending you on a spaceship close to the speed of light, and that's exactly what would happen!
That makes a lot of sense, thanks.
I'm sorry you've said you regret your engagement, since I've found your comments helpful (the link to AISLE's OpenSSL zero days has shifted my view on this a fair bit).
I guess this whole discussion does just feel like a classic example of "All debates are bravery debates".