Comments
Suggested hiring practice tweak
There are typically two ways for organisations of running hiring rounds: deadlined, in which job applications are no longer processed after a publicised date, and rolling in which the organisation will keep allowing submissions until they've found someone they want.
The upside of a deadline is both to an applicant that they know they're not wasting their time on a job that's 99% assigned, and to the organisation, which doesn't have to delay giving an answer to an adequate candidate on the grounds that a potentially better one submits when you're most of the way through the hiring process, and incentivises people to apply slightly earlier than they would have.
The downsides are basically the complement. The individual doesn't get to go for a job that they've just missed and would be really suited to, and the org doesn't get to see as large a pool of applicants.
It occurred to me that an org might be able to get some of the best of both by explicitly giving a mostly-deadline, after which they will explicitly downweight new applications. So if you see the mostly-deadline in time, you're still incentivised to get your application in by the date given, and if it's passed you should rationally apply if and only if you think there's a good chance you're an exceptional fit..
One of the problems with AI benchmarks is that they can't effectively be backcast more than a couple of years. This prompted me to wonder if a more empirical benchmark might be something like 'Ability of a human in conjunction with the best technology available at time t'.
For now at least, humans are still necessary to have in the loop, so this should in principle be at least as good as coding benchmarks for gauging where we are now. When/if humans become irrelevant, it should still work - 'AI capability + basically nothing' = 'AI capability'. And looking back, it gives a much bigger reference class for forecasting future trends, allowing us to compare e.g.
etc.
Thoughts?
One problem is putting everything on a common scale when historical improvements are so sensitive to the distribution of tasks. A human with a computer with C, compared to a human with just log tables, is a billion times faster at multiplying numbers but less than twice as fast at writing a novel. So your distribution of tasks has to be broad enough that it captures the capabilities you care about, but it also must be possible to measure a baseline score at low tech level and have a wide range of possible scores. This would make the benchmark extremely difficult to construct in practice.
I think that's right, but modern AI benchmarks seem to have much the same issue. A human with a modern Claude instance might be able to write code 100x faster than without, but probably less than 2x as fast at choosing a birthday present for a friend.
Ideally you want to integrate over... something to do with the set of all tasks. But it's hard to say what that something would be, let alone how you're going to meaningfully integrate it.