Topic Contributions


What Are Your Software Needs?

Talk to HaonChan and Jehan on the Bountied Rationality Discord. They're trying to build this.

Affectable universe?

Not sure if it's the widely used definition, but I think of affectable as anything in our future light cone, with accessible as just the bits which we could get something (e.g. a von Neumann probe) physically to before the expansion of the universe takes it away from our reach, which is a smaller bubble because we can't send matter at light speed, our probe would have to slow down at the other end, and the expansion of the universe robs our probe of velocity relative to distant galaxies over very long periods.

Edit: The FHI has a relevant paper: https://www.fhi.ox.ac.uk/wp-content/uploads/space-races-settling.pdf 

The Vultures Are Circling

I was considering writing something like this up a a while back, but didn't have enough evidence directly, was mostly working of too few examples as a grantmaker and general models. Glad this concern is being broadcast.

I did come up with a proposal for addressing parts of the problem over on the "Please pitch ideas to potential EA CTOs" post. If you're a software dev who wants to help build a tool which might make the vultures less able to eat at least parts of EA please read over the proposal and ping me if interested.

Please pitch ideas to potential EA CTOs

Additional layer: Have the researchers have a separate "Which infrastructure / support has been most valuable to you?" category of karma, and use that to help direct funding towards the most valuable parts of the infrastructure ecosystem to support alignment. This should be one way, with researchers able to send this toward infrastructure, but not the reverse since research is the goal.

Preserving and continuing alignment research through a severe global catastrophe

The purpose of preserving alignment is not to get back to AI as quickly as possible, but to make it more likely that when we eventually do climb the tech tree we are more likely to be able to align advanced AIs. Even if we have to reinvent a large number of technologies, having alignment research ready represents a (slightly non-standard) form of differential technological development rather than simply speeding up the recovery overall.

Preserving and continuing alignment research through a severe global catastrophe

Agreed that civilization restart manuals would be good, would be happy to have the alignment archives stored alongside those. Would prefer not to hold up getting a MVP of this much smaller and easier archive in place waiting for that to come together though.

Preserving and continuing alignment research through a severe global catastrophe

My guess is these are great for longevity,  but maybe prohibitively expensive[1] if you want to print out e.g. the entire alignment forum plus other papers. 

Could be good for a smaller selected key insights collection, if that exists somewhere?

  1. ^

    Likely reference class is gravestones. I'm getting numbers like:  "Extra characters are approximately $10 thereafter" and "It costs around £1.95 per letter or character", even with a bulk discount that's going to add up.

Please pitch ideas to potential EA CTOs

AI alignment is rapidly scaling funding, and this means grantmakers will be stretched thin and less able to reliably avoid giving money to people who are not mission aligned, not producing useful research, or worst of all just want to extract money from the system. This has the potential to cause massive problems down the road, both by producing distracting low-quality research and by setting up expectations which will cause drama if someone is defunded later on while there's still a lot of money flowing to others.

An attack-resistant EigenKarma[1]-like network for alignment researchers would, if adopted, allow the bottleneck of grantmaker time to be eased and quality of vetting to be improved, by allowing all researchers to participate in the process of vetting people and assessing the quality of their work.  The ideal system would:

  • Allow grantmakers to view any individual's score with an arbitrary initial trusted group, so they could analyze how promising someone seems from the perspective of any subgroup, with a clean UI.
  • Allow people to import their own upvote history from any of AF/LW/EAF to seed their outflowing karma, but adjust it manually via a clean UI.
  • Have some basic tools to flag suspicious voting patterns (e.g. two people only channeling karma to each other), likely by graphing networks of votes.
  • Maybe have some features to allow grants to be registered on the platform, so grantmakers can see what's been awarded already?
  • Maybe have a split between "this person seems competent and maybe we should fund them to learn" vs "this person has produced something of value"?

Rob Miles has some code running on his Discord with a basic EigenKarma system, which is currently being used as the basis for a crypto project by some people from Monastic Academy, and could be used to start your project. I have some thoughts on how to improve the code and would be happy to advise.

I'm imagining a world where researchers channel their trust into the people they think are doing the most good work, which means that grantmakers can go "oh, conditioning on interpretability-focused researchers as the seed group, this applicant scores highly" or "huh, this person has been working for two years but no one trusted thinks what they're doing is useful" rather than relying on their own valuable time to assess the technical details or their much less comprehensive and scalable sense of how they think the person is perceived.

Obviously some pre-research would be to make a sketch of how it'd work and ask grantmakers and researchers if they would use the system, but I for one would if it worked well (and might provide some seed funding to the right person).

  1. ^

    This is Rob Miles's description of his code, I hear the EigenTrust++ model is even better, but have not read the paper yet to verify that it makes more sense here.

The Future Fund’s Regranting Program

Yes: "Regrantors will be compensated for their work based on the quality and volume of their grantmaking."

The Future Fund’s Regranting Program

I'd be very interested  in joining as a regranter, though it may make sense to wait a few years, by which point I will have donated most of my crypto pool and gained a bunch of experience. You can see my current strategy at Being an individual alignment grantmaker.

Edit: Does screening for conflicts of interest mean not allowing regranters to grant to people they know? If yes, I can see the reasoning, but if I was operating under this rule it would have blocked several of my most promising grants, which I found through personal connections. I would propose having these grants marked clearly and the regranter's reputation being more strongly staked on those grants going well, rather than outright banning them.

Edit2: Will there be a network for regranters (e.g. Discord, Slack), and would it be possible for me to join as an independent grantmaker to share knowledge and best practices? Or maybe I should just apply now as I'm keen to learn, just not confident I am ready to direct $250k+/year.

Load More