All of WillPearson's Comments + Replies

Open Thread #45

Hi, I'm thinking about a possibly new approach to AI safety. Call it AI monitoring and safe shutdown.

Safe shutdown, riffs on the idea of the big red button, but adapts it for use in simpler systems. If there was a big red button, who gets to press it and how? This involves talking to law enforcement, legal and policy. Big red buttons might be useful for non learning systems, large autonomous drones and self-driving cars are two system that might suffer from software failings and need to be shutdown safely if possible (or precipitously if the risks fr... (read more)

8 things I believe about climate change

I found this report on adaptation, which suggest adaptation with some forethought will be better than waiting for problems to get worse. Talks about things other than crops too. The headlines

  • Without adaptation, climate change may depress growth in global agriculture yields up to 30 percent by 2050. The 500 million small farms around the world will be most affected.
  • The number of people who may lack sufficient water, at least one month per year, will soar from 3.6 billion today to more than 5 billion by 2050.
  • Rising seas and greater storm surges could force
... (read more)
On Collapse Risk (C-Risk)

I've been thinking for a while civilisational collapse scenarios impact some of the common assumptions about the expected value of movement building or saving for effective altruism. This has knock on implications to when things are most hingeist.

8 things I believe about climate change
That said, I personally would be quite surprised if worldwide crop yields actually ended up decreasing by 10-30%. (Not an informed opinion, just vague intuitions about econ).

I hope they won't too, if we manage to develop the changes we need to make before we need them. Economics isn't magic

But I wanted to point out that there will probably be costs associated with stopping deaths associated with food shortages with adaptation. Are they bigger or smaller than mitigation by reducing CO2 output or geoengineering?

This case hasn't been made either way to my knowledge and could help allocate resources effectively.

1WillPearson2yI found this report [https://gca.org/global-commission-on-adaptation/report] on adaptation, which suggest adaptation with some forethought will be better than waiting for problems to get worse. Talks about things other than crops too. The headlines * Without adaptation, climate change may depress growth in global agriculture yields up to 30 percent by 2050. The 500 million small farms around the world will be most affected. * The number of people who may lack sufficient water, at least one month per year, will soar from 3.6 billion today to more than 5 billion by 2050. * Rising seas and greater storm surges could force hundreds of millions of people in coastal cities from their homes, with a total cost to coastal urban areas of more than $1 trillion each year by 2050. * Climate change could push more than 100 million people within developing countries below the poverty line by 2030. The costs of climate change on people and the economy are clear. The toll on human life is irrefutable. The question is how will the world respond: Will we delay and pay more or plan ahead and prosper?
8 things I believe about climate change

Are there any states that have committed to doing geoengineering, or even experimenting with geoengineering, if mitigation fails?

Having some publicly stated sufficient strategy would convince me that this was not a neglected area.

1Davidmanheim2yFrom what I understand, Geoengineering is mostly avoided because people claim (incorrectly, in my view) it is a signal that the country thinks there is no chance to fix the problem by limiting emissions. In addition, people worry that it has lots of complex impacts we don't understand. As we understand the impacts better, it becomes more viable - and more worrisome. And as it becomes clearer over the next 20-30 years that a lot of the impacts are severe, it becomes more likely to be tried.

Current investment in solar geoengineering is roughly 10 million annually (this may have increased in the last few years), so by most metrics it's really neglected. The main project working on this is the Harvard solar geoengineering research program, which OPP has funded about 2.5 million dollars for a few years in 2016. They've also funded a solar governance program in 2017 for about 2 million dollars. Grants here. Recently, they don't appear to have made any climate-related grants in this space, and its unclear to me what the funding situ... (read more)

8 things I believe about climate change

I'm expecting the richer nations to adapt more easily, So I'm expecting a swing away from food production in the less rich nations as poorer farmers would have a harder time adapting as there farms get less productive (and they have less food to sell). Also farmers with now unproductive land would struggle to buy food on the open market

I'd be happy to be pointed to the people thinking about this and planning on having funding for solving this problem. Who are the people that will be funding the teaching of subsistence rice farmers (of all na... (read more)

8 things I believe about climate change

On 1) not being able to read the full text of the impactlab report, but it seem they just model the link between heat and mortality, but not the impact of heat on crop production causing knock on health problems. E.g. http://dels.nas.edu/resources/static-assets/materials-based-on-reports/booklets/warming_world_final.pdf suggests that each degree of warming would reduce the current crop yields by 5-15%. So for 4 degrees warming (baseline according to https://climateactiontracker.org/global/temperatures/ ), this would be 20-60% of world food supply reduction... (read more)

3Linch2yThe way climate scientists use those terms, I think of safeguarding soil quality and genetically engineering or otherwise modifying new crops for the heat as more of climate change adaption than mainstream mitigation problem. Tony Allan who I quoted in a different comment also believed that there are a bunch of other ecological problems with the future of our current soil quality. This does seem important? I don't know nearly enough about the field to have any opinions on tractability or neglectedness (David Manheim who commented below seems to know more). That said, I personally would be quite surprised if worldwide crop yields actually ended up decreasing by 10-30%. (Not an informed opinion, just vague intuitions about econ).

You'd need to think there was a very significant failure of markets to assume that food supplies wouldn't be adapted quickly enough to minimize this impact. That's not impossible, but you don't need central management to get people to adapt - this isn't a sudden change that we need to prep for, it's a gradual shift. That's not to say there aren't smart things that could significantly help, but there are plenty of people thinking about this, so I don't see it as neglected of likely to be high-impact.

Are we living at the most influential time in history?

As currently defined, long termists have two possible choices.

  1. Direct work to reduce X-risk
  2. Investing for the future (by saving or movement building) to then spend on reduction of x-risk at a later date

There are however other actions that may be more beneficial.

Let us look again at the definition of influential again

a time ti is more influential (from a longtermist perspective) than a time tj iff you would prefer to give an additional unit of resources,[1] that has to be spent doing direct work (rather than investment), to a longtermist altruist living at t
... (read more)
Critique of Superintelligence Part 2
Let's say they only mail you as much protein as one full human genome.

This doesn't make sense. Do you mean proteome? There is not a 1-1 mapping between genome and proteome. There are at least 20,000 different proteins in the human proteome, it might be quite noticeable (and tie up the expensive protein producing machines), if there were 20,000 orders in a day. I don't know the size of the market, so I may be off about that.

I will be impressed if the AI manages to make a biological nanotech that is not immediately eaten up or accidentally sa... (read more)

1Denkenberger3yI'm not a biologist, but the point is that you can start with a tiny amount of material and still scale up to large quantities extremely quickly with short doubling times. As for competition, there are many ways in which human design technology can exceed (and has exceeded) natural biological organisms' capabilities. These include better materials, not being constrained by evolution, not being constrained by having the organism function as it is built, etc. As for the large end, good point about availability of uranium. But the super intelligence could design many highly transmissible and lethal viruses and hold the world hostage that way. Or think of much more effective ways than we can think of. The point is that we cannot dismiss that the super intelligence could take over the world very quickly.
Impact Investing - A Viable Option for EAs?

There might be a further consideration, people might not start or fund impactful startups if there wasn't a good chance of getting investment. The initial investors (if not impact oriented), might still be counting on impact oriented people to buy the investment. So while each individual impact investor is not doing much in isolation, collectively they are creating a market for things that might not get funded otherwise. How you account for that I'm not sure.

Open Thread #40

It might be worth looking at the domains where it might be less worthwhile (formal chaotic systems, or systems with many sign flipping crucial considerations). If you can show that trying to make cost-effectiveness based decisions in such environments is not worth it, that might strengthen your case.

2Milan_Griffes3yYeah, I'm continuing to think about this, and would like to get more specific about which domains are most amiable to cost-effectiveness analysis (some related thinking here [http://effective-altruism.com/ea/1kv/doinggood%20while_clueless/]). I think it's very hard to identify which domains have the most crucial considerations, because such considerations are unveiled over long time frames. -------------------------------------------------------------------------------- A hypothesis that seems plausible: cost-effectiveness is good for deciding about which interventions to focus on within a given domain (e.g. "want to best reduce worldwide poverty in the next 20 years? These interventions should yield the biggest bang for buck...") But not so good for deciding about which domain to focus on, if you're trying to select the domain that most helps the world over the entire course of the future. For that, comparing theories of change probably works better.
Informational hazards and the cost-effectiveness of open discussion of catastrophic risks

Hi Gregory,

A couple of musings generated by your comment.

2: I don’t think there’s a neat distinction between ‘technical dangerous information’ and ‘broader ideas about possible risks’, with the latter being generally safe to publicise and discuss.

I have this idea of independent infrastructure, trying to make infrastructure (electricity/water/food/computing) that is on a smaller scale than current infrastructure. This is for a number of reasons, one of which includes mitigating risks, How should I build broad-scale support for my ideas without talking ... (read more)

Informational hazards and the cost-effectiveness of open discussion of catastrophic risks

For people outside of EA, I think those who are in possession of info hazard-y content are much more likely to be embedded in some sort of larger institution (e.g., a research scientist or a journal editor looking to publish something), where perhaps the best leverage is setting up certain policies, rather than trying to teach everyone the unilateralist's curse.

There is a growing movement of maker's and citizen scientists that are working on new technologies. It might be worth targeting them somewhat (although again probably without the math). I think t... (read more)

-1turchin3yOne more problem with the idea that I should consult my friends first before publishing a text is a "friend' bias": people who are my friends tend to react more positively on the same text than those who are not friends. I personally had a situation when my friends told me that my text is good and non-info-hazardous, but when I presented it to people who didn't know me, their reaction was opposite.
Informational hazards and the cost-effectiveness of open discussion of catastrophic risks

Ah right. I suppose the unilateralist's curse is only a problem insofar as there are a number of other actors also capable of releasing the information; if you are a single actor then the curse doesn't really apply. Although one wrinkle might be considering the unilateralist's curse with regards to different actors through time (i.e., erring on the side of caution with the expectation that other actors in the future will gain access to and might release the information), but coordination in this case might be more challenging.

Interesting idea. This may ... (read more)

1brianwang7123yYeah. I'll have to think about it more. Yeah, for people outside EA I think structures could be set up such that reaching consensus (or at least a majority vote) becomes a standard policy or an established norm. E.g., if a journal is considering a manuscript with potential info hazards, then perhaps it should be standard policy for this manuscript to be referred to some sort of special group consisting of journal editors from a number of different journals to deliberate. I don't think people need to be taught the mathematical modeling behind the unilateralist's curse for these kinds of policies to be set up, as I think people have an intuitive notion of "it only takes one person/group with bad judgment to fuck up the world; decisions this important really need to be discussed in a larger group." One important distinction is that people who are facing info hazards will be in very different situations when they are within EA vs. when they are out of EA. For people within EA, I think it is much more likely to be the case that a random individual has an idea that they'd like to share in a blog post or something, which may have info hazard-y content. In these situations the advice "talk to a few trusted individuals first" seems to be appropriate. For people outside of EA, I think those who are in possession of info hazard-y content are much more likely to be embedded in some sort of larger institution (e.g., a research scientist or a journal editor looking to publish something), where perhaps the best leverage is setting up certain policies, rather than trying to teach everyone the unilateralist's curse. You're right, strict consensus is the wrong prescription. A vote is probably better. I wonder if there's mathematical modeling that you could do that would determine what fraction of votes is optimal, in order to minimize the harms of the standard unilateralist's curse and the curse in reverse? Is it a majority vote? A 2/3s vote? l suspect this will depend on what th
-2turchin3ySometimes, when I work on a complex problem, I feel as if I become one of the best specialists in it. Surely, I know three other people who are able to understand my logic, but one of them is dead, another is not replying on my emails and the third one has his own vision, affected by some obvious flaw. So none of them could give me correct advice about the informational hazard.
Informational hazards and the cost-effectiveness of open discussion of catastrophic risks

My understanding is that it applies regardless of whether or not you expect others to have the same information. All it requires is a number of actors making independent decisions, with randomly distributed error, with a unilaterally made decision having potentially negative consequences for all.

Information determines the decisions that can be made. For example you can't spread the knowledge of how to create effective nuclear fusion without the information on how to make it.

If there is a single person with the knowledge of how to create safe efficient n... (read more)

0brianwang7123yAh right. I suppose the unilateralist's curse is only a problem insofar as there are a number of other actors also capable of releasing the information; if you are a single actor then the curse doesn't really apply. Although one wrinkle might be considering the unilateralist's curse with regards to different actors through time (i.e., erring on the side of caution with the expectation that other actors in the future will gain access to and might release the information), but coordination in this case might be more challenging. Thanks, this concrete example definitely helps. This makes sense. "Release because the expected benefit is above the expected risk" or "not release because the vice versa is true" is a bit of a false dichotomy, and you're right that we should be more thinking about options that could maximize the benefit while minimizing the risk when faced with info hazards. This can certainly be a problem, and is a reason not to go too public when discussing it. Probably it's best to discuss privately with a number of other trusted individuals first, who also understand the unilateralist's curse, and ideally who don't have the means/authority of releasing the information themselves (e.g., if you have a written up blog post you're thinking of posting that might contain info hazards, then maybe you could discuss in vague terms with other individuals first, without sharing the entire post with them?).
Informational hazards and the cost-effectiveness of open discussion of catastrophic risks

The unilateralists curse only applies if you expect other people to have the same information as you right?

You can figure out if they have the same information as you to see if they are concerned about the same things you are. By looking at the mitigation's people are attempting. Altruists should be attempting mitigations in a unilateralist's curse position, because they should expect someone less cautious than them to unleash the information. Or they want to unleash the information themselves and are mitigating the downsides until they think it is safe.

... (read more)
0turchin3yYes, I met the same problem. The best way to find people who are interested and are able to understand the specific problem is to publish the idea openly in a place like this forum, but in that situation, hypothtical bad people also will be able to read the idea. Also, info-hazard discussion applies only to "medium level safety reserachers", as top level ones have enough authority to decide what is the info hazard, and (bio)scientists are not reading our discussions. As result, all fight with infor hazards is applied to small and not very relevant group. For example, I was advised not to repost the a scientific study as even reposting it would create the informational hazard in the form of attracting attention to its dangerous applications. However, I see the main problem on the fact that such scinetific research was done and openly published, and our relactance to discuss such events only lower our strategic understanding of the different risks.
3brianwang7123yMy understanding is that it applies regardless of whether or not you expect others to have the same information. All it requires is a number of actors making independent decisions, with randomly distributed error, with a unilaterally made decision having potentially negative consequences for all. I agree that having dangerous information released by those who are in a position to mitigate the risks is better than having a careless actor releasing that same information –– but I disagree that this is sufficient reason to preemptively release dangerous information. I think a world where everyone follows the logic of "other people are going to release this information anyway but less carefully, so I might as well release it first" is suboptimal compared to a world where everyone follows a norm of reaching consensus before releasing potentially dangerous information. And there are reasons to believe that this latter world isn't a pipe dream; after all, generally when we're thinking about info hazards, those who have access to the potentially dangerous information generally aren't malicious actors, but rather a finite number of, e.g., biology researchers (for biorisks) who could be receptive to establishing norms of consensus. I'm also not sure how the strategy of "preemptively release, but mitigate" would work in practice. Does this mean release potentially dangerous information, but with the most dangerous parts redacted? Release with lots of safety caveats inserted? How does this preclude the further release of the unmitigated info? I'm not sure I'm fully understanding you here. If you're saying that the majority of potentially dangerous ideas will originate in those who don't know what the unilateralist's curse is, then I agree –– but I think this is just all the more reason to try to spread norms of consensus.
Ineffective entrepreneurship: post-mortem of Hippo, the happiness app that never quite was

Thanks for writing this up! I've forwarded it to a friend who was interested in the happiness app space a while back.

I would add to the advice, from my experience, pick something not too far out of people's comfort zones for a startup or research idea. There seems to be a horizon beyond which you don't get feedback or help at all.

An Argument To Prioritize "Positively Shaping the Development of Crypto-assets"

I think it possible that blockchain can help us solve some co-ordination problems. However it also introduces new ones (e.g. which fork of a chain/version of the protocol you should go with).

So I am torn. It would be good to see one successful use/solid proposal of the technology for solving our real world coordination problems using ethereum.

Something I am keeping an eye on is the economic space agency

Doing good while clueless

I would add something likes "Sensitivity" to the list of attributes needed to navigate the world.

This is different from Predictive Power. You can imagine two ships, with the exact same compute power and Predictive Power. One with cameras on the outside and long range sensors, one blind without. You'd expect the first to do a lot better moving about the world

In Effective Altruism's case I suspect this would be things like the basic empirical research about the state of the world and the things important to their goals.

Open Thread #39

I'm thinking about radically more secure computer architectures as a cause area.

  1. Radical architecture changes are neglected because it hard to change computer architecture
  2. Bad Computer security costs a fair amount at the moment
  3. Having a computer architecture that is insecure is making it hard to adopt more useful technology like Internet of Things.

I'd be interested in doing an analysis of whether it is effective altruistic cause. I'm just doing it as a hobby at the moment. Anyone interested in the same region want to collaborate?

Could I have some more systemic change, please, sir?

There are some systemic reforms that seem easier reason about that others. Getting governments to be able to agree a tax scheme such that the Google's and Facebook's of the world can't hide their profits, seems like a pretty good idea. Their money piles suggest that they aren't hurting for cash to invest in innovation. It is hard to see the downside.

The upside is going to be less in developing world than the developed (due to more profits occurring in the developed world). So it may not be ideal. The tax justice network is something I want to follow more.... (read more)

0Michael_PJ4yThere's a sliding scale of what people consider "systematic reform". Often people mean things like "replace capitalism". I probably wouldn't even have classed drug policy reform or tax reform as "systematic reform", but it's a vague category. Of course the simpler ones will be easier to analyze.
Open Thread #39

I'm thinking about funding an analysis of the link between autonomy and happiness.

I have seen papers like

https://academic.oup.com/heapro/article/28/2/166/661129

and http://www.apa.org/pubs/journals/releases/psp-101-1-164.pdf

I am interested in how reproducible and reliable they are and I was wondering if I could convert money into an analysis of the methodology used in (some of) these papers.

As I respect EA's analytical skills (and hope their is a shared interest in happiness and truth), I thought I would ask here.

Inadequacy and Modesty

In the context of the measurement problem: If the idea is that we may be able to explain the Born rule by revising our understanding of what the QM formalism corresponds to in reality (e.g., by saying that some hidden-variables theory is true and therefore the wave function may not be the whole story, may not be the kind of thing we'd naively think it is, etc.), then I'd be interested to hear more details.

Heh, I'm in danger of getting nerd sniped into physics land, which would be a multiyear journey. I'm found myself trying to figure out whether the st... (read more)

Inadequacy and Modesty

Ah, it has been a while since I engaged with this stuff. That makes sense. I think we are talking past each other a bit though. I've adopted a moderately modest approach to QM since I've not touched it in a bit and I expect the debate has moved on a bit.

We started from a criticism of a particular position (the copenhagen interpretation) which I think is a fair thing to do for the modest and immodest. The modest person might misunderstand a position and be able to update themselves better if they criticize it and get a better explanation.

The question is wh... (read more)

1RobBensinger4yI don't think we should describe all instances of deference to any authority, all uses of the outside view, etc. as "modesty". (I don't know whether you're doing that here; I just want to be clear that this at least isn't what the "modesty" debate has traditionally been about.) I don't think there's any general answer to this. The right answer depends on the strength of the object-level arguments; on how much reason you have to think you've understood and gleaned the right take-aways from those arguments; on your model of the physics community and other relevant communities; on the expected information value of looking into the issue more; on how costly it is to seek different kinds of further evidence; etc. In the context of the measurement problem: If the idea is that we may be able to explain the Born rule by revising our understanding of what the QM formalism corresponds to in reality (e.g., by saying that some hidden-variables theory is true and therefore the wave function may not be the whole story, may not be the kind of thing we'd naively think it is, etc.), then I'd be interested to hear more details. If the idea is that there are ways to talk about the experimental data without committing ourselves to a claim about why the Born rule holds, then I agree with that, though it obviously doesn't answer the question of why the Born rule holds. If the idea is that there are no facts of the matter outside of observers' data, then I feel comfortable dismissing that view even if a non-negligible number of physicists turn out to endorse it. I also feel comfortable having lower probability in the existence of God than the average physicist does; and "physicists are the wrong kind of authority to defer to about God" isn't the reasoning I go through to reach that conclusion.
Inadequacy and Modesty

and Eliezer hasn't endorsed any solution either, to my knowledge)

Huh, he seemed fairly confident about endorsing MWI in his sequence here

1RobBensinger4yHe endorses "many worlds" in the sense that he thinks the wave-function formalism corresponds to something real and mind-independent, and that this wave function evolves over time to yield many different macroscopic states like our "classical" world. I've heard this family of views called "(QM) multiverse" views to distinguish this weak claim from the much stronger claim that, e.g., decoherence on its own resolves the whole question of where the Born rule comes from. From a 2008 post [http://lesswrong.com/lw/py/the_born_probabilities/] in the MWI sequence:
Inadequacy and Modesty

Concerning QM: I think Eliezer's correct that Copenhagen-associated views like "objective collapse" and "quantum non-realism" are wrong, and that the traditional arguments for these views are variously confused or mistaken, often due to misunderstandings of principles like Ockham's razor. I'm happy to talk more about this too; I think the object-level discussions are important here.

I don't think the modest view (at least as presented by Gregory) would believe in any of the particular interpretations as there is significant debate sti... (read more)

0RobBensinger4yYeah, I'm not making claims about what modest positions think about this issue. I'm also not endorsing a particular solution to the question of where the Born rule comes from (and Eliezer hasn't endorsed any solution either, to my knowledge). I'm making two claims: 1. QM non-realism and objective collapse aren't true. 2. As a performative corollary, arguments about QM non-realism and objective collapse are tractable, even for non-specialists; it's possible for non-specialists to reach fairly confident conclusions about those particular propositions. I don't think either of those claims should be immediately obvious to non-specialists who completely reject "try to ignore object-level arguments"-style modesty, but who haven't looked much into this question. Non-modest people should initially assign at least moderate probability to both 1 and 2 being false, though I'm claiming it doesn't take an inordinate amount of investigation or background knowledge to determine that they're true. (Edit re Will's question below [http://effective-altruism.com/ea/1g4/inadequacy_and_modesty/cff]: In the QM sequence [http://lesswrong.com/lw/py/the_born_probabilities/], what Eliezer means by "many worlds" is only that the wave-function formalism corresponds to something real in the external world, and that this wave function evolves over time to yield many different macroscopic states like our "classical" world. I've heard this family of views called "(QM) multiverse" views to distinguish this weak claim from the much stronger claim that, e.g., decoherence on its own resolves the whole question of where the Born rule comes from.)
In defence of epistemic modesty

Is there any data on how likely EAs think that explosive progress after HLMI will happen? I would have thought it more than 10%?

I would also have expected more debate about explosive progress, more than just the recent Hanson-Yudkowski flair up, if there was as much doubt in the community as that survey suggests.

In defence of epistemic modesty

Another reason to not have too much modesty within society is that it makes expert opinion very appealing to subvert. I wrote a bit about that here.

Note that I don't think that my views about the things that I believe subverted/unmoored would be necessarily correct, but that the first order of business would be to try and build a set of experts with better incentives.

Why & How to Make Progress on Diversity & Inclusion in EA

Since I've not seen it mentioned here, unconferences seem like a inclusive type of event as described above. I'm not sure how EAG compare.

Why & How to Make Progress on Diversity & Inclusion in EA

Yes, there are a few EA leftists whose main priority is to systemically reform capitalism, but not significantly more than there were in the first place, and they are a tiny group in comparison to the liberals, the conservatives, the vegans, the x-risk people, and so on. As far as I can tell, the impact of all these articles and comments in bringing leftists into active participation with EA was totally nonexistent.

I'm not sure I count or not. My work on autonomy can be seen as investigating systemic change. I've been to a couple of meetups and hung aro... (read more)

5 Types of Systems Change Causes with the Potential for Exceptionally High Impact (post 3/3)

As a data point for how useful ITN is for trying to think about system change I shall talk about my attempt.

My attempt is here.

So the intervention was to try and start a movement of people who shared technological development with each other to help them live (so food/construction/computing), with the goal of creating autonomous communities (capable of space faring or living on other planets would be great, but is not the focus). The main difference between it and the normal open source movement is that it would focus on intelligence augmentation to start... (read more)

Open Thread #39

I've posted about an approach to AGI estimation.

I would love to find collaborators on this kind of thing. In London would be great.

An intervention to shape policy dialogue, communication, and AI research norms for AI safety

I take the point. This is a potential outcome, and I see the apprehension, but I think it's a probably a low risk that users will grow to mistake robotics and hardware accidents for AI accidents (and work that mitigates each) - sufficiently low that I'd argue expected value favours the accident frame. Of course, I recognize that I'm probably invested in that direction.

I would do some research onto how well sciences that have suffered brand dilution do.

As far as I understand it Research institutions have high incentives to

  1. Find funding
  2. Pump out tracti
... (read more)
An intervention to shape policy dialogue, communication, and AI research norms for AI safety

I agree it is worth reconsidering the terms!

The agi/narrow ai distinction is beside the point a bit, I'm happy to drop it. I also have an AI/IA bugbear so I'm used to not liking how things are talked about.

Part of the trouble is we have lost the marketing war before it even began, every vaguely advanced technology we have currently is marketing itself as AI, that leaves no space for anything else.

AI accidents brings to my mind trying to prevent robots crashing into things. 90% of robotics work could be classed as AI accident prevention because they are alw... (read more)

0Lee_Sharkey4yI take the point. This is a potential outcome, and I see the apprehension, but I think it's a probably a low risk that users will grow to mistake robotics and hardware accidents for AI accidents (and work that mitigates each) - sufficiently low that I'd argue expected value favours the accident frame. Of course, I recognize that I'm probably invested in that direction. I think this steers close to an older debate on AI “safety” vs “control” vs “alignment” [https://ai-alignment.com/ai-safety-vs-control-vs-alignment-2a4b42a863cc]. I wasn't a member of that discussion so am hesitant to reenact concluded debates (I've found it difficult to find resources on that topic other than what I've linked - I'd be grateful to be directed to more). I personally disfavour 'motivation' on grounds of risk of anthropomorphism.
An intervention to shape policy dialogue, communication, and AI research norms for AI safety

So what are the risks of this verbal change?

Potentially money gets mis-allocated: Just like all chemistry got rebranded nanotech during that phase in the 2000, if there is money in AI safety, computer departments will rebrand research as AI safety to prevent AI accidents. This might be a problem when governments start to try and fund AI Safety.

I personally want to be able to differentiate different types of work, between AI Safety and AGI Safety. Both are valuable, we are going to living in a world of AI for a while and it may cause catastrophic problems (... (read more)

2Lee_Sharkey4yI think this proposition could do with some refinement. AI safety should be a superset of both AGI safety and narrow-AI safety. Then we don't run into problematic sentences like "AI safety may not help much with AGI Safety", which contradicts how we currently use 'AI safety'. To address the point on these terms, then: I don't think AI safety runs the risk of being so attractive that misallocation becomes a big problem. Even if we consider risk of funding misallocation as significant, 'AI risk' seems like a worse term for permitting conflation of work areas. Yes, it's of course useful to have two different concepts for these two types of work, but this conceptual distinction doesn't go away with a shift toward 'AI accidents' as the subject of these two fields. I don't think a move toward 'AI accidents' awkwardly merges all AI safety work. But if it did: The outcome we want to avoid is AGI safety getting too little funding. This outcome seems more likely in a world that makes two fields of N-AI safety and AGI safety, given the common dispreference for work on AGI safety. Overflow seems more likely in the N-AI Safety -> AGI Safety direction when they are treated as the same category than when they are treated as different. It doesn't seem beneficial for AGI safety to market the two as separate types of work. Ultimately, though, I place more weight on the other reasons why I think it's worth reconsidering the terms.
Personal thoughts on careers in AI policy and strategy

I think an important thing for Ai strategy is to figure out ishow to fund empirical studies into questions that impinge on crucial considerations.

For example funding studies into the nature of IQ. I'll post an article on that later but wanted to flag it here as well.

Personal thoughts on careers in AI policy and strategy

I agree that creativity is key.

I'd would point out that you may need discipline to do experiments based upon your creative thoughts (if the information you need is not available). If you can't check your original reasoning against the world, you are adrift in a sea of possibilities.

1John_Maxwell4yYeah, that sounds about right. Research and idea generation are synergistic processes. I'm not completely sure what the best way to balance them is.
Personal thoughts on careers in AI policy and strategy

I think it is important to note that in the political world there is the vision of two phases of AI development, narrow AI and general AI.

Narrow AI is happening now. The 30+% job loss predictions in the next 20 years, all narrow AI. This is what people in the political sphere are preparing for, from my exposure to it.

General AI is conveniently predicted more that 20 years away, so people aren't thinking about it because they don't know what it will look like and they have problems today to deal with.

Getting this policy response right to narrow AI does hav... (read more)

S-risk FAQ

How do you feel about the mere addition paradox? These questions are not simple.

Personal thoughts on careers in AI policy and strategy

I would broadly agree. I think this is an important post and I agree with most of the ways to prepare. I think we are not there yet for large scale AI policy/strategy.

There are few things that I would highlight as additions. 1) We need to cultivate the skills of disentanglement. Different people might be differently suited, but like all skills it is one that works better with practice and people to practice with. Lesswrong is trying to place itself as that kind of place. It is having a little resurgence with the new website www.lesserwrong.com. For exampl... (read more)

Capitalism and Selfishness

There is still disagreement about how to best donate (to do most good) among individuals which gives support to the argument that profits should be paid out even among altruistic investor base

True, but to if I put myself in the perfect altruist company owner shoes I would really want to delegate the allocation of the my charitable giving, because I am too busy running my company to have much good information about who to donate to.

If I come happen to come in to some information about what good charitable giving is, I should be able to take the informati... (read more)

0tuukkasarvi4yHi! Apologies for response delay. "True, but to if I put myself in the perfect altruist company owner shoes I would really want to delegate the allocation of the my charitable giving, because I am too busy running my company to have much good information about who to donate to." I agree with that usually it is not efficient for same person to take care and optimize 1) (for-profit private) company operations 2) allocation of charitable giving. So person doing 1) would do well to delegate 2) to someone who she trusts. In any case, I reiterate my previous point: I don't think having "benevolent" companies would be something I support (benevolent in the sense that the company commits to donate all profits) because: Firstly, it would decrease the possible investor base because only strictly altruistic investors would be interested and thus it would not likely able to raise as much funding as a "non-benevolent" company (altruistic investors are also interested in "non-benevolent" companies because they can freely donate any profits they make). Secondly, there is disagreement among altruists of how to best donate. Thus, if profits are given to investors, each altruist can choose personally how to donate. So even altruistic investors might be hesitant to invest in a "benevolent" company I outlined here. As far as I am tell, it's best to have a for-profit company optimizing production and maximizing profits which are distributed to investors some of which can be efficient altruists who in turn donate them as they see fit. Charitable givers can delegate their giving to a fund of charities of which I think OpenPhil is an example of.
Capitalism and Selfishness

I don't understand this. Do you suggest that all companies should be trying to fulfill (all) the needs of some collective. It is very useful for companies to specialize.

I expect all benevolent companies to fulfil the needs of others with their profits (if they are not reinvesting them in expansion). For that is the definition of benevolence right? People have an ethos of benevolence insofar as they pursue the interests of others.

There are two aspects of ownership of the means of production

  1. Control over the operations
  2. Control of the profits

I would exp... (read more)

0tuukkasarvi4yI think I have now a better understanding of what you meant. I think there are at least three optimization problems here: 1) what to produce? (investment decision) 2) how to produce? (organization of operations) and 3) how to use the returns , for EA-minded, how to donate? Company traditionally optimizes 2) and 1) in a more restricted manner (within their field of business or local opportunities) I think there might be some problems with a hypothetical "benevolent" company that also commits to donate all the profits to an charity or portfolio of charities. Firstly, it would decrease the possible investor base because only strictly altruistic investors would be interested and thus it would not likely able to raise as much funding as a "non-benevolent" company (altruistic investors are also interested in "non-benevolent" companies because they can freely donate any profits they make). Secondly, there is disagreement among altruists of how to best donate. Thus, if profits are given to investors, each altruist can choose personally how to donate. So even altruistic investors might be hesitant to invest in a "benevolent" company I outlined here. "So I was trying to break down the concept of ownership some more and arguing that in a benevolent world private ownership might only mean keeping control over operations."
Capitalism and Selfishness

I'm having trouble seeing how an individual benevolent person would hold ownership of a company (unless they were the only benevolent person). It would require the owner to think that they knew the best about how to distribute the fruits of the company.

This seems unlikely, as they are trying to help lots of people they need lots of data about what people need to meet their interests. This data collection seems like it would be best be done in a collective manner (as getting more data from more people about what is needed should give a more accurate view o... (read more)

0tuukkasarvi4y"So why wouldn't the benevolent individual give their share of the company to whatever collective system that determined the needs of the world? They could still be ceo, so that they could manage the company better (as they have good data about that). It seems like the capitalist system would morph into either socialism or charity-sector owned means of production, if everyone were benevolent." I don't understand this. Do you suggest that all companies should be trying to fulfill (all) the needs of some collective. It is very useful for companies to specialize. I think charityism and capitalism can work well together. The question of how to best produce something (1) is separate from what to produce (2), and these are in turn separate from the question how to best donate (3). Basically, in a market economy, investors (capital owners) are making bets on how to answer the first two questions. Consumers (which includes also capital owners) are collectively affecting and largely determining (2) and also (3) (to the extent they decide to donate). If you are a large investor (own a lot of capital) you can donate all your returns if you wish, EA-style. "So why wouldn't the benevolent individual give their share of the company to whatever collective system that determined the needs of the world?"
EA Survey 2017 Series: Cause Area Preferences

My personal idea of it is a broad church. So the systems that govern our lives, government and the economy distribute resources in a certain way. These can have a huge impact on the world. They are neglected because it involves fighting an uphill struggle against vested interests.

Someone in a monarchy campaigning for democracy would be an example of someone who is aiming for systemic change. Someone who has an idea to strengthen the UN so that it could help co-ordinate regulation/taxes better between countries (so that companies don't just move to low tax, low worker protection, low environmental regulation areas) is aiming for systemic change.

2Michelle_Hutchinson4yWill, you might be interested in these conversation notes between GiveWell and the Tax Justice Network: http://files.givewell.org/files/conversations/Alex_Cobham_07-14-17_(public).pdf [http://files.givewell.org/files/conversations/Alex_Cobham_07-14-17_(public).pdf] (you have to c&p the link)
EA Survey 2017 Series: Cause Area Preferences

I'm not sure if there are many EAs interested in it, because of potential low tractability. But I am interested in "systemic change" as a cause area.

1Peter Wildeford4yWhat does "systemic change" actually refer to? I don't think I ever understood the term.
EA Survey 2017 Series: Cause Area Preferences

Just a heads up "technological risks" ignores all the non-anthropogenic catastrophic risks. Global catastrophic risks seems good.

Ideological engineering and social control: A neglected topic in AI safety research?

I think this is part of the backdrop to my investigation into the normal computer control problem. People don't have control over their own computers. the bad actors that do get control could be criminals or a malicious state (or AIs).

Ideological engineering and social control: A neglected topic in AI safety research?

The increasing docility could be a stealth existential risk increaser, in that people would be less willing to challenge other peoples ideas and so slow or stop entirely technological progress we need to save ourselves from super volcanoes and other environmental threats

Looking at how Superforecasting might improve some EA projects response to Superintelligence

Thanks for the links. It would have been nice to have got them when I emailed OPP a few days ago with a draft of this article.

I look forward to seeing the fruits of "Making Conversations Smarter, Faster"

I'm going to dig into the AI timeline stuff, but from what I have seen from similar things, there is an inferential step missing. The question is "Will HLMI (of any technology) might happen with probability X by Y" and the action is then "we should invest in most of the money in a community for machine learning people and people wo... (read more)

Load More