User Comment Replies

Academics must gain something from spending ages thinking and studying ethics, be it understanding of the arguments, knowledge of more arguments or something else. I think this puts them in a better position than others and should make others tentative in saying that they're wrong.

Btw, I agree with this in the sense that I'd rather have a random ethicist make decisions about an ethical question than a random person.

I'd definitely be interested to hear more :)

Great! I'm writing a text about this, and I'll add a comment with a reference to it when the... (read more)

EA's Image Problem

Tor_Barstad9y1

Thanks for a thoughtful response.

Likewise :)

My worry is the idea we can round this problem by evaluating the arguments ourselves. We're not special. Academics just evaluate the arguments, like we would, but understand them better. The only way i can see myself being justified in rejecting their views is by showing they're biased. So maybe my point wasn't "the academics are right, so narrow consequentialism is wrong" but "most people who know much more about this than us don't think narrow consequentialism is right, so we don't know its

... (read more)

Tom_Davidson

Thanks for that. My basic worries are: -Academics must gain something from spending ages thinking and studying ethics, be it understanding of the arguments, knowledge of more arguments or something else. I think this puts them in a better position than others and should make others tentative in saying that they're wrong. -Your explanation for disagreeing with certain academics is that they have different starting intuitions. But does this account for the fact that academics can revise/abandon intuitions because of broader considerations. Even if you're right, why you think your intuitions are more reliable than theirs? I'd definitely be interested to hear more :)

EA's Image Problem

Tor_Barstad9y1

Starting a long debate about moral philosophy would be relevant here, but also out of place, so I'll refrain myself.

But what do you mean by "Refrain from posting things that assume that consequentialism is true"? That its best to refrain from posting things that assume that values like e.g. justice aren't ends-in-themselves, or refrain from posting things that assume that consequences and their quantity are important?

If it is something more like the latter, I would ask myself if this would be to pursue the goal of popularity by diminishing a part... (read more)

Tom_Davidson

Thanks for a thoughtful response. Definitely the former. I find it hard to get my head round people who deny the latter. I suspect only people committed to a weird philosophical theories would do it. I thought modern Kantians were more moderate. Let's remember that most people don't have a "moral theory" but care about consequences and a cluster of other concerns: it's these people I don't want to alienate. I think philosophers who reject consequentialism (as the claim that consequences are the only morally relevant thing) might be correct, and personally find it annoying when everyone speak assumes that any such philosopher is obviously mistaken. I certainly agree there's no need to talk as if consequences might be irrelevant! I'm sympathetic with your comments about rationality. I wonder if an equally informative way of phrasing it would be "carefully investigating about which actions help the most people". For people who disagree, reading EA's describe itself as "rational" will be annoying because it implies that they are irrational. This is a really interesting point. We could see history as a reductio on the claim that the academic experts reach even roughly true moral conclusions. So maybe the academics are wrong. My worry is the idea we can round this problem by evaluating the arguments ourselves. We're not special. Academics just evaluate the arguments, like we would, but understand them better. The only way i can see myself being justified in rejecting their views is by showing they're biased. So maybe my point wasn't "the academics are right, so narrow consequentialism is wrong" but "most people who know much more about this than us don't think narrow consequentialism is right, so we don't know its right".

What is the expected effect of poverty alleviation efforts on existential risk?

Tor_Barstad9y2

I'm not aware of careful analysis having been done on the topic.

One thing speaking in favour of it increasing existential risk is if it leads to faster technological progress, which in turn could give less time to research on things that specifically benefit safety, of the kind that MIRI and FHI are doing. I'm thinking that more rich people in previously poor countries would make it more profitable for western countries to invest in R&D and that these previously would fund proportionally less x-risk-research than what takes place in the west (this is n... (read more)

Might wireheaders turn into paperclippers?

Tor_Barstad9y0

“I think the major issue here is that you seem to be taking moral realism for granted and assume that if we look hard enough, morality will reveal itself to us in the cosmos. I'm a moral anti-realist, and I'm unable to conceive of what evidence for moral realism would even look like.”

That may be a correct assessment.

I think that like all our knowledge about anything, statements about ethics rest on unproven assumptions, but that there are statements about some states of the world being preferable to others that we shouldn’t have less confidence in than man... (read more)

Might wireheaders turn into paperclippers?

Tor_Barstad9y0

So a bit of a late answer here :)

"Is this a problem? I don't think humor is inherently valuable. It happens to be valuable to humans, but an alternate world in which it weren't valuable seems acceptable."

If a species has conscious experiences that all are of a kind that we are familiar with, but they lack our strongest and most valued experiences, and devalue these because they follow a strict the-less-similar-to-us-the-less-valuable-policy, then I think that’s regrettable. If they themselves and/or beings they create don’t laugh at jokes but hav... (read more)

Might wireheaders turn into paperclippers?

Tor_Barstad9y2

It appears to me that if we were a species that didn't have [insert any feeling we care about, e.g. love, friendship, humour or the feeling of eating tasty food], and someone then invented it, then many people would think of it as not being valuable. The same would go for some alien species that has different kinds of conscious experiences from us trying to evaluate our experiences. I'm convinced that they would be wrong in not valuing our experiences, and I think this shows that that way of thinking leads to mistakes. Would you agree with this (but perhap... (read more)

Lila

"It appears to me that if we were a species that didn't have [insert any feeling we care about, e.g. love, friendship, humour or the feeling of eating tasty food], and someone then invented it, then many people would think of it as not being valuable." Is this a problem? I don't think humor is inherently valuable. It happens to be valuable to humans, but an alternate world in which it weren't valuable seems acceptable. "I'm convinced that they would be wrong in not valuing our experiences, and I think this shows that that way of thinking leads to mistakes. Would you agree with this (but perhaps still think it's the best policy because there's no better option)?" Completely disagree. They'd be in disagreement with my values, but there's no way to show that they're objectively wrong. "Strong indicators could include what the minds want themselves, how different chemical occurrences in our brains correlate with which experiences we value/prefer, etc." What they "want"? Just like paperclippers "want" paperclips? "Chemical occurrences" is an even more implausible framing. I doubt they'd have any analogue of dopamine, etc. "While similarities to our own minds makes it easier for us to make judgments about the value of a minds consciousness with confidence, it could be that we find that there are states of being that probably are more valuable than that of a biological human. Would you agree?" No, I don't think I agree. Maybe some states are better but only because of degree, e.g. developing purer heroin. I don't think anyone could convince me that a certain configuration of, say, helium is more valuable than a human mind. "It seems entirely plausible that there are conscious experiences that can be perceived to be much more profound/meaningful than anything experienced by current biological humans, and that there could be experiences that are as intensively positive as the experiences of torture are negative to us. Would you agree?" Not sure what you mean by mean

Permanent Societal Improvements

Tor_Barstad9y1

An important topic!

Potentially influencing lock-in is certainly among my motivations for wanting to work on AI friendliness, and doing things that could have a positive impact of a potential lock-in has a lot speaking for it I think (and many of these things, such as improving the morality of the general populous, or creating tools or initiatives for thinking better about such questions, are things that could have significant positive effects also if no lock-in occurs).

As to example of having-more-children out of far-future concerns, I think this could go ... (read more)

How much does work in AI safety help the world?

Tor_Barstad9y3

Cool idea and initiative to make such a calculator :) Although it doesn't quite reflect how I make estimations myself (I might make a more complicated calculator of my own at some point that does).

The way I see it, the work that is done now will be the most valuable per person, and the amount of people working on this towards the end may not be so indicative (nine women cannot make a baby in a month, etc).

Owen Cotton-Barratt

Agree that we are missing some things here. My guess is that this one is not too large (less than an order of magnitude), in significant part because we're just using 'eventual size' as a convenient proxy, and increases to the size there seem likely to be highly correlated with increases at intermediate times. That said, it's great to look for things we may be missing, and see if we can reach consensus about which could change the answer crucially.

Daniel_Dewey

Thanks! :) After our conversation Owen jumped right into the write-up, and I pitched in with the javascript -- it was fun to just charge ahead and execute a small idea like this. It's true that this calculator doesn't take field-steering or paradigm-defining effects of early research into account, nor problems of inherent seriality vs parallelizable work. These might be interesting to incorporate into a future model, at some risk of over-complicating what will always be a pretty rough estimate.

I am Nate Soares, AMA!

Tor_Barstad9y1

So as I understand it, what MIRI is doing now is to think about theoretical issues and strategies and write papers about this, in the hope that the theory you develop can be made use of by others?

Does MIRI think of ever:

Developing AI yourselves at some point?
Creating a goal-alignment/safy-framework to be used by people developing AGI? (Where e.g. reinforcement learners or other AI-compinents can be "plugged in", but in some sense are abstracted away.)

Also (feel free to skip this part of the question if it is too big/demanding):

Personally, I ... (read more)

So8res

Kinda. The current approach is more like "Pretend you're trying to solve a much easier version of the problem, e.g. where you have a ton of computing power and you're trying to maximize diamond instead of hard-to-describe values. What parts of the problem would you still not know how to solve? Try to figure out how to solve those first." (1) If we manage to (a) generate a theory of advanced agents under many simplifying assumptions, and then (b) generate a theory of bounded rational agents under far fewer simplifying assumptions, and then (c) figure out how to make highly reliable practical generally intelligent systems, all before anyone else gets remotely close to AGI, then we might consider teching up towards designing AI systems ourselves. I currently find this scenario unlikely. (2) We're currently far enough away from knowing what the actual architectures will look like that I don't think it's useful to try to build AI components intended for use in an actual AGI at this juncture. (3) I think that making theorem provers easier to use is an important task and a worthy goal. I'm not optimistic about attempts to merge natural language with Martin-Lof type theory. If you're interested in improving theorem-proving tools in ways that might make it easier to design safe reflective systems in the future, I'd point you more towards trying to implement (e.g.) Marcello's Waterfall in a dependently typed language (which may well involve occasionally patching the language, at this stage).

I am Nate Soares, AMA!

Tor_Barstad9y6

To what degree is MIRI now restricted by lack of funding, and is there any amount of funding beyond which you could not make effective use of it?

Among recruiting new talent and having funding for new positions, what is the greatest bottleneck?

So8res

Right now we’re talent-constrained, but we’re also fairly well-positioned to solve that problem over the next six months. Jessica Taylor is joining us in august. We have another researcher or two pretty far along in the pipeline, and we’re running four or five more research workshops this summer, and CFAR is running a summer fellows program in July. It’s quite plausible that we’ll hire a handful of new researchers before the end of 2015, in which case our runway would start looking pretty short, and it’s pretty likely that we’ll be funding constrained again by the end of the year.

mhpage

A modified version of this question: Assuming MIRI's goal is saving the world (and not MIRI), at what funding level would MIRI recommend giving elsewhere, and where would it recommend giving?

I am Nate Soares, AMA!

Tor_Barstad9y2

Is MIRIs hope/ambition that that CEV (http://wiki.lesswrong.com/wiki/Coherent_Extrapolated_Volition) or something resemblant of CEV will be implement, or is this not something you have a stance on?

(I'm not asking whether you think CEV should be the goal-system of the first superintelligence. I know it's possible to have strategies such as first creating an oracle and then at some later point implement something CEV-like.)

So8res

First, I think that civilization had better be really dang mature before it considers handing over the reins to something like CEV. (Luke has written a bit about civilizational maturity in the past.) Second, I think that the CEV paper (which is currently 11 years old) is fairly out of date, and I don't necessarily endorse the particulars of it. I do hope, though, that if humanity (or posthumanity) ever builds a singleton, that they build it with a goal of something like taking into account the extrapolated preferences of all sentients and fulfilling some superposition of those in a non-atrocious way. (I don't claim to know how to fill in the gaps there.)

I am Nate Soares, AMA!

Tor_Barstad9y4

Unless there are strategic concerns I don't fully understand I second this. I cringe a little every time I see such goal-descriptions.

Personally I would argue that the issue of largest moral concern is ensuring that new beings that can have good experiences and have a meaningful existence are put into existence, as the quality and quantity of consciousness experienced by such not-yet-existant beings could dwarf what is experienced by currently existing beings on our small planet.

I understand that MIRI doesn't want to take stance on all controversial ethica... (read more)

Effective Altruism Forum
EA Forum

All of Tor_Barstad's Comments + Replies