All of Jan_Kulveit's Comments + Replies

I was considering hypothetical scenarios of the type "imagine this offer from MIRI arrived, would a lab accept" ; clearly MIRI is not making the offer because the labs don't have good alignment plans and they are obviously high integrity enough to not be corrupted by relatively tiny incentives like $3b

I would guess there are ways to operationalise the hypothethicals, and try to have, for example, Dan Hendrycks guess what would xAI do, him being an advisor. 

With your bets about timelines - I did 8:1 bet with Daniel Kokotajlo against AI 2027 being as ac... (read more)

The operationalisation you propose does not make any sense, Yudkowsky and Soares do not claim ChatGPT 5.2 will kill everyone or anything like that. 

What about this:

MIRI approaches [a lab] with this offer: we have made some breakthrough in ability to verify if the way you are training AIs leads to misalignment in the way we are worried about. Unfortunately the way to verify requires a lot of computations (ie something like ARC), so it is expensive.  We expect your whole training setup will pass this, but we will need $3B from you to run this; if o... (read more)

Travel: mostly planned (conferences, some research retreats).

We expect closely coordinated team work on the LLM psychology direction, with a bit looser connections to the gradual disempowerment / macrostrategy work. Broadly ACS is small enough that anyone is welcome to participate in anything they are interested in, and generally everyone has idea what others work on.

My impression is EAGx Prague 22 managed to balance 1:1s with other content simply by not offering SwapCard 1:1s slots part of the time, having a lot of spaces for small group conversations, and suggesting to attendees they should aim for something like balanced diet. (Turning off SwapCard slots does not prevent people from scheduling 1:1, just adds a little friction; empirically it seems enough to prevent the mode where people just fill their time by 1:1s).

As far as I understand this will most likely not happen, because weight given to / goodharting on met... (read more)

Thanks for explanation. My guess is this decision should not be delegated to LLMs but mostly to authors (possibly with some emphasis on correct classification in the UI).

I think the "the post concerns an ongoing conversation, scandal or discourse that would not be relevant to someone who doesn't care about the EA community" should not be interpreted extensively, otherwise it can easily mean "any controversy or criticism". I will repost it without the links to current discussions - these are non-central, similar points are raised repeatedly over the years and it is easy to find dozens of texts making them.

I wrote a post on “Charity” as a conflationary alliance term. You can read it on LessWrong, but I'm also happy to discuss it here.

If wondering why not post it here: Originally posted it here with a LW cross-post. It was immediately slapped with the "Community" tag, despite not being about community, but about different ways people try to do good, talk about charity & ensuing confusions. It is about the space of ideas, not about the actual people or orgs.

With posts like OP announcements about details of EA group funding or EAG admissions bar not being m... (read more)

Seems plausible the impact of that single individual act is so negative that aggregate impact of EA is negative.

I think people should reflect seriously upon this possibility and not fall prey to wishful thinking (let's hope speeding up the AI race and making it superpower powered is the best intervention! it's better if everyone warning about this was wrong and Leopold is right!).

The broader story here is that EA prioritization methodology is really good for finding highly leveraged spots in the world, but there isn't a good methodology for figuring out what to do in such places, and there also isn't a robust pipeline for promoting virtues and virtuous actors to such places.

I don't think so. I think in practice

I. - Some people don't like the big R community very much.

AND

2a. - Some people don't think improving the EA community small-r rationality/epistemics should be one of top ~3-5 EA priorities. 
OR
2b.  - Some people do agree this is important, but don't clearly see the extent to which the EA community imported healthy epistemic vigilance and norms from Rationalist or Rationality-adjacent circles

=>

- As a consequence, they are at risk of distancing from small r rationality as a collateral damage / by neglect


Also I... (read more)

Reducing rationality to "understand most of Kahneman and Tversky's work" and cognitive psychology would be extremely narrow and miss most of the topic.

To quickly get some independent perspective, I recommend reading the Overview of the handbook part of "The Handbook of Rationality"  (2021, MIT Press, open access). For an extremely crude calibration: the Handbook has 65 chapters. I'm happy to argue at least half of them cover topics relevant to the EA project. About ~3 are directly about Kahneman and Tversky's work. So, by this proxy, you would miss about 90% of whats relevant.


 

Sorry for sarcasm, but what about returning to the same level of non-involvement and non-interaction between EA and Rationality as you describe was happening in Sydney? I.e. EA events are just co-hosted with LW Rationality and Transhumanism, and the level of Rationality idea non-influence is kept on par with Transhumanism? 

It would be indeed very strange if people made the distinction, thought about the problem carefully, and advocated for distancing from 'small r' rationality in particular.

I would expect real cases to look like
- someone is deciding about an EAGx conference program; a talk on prediction markets sounds subtly Rationality-coded, and is not put on schedule
- someone applies to OP for funding to create rationality training website; this is not funded because making the distinction between Rationality and rationality would require too much nuance
- someone is decid... (read more)

(crossposted from twitter) Main thoughts: 
1. Maps pull the territory 
2. Beware what maps you summon 

Leopold Aschenbrenners series of essays is a fascinating read: there is a ton of locally valid observations and arguments. Lot of the content is the type of stuff mostly discussed in private. Many of the high-level observations are correct. 

At the same time, my overall impression is the set of maps sketched pulls toward existential catastrophe, and this is true not only for the 'this is how things can go wrong' part, but also for the 'thi... (read more)

FWIW ... in my opinion, retaining the property might have been a more beneficial decision. 

Also, I think some people working in the space should not make an update against plans like "have a permanent venue", but plausibly should make some updates about the "major donors". My guess this almost certainly means Open Philanthropy, and also likely they had most of the actual power in this decision. 

Before delving further, it's important to outline some potential conflicts of interest and biases:
- I did co-organize or participated at multiple events a... (read more)

In my view the basic problem with this analysis is you probably can't lump all the camps together as one thing and evaluate them together as one entity. Format, structure, leadership and participants seem to have been very different.

Based on public criticisms of their work and also reading some documents about a case where we were deciding whether to admit someone to some event (and they forwarded their communication with CH). It's a limited evidence, but still some evidence.

 

This is a bit tangential/meta, but looking at the comment counter makes me want to express gratitude to the Community Health Team at CEA. 

I think here we see a 'practical demonstration' of the counterfactuals of their work:
- insane amount of attention sucked by this
- the court of public opinions on fora seems basically strictly worse at all relevant dimensions like fairness, respect of privacy or compassion to people involved

As 'something like this'  would be quite often the counterfactual to CH to trying to deal with stuff ...it makes it clear how much value they are creating by dealing with these problems, even if their process is imperfect

Sorry for the delay in response.

Here I look at it from a purely memetic perspective - you can imagine thinking as a self-interested memplex. Note I'm not claiming this is the main useful perspective, or this should be the main perspective to take. 

Basically, from this perspective

* the more people think about AI race, the easier is to imagine AI doom. Also the specific artifacts produced by AI race make people more worried - ChatGPT and GPT-4 likely did more for normalizing and spreading worried about AI doom than all the previous AI safety outreach to... (read more)

Personally, I think the 1:1 meme is deeply confused.

A helpful analogy (thanks to Ollie Base) is with nutrition. Imagine someone hearing that "chia seeds are the nutritionally most valuable food, top rated in surveys" ... and subsequently deciding to eat just chia seeds, and nothing else! 

In my view, sort of obviously, intellectual conference diet consisting just of 1:1s is poor and unhealthy for almost everyone. 

In my view this is a bad decision. 

As I wrote on LW 

Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball - and the style of somewhat strawmanning everyone & strong words is a bit irritating.

In particular I don't appreciate the epistemic of these moves together

1. Appeal to seeing thinks from close proximity. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this o... (read more)

Copy-pasting here from LW.

Sorry but my rough impression from the post is you seem to be at least as confused about where the difficulties are as average of alignment researchers you think are not on the ball - and the style of somewhat strawmanning everyone & strong words is a bit irritating.

Maybe I'm getting it wrong, but it seems the model you have for why everyone is not on the ball is something like "people are approaching it too much from a theory perspective, and promising approach is very close to how empirical ML capabilities research works" &a... (read more)

(crossposted from Alignment Forum)

While the claim - the task ‘predict next token on the internet’ absolutely does not imply learning it caps at human-level intelligence - is true, some parts of the post and reasoning leading to the claims at the end of the post are confused or wrong. 

Let’s start from the end and try to figure out what goes wrong.

GPT-4 is still not as smart as a human in many ways, but it's naked mathematical truth that the task GPTs are being trained on is harder than being an actual human.

And since the task that GPTs are be

You are correct with some of the criticism, but as a side-note, completeness is actually crazy. 

All real agents are bounded, and pay non-zero costs for bits, and as a consequence, don't have complete preferences. Complete agents in real world do not exist. If they existed, correct intuitive model of them wouldn't be 'rational players' but 'utterly scary god, much bigger than the universe they live in'. 

1. a. 
   Sequoia led FTX round B in Jul 2021 and had notably more time to notice any irregularities than grant recipients. 
   
   b.
   I would expect the funds to have much better expertise in something like "evaluating the financial health of a company".  
   
   Also it seem you are somewhat shifting the goalposts: Zoe's paragraph with "On Halloween this past year, I was hanging out with a few EAs." It is reasonable to assume the reader will interpret it as hanging out with basically random/typical EAs, and the argument should hold for these people.  Your ar

In my view this is an example of a mistake in bounded/local consequentialism

From deontic perspective, there is a coordination problem, where "at least consistent handle" posts can be somewhat costly for the poster, but an atmosphere of an earnest discussion of real people  has large social benefits. Vice versa,  discussion with a large fraction of anonymous accounts - in particular if they are sniping at real people and each other - decreases trust, and is vulnerable to manipulation by sock puppets and nefarious players. 

Also, I think there ... (read more)

Seems worth trying

At the same time, I don't think the community post / frontpage attention mechanism is the core of what's going on. Which is, in my guess, often best understood as a fight between memeplexes about hearts and minds

The quality of reasoning in the text seems somewhat troublesome. Using two paragraphs as example 

On Halloween this past year, I was hanging out with a few EAs. Half in jest, someone declared that the best EA Halloween costume would clearly be a crypto-crash — and everyone laughed wholeheartedly. Most of them didn’t know what they were dealing with or what was coming. I often call this epistemic risk: the risk that stems from ignorance and obliviousness, the catastrophe that could have been avoided, the damage that could have been abated, by simply kno

Also: Confido works for intuitively eliciting probabilities

I think this is a weird response to what Buck wrote. Buck also isn't paid  either to reform EA movement, or to respond to criticism on EA forum, and decided to spend his limited time to express how things realistically look from his perspective.

I think it is good if people write responses like that, and such responses should be upvoted, even if you disagree with the claims. Downvotes should not express 'I disagree', but 'I don't want to read this'.

Even if you believe EA orgs are horrible and should be completely reformed, in my view, you should be gla... (read more)

Thanks for all the care and effort which went into writing this!

At the same time, while reading, my reactions were most of the time "this seems a bit confused", "this likely won't help" or "this seems to miss the fact that there is someone somewhere close to the core EA orgs who understands the topic pretty well, and has a different opinion".

Unfortunately, to illustrate this in detail for the whole post would be a project for ...multiple weeks.

At the same time I thought it could be useful to discuss at least one small part in detail, to illustrate how the ... (read more)

In the spirit of communication style you advocate for... my immediate emotional reaction to this is "Eternal September has arrived".

I dislike my comment being summarized as "brings up the "declining epistemics" argument to defend EA orgs from criticism".  In the blunt style you want, this is something between distortion and manipulation. 

On my side, I wanted to express my view on the Wytham debate. And I wrote a comment expressing  my views on the debate.

I also dislike the way my comment is straw-manned by selective quotation.

In the next bul... (read more)

I  will try to paraphrase, please correct me if I'm wrong about this: the argument is, this particular bikeshed is important because it provides important evidence about how EA works, how trustworthy the people are, or what are the levels of transparency. I think this is a fair argument.

At the same time I don't think it works in this case, because while I think EA has important issues, this purchase does not really illuminate them.

Specifically, object level facts about this bikeshed

• do not provide that that much evidence, beyond basic facts like "peopl

For me, unfortunately, the discourse surrounding Wytham Abbey, seems like a sign of epistemic decline of the community, or at least on the EA forum.
 

• The amount of attentions spent on this seems to be a textbook example of bikeshedding. 
  
  Quoting Parkinson :"The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved." A reactor is so vastly expensive and complicated that an average person cannot understand it (see ambiguity aversion), so one assumes that those who work on it understand it. However, everyone

Nice post, but my rough take is there is 
 

• it's relatively common markets are inefficient, but unexploitable; trading on "everyone dies" seems a clear case of hard-to-exploit inefficiency
• markets are not magic; impacts of one-off events with complex consequences are difficult to price in, and what all the magical market aggregation boils down to are a bunch of  human brains doing the trades; e.g. I was able to beat the market and get n-times return at point where markets were insane about covid; later, I talked about it with someone in one of

When the discussion is roughly at the level  'seem to me obviously worth doing ' it seem to me fine to state dissent of the form 'often seems bad or not working to me'.

Stating an opinion is not 'appeal to authority'. I think in many cases it's useful to know what people believe, and if I have to choose between a forum where people state their beliefs openly and more often, and a forum, where people state beliefs only when they are willing to write a long and detailed justification, I prefer the first.

I'm curious in which direction you think the suppos... (read more)

• If we had incentivised whistle-blowers to come forward around shady things happening at FTX, would we have known about FTX fraud sooner and been less reliant on FTX funding? Very plausibly yes. She says "likely" which is obviously not particularly specific, but this would fit my definition of likely.

Why do you think so? Whistleblowers inside of FTX would have been protected under US law, and US institution like SEC offer them multi-million dollar bounties. Why would EA scheme create stronger incentive?

Also: even if the possible whistleblowers inside ... (read more)

I was reacting mostly  to this part of the post

I’ve honestly been pretty surprised there has not been more public EA discussion post-FTX of adopting a number of Cremer's proposed institutional reforms, many of which seem to me obviously worth doing 
...
Also, insofar as she'd be willing (and some form of significant compensation is clearly merited), integrally engaging Cremer in whatever post-FTX EA institutional reform process emerges would be both directly helpful and a public show of good faith efforts at rectification.

I think it's fine for a co... (read more)

I think you lack part of the context where Zoe seems to claim to media the suggested reforms would help

- this Economist piece, mentioning Zoe about 19 times
- WP
- this New Yorker piece, with Zoe explaining "My recommendations were not intended to catch a specific risk, precisely because specific risks are hard to predict”  but still saying ... “But, yes, would we have been less likely to see this crash if we had incentivized whistle-blowers or diversified the portfolio to be less reliant on a few central donors? I believe so.”
-this twitter thread
 ... (read more)

I'm not confident what the whole argument is. 

In my reading, the OP updated toward the position  "it’s plausible that effective altruist community-building activities could be net-negative in impact, and I wanted to explore some conjectures about what that plausibility would entail" based on FTX causing large economic damage.  One of the conjectures based on this is "Implement Carla Zoe Cremer’s Recommendations".

I'm mostly arguing against the position that 'the update of probability mass on EA community building being negative due to FTX evi... (read more)

Unfortunately not in detail  - it's a lot of work to go through the whole list and comment on every proposal. My claim is not 'every item on the list is wrong', but 'the list is wrong on average' so commenting on three items does not solve possible disagreement. 

To discuss something object-level, let's look at the first one

'Whistleblower protection schemes' sound like a good proposal on paper, but the devil is in detail:

1. Actually, at least in the EU and UK, whistleblowers pointing out things like fraud or illegal stuff are protected by the law.... (read more)

Just wanted to flag that I personally believe
- most of Cremer's proposed institutional reforms are either bad or zero impact, this was the case when proposed, and is still true after updates from FTX
- it seems clear proposed reforms would not have prevented or influenced the FTX fiasco 
- I think part of Cremer's reaction after FTX is not epistemically virtuous; "I was a vocal critic of EA" - "there is an EA-related scandal" - "I claim to be vindicated in my criticism" is not sound reasoning, when the criticisms are mostly tangentially related to the s... (read more)

Sorry for critical feedback, but

• Not sure if this is best run as a non-profit on a grant. I think people who want to buy this should preferably pay for it in price of the merch.
• Also, the products are somewhat expensive, compared to manufacturing costs. This is because it's actually for profit, just the profits are with Printful
• The design of the website is quite bad, in my opinion. Have you considered hiring a professional?
• Also the designs of majority products which try to do something more complicated than just print the logo on something is ...mediocre at

Could be time as well: EAGx Prague had dedicated time where it was not possible to book 1:1s on SwapCard, and I think it worked well.

For example, the OP could have contacted the person he talked to at the EAGx and asked her whether his interpretation of what she said were correct. If you read OP's other post about a conflict resulting from asking for feedback about the grant application, you have one datapoint where someone was talking with him about the unsuccessful grant application, and was surprised by OPs interpretation. 

As I mentioned in the comment, in case of this post, unless the person he talked to decides to reveal herself (and she may have good reasons not to do that), ... (read more)

I don't know what the actual grantmakers think, but if I was deciding about the funding

- you can get funding to do EA localization basically in any country, if you come up with a reasonable strategy & demonstrate competence and understanding of EA
- difficulty of coming up with a reasonable strategy in my view varies between places; e.g., if you wanted to create, for the sake of discussion, EA Norway, a reasonable strategy may be 'just' supporting people in attempts to do impactful work, supporting uni groups, routing donations and maybe engaging with N... (read more)

While I don't want to go into discussing specific details, I want to make clear I don't think 'not writing something egregious in the forum' is the standard which should be applied here.

In my view, national-level groups are potentially important and good but need to be able to do a lot of complex work to be really successful. This means my bar for 'who should do this as a funded  full-time job' is roughly similar to 'who should work at median-impact roles at CEA' or 'work as a generalist at Rethink Priorities' or similar.

I don't think the original pos... (read more)

Downvoted. Let me explain why:

1. I'm not really convinced by your post that what actually happened with your grant applications was at all caused by you applying from Romania. (Personally, if I was a grantmaker, based on your EA Forum presence, I would reject your application to create/grow a new national-level group if you applied from basically anywhere. You can read more of my thinking about the topic of national EA groups in this post) 

2. I really dislike the type of discourse exemplified by this paragraph: "I said that I felt Romania was being di... (read more)

I do agree net positive is too soft, but I don't think this is what anyone is seriously advocating for here.

The main implicit theory of impact for event venues is 
venues -> events -> {ideas / intellectual progress / new people engaging with the ideas / sharing of ideas}

I think in domains like "thinking about future with powerful AI" or "how to improve humanity's epistemics" or "what are some implications of digital minds existing" it seems the case that noticeable amount of thinking and discussion is happening at various in-person gatherings.&nb... (read more)