Relatedly, what is the likelihood that future iterations of the fellowship might be less US-centric, or include Visa sponsorship?
The job posting states:
"All participants must be eligible to work in the United States and willing to live in Washington, DC, for the duration of their fellowship. We are not able to sponsor US employment visas for participants; US permanent residents (green card holders) are eligible to apply, but fellows who are not US citizens may be ineligible for placements that require a security clearance."
So my impression would be that it would be pretty difficult to participate for non-US citizens who do not already live in the US.
https://en.wikipedia.org/wiki/Technological_transitions might be relevant.
The Geels book cited in the article (Geels, F.W., 2005. Technological transitions and system innovations. Cheltenham: Edward Elgar Publishing.) has a bunch of interesting case studies I read a while ago and a (I think popular) framework for technological change, but I am not sure the framework is sufficiently precise to be very predictive (and thus empirically validatable).
I don't have any particular sources on this, but the economic literature on the effects of regulation might be quite relevant. In particular, I do remember attending a lecture arguing that limited liability played an important role for innovation during the industrial revolution.
Facebook has at least experimented with using deep reinforcement learning to adjust its notifications according to https://arxiv.org/pdf/1811.00260.pdf . Depending on which exact features they used for the state space (i.e. if they are causally connected to preferences), the trained agent would at least theoretically have an incentive to change user's preferences.
The fact that they use DQN rather than a bandit algorithm seems to suggest that what they are doing involves at least some short term planning, but the paper does not seem to analyze the experiments in much detail, so it is unclear whether they could have used a myopic bandit algorithm instead. Either way, seeing this made me update quite a bit towards being more concerned about the effect of recommender systems on preferences.
Depending on your intended audience, it might make sense to add more details for some of the proposals. For example, why is scenario planning a good idea compared to other methods of decision making? Is there a compelling story, or strong empirical evidence for its efficacy?
Some small nitpicks:
There seems to be a mistake here:
"Bostrom argues in The Fragile World Hypothesis that continuous technological development will increase systemic fragility, which can be a source of catastrophic or existential risk. In the Precipice, he estimates the chances of existential catastrophe within the next 100 years at one in six."
I also find this passage a bit odd:
"One example of moral cluelessness is the repugnant conclusion, which assumes that by adding more people to the world, and proportionally staying above a given average in happiness, one can reach a state of minimal happiness for an infinitely large population."
The repugnant conclusion might motivate someone to think about cluelessness, but it does not really seem to be an example of cluelessness (the question whether we should accept it might or might not be).
Most of the links to the papers seem to be broken.
So for the maximin we are minimizing over all joint distributions that are κ-close to our initial guess?
"One intuitive way to think about this might be considering circles of radius κ>0 centered around fixed points, representing your first guesses for your options, in the plane. As κ becomes very large, the intersection of the interiors of these circles will approach 100% of their interiors. The distance between the centres becomes small relative to their radii. Basically, you can't tell the options apart anymore for huge κ. (I might edit this post with a picture...)"
If I can't tell the options apart any more, how is the 1/n strategy better than just investing everything into a random option? Is it just about variance reduction? Or is the distance metric designed such that shifting the distributions into "bad territories" for more than one of the options requires more movement?
I wrote up my understanding of Popper's argument on the impossibility of predicting one's own knowledge (Chapter 22 of The Open Universe) that came up in one of the comment threads. I am still a bit confused about it and would appreciate people pointing out my misunderstandings.Consider a predictor:
A1: Given a sufficiently explicit prediction task, the predictor predicts correctly
A2: Given any such prediction task, the predictor takes time to predict and issue its reply (the task is only completed once the reply is issued).
T1: A1,A2=> Given a self-prediction task, the predictor can only produce a reply after (or at the same time as) the predicted event
T2: A1,A2=> The predictor cannot predict future growth in its own knowledge
A3: The predictor takes longer to produce a reply, the longer the reply is
A4: All replies consist of a description of a physical system and use the same (standard) language.
A1 establishes implicit knowledge of the predictor about the task. A2, A3 and A4 are there to account for the fact that the machine needs to make its prediction explicit.
A5: Now, consider two identical predictors, Tell and Told. At t=0 give Tell the task to predict Told's state (including it's physically issued reply) at t=1 from Told's state at t=0. Give Told the task to predict a third predictor's state (this seems to later be interpreted as Tell's state) at t = 1 from that predictor's state at t=0 (such that Tell and Told will be in the exact same state at t=0).
T3: If Told has not started issuing its reply at t=1, Tell won't have completed its task at t=1
T4: If Told has completed its task at t=1, Tell will complete its task at t=1.
T5: Tell cannot predict its own future growth in knowledge
A6: The description of the physical state of another description (that is for example written on a punch card) cannot be shorter than said other description.
T6: If Told has completed its task at t=1, Tell must have taken longer to complete its task
T6 contradicts T4, so some of the assumptions must be wrong.
This seems to establish too little, as it is about deterministic predictions. Also, the argument does not seem to preclude partial predictions about certain aspects of the world's state (for example, predictions that are not concerned with the other predictor's physical output might go through). Less relevantly, the argument heavily relies on (pseudo) self-references and Popper distinguishes between explicit and implicit knowledge and only explicit knowledge seems to be affected by the argument. It is not clear to me that making an explicit prediction about the future necessarily requires me to make all of the knowledge gains I have until then explicit (If we are talking about determinstic predictions of the whole world's state, I might have to, though, especially if I predict state-by-state ).
Then, if all of my criticism was invalid and the argument was true, I don't see how we could predict anything in the future at all (like the sun's existence or the coin flips that were discussed in other comments). Where is the qualitative difference between short- and long-term predictions? (I agree that there is a quantitative one, and it seems quite plausible that some lontermists are undervaluing that.)
I am also slightly discounting the proof, as it uses a lot of words that can be interpreted in different ways. It seems like it is often easier to overlook problems and implicit assumptions in that kind of proof as opposed to a more formal/symbolic proof.
Popper's ideas seem to have interesting overlap with MIRI's work.
They are, but I don't think that the correlation is strong enough to invalidate my statement. P(sun will exist|AI risk is a big deal) seems quite large to me.
Obviously, this is not operationalized very well...
It seems like the proof critically hinges on assertion 2) which is not proven in your link. Can you point me to the pages of the book that contain the proof?
I agree that proofs are logical, but since we're talking about probabilistic predictions, I'd be very skeptical of the relevance of a proof that does not involve mathematical reasoning,