All of Rory Greig's Comments + Replies

Thanks for the great list of resources!

Coincidentally I just discovered the Jim Rutt Show podcast recently and I've been enjoying it.

2
annaleptikon
10mo
Glad you enjoy it!

I just wrote a relevant forum post on how simulation models / Agent-based models could be highly impactful for pandemic preparedness: https://forum.effectivealtruism.org/posts/2hTDF62hfHAPpJDvk/simulation-models-could-help-prepare-for-the-next-pandemic

A crucial aspect of this is better software tools for building large scale simulations, so I would say this is a large opportunity for someone who wants to work in software engineering. 

Even just working as a research engineer in an existing academic group building epidemiological models would be impactf... (read more)

Thanks for writing this, in my opinion the field of complex systems provides a useful and under-explored perspective and set of tools for AI safety. I particularly like the insights you provide in the "Complex Systems for AI Safety" section, for example that ideas in complex systems foreshadowed inner alignment / mesa-optimisation.  

I'd be interested in your thoughts on modelling AGI governance as a complex system, for example race dynamics.

I previously wrote a forum post on how complex systems and simulation could be a useful tool in EA for improving... (read more)

I can think of a few other areas of direct impact which could particularly benefit from talented software engineers:

Improving climate models is a potential route for high impact on climate change, there are computational modelling initiatives such as the Climate Modeling Alliance and startups such as Cervest. It would also be valuable to contribute to open source computational tools such as the Julia programming language and certain Python libraries etc.

There is also the area of computer simulations for organisational / government decision making, such as ... (read more)

Is there a list of the ideas that the fellows were working on? I'd be curious. 

It's not surprising to me that there aren't many "product focused" traditional startup style ideas in the longtermist space, but what does that leave? Are most of the potential organisations research focused? Or are there some other classes of organisation that could be founded? (Maybe this is a lack of imagination on my part!)

4
Clifford
2y
Hi Rory, thanks for the comment! We haven’t published those ideas. In terms of classes of organisation, one way to carve up the space is to think about Object-level and Meta-level approaches to generating ideas. Object-level approaches focus on doing direct work to solve the problem at hand. For example: * developing and deploying technologies * conducting research * advocating for policy change The main type of impact here comes in the form of tangible changes in actions taken in the real world, in whatever form that might take. Meta-level approaches focus on improving the capacity for others to solve the problem. This can be done on the EA/longtermist wide-level (building up the movements) or in a specific domain, e.g. building a talent pipeline specifically for bio policy experts. Concrete types of meta work include, for example: * community and field building * the dissemination of ideas and knowledge and values * increasing the resources available to work on object-level approaches The main type of impact here comes in the form of the change in likelihood that object-level approaches will be impactful. Hope that's useful!

Very useful to know, thanks for the context!

Congratulations, this is really great to hear, and seems like a fantastic opportunity!

Out of interest, what was the sequence of events, did you already have a PhD program lined up when you applied for funding? Or are you going to apply for one now that you have the funding? Also had you already discussed this with your current employer before applying for funding?

I only ask because I have been considering attempting to do something similar!

4
Aryeh Englander
3y
1. I did not have an advisor when I sent the original email, but I did have what amounted to a standing offer from my undergrad ML professor that if I ever wanted to do a PhD he would take me as a grad student. I spent a good amount of time over the past three months deciding whether I should take him up on that or if I should apply elsewhere. I ended up taking him up on the offer. 2. I did not discuss it with my employer before sending the original email. It did take some work to get it through bureaucratic red tape though (conflict of interest check, etc.).

This is a good point, although I suppose you could still think of this in the framing of "just in time learning", i.e. you can attempt a deep RL project, realise you are hopelessly out of your depth, then you know you'd better go through Spinning Up in Deep RL before you can continue. 

Although the risk is that it may be demoralising to start something which is too far outside of your comfort zone.

 you can attempt a deep RL project, realise you are hopelessly out of your depth, then you know you'd better go through Spinning Up in Deep RL before you can continue. 

Tbc, I do generally like the idea of just in time learning. But:

  • You may not realize when you are hopelessly out of your depth ("doesn't everyone say that ML is an art where you just turn knobs until things work?" or "how was I supposed to know that the algorithm was going to silently clip my rewards, making all of my reward shaping useless?")
  • You may not know what you don't know. In
... (read more)

I massively agree with the idea of "just do a project", particularly since it's a better way of getting practice of the type of  research skills (like prioritisation and project management) that you will need to be a successful researcher.

I suppose the challenge may be choosing a topic for your project, but reaching out to others in the community may be one good avenue for harvesting project ideas. 

What are your thoughts on re-implementing existing papers? It can be a good way to develop technical skills, and maybe a middle ground between learning pre-requisites and doing your own research project? Or would you say it's better to just go for your own project?

5
JJ Hepburn
3y
The best thing to do is the thing that works for YOU. Yes, reimplementing existing papers is great. Talking to others in the community for ideas is great if you can.  I don't think there is a right way for everyone. So if you are already making learning a lot through re-implementing or something else then just ignore most of my advice. Also, if my advice isn't helpful for you then try one of the other ideas. 

These links are excellent! I hadn't come across these before, but I am really excited about the idea of using roleplay and table top games as a way of generating insight and getting people to think through problems. It's great to see this being applied to AI scenarios.

@djbinder Thanks for taking the time to write these comments. No need to worry about being negative, this is exactly the sort of healthy debate that I want to see around this subject.

I think you make a lot of fair points, and it’s great to have these insights from someone with a background in theoretical physics, however I would still disagree slightly on some of them, I will try to explain myself below.

I don’t think the only meaningful definition of complex systems is that they aren’t amenable to mathematical analysis, that is perhaps a feature of them, b... (read more)

6
djbinder
3y
Thanks for the reply Rory! I think at this point it is fairly clear where we agree (quantitative methods and ideas from maths and physics can be helpful in other disciplines) and where we disagree (whether complexity science has new insights to offer, and whether there is a need for an interdisciplinary field doing this work separate from the ones that already exist), and don't have any more to offer here past my previous comments. And I appreciate your candidness noting that most complexity scientists don't mention complexity or emergence much in their published research; as is probably clear I think this suggests that, despite their rhetoric, they haven't managed to make these concepts useful. I do not think the SFI, at least judging from their website, and from the book Scale which I read a few years ago, is a good model of public relations that EAs should try to emulate. They make grand claims about what they have achieved which seems to me to be out of proportion to their actual accomplishments. I'm curious to hear what you think the great success stories of SFI are. The one I know the most about, the scaling laws, I'm pretty skeptical of for the reasons outlined previously. I had a look at their "Evolution of Human Languages" program, and it seems to be  fringe research by the standards of mainstream comparative linguistics. But there could well be success stories that I am unfamiliar with,  particularly in economics.

I would add the New England Complex Systems Institute, particularly Yaneer Bar Yam: https://necsi.edu/corona-virus-pandemic 

In this article from January 2020, which has aged very well, they were advocating for restrictions on international movement and warning of the effect of superspreader events on estimates of R0. 

Yaneer Bar Yam also started this multidisciplinary effort to tackle covid: https://www.endcoronavirus.org/ 

Hey Alex, thanks for writing this, loads of useful advice in here that I want to try!

I have had similar (but seemingly milder than yours) problems with low energy, where I just felt very lethargic and drained periodically (about once a month). I would compare it to how you feel in the first day of getting a flu or cold, with low energy and mild muscle aches. I went to the doctor and had a similar story to you, they ran some blood tests and found nothing wrong, and that was it.

The answer: it was almost definitely stress. I was in a management position at wo... (read more)

5
Kirsten
3y
Yes, I've also gotten really sick from stress in the past! Both times, my doctors asked if I was under a lot of stress, and both times I said no, even though in retrospect it seems obvious that I was.

Hey Venkatesh, I am also really interested in Complexity Science, in fact I am going to publish a blog post on here soon about Complexity Science and how it relates to EA.

I've also read Bookstaber's book, in fact Doyne Farmer has a similar book coming out soon which looks great, you can read the intro here.

I hadn't heard of the Complexity Weekend event but it looks great, will check that out!

This is an interesting thought experiment and I like the specific framing of the question.

My initial thoughts are that this clearly would have been a good thing to try to work on, mainly due to the fact that the 2008 financial crisis cost trillions of dollars and arguably also led to lots of bad political developments in the western world (eg. setting the stage for Trumpism). If you buy the Tyler Cowen arguemnts that economic growth is very important for the long term then that bolsters this case. However a caveat would be that due to moral uncertainty it'... (read more)

2
Venkatesh
3y
As someone interested in Complexity Science I find the ABM point very appealing. For those of you with a further interest in this, I would highly recommend this paper by Richard Bookstaber as a place to start. He also wrote a book on this topic and was also one of the people to foreshadow the crisis. Also if you are interested in Complexity Science but never got a chance to interact with people from the field/learn more about it, I would recommend signing up for this event.

Hey JP, thanks for your question, here are some questions that may be useful in your search, and may help people other provide you more advice: 


1. Do you have any criteria for what you consider a "job within EA"? There are many types of job which could be considered EA related, from jobs within EA organisations, to jobs which have a large potential impact but are not directly for EA orgs (for example working in certain government departments or private companies). It might be worth reframing how you think about this as "how can I find a job that has t... (read more)

All this advice seems realy good, and I want to particularly echo this bit:

It might be worth reframing how you think about this as "how can I find a job that has the biggest impact", rather than "how can I get an EA job".

Regarding AI alignment and existential risk in general, Cummings already has a blog post where he mentions these: https://dominiccummings.com/2019/03/01/on-the-referendum-31-project-maven-procurement-lollapalooza-results-nuclear-agi-safety/

So he is clearly aware and responsive to the these ideas, it would be great to have an EA minded person on his new team to emphasise these.


0
Arran McCutcheon
4y
Exactly, he has written posts about those topics, and about 'effective action', predictions and so on. And there is this article from 2016 which claims 'he is an advocate of effective altruism', although it then says 'his argument is mothball the department (DFID)', which I'm fairly sure most EAs would disagree with. But as he's also written about a huge number of other things, day-to-day distractions are apparently the rule rather than the exception in policy roles, and value drift is always possible, it would be good to have someone on his team, or with good communication channels to them, who can re-emphasise these issues (without publicly associating EA with Cummings or any other political figure or party).