All of jacquesthibs's Comments + Replies

I shared the following as a bio for EAG Bay Area 2024. I'm sharing this here if it reaches someone who wants to chat or collaborate.

Hey! I'm Jacques. I'm an independent technical alignment researcher with a background in physics and experience in government (social innovation, strategic foresight, mental health and energy regulation). Link to Swapcard profile. Twitter/X.

CURRENT WORK

• Collaborating with Quintin Pope on our Supervising AIs Improving AIs agenda (making automated AI science safe and controllable). The current project involves a new method allowi

Another data point: I got my start in alignment through the AISC. I had just left my job, so I spent 4 months skilling up and working hard on my AISC project. I started hanging out on EleutherAI because my mentors spent a lot of time there. This led me to do AGISF in parallel.

After those 4 months, I attended MATS 2.0 and 2.1. I've been doing independent research for ~1 year and have about 8.5 more months of funding left.

More information about the alleged manipulative behaviour of Sam Altman

Source

Update, board members seem to be holding their ground more than expected in this tight situation:

My current speculation as to what is happening at OpenAI

How do we know this wasn't their best opportunity to strike if Sam was indeed not being totally honest with the board?

Let's say the rumours are true, that Sam is building out external orgs (NVIDIA competitor and iPhone-like competitor) to escape the power of the board and potentially go against the charter. Would this 'conflict of interest' be enough? If you take that story forward, it sounds more and more like he was setting up AGI to be run by external companies, using OpenAI as a fundraising bargai... (read more)

Quillette founder seems to be planning to write an article regarding EA's impact on on tech:

"If anyone with insider knowledge wants to write about the impact of Effective Altruism in the technology industry please get in touch with me claire@quillette.com. We pay our writers and can protect authors' anonymity if desired."

It would probably be impactful if someone in the know provided a counterbalance to whoever will undoubtedly email her to disparage EA with half-truths/lies.

To share another perspective: As an independent alignment researcher, I also feel really conflicted. I could be making several multiples of my salary if my focus was to get a role on an alignment team at an AGI lab. My other option would be building startups trying to hit it big and providing more funding to what I think is needed.

Like, I could say, "well, I'm already working directly on something and taking a big pay-cut so I shouldn't need to donate close to 10%", but something about that doesn't feel right... But then to counter-balance that, I'm constantly worried that I just won't get funding anymore at some point and would be in need of money to pay for expenses during a transition.

I've also started working on a repo in order to make Community Notes more efficient by using LLMs.

Don't forget that we train language models on the internet! The more truthful your dataset is, the more truthful the models will be! Let's revamp the internet for truthfulness, and we'll subsequently improve truthfulness in our AI systems!!

I shared a tweet about it here: https://x.com/JacquesThibs/status/1724492016254341208?s=20

Consider liking and retweeting it if you think this is impactful. I'd like it to get into the hands of the right people.

If you work at a social media website or YouTube (or know anyone who does), please read the text below:

Community Notes is one of the best features to come out on social media apps in a long time. The code is even open source. Why haven't other social media websites picked it up yet? If they care about truth, this would be a considerable step forward beyond. Notes like “this video is funded by x nation” or “this video talks about health info; go here to learn more” messages are simply not good enough.

If you work at companies like YouTube or know someone who... (read more)

Attempt to explain why I think AI systems are not the same thing as a library card when it comes to bio-risk.

To focus on less of an extreme example, I’ll be ignoring the case where AI can create new, more powerful pathogens faster than we can create defences, though I think this is an important case (some people just don’t find it plausible because it relies on the assumption that AIs being able to create new knowledge).

I think AI Safety people should make more of an effort to walkthrough the threat model so I’ll give an initial quick first try:

1) Library.... (read more)

I'm working on an ultimate doc on productivity I plan to share and make it easy, specifically for alignment researchers.

Let me know if you have any comments or suggestions as I work on it.

Roam Research link for easier time reading.

Google Docs link in case you want to leave comments there.

From what I understand, Amazon does not get a board seat for this investment. Figured that should be highlighted. Seems like Amazon just gets to use Anthropic’s models and maybe make back their investment later on. Am I understanding this correctly? 

As part of the investment, Amazon will take a minority stake in Anthropic. Our corporate governance structure remains unchanged, with the Long Term Benefit Trust continuing to guide Anthropic in accordance with our Responsible Scaling Policy. As outlined in this policy, we will conduct pre-deployment tests

I would, however, not downplay their talent density.

Fantastic news. Note: don’t forget to share it on LessWrong too.

Thanks for sharing. I think the above are examples of things people often don't think of when trying new ways to be more productive. Instead, the default is trying out new productivity tools and systems (which might also help!). Environment and being in a flux period can totally change your behaviour in the long term; sometimes, it's the only way to create lasting change.

Perfect, thanks!

Is someone planning on doing an overview post of all the AI Pause discussion? I’m guessing some people would appreciate it if someone took the time to make an unbiased synthesis of the posts and discussions.

Are you or any other EA lawyer still doing this?

Either way, I’m seeking advice to figure out how I can save money on taxes once I move to the UK (I’m from Canada) and receive funding for my independent AI Safety research. I’ll be going to the UK on a Youth Mobility visa. I’m wondering if it’s possible for me to setup something so that I can save tax on ‘business’ expenses (office space, laptop, monitor, etc.).

I’m happy to pay if someone can help with this (otherwise I will reach out to non-EA lawyers).

Would newer people find it valuable to have some kind of 80,000 hours career chatbot that had access to the career guide, podcast notes, EA forum posts, job postings, etc, and then answered career questions? I’m curious if it could be designed to be better than just a raw read of the career guide or at least a useful add-on to the career guide.

Potential features:

• It could collect your conversation and convert most of it into an application for a (human) 1-on-1 meeting.
• You could have a speech-to-text option to ramble all the things you’ve been thinking of.
• ???

If anyone from 80k is reading this, I’d be happy to build this as a paid project.

Love this idea, thanks for organizing this.

Thanks for all your work, JJ! Good luck with whatever you end up doing next!

(Note in case anybody else wants to pick up where AISS left off: This is a bit unfortunate for me given that not having an org to sponsor work visas in the UK might affect my decision for moving to there. We had talked about AISS trying to do the work to get that setup in the next 1-2 years.)

In this framework, I propose that Xnder ³amoXnt of mone\ to donate ́ Ze bXndle
considerations relating to taxes, weakness of will and uncertainty as well as financial
investment.

One thing I think is missing from the "how much you should donate" section above is a discussion about what kind of job the person is doing. Should the percentage be the same for someone doing Earning to Give vs someone working on a direct cause area?

Added mine. :)

I recently sent in some grant proposals to continue working on my independent alignment research. It gives an overview of what I'd like to work on for this next year (and more really). If you want to have a look at the full doc, send me a DM. If you'd like to help out through funding or contributing to the projects, please let me know.

Here's the summary introduction:

12-month salary for building a language model system for accelerating alignment research and upskilling (additional funding will be used to create an organization), and studying how to&nbs... (read more)

I gave talk about my Accelerating Alignment with LLMs agenda about 1 month ago (which is basically a decade in AI tools time). Part of the agenda covered (publicly) here.

I will maybe write an actual post about the agenda soon, but would love to have some people who are willing to look over it. If you are interested, send me a message. I am currently applying for grants and exploring the possibility of building an org focused on speeding up this agenda and avoid spreading myself too thin.

I agree with this post. I've been reading many more papers since first entering this field because I've been increasingly convinced of the value of treating alignment as an engineering problem and pulling insights from the literature. I've also been trying to do more thinking about how to update on the current paradigm from the classic Yud and Bostrom alignment arguments. In this respect, I applaud Quintin Pope for his work.

This week, I will send a grant proposal to continue my work in alignment. I'd be grateful if you could look at my proposal and provide... (read more)

So, things have blown up way more than I expected and things are chaotic. Still not sure what will happen or if a treaty is actually in the cards, but I’m beginning to see a path to tons of more investment in alignment potentially. One example why is that Jeff Bezos just followed Eliezer on Twitter and I think it may catch the attention of pretty powerful and rich people who want to see AI go well. We are so off-distribution, could go in any direction.

Thanks for reporting back! I’m sharing it with my friends as well (none of which are in tech and most of them live in fairly rural parts in Canada) to see their reaction.

In case we have very different feeds, here’s a set of tweets critical about the article:

• https://twitter.com/mattparlmer/status/1641230149663203330?s=61&t=ryK3X96D_TkGJtvu2rm0uw (lots of quote-tweets on this one)
• https://twitter.com/jachiam0/status/1641271197316055041?s=61&t=ryK3X96D_TkGJtvu2rm0uw 
• https://twitter.com/finbarrtimbers/status/1641266526014803968?s=61&t=ryK3X96D_TkGJtvu2rm0uw 
• https://twitter.com/plinz/status/1641256720864530432?s=61&t=ryK3X96D_TkGJtvu2rm0uw 
• https://twitter.com/perrymetzger/status/1641280544007675904

Here’s a comment I shared on my LessWrong shortform.

——

I’m still thinking this through, but I am deeply concerned about Eliezer’s new article for a combination of reasons:

• I don’t think it will work.
• Given that it won’t work, I expect we lose credibility and it now becomes much harder to work with people who were sympathetic to alignment, but still wanted to use AI to improve the world.
• I am not convinced as he is about doom and I am not as cynical about the main orgs as he is.

In the end, I expect this will just alienate people. And stuff like this concerns me... (read more)

Yeah, that’s where I asked first. No responses in the thread, but someone DMd me to suggest looking at universities: https://www.lesswrong.com/posts/kH2uJeQZMnEBKG9GZ/can-independent-researchers-get-a-sponsored-visa-for-the-us.

The pay difference between working in industry and doing a PhD was a big factor for me to avoid getting a PhD a few years ago.

These days it still plays a role, though as an independent researcher I’d like to connect with more academics so that I can get better at doing research with more rigour and publish more papers. Avoiding the PhD has made this hard and I kind of have to have a lot more initiative to develop these skills that PhD students typically develop. That said, being able to selectively learn the skills that are actually useful for solving alig... (read more)

When I say "true," I simply mean that it is inevitable that these things are possible by some future AI system, but people have so many different definitions of AGI they could be calling GPT-3 some form of weak AGI and, therefore incapable of doing the things I described. I don't particularly care about "true" or "fake" AGI definitions, but just want to point out that the things I described are inevitable, and we are really not so far away (already) from the scenario I described above, whether you call this future system AGI or pre-AGI.

Situational awarenes... (read more)

A "true" AGI will have situational awareness and knows its weights were created with the help of code, eventually knows its training setup (and how to improve it), and also knows how to rewrite its code. These models can already write code quite well; it's only a matter of time before you can ask a language model to create a variety of architectures and training runs based on what it thinks will lead to a better model (all before "true AGI" IMO). It just may take it a bit longer to understand what each of its individual weights do and will have to rely on ... (read more)

Since I expect some people to be a bit confused as to what exactly was the bad thing that has happened after reading this post, I think it would be great if the community health team could write a post explaining and pointing out exactly what was bad here and in other similar instances.

I think there is value in being crystal clear about what were the bad things that happened because I expect people will takeaway different things from this post.

I honestly didn’t know how to talk about it either, but wanted to point at general vibes I was getting. While I’m still confused about what‘s the issue exactly, contrary to my initial comment, I don’t really think polyamory within the community is a problem anymore. Not because of Arepo’s comment specifically, but because there are healthy ways to do polyamory just like other forms of relationships. It’s something that I thought was true before writing the comment, but was a bit confused about the whole mixing of career and “free love” with everyone in the... (read more)

People have some strong opinions about things like polyamory, but I figured I’d still voice my concern as someone who has been in EA since 2015, but has mostly only interacted with the community online (aside from 2 months in the Bay and 2 in London):

I have nothing against polyamory, but polyamory within the community gives me bad vibes. And the mixing of work and fun seems to go much further than I think it should. It feels like there’s an aspect of “free love” and I am a little concerned about doing cuddle puddles with career colleagues. I feel like all ... (read more)

Consider using Conjecture’s new Verbalize (https://lemm.ai/home) STT tool for transcriptions! They’ll be adding some LLM features on top of it, and I expect it to have some cool features in coming out this year.

I’ve been also pushing (for a while) for more people within EA to start thinking of ways to apply LLMs to our work. After ChatGPT, some people started saying similar stuff so I’m glad people are starting to see the opportunity.

Are there any actual rigorous calculations on this? It's hard for me to believe someone making $2M/year and donating $1M/year (to AI Safety or top GW charities) would have less counterfactual impact than someone working at CEA.

Edit: Let's say you are donating $1M/year to AI Safety, that might be about enough to cover the salary for about 9 independent alignment researchers. Though, those 9 researchers might not be yet comparable to top-level researchers who would get funding regardless. So, it would probably end up as additional funding for getting more yo... (read more)

Thanks for doing this!

In terms of feedback: the most annoying thing so far is that as soon as you click on any grant, going 'back' to the previous page puts you back at a completely fresh search. You can't click to open up new tabs either.

I want to say that I appreciate posts like this by parents in the community. I'm an alignment researcher and given how fast things are moving, I do worry that I'm under-weighting the amount of impact I could lose in the next 10 years if I have kids. I feel like 'short timelines' make my decision harder even though I'm convinced I want kids in 5 or so years from now.

Some considerations I've been having lately:

• Should I move far away from my parents, which would make it harder to depend on someone for childcare on the weekends and evenings? Will we be close t

Yeah, I agree. Though I feel I can imagine a lot of startups or businesses that require a lot of work, but don't require as much brain power. I could have chosen a sector that doesn't move as fast as AI (which has an endless stream of papers and posts to keep track of) and just requires me to build a useful product/service for people. Being in a pre-paradigmatic field where things feel like they are moving insanely fast can feel overwhelming.

I don't know, being in a field that I think feels much more high-risk and forces me to grapple with difficult concep... (read more)

As an example: I specifically chose to start working on AI alignment rather than trying to build startups to try to fund EA because of SBF.  I would probably be making a lot more money had I took a different route and would likely not have to deal with being in such a shaky, intense field where I’ve had to put parts of my life on hold for.

I mostly agree with Buck’s comment and I think we should probably dedicate more time to this at EAGs than we have in the past (and probably some other events). I’m not sure what is the best format, but I think having conversations about it would allow us to feel it much more in our bones rather than just discussing it on the EA forum for a few days and mostly forget about it.

I believe that most of these in-person discussions seem to have mostly happened at the EA Leaders Forum, so we should probably change that.

That said, I’m concerned that a lot of people... (read more)

Here's a relevant tweet thread.

One response from Ollie:

I feel like we need stronger communication coming from event organizers regarding these things. Even though it doesn't affect me personally.