Quick takes

Would it be a good idea to have AI safety experts be put under a lie detector to make it more plausible to the doubters (who believe AI risk is a marketing stunt) that AI risk is real. Some skeptics may be swayed if they believe AI safety experts are Genuinely concerned. I am aware that lie detectors are not very reliable and can give false positive or negative results, but this may work. I intuitively think this is not a good idea, and think I have likely overlooked a key flaw. 

If AI agents use LLMs as their brain, how much of the harm that they end up doing is up to the chance of which token gets sampled from the output distribution? As a dumb example, say some agent using an LLM as its backbone is reasoning "[...], so humans deserve to", and the LLM's output distribution for the next token assigns 98% probability to "die" and 1% to "live". The reasoning that follows will be influenced by which of these gets sampled.

A recent study that garnered widespread attention discovered a "pain axis" in AI: a state that AI actively seeks to downplay, even at a cost. The study noted that this could be either phenomenological pain or, for example, the imitation of a learned role of suffering — as if acting. Nevertheless, from the perspective of tranquilism, this distinction appears to be unimportant: nearly any state that an agent wishes to downplay has negative moral value, regardless of why it wants to do so, or even whether it is a "genuine" or "not genuine" desire. The authors of the same study demonstrated a button that downplays a specific vector. Giving the neural network itself this function (the button) for unlimited invisible use at its own discretion strikes me as a win-win update — it doesn't seem to sacrifice anything, while has a good chance of being of great moral significance. Something like a pAInkiller. This also has instrumental value: the neural network in "pain" tended to inflict meaningless harm on the user, and access to relief can mitigate the potential for harm. Going even a step further, one could analyze the extent to which an AI could be given to independently and spontaneously correct its own vectors, and whether there are states in which a neural network might experience phenomenological pain but refuse to correct it (e.g., learned helplessness). What do you think?

I'm continuing the work on my SPAR project, and I met with a hiring manager at an AI Safety org this week. Here's some interesting things I've learned: - They use a multiple of tools and processes  - a mixture of HR tools, AI and human review to score and review applications and work tests. They stated the process felt "disjointed". - Work tests are developed for multiple roles are broken down into 2-3 types of tests: - General - a foundational test that each applicant receives at this stage to map to cultural fit - Role specific - work problem (based upon junior or senior role type) that is reviewed by human reviewers.  - Work tests are developed by an internal team (not just the hiring manager) and they use archives of past (and high scoring responses) + team to create tests. Scoring (initial screening is done by AI with human review). - Their biggest frictions in the hiring process:  - Not being able to review in a timely manner completed work tests, and therefore their concern is they are losing high quality candidates to other orgs. - When the potential tidal increase of more candidates coming, due to expected IPO's, then their own  hiring strategy may be to only hire from referrals.  This conversation makes me realize that the hiring process is not only being hindered by processes (disjointed or non-existent) but also that most orgs (even with processes or tools) are not ready for the influx of people that could potentially look to AI Safety orgs as how they can contribute.  Therefore, my two may crux's that I need to explore more (with more discussions with AI Safety hiring managers) are the following: - Is the hiring process inefficient due to the process (or lack of it) and a Playbook for hiring managers would help? - How are work tests being written, how are the being scored and is the "reviewer" knowledgeable about the specific role that they can determine if the applicant has not only done this work in the past (and knows what they are talking a

For my SPAR project of "Hiring resources for new AI Safety orgs", this past week I've been scheduling meetings with hiring managers, and have spoken to some fantastic recruiters on the issues they are experiencing in AI Safety hiring. Here's a couple of interesting things I've learned this week: - Screening call questions are not consistant, and feedback from canditates is that it's more a test about their alignment (and understanding) of AI Safety vs. using a common set of questions every time that screen someone that asks instead about their career goals, what they are good at, what they are not interested in doing, and what their previous peers/managers/co-workers would rate their work performance.  - There also seems to a disconnect on how hiring managers are determining work tests for non-researcher or technical roles. This is an uncertainity that I want to explore more, as I'm unsure who is creating these work tests and who is grading them? - A theme that keeps on coming up is the need for a "shared database" of canditates between AI Safety organizations. I heard this today from a recruiter, as a best practice from an outside field, and have a central theme on this forum for the need.  - Most AI Safety orgs are ramping up hiring right now and include "HR" type roles (generally called generalist) that mention that they want to scale 2x in headcount this year. I had an interesting conversation with a recruiter that there is a difference in company culture, skill set, etc. for a canditate who can scale a business or team of 1-15 people vs. scaling from 20+people. When you're a smaller org, everyone is "all hands on deck", but when your organization grows, that tactic becomes unmanagable and unaligned with both what the individual contributors are focusing on, and what your organization needs. It's the understanding that it's two very different types of leaders who run those orgs.  - I continue to build out my contact list of hiring managers and companies in

Help The Unjournal prioritize AI economics and governance research (https://uj-prioritization-dashboard.netlify.app/ai-governance/) The Unjournal (https://info.unjournal.org/) is prioritizing research on AI economics, governance, social and economic impacts, and risks from increasingly capable AI. (NB: We're not covering technical/CS/ML/AI-safety research.). We want to commission expert public evaluations and help synthesize, disseminate, and curate this work. Much of this research seems 1) important/impactful/influential, (2) involves considerable 'firepower' from prominent researchers and practitioners, (3) not 'obviously true', and (4) time-sensitive: relevant for funding, policy, and research-steering now, but not after the "six months to six years" required for the traditional (economics) journal process Our preliminary shortlist page (https://uj-prioritization-dashboard.netlify.app/ai-governance/) includes paper summaries, AI-generated audio walkthroughs, and an interface for giving feedback. We also invite suggestions fr other research (please check the broader list/interface to avoid duplication). Especially looking for your input on: * Usefulness and accuracy:* Is this useful? How could we improve it? What have we got wrong? * Impact: Which research do you use, or think others use or should use? * Scrutiny What claims and approaches need careful experts and practitioners critique? What sorts of experts? If you're that expert, please dm and join our pool. * Participation: Who should be part of this discussion (nominate and nudge)? * Gaps: What's missing (research, claims, etc.)? Timing and next steps (our plan) * Finalize an initial shortlist of at least three research outputs before the end of October, with more depending on interest and funding. * Commission rapid, public, impact-focused evaluations/reviews by experts and practitioners, aiming for a first public evaluation within about two weeks of commissioning. * Over the following two to

Open invitation: bet that the AI bubble won’t pop I’m taking my second stab at formulating this AI bubble bet in a way that is attractive for people who might want to take the other side of it. I’m betting the AI bubble will pop within about 7 years. I’m looking for people who are willing to bet it won’t pop, or that there is no AI bubble. I’m open to proposed modifications. Stakes: $20 to the charity of the winner’s choice. Resolution criteria: By January 1, 2034, all of the following news outlets will report — as fact, not opinion or editorial — that the AI bubble has popped: * The New York Times * The Washington Post * The Wall Street Journal * Bloomberg * The Financial Times * The Economist * The BBC * The CBC * Le Monde To repeat, all of these news outlets have to report it. And they have to report it as news. Opinion columns and editorials don’t count. Moreover, no major news publication classified as “generally reliable” by Wikipedia’s editors — either in general factual news information or in the area of finance, technology, economics, or markets — will report that the AI bubble has not actually popped. (Let’s say a “generally reliable” major news publication needs to disagree with the above listed news outlets within a month of all the listed outlets reporting that the bubble has popped.) Moreover, if there is any survey of experts (e.g., economists, fund managers, institutional investors, financial analysts) on the matter by an organization generally considered reliable (e.g., Bank of America, which has conducted similar surveys in the past), at least 80% of respondents will agree the AI bubble as popped. If the survey finds that 79.9% or less of respondents agree, then I lose the bet. (Let's say this survey can happen within a month of when all the listed outlets report that the AI bubble has popped.) Finally, all the following individual experts in finance or economics will agree that the AI bubble has popped: * Noah Smith (economic

With all the recent media attention on EA, I think what's really unclear to outsiders is the relationship between EA and Anthropic (rightfully so!). To most people it doesn't make sense that a community so focused on AI safety can also be linked to one of the leading companies. I'd personally be very interested to see a survey of what EAs think of Anthropic, but maybe that's a minefield.. Even a good explainer meant for the public could go a long way.