Hi Leonie - great post! It's really valuable to see people give an honest account of their learning journey in a public forum like this one. 

I am a data scientist and work with machine learning in some capacity for my job, so have plenty of more mathematical textbooks I could recommend, but I won't do that. My background is actually philosophy, so I have had a journey moving from an essay-writing undergraduate student to graduate data scientist, and I know what it's like to not feel like you know anything about this stuff. 

With that said, here are three books I would recommend to a non-technical person wanting to learn more about AI, for AI governance or otherwise. These are not AI safety books, or AI policy books, but are merely introductory books for someone with close to zero starting knowledge about AI.
 

1. The Hundred-Page Machine Learning Book by Andriy Burkov. This one is commonly suggested as a quick overview of machine learning and tries to go deep without going too technical. It has glowing recommendations from many experienced people in the field, such as Peter Norvig. https://www.amazon.de/Hundred-Page-Machine-Learning-Book/dp/199957950X/
2. Artificial Intelligence: A Guide for Thinking Humans by Melanie Mitchell. This is in the Pelican series, which is a series of short explanatory books on a single topic for a general reader (another one is the book The Art of Statistics by David Spiegelhalter, also recommended for learning the intuition behind statistics). I have heard good things about this book and Melanie Mitchell has a well-established reputation in the field of AI research. https://www.amazon.de/Artificial-Intelligence-Thinking-Humans-Pelican/dp/0241404835/
3. The Quest for Artificial Intelligence: A History of Ideas and Achievements by Nils J. Nilsson. This book is unique. The author has lived through some of the most important developments in the history of artificial intelligence and has often directly worked with many of the key characters in the story. For someone with a more arts or humanities background, getting to know a technical field by its history is sometimes a really good tactic. I read this a few years ago and it really gave me a high-level sense of where the field has come from and where the field might be going. This one is highly recommended. https://www.amazon.de/Quest-Artificial-Intelligence-Nils-Nilsson/dp/0521122937/ 
   
    

I hope your learning journey goes well and that you continue to write down things that you learn (as you have already done so with this post), as I'm sure it will be really useful to others in a similar position. 

Best of luck!

----

EDIT: I was reflecting on what you wrote in the post regarding math and felt like I should say something else just about learning mathematics. Math can definitely feel like something that people are either good at or not, with no in-between. I am sure that you've already done some thinking about this yourself(!) It's a personal choice for everyone but I think trying to learn the basics of geometry, algebra, calculus, etc, is so important for everyone that you should at least give it multiple attempts, kind of like learning to drive a car! Here are some resources that I have found useful to motivate myself:

https://brilliant.org/courses/ - Brilliant is a math and science education website with a free and paid-for tier. I have the paid tier at the moment. It has everything from daily problems, games, and many short courses that range in difficulty from high school through to approximately first-year university/college level. I have particularly enjoyed some of the linear algebra and probability courses, which I did this year, because they were relevant to my job, but I would recommend to not choose the course by its 'utility' as such - try to choose a course that interests you!

Also, here's a TED talk I enjoyed by math educator Jo Boaler on the idea of a 'math person':

Great post.

Regarding the section on software engineering for biosecurity:

"...potentially on biosecurity and pandemic preparedness (I don't currently know any examples of the latter, but think it's reasonably likely there will be some down the line)."

— I have some experience with this having worked for the UK Joint Biosecurity Centre during the height of the pandemic (albeit briefly) in a data science role. I think it's fair to say we had a reasonably sized influence on the analysis that went into government policy relating to the pandemic, with my seniors often reporting straight to the Prime Minister's Office, and where 'reasonably sized' means JBC technical reports or slide decks might have made it into the top ten or even top five most influential policy documents that the most senior health officials would look at that day (very rough guess).

I would argue that data engineering was a reasonably sized bottleneck (that could have been addressed by having access to more good software engineers to help improve our data platforms) but that there were also difficulties in knowing what was relevant data etc, which was more of a data science problem. So really there were many bottlenecks to growth/research of which data engineering was just one (personal opinion).

As a piece of general career advice I would say that software engineers thinking of data engineering as a career would probably find their skills remain in the demand or possibly increase in the following decades, which might make it a good bet. Just as research software engineering is a thing, research data engineering is definitely a thing (if not always given that name) and more talented people working in this area would probably be good.

JBC might not exist in quite the same way for much longer because of how much the public health infrastructure in the UK is changing at the moment (personal opinion), but I think data (software) engineering in biosecurity and pandemic preparedness is definitely a thing (for as long as these institutions persist after covid). If you're interested it helps to have some domain experience of what existing public health data infrastructure exists in your country or region, so that you know where to actually search for jobs. Alternatively you could go in through the contractor route although this seems like a less efficient way of working on the things you are actually interested in.