I've been chatting to Vasco via email about this. I guess I'll import some of my critiques here.

I feel like Brian's right that consciousness has some fuzzy edges (maybe it's a combination of many hard-to-recognise-in-isolation subphenomena), but his actual takes on this and most other philosophical subjects seem to me to reduce to an equivocation between nihilism and extreme egoism - they always seem to boil down to 'what "I" (the author) "care" about' as some kind of acceptance test. In the case of consciousness, this seems like you could understand it in a couple of ways, neither of which seem very informative to the reader:

1. Brian defines consciousness  as what Brian/perhaps the reader cares about, and we are to assume that Brian's 'caring' is a real phenomenon, not a meaningless string. Then his definition is circular or at least unfinished, since the reader has no access to what it means for Brian to care (and possibly Brian doesn't either), or how one could predict what future and other Brian- and reader-likes will care about. Nonetheless we are to assume that morality will perenially boil down to internecine struggles among different spheres of caring among different individuals
2. Brian defines consciousness as what Brian/perhaps the reader cares about, and Brian recognises the circularity and implies that there's really nothing resolving it - his strings are ultimately meaningless, produced, for all anyone including he can tell, by an automaton. Then either the reader is an automaton too, and their actions are equally meaningless, or they are not, and have no cause to read the arbitrarily generated output of an automaton

Brian subsequently wrote the essay My Confusions about the Hard Problem of Consciousness, which I think (though Vasco disagrees) basically contradicts this one. 

Though to do that I need to digress into an extended caveat that any purported contradiction comes with the caveat that on interpretation two, Brian's words and all words are referentless. At the very least, if we abandon the notion of consciousness, I think we most likely lose our entire vocabulary of non-physically-descriptive nouns, including but not limited to

• Contradiction/validity
• Reference classes
• Prediction
• Moral norms
• Metaethics
• Rational norms
• (is metarationality an analogous thing? If not, why not, and does that suggest something about metaethics and normative ethics)
• Decision theory
• Axiology (‘values’)
• Causality
• Free will
• Meaning
• Need
• Epistemology
• Internal (non revealed) preferences
• Beliefs
• Desires
• Goal-seeking
• Personal identity/the self
• Analyticity/maths
• Syntheticity/science (these last two already unified courtesy of Quine)
• Valence (though this maybe has two distinct phenomena)
• Ontology
• Revealed preferences
• Theology
• Moral uncertainty
• Political science
• Truth/falsity

(not to say I would miss all of these) (also heads up so it doesn't catch Vasco off guard, I added a couple of key ones since our email discussion)

And I think Vasco is arguing for interpretation two. So if so I think there's no discussion to be had - we are both just flapping our meat. It might turn out somehow that one of us counterfactually changes their behaviour if in range of the others' meat-flapping, but we can't ascribe that to causality - since that's a non-physically-descriptive noun - and we certainly can't ascribe it to anything like semantic properties of the words.

So I have to proceed by assuming that the first interpretation was correct, or that I somehow misunderstand the nature of the absence of consciousness, which is a strange position to be in. Perhaps Vasco can help me understand a different way of 'thinking about' it, but I am sceptical.

On this first-and-a-halfth interpretation Brian has gone from defining consciousness as 'thing Brian cares about' to 'we need to know what other things have raw feelings'. If the first essay were correct, 'need' wouldn't mean much, and if it did, the answer would be 'whatever things Brian cares about'. So perhaps Brian is saying there he anticipates caring about future things and cares about that anticipation. But learning information about the world couldn't give him information about other consciousnesses, because that information would be entirely contained within the stochastic process of him caring about things or not.

In the latter essay, Brian writes:

Reductionism: The world ultimately reduces to physics—quarks, leptons, etc., or probably something else more fundamental. "Consciousness" is not one of those physical primitives. Thus, consciousness must be an emergent phenomenon—a process that involves many parts working together in various ways. But if consciousness is emergent, then it must have fuzzy boundaries just like any concept does. There's no "true essence" underlying an emergent phenomenon. It just is what its components are, and we decide the boundaries of labeling which emergent phenomena belong to which categories that we make up. Consciousness must be like this.

Which Vasco approves of. I think these claims are far too confident:

• Depending on what we mean by a physical primitive, consciousness could be a property of one or more physical primitives. Perhaps panpsychists are correct and it's a property of all of them (I find this view weird, but easier to parse than the entirely dissolutionary view of consciousness)
• 'if consciousness is emergent, then it must have fuzzy boundaries just like any concept does.' - doesn't seem to follow. It might be a crisp subset that emerges from a specific minimum set of objects given by some as-yet unknown parameter; it might be inherent in a well-defined mathematical property, as the QRI guys think; it might be fuzzy-in-principle but so tightly defined in practice that for almost all practical purposes we can treat it as discrete at the macroscropic level (as e.g. the position of particles)

Ultimately I think that consciousness, despite being a philosophical black box and a neuroscientifically grey one, is too powerful a high level theory to abandon without a much more solid replacement. If you were to put me on a rack for long enough, I would recant the positions I'm arguing for here. But my prior explanation of why I would recant refers to a relatively simple, localisable statement that some -currently but not inevitably hard to specify - measurable quantity had gone negative, and that 'caused' my action. 

I won't put words into Vasco's mouth, but I wonder whether he thinks there's a way of describing that process without using 'reference classes' that's simpler than '{full mathematical derivation that} that's the all the interacting fields and forces in that local area'.

I also want to ask Vasco how he can be confident that, even if it's true that he has no consciousness (and it does seem to be a contingent and disablable property in humans), that lack of a light would generalise to those of us who profess to have one.

Thanks for sharing this opportunity!

I found a few more EA-aligned animal welfare organizations that appear eligible based on Every.org's official criteria:

• EA Animal Welfare Fund
• The Humane League
• Wild Animal Initiative
• Fish Welfare Initiative
• Animal Ethics
• Mercy For Animals

I'd always understood instrumental convergence as the claim about what is useful for achieving goals rather than a claim about what models would inevitably do: i.e. that seizing power and resource is useful towards virtually any goal. I know people went on to claim this implied actual AIs would pursue power and resources fanatically, but strictly speaking, I thought that was meant to be a likely consequence of "instrumental convergence" and not "instrumental convergence" itself. This is all based on my memory of reading Bostrom >10 years ago though, so I could be wrong.