Tyler Cowen had an interview with Sam Bankman-Fried in which he proposed a challenge to utilitarianism. 

COWEN: Should a Benthamite be risk-neutral with regard to social welfare?

BANKMAN-FRIED: Yes, that I feel very strongly about.

COWEN: Okay, but let’s say there’s a game: 51 percent, you double the Earth out somewhere else; 49 percent, it all disappears. Would you play that game? And would you keep on playing that, double or nothing?

BANKMAN-FRIED: With one caveat. Let me give the caveat first, just to be a party pooper, which is, I’m assuming these are noninteracting universes. Is that right? Because to the extent they’re in the same universe, then maybe duplicating doesn’t actually double the value because maybe they would have colonized the other one anyway, eventually.

COWEN: But holding all that constant, you’re actually getting two Earths, but you’re risking a 49 percent chance of it all disappearing.

BANKMAN-FRIED: Again, I feel compelled to say caveats here, like, “How do you really know that’s what’s happening?” Blah, blah, blah, whatever. But that aside, take the pure hypothetical.

COWEN: Then you keep on playing the game. So, what’s the chance we’re left with anything? Don’t I just St. Petersburg paradox you into nonexistence?

BANKMAN-FRIED: Well, not necessarily. Maybe you St. Petersburg paradox into an enormouslyvaluable existence. That’s the other option.

Perhaps it is the morally correct choice to take risky bets to maximize welfare, and advanced civilizations realize this. Perhaps sufficiently advanced civilizations face a potentially high payoff bet and accept it, leading to their non-existence. I'm not sure what this would look like, but it seems possible that these sorts of bets exist. Perhaps even a morally advanced civilization concerned about existential risk could St. Petersburg paradox itself out of existence in one way or another.