It turns out sending a closed FD via XCB doesn't fail. The other side (X.Org server) just receives a random FD number, which I think by chance referred to an unrelated FD which was opened.
But why was I sending a closed FD via XCB in the first place? It turns out sending a FD via XCB also closes it. So it worked fine the first time, and caused the bug the second time.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!