It turns out sending a closed FD via XCB doesn't fail. The other side (X.Org server) just receives a random FD number, which I think by chance referred to an unrelated FD which was opened.
But why was I sending a closed FD via XCB in the first place? It turns out sending a FD via XCB also closes it. So it worked fine the first time, and caused the bug the second time.
@emersion does it only close it when it succeeds?
@wolf480pl no, thankfully it closes unconditionally
So, yeah, *&%*($ to silently ignoring errors, XCB.