Of course, this has spawned numerous blog posts and tweet threads, but what no one has just said (and I wish they would), was that Black Roman Britains were too upper class (generals, governors, etc., brought in from elsewhere to rule over the Britains) to have stuck around when the empire collapsed. Soldiers would have been needed elsewhere and bureaucrats had to go somewhere where they could be paid.