Just spent far too long working out why Rails was capturing exceptions when running tests... the tests were running in the development environment, so much for convention over configuration :-\