I implemented the UGen directly, takes 5 seconds now, so that's viable for the video project, and I still have a variable kernel definition, not a bad compromise.
I checked that into FScape v2.14.2
https://github.com/Sciss/FScape-next/commit/49d27925274e001eef2309d18798f107c35fdd09