Look, Wiley E. Coyote Super Genius shell programmer, bash + awk is not the right combination for heavy duty statistical stuff, especially if you have to process a few thousand files of over 100MB each. And particularly don't fork each file's handling in an awk process inside a bash for loop. We've got languages & libraries intended for this sort of thing: octave and matlab and scipy and R and more, all installed and ready to use.