So after some testing it looks like this happens when there are too many sounds being played simultaneously (such as skills, effects, monster attacks..etc) it creates problems in the call stack for queued up .wav files since it's trying play audio asynchronously.
This can be solved by extracting all the .wav files in the GRF and using bash to replace them with millisecond .wav files so they are removed from the queue quicker.
for i in ~/data/wav/* ; do cp ~/file.wav ; done
