Hello, the final GRF size is the problem here and your texture GRF being 800MB on its own doesn't mean much without knowing the size of the base GRF. That error is shown if the file table offset in the final GRF goes past the 4GB size limit. There's not much I can tell without more information on your end though.
Perhaps your GRFs are compacted using the "redirect indexes" option, which makes them seem smaller than they really are. When merging, if the conditions aren't perfect, the GRFs will be unpacted first and then merged together which can trigger the 4GB limit more easily.
It's either that or the base GRF size is near 3.4+GB.
None of that is helpful at the end of the day though, it seems like you will have to make a new GRF alongside instead of trying to keep compressing/removing stuff in it. There's really not much to do once you reach the size limit and the obvious solution is simply to have another GRF in your RO folder to save a lot of future headaches.