LessGRF - Slim down your GRF

[curiosity]

LessGRF

 

What?

LessGRF (LGRF) is an extension to the GRF format which adds a new algorithm for file compression. Normally the GRF will compress files using the DEFLATE algorithm which is the compression method typically used in ZIP files, but LessGRF adds the LZMA algorithm as an option.

 

Why?

Ragnarok's GRF files can get quite big. Back in the days the game data used to take up a few hundred megabytes, but today the GRF archives are about 2GB and contain tens of thousands of files. Since the data is starting to get pretty big it's possible to save quite a bit of space simply by re-compressing the game files with a more efficient algorithm. In my test on a full data.grf the file was reduced to about 60% of its former size. Maybe someone will find it useful if they want to make a mini client or something? Honestly, this was just a little experiment I did out of curiosity. 

 

How?

Two simple steps are all that's necessary to use the LGRF format. First, download this file.

 

1. Installation

To upgrade your client, replace the cps.dll in your Ragnarok folder with the one from the download. (Make a backup of the old DLL file first.)

 

2. Recompression

Extract GRFCrush.exe somewhere. Run it from command line or just drag and drop your GRF file onto it. It will generate a new GRF file which you can use instead of the old one. The recompression process will likely take about 20 minutes on a moderately powerful computer.

 

Q&A

 

Q: Since the compression rate is better, will files load slower?

A: I've not notice any different in map load times. LZMA is more expensive to decompress, but modern processors are so fast the load time might just as well be reduced since less data has to be read from the hard disk.

 

Q: Can I use both GRF files and LGRF files at the same time?

A: Yes. The new cps.dll supports the old GRF compression as well. You can mix LGRF and GRF

 

Q: Can patchers or GRF tools update LGRF archives?

A: Very likely, but only if they don't attempt to decode the data in the process.

 

Q: Can current GRF tools extract files from LGRF archives?

A: No.

 

Q: ...Does that mean it can also be used for GRF protection?

A: I wouldn't recommend it. (That being said I guess this is no less "secure" than GRF protectors like SecureGRF.)

 

Q: Are there any penalties involved with using this format? 

A: Incompatibility with GRF extractors is currently the only thing I've been able to think of. Let me know if you notice something else.

 

[Euphy]

This is a great idea; I had no idea GRFs still used DEFLATE. I can see this being very useful, especially for small private servers, since they'll no longer have to find a way to host a 2GB+ file with limited bandwidth or space.

It'd be helpful if some users tried testing what tools currently work or don't work with this, to encourage faster adoption of it.

Questions, though: do you have a list of the command-line options (are there any?), and can the tool also decompress the GRFs?

[Valo]

Very cool and useful project.

 

I got 2 errors though.

 

Did you try to profile your application? I have a feeling that a lot of time is wasted in memory allocations and thread creation. Also multithreading would be very cool.

Edited May 19, 2014 by Valo

[Angelisk]

I'll send it to a dev friend for testing in real server =o, thank you, I'll reply here with more info later!

[Skorm]

This is great! I've struggled trying to sift and sort content because of size restrictions. This will help me greatly, thank you very much.

[QQfoolsorellina]

sounds greart! ,but I don't wanna use it  unless  there has been a grf tool  can edit  LGRF

[Zell]

Using this with SecureGRf works?

[KeyWorld]

Using this with SecureGRf works?

SecureGRF works with its own cps.dll, same as this tool. So you can't since you have to choose which .dll you want the client to load.

 

Great work as always curiosity, I don't know why Gravity didn't implement something similar yet.

Hope this will be used soon in a lot of servers : boring to download 2go of data.

[Zell]

Using this with SecureGRf works?

SecureGRF works with its own cps.dll, same as this tool. So you can't since you have to choose which .dll you want the client to load.

Great work as always curiosity, I don't know why Gravity didn't implement something similar yet.

Hope this will be used soon in a lot of servers : boring to download 2go of data.

Thanks. Make sense. but i think its the same to make a winrar compressed a lot. But its very useful.

Thanka for release

[curiosity]

Thanks for all the kind words. I didn't really expect it to get much attention.

 

Questions, though: do you have a list of the command-line options (are there any?), and can the tool also decompress the GRFs?

No options so far. I considered adding an option for compression level, but decided to go for always max. It can not revert an archive at the moment, but I that could be implemented with ease if there's demand for it.

 

 

I got 2 errors though. (...)

 

Did you try to profile your application? I have a feeling that a lot of time is wasted in memory allocations and thread creation. Also multithreading would be very cool.

Welcome to rAthena

 

Yeah, I got those too. Actually I was certain that both are junk entries, but taking a second glance I see that worm_tail_damage.wav is not. Looks like the compressed data is malformed at the very end, which makes the decoder complain even though it manages to decompress the whole thing. I don't know if this is due to my DES decryption or not, but I've noticed in the client that they don't bother to check if decompression fails, so it wouldn't really care either way.

 

You hit the nail on the head in regards to memory allocations and threads. The converter is very quick and dirty, so it's not been optimized in any way and there are plenty of wasteful allocation and memory copying going on. Current it will actually use two threads while encoding, but since the files are so small this likely introduces a large overhead. It could definitely be improved. Tell me if you're interesting in the source code.

 

 

sounds greart! ,but I don't wanna use it  unless  there has been a grf tool  can edit  LGRF

That's understandable. However to be clear, let me reinstate that very likely the only thing that won't work with current tools is extraction. Adding new files, merging archives, rebuilding etc. should all work with current tools.

[trinity]

http://gyazo.com/f2107939307d35abb28ca02b0ed4cfd8

 

tried it on the full data.grf 50minutes XD

[Skorm]

Does it need to be inside of my client folder before I can compress GRF files? Because I'm getting this atm.

 

 

:/

[curiosity]

You probably just don't have the newest version of the Microsoft Visual C++ runtime distributable. However it's not really necessary, so I made a version which will hopefully work without you having to install anything.

[Skorm]

You probably just don't have the newest version of the Microsoft Visual C++ runtime distributable. However it's not really necessary, so I made a version which will hopefully work without you having to install anything.

 

It's working for me now thank you very much.

[Neo-Mind]

Curiosity killed the GRF!!! Size that is . Excellent work again Curiosity. Thanks for this great tool.

 

Honestly, this was just a little experiment I did out of curiosity.

 

Curioser and Curioser eh 

 

Tell me if you're interesting in the source code.

I most certainly am interested .

[Aeomin]

Interesting idea, though I have been thinking if possible to shrink down grf by removing files no longer in use.

[Tokei]

Hey Curiosity,

 

I liked your idea, so I've added a new feature in GRF Editor to load GRF files with a custom DLL (this can be done in Tools > Settings > Compression method > Custom compression... and simply select the cps.dll provided with your download). That way, you can extract files and treat the GRF just like any other one xD.

 

There is however one issue that I can't really do much about. New files added cannot be compressed with LZMA because the compress method in the DLL does a normal zlib compression. It would be quite useful if you could implement it though, but that's entirely up to you . (Oh also, I've added a repack feature which is basically meant for repacking the GRF with a custom DLL, so that could be useful to you).

 

Thank you for the release!

Edited May 21, 2014 by Tokei

[Olrox]

Another awesome release, thank you for all these tools you are developing.

[curiosity]

@Tokei

Hey, that's great! I hadn't seen your GRF tool before, but it looks really good. Unfortunately the compressor function must encode ZLIB to stay compatible with the server. The client will use the compress function to encode guild emblems before they're sent to the server, and naturally Athena isn't able to decode LZMA. However I compiled a new DLL which also exports a function called compress2 that encodes LZMA (untested, but hopefully it works).

 

@NeoMind

Cheerful as always, eh?

I uploaded the code for the DLL at github. To compile you'll need to download the ZLIB SDK and fix the include/lib directories in the solution properties. I didn't upload the compressor tool code since it isn't really needed anymore if Tokei's editor can do it. It wasn't very good anyway, so not a big loss there to be honest...

 

@Aeomin

Yes, I think that's an idea worth exploring as well. Though there would be some manual effort required, which is a little boring. What really takes up space (while compressed) are textures and map files (particularly GNDs). I think most map files are in use, but selecting based on a map config would be simple. However I think a lot of textures aren't referenced by GNDs or RSMs, but hardcoded into the client, so these would have to be manually included.

[Ai4rei]

Why not both?

zlib side-by-side with lzma

[Aeomin]

Why not both?

zlib side-by-side with lzma

Twice the space, twice the fun!

I think you mean by keep gat files in zlib so that server can read it?

[Ai4rei]

You cannot use the official GRF without re-compressing it after every update, if the cps.dll does not support zlib; unless you can force Gravity to ship patches with lzma compression.

[curiosity]

@Ai4Rei

 

My version of cps.dll is already able to handle both ZLIB and LZMA because I use a simple mechanism to detect the format when uncompressing. So it's no problem to mix LZMA and ZLIB encoded entries in the same GRF. What I was talking about in my previous post was cps.dll's compress function. It will only encode to ZLIB because there's no obtrusive way to know if the caller is repacking the GRF or encoding a guild emblem to be sent to the server. However by now I've added a second compression function which GRF editors can use to encode LZMA.

 

Or maybe I misunderstood what you meant?

[Tokei]

Heya,

 

Alright, I've modified my editor a bit, it now reads "compress2" before "compress", so you can repack a GRF with your custom compression directly, thank you ! It will always compress the file table with the normal zlib library for obvious reasons.

 

On an unrelated note though, I was a bit curious and I checked worm_tail_damage.wav. It is indeed not compressed properly and almost all GRF editors can't read it. The original cps.dll can read it perfectly fine however, so that's what I rely on when repacking!

[Aeomin]

Heya,

 

Alright, I've modified my editor a bit, it now reads "compress2" before "compress", so you can repack a GRF with your custom compression directly, thank you ! It will always compress the file table with the normal zlib library for obvious reasons.

 

On an unrelated note though, I was a bit curious and I checked worm_tail_damage.wav. It is indeed not compressed properly and almost all GRF editors can't read it. The original cps.dll can read it perfectly fine however, so that's what I rely on when repacking!

My guess is that it is corrupted and cps's implementation is to ignore zlib error. That may not be the case though.