Jump to content
curiosity

LessGRF - Slim down your GRF

Recommended Posts

LessGRF

 

What?

LessGRF (LGRF) is an extension to the GRF format which adds a new algorithm for file compression. Normally the GRF will compress files using the DEFLATE algorithm which is the compression method typically used in ZIP files, but LessGRF adds the LZMA algorithm as an option.

 

Why?

Ragnarok's GRF files can get quite big. Back in the days the game data used to take up a few hundred megabytes, but today the GRF archives are about 2GB and contain tens of thousands of files. Since the data is starting to get pretty big it's possible to save quite a bit of space simply by re-compressing the game files with a more efficient algorithm. In my test on a full data.grf the file was reduced to about 60% of its former size. Maybe someone will find it useful if they want to make a mini client or something? Honestly, this was just a little experiment I did out of curiosity. 

 

How?

Two simple steps are all that's necessary to use the LGRF format. First, download this file.

 

1. Installation

To upgrade your client, replace the cps.dll in your Ragnarok folder with the one from the download. (Make a backup of the old DLL file first.)

 

2. Recompression

Extract GRFCrush.exe somewhere. Run it from command line or just drag and drop your GRF file onto it. It will generate a new GRF file which you can use instead of the old one. The recompression process will likely take about 20 minutes on a moderately powerful computer.

 

Q&A

 

Q: Since the compression rate is better, will files load slower?

A: I've not notice any different in map load times. LZMA is more expensive to decompress, but modern processors are so fast the load time might just as well be reduced since less data has to be read from the hard disk.

 

Q: Can I use both GRF files and LGRF files at the same time?

A: Yes. The new cps.dll supports the old GRF compression as well. You can mix LGRF and GRF

 

Q: Can patchers or GRF tools update LGRF archives?

A: Very likely, but only if they don't attempt to decode the data in the process.

 

Q: Can current GRF tools extract files from LGRF archives?

A: No.

 

Q: ...Does that mean it can also be used for GRF protection?

A: I wouldn't recommend it. (That being said I guess this is no less "secure" than GRF protectors like SecureGRF.)

 

Q: Are there any penalties involved with using this format? 

A: Incompatibility with GRF extractors is currently the only thing I've been able to think of. Let me know if you notice something else.

 

  • Upvote 10

Share this post


Link to post
Share on other sites

This is a great idea; I had no idea GRFs still used DEFLATE. I can see this being very useful, especially for small private servers, since they'll no longer have to find a way to host a 2GB+ file with limited bandwidth or space.

It'd be helpful if some users tried testing what tools currently work or don't work with this, to encourage faster adoption of it.

Questions, though: do you have a list of the command-line options (are there any?), and can the tool also decompress the GRFs?

  • Upvote 2

Share this post


Link to post
Share on other sites

Very cool and useful project.

 

I got 2 errors though.

mssnDdb.png

 

Did you try to profile your application? I have a feeling that a lot of time is wasted in memory allocations and thread creation. Also multithreading would be very cool.

Edited by Valo

Share this post


Link to post
Share on other sites

I'll send it to a dev friend for testing in real server =o, thank you, I'll reply here with more info later!

  • Upvote 2

Share this post


Link to post
Share on other sites

This is great! I've struggled trying to sift and sort content because of size restrictions. This will help me greatly, thank you very much.

Share this post


Link to post
Share on other sites

sounds greart! ,but I don't wanna use it  unless  there has been a grf tool  can edit  LGRF

Share this post


Link to post
Share on other sites

Using this with SecureGRf works?

Share this post


Link to post
Share on other sites

Using this with SecureGRf works?

SecureGRF works with its own cps.dll, same as this tool. So you can't since you have to choose which .dll you want the client to load.

 

Great work as always curiosity, I don't know why Gravity didn't implement something similar yet.

Hope this will be used soon in a lot of servers : boring to download 2go of data.

  • Upvote 1

Share this post


Link to post
Share on other sites

Using this with SecureGRf works?

SecureGRF works with its own cps.dll, same as this tool. So you can't since you have to choose which .dll you want the client to load.

Great work as always curiosity, I don't know why Gravity didn't implement something similar yet.

Hope this will be used soon in a lot of servers : boring to download 2go of data.

Thanks. Make sense. but i think its the same to make a winrar compressed a lot. But its very useful.

Thanka for release :)

Share this post


Link to post
Share on other sites

Thanks for all the kind words. I didn't really expect it to get much attention.

 

Questions, though: do you have a list of the command-line options (are there any?), and can the tool also decompress the GRFs?

No options so far. I considered adding an option for compression level, but decided to go for always max. It can not revert an archive at the moment, but I that could be implemented with ease if there's demand for it.

 

 

I got 2 errors though. (...)

 

Did you try to profile your application? I have a feeling that a lot of time is wasted in memory allocations and thread creation. Also multithreading would be very cool.

Welcome to rAthena :)

 

Yeah, I got those too. Actually I was certain that both are junk entries, but taking a second glance I see that worm_tail_damage.wav is not. Looks like the compressed data is malformed at the very end, which makes the decoder complain even though it manages to decompress the whole thing. I don't know if this is due to my DES decryption or not, but I've noticed in the client that they don't bother to check if decompression fails, so it wouldn't really care either way.

 

You hit the nail on the head in regards to memory allocations and threads. The converter is very quick and dirty, so it's not been optimized in any way and there are plenty of wasteful allocation and memory copying going on. Current it will actually use two threads while encoding, but since the files are so small this likely introduces a large overhead. It could definitely be improved. Tell me if you're interesting in the source code.

 

 

sounds greart! ,but I don't wanna use it  unless  there has been a grf tool  can edit  LGRF

That's understandable. However to be clear, let me reinstate that very likely the only thing that won't work with current tools is extraction. Adding new files, merging archives, rebuilding etc. should all work with current tools.

Share this post


Link to post
Share on other sites

Does it need to be inside of my client folder before I can compress GRF files? Because I'm getting this atm.

 

238a0d1885da8d700adb124cabed6a80.png

 

:/

Share this post


Link to post
Share on other sites

You probably just don't have the newest version of the Microsoft Visual C++ runtime distributable. However it's not really necessary, so I made a version which will hopefully work without you having to install anything.

Share this post


Link to post
Share on other sites

You probably just don't have the newest version of the Microsoft Visual C++ runtime distributable. However it's not really necessary, so I made a version which will hopefully work without you having to install anything.

 

It's working for me now thank you very much. :)

Share this post


Link to post
Share on other sites

Curiosity killed the GRF!!! Size that is :D. Excellent work again Curiosity. Thanks for this great tool.

 

Honestly, this was just a little experiment I did out of curiosity.

 

Curioser and Curioser eh  /gg

 

Tell me if you're interesting in the source code.


I most certainly am interested :D.

Share this post


Link to post
Share on other sites

Interesting idea, though I have been thinking if possible to shrink down grf by removing files no longer in use.

Share this post


Link to post
Share on other sites
Hey Curiosity,

 

I liked your idea, so I've added a new feature in GRF Editor to load GRF files with a custom DLL (this can be done in Tools > Settings > Compression method > Custom compression... and simply select the cps.dll provided with your download). That way, you can extract files and treat the GRF just like any other one xD.

 

There is however one issue that I can't really do much about. New files added cannot be compressed with LZMA because the compress method in the DLL does a normal zlib compression. It would be quite useful if you could implement it though, but that's entirely up to you :P. (Oh also, I've added a repack feature which is basically meant for repacking the GRF with a custom DLL, so that could be useful to you).

 

Thank you for the release!

Edited by Tokei
  • Upvote 2

Share this post


Link to post
Share on other sites

Another awesome release, thank you for all these tools you are developing.

Share this post


Link to post
Share on other sites

@Tokei

Hey, that's great! I hadn't seen your GRF tool before, but it looks really good. Unfortunately the compressor function must encode ZLIB to stay compatible with the server. The client will use the compress function to encode guild emblems before they're sent to the server, and naturally Athena isn't able to decode LZMA. However I compiled a new DLL which also exports a function called compress2 that encodes LZMA (untested, but hopefully it works).

 

@NeoMind

Cheerful as always, eh? :)

I uploaded the code for the DLL at github. To compile you'll need to download the ZLIB SDK and fix the include/lib directories in the solution properties. I didn't upload the compressor tool code since it isn't really needed anymore if Tokei's editor can do it. It wasn't very good anyway, so not a big loss there to be honest...

 

@Aeomin

Yes, I think that's an idea worth exploring as well. Though there would be some manual effort required, which is a little boring. What really takes up space (while compressed) are textures and map files (particularly GNDs). I think most map files are in use, but selecting based on a map config would be simple. However I think a lot of textures aren't referenced by GNDs or RSMs, but hardcoded into the client, so these would have to be manually included.

Share this post


Link to post
Share on other sites

Why not both?

zlib side-by-side with lzma

Share this post


Link to post
Share on other sites

Why not both?

zlib side-by-side with lzma

Twice the space, twice the fun!

I think you mean by keep gat files in zlib so that server can read it?

Share this post


Link to post
Share on other sites

You cannot use the official GRF without re-compressing it after every update, if the cps.dll does not support zlib; unless you can force Gravity to ship patches with lzma compression.

Share this post


Link to post
Share on other sites

@Ai4Rei

 

My version of cps.dll is already able to handle both ZLIB and LZMA because I use a simple mechanism to detect the format when uncompressing. So it's no problem to mix LZMA and ZLIB encoded entries in the same GRF. What I was talking about in my previous post was cps.dll's compress function. It will only encode to ZLIB because there's no obtrusive way to know if the caller is repacking the GRF or encoding a guild emblem to be sent to the server. However by now I've added a second compression function which GRF editors can use to encode LZMA.

 

Or maybe I misunderstood what you meant? :)

Share this post


Link to post
Share on other sites

Heya,

 

Alright, I've modified my editor a bit, it now reads "compress2" before "compress", so you can repack a GRF with your custom compression directly, thank you ;)! It will always compress the file table with the normal zlib library for obvious reasons.

 

On an unrelated note though, I was a bit curious and I checked worm_tail_damage.wav. It is indeed not compressed properly and almost all GRF editors can't read it. The original cps.dll can read it perfectly fine however, so that's what I rely on when repacking!

  • Upvote 2

Share this post


Link to post
Share on other sites

Heya,

 

Alright, I've modified my editor a bit, it now reads "compress2" before "compress", so you can repack a GRF with your custom compression directly, thank you ;)! It will always compress the file table with the normal zlib library for obvious reasons.

 

On an unrelated note though, I was a bit curious and I checked worm_tail_damage.wav. It is indeed not compressed properly and almost all GRF editors can't read it. The original cps.dll can read it perfectly fine however, so that's what I rely on when repacking!

My guess is that it is corrupted and cps's implementation is to ignore zlib error. That may not be the case though.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use and Privacy Policy.