Understanding GRF Structure

[Neji]

Hi guys,

im trying to understand the grf structure and how it works. i read some information on that forum here. Like:

 

https://rathena.org/board/topic/57175-description-of-the-grf-file-format/

 

 

But some Bytes are not clear for me, maybe someone can help me out what this bytes are for?

 

what i understood so far:

Total 304 Bytes of test.grf
Header
--------------------------------------------------------------------------------------------------------------------
Master of Magic
4D 61 73 74 65 72 20 6F 66 20 4D 61 67 69 63 00 

Key
01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 

TableOffset
50 00 00 00 

Seed
00 00 00 00 

FilesCount
0C 00 00 00 

Version
00 02 00 00 

Content
--------------------------------------------------------------------------------------------------------------------
 ?  ?  ?  4  ?  ?  ?  t  s  e  T  ?  ?  ?  ?  ?
78 01 01 04 00 FB FF 74 73 65 54 04 4B 01 A1 00
 ?  ?  ?  5  ?  ?  ?  1  t  s  e  T  ?  ?  ?  ?
78 01 01 05 00 FA FF 31 74 73 65 54 05 41 01 D2
 ?  ?  ?  5  ?  ?  ?  2  t  s  e  T  ?  ?  ?  ?
78 01 01 05 00 FA FF 32 74 73 65 54 05 46 01 D3
 ?  ?  ?  5  ?  ?  ?  3  t  s  e  T  ?  ?  ?  ?
78 01 01 05 00 FA FF 33 74 73 65 54 05 4B 01 D4
 ?  ?  ?  5  ?  ?  ?  4  t  s  e  T  ?  ?  ?  ?
78 01 01 05 00 FA FF 34 74 73 65 54 05 50 01 D5

TableSize Compressed
AA 00 00 00 

TableSize Uncompressed
9F 00 00 00 

 ?  ?  ?  ?  ?  ?  ?
78 01 01 9F 00 60 FF 

 d  a  t  a  \  t  e  s  t  .  t  x  t  0        zSize           zSizeA          Size            Fg     Offset
64 61 74 61 5C 74 65 73 74 2E 74 78 74 00        0F 00 00 00     10 00 00 00     04 00 00 00     01     00 00 00 00 
 d  a  t  a  \  t  e  s  t  1  .  t  x  t  0
64 61 74 61 5C 74 65 73 74 31 2E 74 78 74 00     10 00 00 00     10 00 00 00     05 00 00 00     01     10 00 00 00 
 d  a  t  a  \  t  e  s  t  2  .  t  x  t  0
64 61 74 61 5C 74 65 73 74 32 2E 74 78 74 00     10 00 00 00     10 00 00 00     05 00 00 00     01     20 00 00 00 
 d  a  t  a  \  t  e  s  t  3  .  t  x  t  0 
64 61 74 61 5C 74 65 73 74 33 2E 74 78 74 00     10 00 00 00     10 00 00 00     05 00 00 00     01     30 00 00 00 
 d  a  t  a  \  t  e  s  t  4  .  t  x  t  0
64 61 74 61 5C 74 65 73 74 34 2E 74 78 74 00     10 00 00 00     10 00 00 00     05 00 00 00     01     40 00 00 00 

 ?  ?  ?  ?
BB CB 1C 7B 

--------------------------------------------------------------------------------------------------------------------

[Tokei]

The GRF format, for version 0x200, goes pretty much as you mentionned above. Something to always keep in mind is that the byte 0x78 is generally the zlib compression header (and the byte 0x00 is the unofficial lzma compression header). If you see some bytes starting with that value, it most likely means it needs to be uncompressed (which is always the case for the GRF's content).

 

Just as an example, if you take the entry "data\test1.txt" and read the offset, you'll get 16. The entries' offsets are relative to the data offset, so you have to add 46 (GRF header size). You'll get 62 which is the 'absolute' offset, and reading the content you'll get "78 01 01 05 00 FA FF 31 74 73 65 54 05 41 01 D2". After uncompressing this with zlib, you'll get "1tseT".

 

{zlib header} {compressed data} {int checksum} - simplified version for zlib compressed data.

 

The 'issue' is that... 0x78 0x01 is the zlib header for non-compressed data. That means when you made your GRF, you compressed it with the lowest compression level possible and that is the only reason why you're able to see the content in plain-text. It's definitely not something you want.

 

In the version 0x200, the file table is compressed, which is why you get the 0x78 byte after reading the table size uncompressed property. Since you know the compressed table size is 170, you pretty much read 170 bytes after you read the table size uncompressed value, and uncompressing that will give you the actual file table data.

 

Edit : You might want to have a look into KeyWorld's project https://github.com/vthibault/roBrowser/blob/master/src/Loaders/GameFile.js or even the OpenRagnarok project http://code.openhub.net/file?fid=D9P_ZHJdGOjvzVgQOe31Wor_U1U&cid=xmpIxx6iTvE&s=&fp=92163&projSelected=true#L0 (they'll become useful if you ever want to read versions 0x100, for the DES encryption).

Edited April 15, 2015 by Tokei

[Neji]

Hi Tokei,

 

thank you for the explanation. I forgot, even if you choose no compression, zlib is part of the progress so even it is uncompressed, the zlib will stored in the grf.

 

so the 7 first bytes right after the header are zlib header information, followed by the data and checksum. i see

the last 4 bytes of the grf is also part of zlib? because 7 bytes from the Table {zlib header} {filesnames}{checksum}, i guess?

 

actuelly im not interesseted in des-encryption, until i understood the 0x200 verion, but thank you for sharing the link.

i choosed uncompressed, its easier to study how the structure is build.

[Tokei]

Hi Tokei,

 

thank you for the explanation. I forgot, even if you choose no compression, zlib is part of the progress so even it is uncompressed, the zlib will stored in the grf.

 

so the 7 first bytes right after the header are zlib header information, followed by the data and checksum. i see

the last 4 bytes of the grf is also part of zlib? because 7 bytes from the Table {zlib header} {filesnames}{checksum}, i guess?

 

actuelly im not interesseted in des-encryption, until i understood the 0x200 verion, but thank you for sharing the link.

i choosed uncompressed, its easier to study how the structure is build.

 

Well... that's one way of seeing it, I guess. You can't get rid of zlib (and that's a good thing!), you can't ignore it while understanding the GRF format either. Let me put that differently, maybe it'll be clearer.

 

GRF = {grf header} {grf content} {grf table}

 

{grf content} = {compressed_file1} {compressed_file2} {compressed_file3} ...

{grf table} = {compressed table size} {uncompressed table size} {compressed table}

 

{compressed table} = zlib_uncompress(compressed table, uncompressed table size) = filenames, no need to read the zlib header, skip or ignore bytes, or anything else like that.

 

The 4 bytes at the end are indeed from zlib, but... you should simply uncompress the data, and read that instead. What you're doing right now is reading from the zlib stream, which happens to be already uncompressed. It's, hmm... 'dangerous'; the number of bytes at the beginning or the end can vary. But otherwise, the rest of the information is correct.

Edited April 15, 2015 by Tokei

[Neji]

well..  i wasnt goin to ignore the zlib bytes. i just thought if you choose 'no compression' the grf-data were just plain.

 

that was my missunderstanding, zlib is part of the progress, after getting the table zlib returns the plain src or throw an exception, so its known whether the data are corrupt or not of the zlib-stream.

 

i guess i clearly understand now how it works, thank you.