WDF (Wii Disc File, file format)

1. Introduction and History

When I started with the development of wwt in 2009, I missed a file format to store scrubbed Wii images efficiently as regular files. The idea with WBFS files was born later, and CISO has some disadvantages.

One way to reduce the disc space is to compress the data. But compressing works only for the holes, but not for encrypted data. And random access is very difficult for compressed data. The direct support of holes seems to be much better.

So I decided to create the file format WDF. WDF is an abbreviation of either »Wiimms Disc File« or shorter »Wii Disc File«. The file format manage holes (scrubbed areas) and stores only non zero data. The result is, that WDF images are smaller than WBFS or with RAR, ZIP or 7Z compressed ISO images.

1.1 WDF Versions

In October 2009, WDFv1 was implemented. The first file header contains data members to support an internal management of split files.

In early 2010 it was clear, that the internal split file support will never be used. It is more practical to search for the continuation file, if the first opened file is to small to hold the complete WDF. The advantage of this external management is, that joining or splitting can be done at any file positions and does not need a change of the file data.

In September 2012 I thought about a new version of WDF without internal split file support. The new format should also support alignment, so that USB loaders are able to use this format like WBFS files. Because of this thoughts, the behaviour of the tools were changed in October 2012 with release v2.10a: The tools ignore the split members for the validation checks.

In January 2014 I decided to complete the plans of WDFv2 and implemented it. WDFv2 files are a little bit smaller than v1 files, because of the removed internal split file support. In the following release of the tools (v2.27a), only the reading support was enabled. The tools (wit, wwt and wdf) are able to detect and read WDF files in versions v1 and v2, but can only create v1.

At the same time, I wrote a WDF library class for C++. Third party projects may use this source code library to implement read access of WDF files. Based on this class, I also developed WDF support for the Dolphin emulator.

With beginning of v2.28a (2014-03-01), the tools will support a full featured WDF version 2 with aligning and some other optimizations. With beginning of v2.30a (2014-11-15), WDF version 2 is the default mode if creating a WDF file.

2. File Format

All WDF support data is stored in network byte order (big endian). So the usage of endian conversion functions like ntohl() is mandatory.

The following table shows the general layout of a WDF file:

General WDF file layout
Size	Name	Description
0x38	header	The file header with a magic and basic information about the WDF file.
*	chunk_table	The chunk tables consist of a magic (8 bytes) and a list with the chunk entries. The magic should be used to validate the beginning of the chunk table. Only WDFv2 allows this place for the chunk table. It is only used, if there is enough space because of aligning the data.
*	padding	It is possible to add padding data here to force an alignment of the data blocks.
*	data	The data blocks. The blocks are referenced by the elements of the chunk list. Unused areas are also possible.
*	chunk_table	The chunk tables consist of a magic (8 bytes) and a list with the chunk entries. The magic should be used to validate the beginning of the chunk table. WDFv2 alternatively allows to place the chunk table before the data; see above.

If a WDF file grows, the header and the chunk table is read, new data appended overwriting the second magic and the chunk table. On closing, the new chunk table is written to the end of the file and a fixed header at the beginning.

2.1 WDF v1

WDFv1 use the following file header since its first implementation. The split file members were never used, are obsolete and should not longer used for validation.

File header of WDFv1 (big endian)
Offset	Size	Type	Name	Description
0x00	8	char[8]	magic	8 characters to identify a WDF file. The string is »WII\1DISC«. The »\1« in the middle is the C notation for ASCII character 1. The hex dump of the string is: 57 49 49 01 44 49 53 43.
0x08	4	u32	version	The WDF version. Always 1 for WDFv1.
0x0c	4	u32	split_file_id	This member were planned for the split file support. It should help to find the correct continuation files. The value is a random number for split files. Because split support was never implemented, the value is always 0 and ignored on reading. Since WDFv2, it has a new meaning.
0x10	4	u32	split_file_index	This member were planned for the split file support. It contains the zero based split index of the file (always 0). The value is ignored on reading. Since WDFv2, it has a new meaning.
0x14	4	u32	split_file_total	This member were planned for the split file support. It contains the total number of split files (always 1). The value is ignored on reading. Since WDFv2, it has a new meaning.
0x18	8	u64	image_size	The total size of the unpacked file (virtual image size).
0x20	8	u64	data_size	The size of all used data areas (a statistical value).
0x28	4	u32	chunk_split_file	This member were planned for the split file support. It contains the index of the file, that contains the chunk list (always 0). The value is ignored on reading. Since WDFv2, it has a new meaning.
0x2c	4	u32	chunk_n	Number of elements in the chunk list.
0x30	8	u64	chunk_off	File offset of the magic before the chunk list. All offsets are always relative to the beginning of the file. This is also the end-of-data position.
0x38	56 bytes total

The chunk list is placed at the end of the file. Directly before the chunk list, a magic is inserted with 8 bytes.

Chunk list element of WDFv1 (big endian)
Offset	Size	Type	Name	Description
0x00	4	u32	split_file_index	This member were planned for the split file support. It is ignored and always 0. It is removed in WDFv2.
0x04	8	u64	file_pos	Virtual offset into the unpacked file.
0x0c	8	u64	data_off	File offset to the related data block. All offsets are always relative to the beginning of the file.
0x14	8	u64	block_size	Size of the data block.
0x1c	28 bytes total

2.2 WDF v2

WDFv2 use the following file header. There are some unneeded members. But I decided to don't shrink the header, so that it looks like a minimal modified v1 header. The changed member names have a red background.

File header of WDFv2 (big endian)
Offset	Size	Type	Name	Description
0x00	8	char[8]	magic	8 characters to identify a WDF file. The string is »WII\1DISC«. The »\1« in the middle is the C notation for ASCII character 1. The hex dump of the string is: 57 49 49 01 44 49 53 43.
0x08	4	u32	version	The WDF version. Always 2 for WDFv2.
0x0c	4	u32	head_size	The size of this WDF header. For WDFv2 it is always 0x38 (56 bytes).
0x10	4	u32	align_factor	If not NULL, it gives an information about the user wanted alignment of the WDF file on creation time.
0x14	4	u32	compatible	This version number is planned for future versions of WDF. If the number is 2, then reading is fully compatible to WDFv2. For WDFv2 it is always 2 (that means: not compatible with WDFv1).
0x18	8	u64	image_size	The total size of the unpacked file (virtual image size).
0x20	8	u64	data_size	The size of all used data areas (a statistical value).
0x28	4	u32	reserved	Always 0.
0x2c	4	u32	chunk_n	Number of elements in the chunk list.
0x30	8	u64	chunk_off	File offset of the magic before the chunk list. All offsets are always relative to the beginning of the file. This is also the end-of-data position.
0x38	56 bytes total

The chunk list is placed at the end of the file. Directly before the chunk list, a magic is inserted with 8 bytes.

The only change since v1 is, that the used split file index is removed and a chunk element is now 4 bytes smaller.

Chunk list element of WDFv2 (big endian)
Offset	Size	Type	Name	Description
0x00	8	u64	file_pos	Virtual offset into the unpacked file.
0x08	8	u64	data_off	File offset to the related data block. All offsets are always relative to the beginning of the file.
0x10	8	u64	block_size	Size of the data block.
0x18	24 bytes total

3. Split file support

A WDF file can be split into any number of pieces. A good reason for split files is, that the file system don't allow large files. Generally, 10 pieces with each 1 GiB are enough to store any Wii image. The size of each piece may differ, but at least the first piece must hold the complete file header of 56 bytes.

The naming scheme is:

The first file has the same file name like a non split file, for example »image.wdf«.
Continuation files use the same full file name as the first file, but with a suffix. The suffix format is either ».%u« (preferred) or ».%02u« or ».%03u«.
Following the rules above, the file name of the first continuation file is either »image.wdf.1« or »image.wdf.01« or »image.wdf.001«.
All following continuation files use the same suffix with the same number of digits as the first continuation file, only the numbered index is changed. However, if the field width of the index needs more digits, the number is extended (see examples below).

Example with 1000 split files and one mandatory digit.
This is also the preferred naming scheme:

image.wdf
image.wdf.1
...
image.wdf.9
image.wdf.10
...
image.wdf.99
image.wdf.100
...
image.wdf.999
image.wdf.1000

Same example with 2 mandatory digits:

image.wdf
image.wdf.01
...
image.wdf.99
image.wdf.100
...
image.wdf.999
image.wdf.1000

Same example with 3 mandatory digits:

image.wdf
image.wdf.001
...
image.wdf.999
image.wdf.1000

4. Operations

The following paragraphs will give you a little introduction, how to implement the WDF functionality.

4.1 Reading a WDF

For reading a WDF do the following steps. Abort the operation if reading data or a test fails.

Read the header and convert it to local endian.
Verify the magic.
Check the version number. It must be >0. If it is larger than expected, check the compatible member. It may tell you, that the new version is read compatible with an old version.
Make some plausibility tests with the offset and size members.
Load the magic before the chunk table and verify it.
Load the chunk table convert it to local endian.
Make some plausibility tests with the offset and size members of all chunk elements.

4.2 Writing a WDF

Create a WDF header and an empty chunk list in the memory. The chunk list must be sorted by data_off and may grow while adding new blocks. Use a dynamic data structure for it.
Set the writing file offset behind the header.
For each write operation:
1. Check, if parts of the data are already defined. Modify that blocks.
2. Add new blocks at the end of file. Create for each new block a new chunk element and insert it at the correct place in the chunk list.
On closing, write the magic2 and the chunk list to the end of the file. Update the WDF header and write it to the beginning of the file.
If for any reasons the file becomes smaller, don't forget to truncate it behind the chunk list.

4.3 Modifying a WDF

Open the WDF in the same way as the reading operation.
Switch to the writing operation, but skip step 1.

5. Source Code

5.1 WIT Implementation

You can find the source of the WIT implementation (C) in the open SVN repository. Bit it is hard to read, because it is implemented to support the virtual file system of WIT:

5.2 C++ source code library

Here you can find a C++ source code library for reading WDF files:

src-lib/wdf/

Files lib-wdf.cpp and lib-wdf.h are the source code library. Please use only unmodified versions of these files and write a wrapper around to allow easy updates of the library.

Files example-wdf.cpp contains some examples, how to use the 2 library files files.