ISO Images (file formats)

1. Introduction and History

When I started with USB loaders and backup copies in spring 2009. I have used the linux tool wbfuse and the USB loaders to manage my WBFS partition. In summer 2009 my WBFS was corrupted. So I had to dump all my games again. Because this failure and because disc space is cheap I made a copy of all my about 40 Wii discs to a linux ext3 disc.

The tool wbfuse exports all Wii ISO images as PLAIN ISO. It size is always 4.7GB (double layer are larger) independent from the real data size. Because of the sparse effect of scrubbed images they take less disc space. But copying such sparse file with simple copy tools removes the sparse effect.

In August 2009 my WBFS was corrupted again. I thought about that problem and decided to create my own tool with a more handy file format: The tool wwt and the file format WDF was born, the first commit into the repository was at September 23 and the first alpha was released at September 28. WDF manage the scrubbed holes. This was more efficient than packing with tools like zip or rar: The WDF images are a little bit smaller than compressed images and WDF images can be read and written as fast as plain ISO images.

WDF is a universal packing container. Any file can be packed with WDF. WDF allows random access and is as fast as plain ISO. Since October 2009 it is my standard archive format.

Since the beginning I worked with WBFS files instead with WBFS partitions for testing. It was oggzee from GBAtemp who discovered WBFS files as data format for single ISO images. This was the third ISO image file format in my personnel history. WBFS can be used by USB loaders because of the large chunk size (greater equal 2 MiB, always power of 2).

In March 2010 I noticed, that there was an other file format: CISO (=WBI). CISO manages hole like WDF but with larger chunks (standard 2 MiB). It has some design impairments: No network byte order, no info about the original file size, blocks must be stored in ascending order. But it has also a great advantage: It can be used by USB loaders.

In August 2010 I developed a new format: WIA (Wii ISO Archive). First it was more a proof of concept to find out a format that reduce the needed disc space and can be handled with my tools directly. The beta test of WIA ended in October 2010.

In January 2014, I added support for GCZ files (Dolphins file format). Dolphin supports also ISO and WBFS. Comparing GCZ and WBFS, WBFS ist faster (no compression) and produce smaller Wii images files. Only GameCube images become much smaller if using GCZ.

There is an alternate Format: FST. FST is not an image, it is an extracted file system.

2. File formats

All WIT tools support different file formats for Wii ISO Images:

Plain ISO files (.iso)
WDF files in versions 1 and 2 (.wdf)
WIA files (.wia)
CISO files (.ciso, .wbi)
WBFS files (.wbfs and disk partitions)
GCZ files (.gcz)
FST (extracted file systems)

2.1 ISO : Plain ISO

A Plain ISO file is a 1:1 copy of a Wii disc, may be scrubbed. It is not compressed and not part of any container. The standard extension is ".iso".

To save disk space all tools store Plain ISO as sparse files.

2.2 WDF : Wii(mms) Disc Format

When I start to implement my WBFS manager wwt I want to archive my Backups. Because Plain ISO are large and unhandy when copying I developed a new file format: WDF.

A WDF container manage the unneeded ranges (holes with only zero data) of the Wii discs and of any other files. All other formats needs more space than WDF to store the same content. The standard extension is ".wdf".

WDF is clearly better than packing images with RAR or ZIP: The WDF images are a little bit smaller than compressed images and WDF images can be read and written as fast as plain ISO images.

There are 2 similar versions of WDF: WDFv1 and WDFv2. For details, see »WDF (Wii Disc File, file format)«.

2.3 WIA : Wii(mms) ISO Archive

WIA is the slowest supported data format because it compresses the data. The tools supports several compression methods like BZIP2 or LZMA. If compession with mode BEST the images are about 10-40% smaller than the same file in WDF. USB loaders can't use WIA.

If you want minimal disc usage use WIA with the compression mode BEST. In general you save 25-50% of disc space compared to WDF or RAR.

For more details read the article »WIA (Wii ISO Archive)«.

2.4 CISO : Compact ISO (=WBI)

CISO is an other compact file format. It is also known as WBI. CISO stores only WBFS sectors that are needed. Like WDF, the CISO container can be used for any files and not only for Wii discs. The standard extensions are ".ciso" (default) and ".wbi".

CISO images can be used directly by USB loaders, when large chunk sizes with ≥1 MiB are used.

2.5 WBFS : Wii Backup File System

WBFS is a container for multiple Wii discs. It will be used by most of the WBFS loaders. To manage games the users need a WBFS manager like wwt. The standard extension for WBFS files is ".wbfs".

2.6 GCZ : GameCube Zip

GCZ is Dolphins (a GameCube and Wii emulator for PC) native image format. It uses chunks with 16 KiB and each chunk is deflated (compressed like ZIP). The compression is only good for GameCube images. Because of the encrypted data of Wii images, compression is useless.

2.7 FST : extracted File System

An extracted file system is a directory that contains all real and virtual files of a Wii discs as single files in the local files system.

When using a FST as source the internal file system builds an internal virtual ISO image. All tools and commands can use this virtual image transparently.

Read the article »Composing Wii ISO Images« for more details.

3. Reading a Wii disc image

All WIT tools auto detect the file format independet of the file name, they analyze the first bytes of the file. Split files are joined together. FST directories (see composing) are handled transparently as ISO image if option --ignore-fst is not set.

All tools and commands accept all file formats when reading a Wii ISO Image.

Options
Option		Param	Description
	`--ignore-fst`		Disable composing and ignore FST directories as input. This legacy option is a shortcut for --allow-fst=off.

3.1 WBFS disc selectors

To access a disc within a WBFS the user can add a selector at the end of the path. All WIT tools support 4 kinds of selectors when accessing a source image. In the following examples /path/to/wbfs is the path to the WBFS file or partition:

/path/to/wbfs/ID6
The first disc with the entered ID6 is used as source. Letters are tranformed to uppercase. Usually but not always the ID6 is unique.
/path/to/wbfs/index
The zero based index of the disc. The range goes from zero up to the number of discs in the WBFS minus 1. The index is any decimal number with 1 to 5 or 7 or more digits. 6 digits are not possible because they are interpreted as ID6. Because of adding and removing games the disc index may change for a single disc.
/path/to/wbfs/#slot
The decimal slot number within the WBFS. The range goes from zero up to the maximal number of possible discs in the WBFS minus 1. The slot number is the only stable selector: Adding and removing discs will not change the slot number of other discs.
/path/to/wbfs (no selector)
All discs of the WBFS are accepted as source image.

Examples:

If you have an empty WBFS and add 3 discs with IDs AAAAAA, BBBBBB and CCCCCC in this order, the slot usage is this:

 slot  index   ID6    remark
------------------------------------------------
  #0     0    AAAAAA
  #1     1    BBBBBB
  #2     2    CCCCCC
  #3     -    -       slot not used

If you remove the disc with ID6 BBBBBB the slot usage changes to:

 slot  index   ID6    remark
------------------------------------------------
  #0     0    AAAAAA
  #1     -    -       slot not used, was BBBBBB
  #2     1    CCCCCC  index changed from 2 to 1
  #3     -    -       slot not used

To access the disc with ID6 CCCCCC you have the following choices:

/path/to/wbfs/CCCCCC  : access via ID6
/path/to/wbfs/cccccc  : access via ID6, case ignored
/path/to/wbfs/1       : access via disc index
/path/to/wbfs/#2      : access via slot number

4. Writing a Wii disc image

If writing ISO files the options --wdf, --wdf1, --wdf2, --wia, --iso, --ciso, --wbfs, --gcz and --fst control the output format. The option --fst is only for some command available and switch to extract mode (see »wit EXTRACT«). If writing a WBFS file, the WBFS is truncated and contains exactly one ISO image. The default file name of this WBFS is '.wbfs'.

If none of the above options is set, the destination filename will be analyzed. If the extension (ignoring case) is .wdf, .wdf1, .wdf2, .wia, .iso, .ciso, .wbi (an alternative for .ciso), .gcz or .wbfs, the specific output format is used. The default is WDF if all other fails.

Options
Option		Param	Description
`-W`	`--wdf`	[=param]	Set the image output file type to WDF (Wii Disc Format). The output format is either WDFv1 or WDFv2 (default). It depends of the input file format and of the aligning. --wdf=param is a short cut for »--wdf --align-wdf=param«.
	`--wdf1`	[=param]	Set image output file type to WDF and force version 1. --wdf1=param is a short cut for »--wdf1 --align-wdf=param«.
	`--wdf2`	[=param]	Set image output file type to WDF and force version 2. --wdf2=param is a short cut for »--wdf2 --align-wdf=param«.
	`--wia`	[=compr]	Set image output file type to WIA (Wii ISO Archive). The optional parameter is a compression mode and --wia=mode is a shortcut for »--wia --compression mode«.
`-I`	`--iso`		Set image output file type to PLAIN ISO.
`-C`	`--ciso`		Set image output file type to CISO (Compact ISO, same as WBI).
`-B`	`--wbfs`		Set image output file type to WBFS (Wii Backup File System, default).
`-G`	`--gcz`		Set image output file type to GCZ (Dolphins GameCube Zip).
	`--fst`		Set image output mode to 'file system' (extracted ISO).

4.1 Escape sequences

Destination filenames are scanned for escape sequences beginning with the escape character '%'. The accepted format is:

'%cX' or '%mcX' or '%n-mcX'

'n' is the number of skipped characters of the field.
'm' is the zero based index of the last copied character.
'c' is an optional character. If c is 'u' then the source will be converted into upper case and if c is 'l' into lower case.
'X' selects the source and is one of (ignoring case):
- 'I' : The ID6 of the image.
- 'N' : Disc name. If no disc name was found, the ID6 (if not empty) or '%G' (e.g. for WBFS archvies) is used.
- 'T' : Title of title db. If no title is found, then the disc name is used.
- 'E' : The default extension of the image ('wdf', 'wia', 'iso', 'ciso', 'wbfs' or 'gcz').
- 'P' : The path (all up to the last '/') of the source file.
- 'F' : The filename (starts behind the last '/') of the source file. '%G%H' is always the same as '%F'.
- 'G' : The name part of '%F' (extension removed).
- 'H' : The file extension of '%F' including the point.
- 'X' : Extended filename: A shortcut for '%T [%I].%E'.
- 'Y' : Extended filename: A shortcut for '%T [%I]' (no file extension).
- '+' : The default filename: WBFS='%I.%E', all other='%X'.
To use the '%' sign itself just type '%%'.

4.1.1 Examples

'%4I.%E'
Store the disc with ID4 as name.
'%1uT/%+'
Store the disc into a subdirectory named with the first character (upper case) of the title.
'%3-4lI/%+'
Store the disc into a subdirectory named with the language code of the ID using the lower case letter.
'%P/%X'
Store the disc into the directory of the source with the extended filename.

4.1.2 Alternative escape character

Instead of '%' an alternative escape character can be used. It is defined by the option --esc (-E). This makes live easier if using the cygwin version together with the windows shell 'cmd'. Define the environment variables 'WIT_OPT' and/or 'WWT_OPT' for a new default definition.

Example for Unix bash: export WIT_OPT="--esc=$"
Example for Windows: set WIT_OPT=--esc=$

4.2 File Splitting

On filesystems, that support only files smaller than 2 or 4 GiB, images must be split in 2 or more files. The FAT filesystem is an example: It supports only file up to 4 GiB minus 32 KiB. Modern USB-Loader support split WBFS files.

The following options control, how the tools split the images:

Options
Option		Param	Description
	`--auto-split`		Enable auto split modus: Split only if necessary and determine the split size automatically. THIS OPTION IS EXPERIMENTAL. In future versions it becomes the default.
	`--no-split`		Disable output file splitting. This is the default, but in future versions, the new option --auto-split becomes the default.
`-z`	`--split`		Enable output file splitting. The default split size is 4 GB.
`-Z`	`--split-size`	sz	Enable output file splitting and define a split size. The parameter 'sz' is a floating point number followed by an optional unit factor (one of 'cb' [=1] or 'kmgtpe' [base=1000] or 'KMGTPE' [base=1024]). The default unit is 'G' (GiB).

4.2.1 Auto Split

--auto-split is based on tests with seek() functions. Linux throws an error if seeking to an inpossible offset. So the maximum file size detection works fast and good.

Cygwin never throws an error and all seeks are successful. So a needed split can't be detected in this way and --auto-split becomes useless.

I have no access to a Mac hardware. So I can't make any tests with a limited filesystem.

4.3 Preallocation and Fragmentation

If creating a new image it is possible to preallocate the needed disc space. The operating system has a chance to allocate all space in only one or few fragments.

Experiments have shown, that this is a clear advantage for Linux, Mac and Windows systems in respect to the number of fragments and the writing speed. So preallocation is enabled by default and can be controlled by the option »--prealloc mode«.

4.3.1 --prealloc [=mode]

This option enables or disables the disc space preallocation. If enabled the tools try to allocate disc space for the new files before writing the data. This reduces the fragmentation but also disables the sparse effect for preallocated areas. The optional parameter decides the preallocation mode: OFF (or 0), SMART (or 1), ALL (or 2). If no parameter is set, ALL is used. Mode 'OFF' disables the preallocation. Mode 'SMART' looks into the source disc to find out the writing areas. SMART is only available for ISO, CISO and WBFS file types. For other file types ALL is used instead. Mode 'ALL' (the default) preallocate the whole destination file. Because of the large holes in plain ISO images, the SMART mode is used for ISOs instead.

Prellocation can be controlled by the option --prealloc. If set --prealloc=SMART the tools discover the needed blocks (non scrubbed blocks) and will preallocate the disc space in peaces. Tests have shown the only plain ISO images has an advantage in this mode. WBFS and CISO images ignores large non needed blocks because of their design. WDF, WIA and GCZ images manage such holes by it self and so SMART is the same as ALL.

The default mode --prealloc=ALL will preallocate the whole destination file. This is very bad for plain ISO images because of the large holes. Therefor the mode SMART is used instead of ALL for plain ISO images. For WDF, WIA and GCZ images the disc space is estimated, rounded up and truncated on closing.

Command reference

»wdf pack«, »wdf unpack«, »wit convert«, »wit copy«, »wit extract«, »wit mix«, »wwt extract«.