1. Introduction and History
In spring 2010 I have done some experiments with decrypted Wii ISO images.
The data of encrypted ISO images looks like random data.
Because of this effective compressing (with 7Z, RAR, BZIP2,...) is not possible.
The compressors only reduce the size of the holes (=
scrubbed data areas).
WDF manage such holes with the result of smaller files than the compressed ones.
But decrypted images are good for compression.
Additional it is possible to ignore the reproducible HASH values (3% of image size).
In August 2010 I have remembered these ideas and I have begun to develop the
file format WIA. First it was more a proof of concept to find out a format
that reduce the needed disc space and can be handled with my tools directly.
The disc data is stored in peaces (chunks) of a multiple of 2 MiB.
Each chunk is compressed (with BZIP2 or LZMA) independent from the other chunks.
This allows random access reading.
My first choice was BZIP2:
It is good because it works already in chunks between 100KB and 900KB.
The resulting WIA file is about 10% .. 40% smaller than the WDF file.
The original ISO can be restored from WIA completly including HASH errors.
The disadvantage of WIA is the multiple time needed for writing and reading.
In a second step I take the LZMA source
from the 7zip project and implement the
compression methods LZMA and LZMA2. Both produce smaller WIA images than BZIP2.
There are 2 more compression methods: NONE and PURGE.
Both do not really compress the data, but they prepare the data for an
external packer. This combination is the most effektive packing method,
but the best internal mode (BEST = LZMA.7@50) have similar packing rates.
WIA is the slowest supported data format. If compression is done with mode BEST
the images are about 15-50% smaller than the same file in WDF.
If copying a WIA file to any other format the new file is a 1:1 copy of
the original source. Bad hash values are stored to guarantee the 1:1 copy.
For scrubbed discs this is only true for non scrubbed sectors.
USB loaders will never use WIA because of the complex structure of WIA
and the needed main memory for decompression.
1.1 Some statistics
Here are some compression statistics.
The tests have run in 2010 on a Linux system with a
AMD Athlon(tm) II X2 235e Processor, 2.7GHz.
The test system was almost idle.
The percent value is the compression rate against the WDF format.
The size of a compressed plain ISO with an external packer is similar
(a little bit larger) to the size of a WDF.
The time values are the time needed for the compression (converting from WDF to WIA)
and decrompression (converting from WIA to WDF).
The source images are scrubbed WDF files and only data partitions are copied.
size creation read compr. disc
in bytes WDF% time time mode id6 name of the game
----------------------------------------------------------------------------
61577490 17.45% 3:14 29 LZMA.7@50 RUUP01 Animal Crossing
6000758 23.83% 26 10 LZMA.7@50 RTYP01 Wii Chess
221789150 33.48% 5:18 43 LZMA.7@50 RSPP01 Wii Sports
54518891 53.62% 56 14 LZMA.7@50 RHAP01 Wii Play
78482721 55.87% 1:25 19 LZMA.7@50 R64P01 Wii Music
2073756489 62.43% 32:19 4:37 LZMA.7@50 RLRP4F T.Raider: Anniversary
1710692161 62.96% 25:15 3:48 LZMA.7@50 R3RP8P S&S All-Stars Racing
789065334 68.54% 11:30 1:46 LZMA.7@50 RZDP01 Twilight Princess
4853083788 67.06% 1:11:48 10:18 LZMA.7@50 RSBP01 Super Smash Bros. Brawl
5655712840 72.59% 1:33:02 11:36 LZMA.7@50 R3OP01 Metroid: Other M
1095283049 78.96% 16:12 2:20 LZMA.7@50 SB4P01 Super Mario Galaxy 2
2258986334 81.47% 28:33 4:27 LZMA.7@50 RMCP01 Mario Kart Wii
----------------------------------------------------------------------------
Analyzing the rates of the larger games
the creation speed is about 1.6 MiB/s and the read speed is about 11 MiB/s.
The creation of WIA files is slow because
of decryption, hash calculations and compressing.
Reading is much faster but slower than reading from a WDF because
of encryption and hash calculations.
1.2 Compression rates
The follwong table shows the compression rates and the resulting size for the game
Super Mario Galaxy 2 (SB4P01) compressed with different variants.
The first part shows the result of the scrubbed image, the second part of the raw image.
The percent value is the compression rate against the
WDF format.
Some image are compressed with an external packer after the conversion.
The tests are done on a Linux system with an AMD quad core (2.2GHz)
and 7zip used 2 CPUs for compressing.
size scrubbed storing post total
in bytes WDF% or raw mode compr. time
-----------------------------------------------------------
1163400363 73.13% scrubbed wia=non rar -m5 11 min
1282548222 80.62% scrubbed wia=non 7z -mx9 14 min
1294777300 81.39% scrubbed wia=best 20 min
1298114999 81.60% scrubbed wia=best rar -m5 38 min
1312530034 82.50% scrubbed wia=best 7z -mx9 39 min
1558919812 97.99% scrubbed wia=non 56 sec
1590904724 100.00% scrubbed wdf 34 sec
1595019594 100.26% scrubbed wdf rar -m5 22 min
1595231049 100.27% scrubbed iso rar -m5 25 min
1612414604 101.35% scrubbed wdf 7z -mx9 15 min
1612876679 101.38% scrubbed iso 7z -mx9 20 min
4699979776 295.43% scrubbed iso 32 sec
-----------------------------------------------------------
4127056954 88.92% raw wia=non rar -m5 53 min
4272920912 92.06% raw wia=non 7z -mx9 43 min
4287339110 92.37% raw wia=best 56 min
4298440747 92.61% raw wia=best rar -m5 115 min
4347309875 93.66% raw wia=best 7z -mx9 96 min
4499644820 96.95% raw wia=non 161 sec
4641376080 100.00% raw wdf 125 sec
4653392026 100.26% raw wdf rar -m5 67 min
4653395984 100.26% raw iso rar -m5 65 min
4699979776 101.26% raw iso 112 sec
4705693014 101.39% raw iso 7z -mx9 44 min
4706027990 101.39% raw wdf 7z -mx9 44 min
-----------------------------------------------------------
You can see clearly, that packing an image with RAR/7ZIP doesn't help to
compress the image because of the encryption.
WDF is here the much faster and better solution.
But if converting to WIA the compression rates are much better.
Using WIA/NONE and a post compressor gives the best packing rates,
but you must unpack the image before any access.
Using WIA/BEST allow you to access some parts of the disc without
decompressing the whole image.
1.3 Creating and reading a WIA file
Creating a WIA file is very simple.
Use the option
--wia to force WIA output.
Or use a filename with the extension
.wia.
See for »
Writing a Wii disc image« for details.
The options
--compression and
--mem controll the compression mode.
When reading a WIA file the format is detected automatically.
See for »Reading a Wii disc image« for details.
2. Compression modes
There are 5 compression methods and 5 meta compression modes.
A compression
mode is a combination of
method,
level and
chunk factor.
2.1 Compression methods
When creating a WIA the data is split into chunks:
Data of selected sectors (all if
--raw is set,
scrubbed else) are stored 1:1.
The hash values are removed if they can be calculated from the source.
In a second step theses chunks will be compressed by one of the following methods:
- --compression NONE
-
No compression is done.
This format is very good for an external packer.
Nearly no management data and hash values for the chunks are stored.
Compression levels are not supported
and the default chunk factor is 20 (chunk size 40 MiB).
- --compression PURGE
-
Data is purged like WDF: Areas with zeros are detected and skipped.
The image is smaller than 'NONE' but contains more management info.
There is a SHA1 hash for each data chunk so that data errors can be noticed.
Compression levels are not supported
and the default chunk factor is 20 (chunk size 40 MiB).
This is more an experimental mode.
- --compression BZIP2
-
Each chunk is compressed with BZIP2.
BZIP2 has its own data verification code.
The default compression level is 5
and the default chunk factor is 20 (chunk size 40 MiB).
- --compression LZMA
-
Each chunk is compressed with LZMA.
LZMA has its own data verification code.
The default compression level is 5
and the default chunk factor is 20 (chunk size 40 MiB).
- --compression LZMA2
-
Each chunk is compressed with LZMA2 level 7.
LZMA2 has its own data verification code.
The default compression level is 5
and the default chunk factor is 20 (chunk size 40 MiB).
Compression rates and times of LZMA2 are nearly equal to LZMA.
2.2 Compression level
The compression level can be changed by appending a point and a digit
at the end of the compression method (e.g.
bzip2.3 or
lzma.7).
The general default is level
.5.
Compression level
.0 means: Use the default level for the selected method.
2.3 Compression chunk factors
The compression is done in chunks.
Because of the Wii disc design the minimal chunk size is 2 MiB.
The default chunk size for all compression methods is 40 MiB (factor
@20).
The factor can be changed by appending a
'@' and the chunk size factor.
Compression factor
@0 means: Use the default factor for the selected method.
Examples:
- 'BZIP2@10' means: use BZIP2, default level, chunk size 10*2 MiB.
- 'LZMA.7@25' means: use LZMA, level .7, chunk size 25*2 MiB.
2.4 Meta compression modes
There are some additional meta compressions methods.
The assignment to the real modes may change in the future:
- --compression FAST (BZIP2.3@10)
-
This is a fast compression mode.
- --compression GOOD (LZMA.5@20)
-
This is a good compression mode.
It is biased towards high compression at expense of speed.
- --compression BEST (LZMA.7@50)
-
This is the best compression mode.
The images becomes small like the external 7zip packer,
but the compression needs a lot of time and very much system memory (about 800 MiB).
(Compression level 7 is the maximum supported level by LZMA)
- --compression MEM
-
This is a generic compression mode.
The tools try to detect the RAM size of the computer
and to select a good compression method, level and chunk size.
The user can define a good memory usage value by using the option --mem
or setting the environment variable WIT_MEM.
The MEM mode can be tested with: »wit compr mem --long --long«
- --compression DEFAULT (GOOD = LZMA.5@20)
-
This is the default compression mode.
It will be used if option --compression is not set.
These meta compression modes can also combined with compression level
or chunk factor to override the default values.
For example:
BEST.6 switches the compression level from
.7
(default for
BEST) to level
.6.
Instead of names the user can also enter a meta compressions level
between 1 (fast) and 9 (best).
2.5 Memory usage
The LZMA compression needs very much memory when creating a WIA file,
much more than the BZIP2 compression.
To handle the chunks and support random access the WIT tools need addionally
4.1 MiB for each chunk factor. For the default chunk factor
@20
this is 84 MiB.
The following table shows the read and write memory usages for reading
and writing WIA files.
It is only the memory needed especcially for the (de-)compression:
»wit compr --long --long«
real memory usage input
mode reading writing mode
-------------------------------------------
NONE 40 MiB 40 MiB NONE
PURGE 40 MiB 40 MiB PURGE
BZIP2 42 MiB 44 MiB BZIP2
LZMA 56 MiB 225 MiB LZMA
LZMA2 56 MiB 225 MiB LZMA2
-------------------------------------------
LZMA 56 MiB 225 MiB DEFAULT
BZIP2.3@10 21 MiB 23 MiB FAST
LZMA 56 MiB 225 MiB GOOD
LZMA.7@50 164 MiB 773 MiB BEST
LZMA.6@50 132 MiB 469 MiB MEM
-------------------------------------------
LZMA.1@5 10 MiB 12 MiB 1
LZMA.2@10 20 MiB 23 MiB 2
LZMA.3@30 61 MiB 69 MiB 3
LZMA.4@30 64 MiB 91 MiB 4
LZMA.4@50 104 MiB 131 MiB 5
LZMA.5@20 56 MiB 225 MiB 6
LZMA.5@50 116 MiB 285 MiB 7
LZMA.6@50 132 MiB 469 MiB 8
LZMA.7@50 164 MiB 773 MiB 9
-------------------------------------------
MEM is a generic compression method.
The tools try to detect the RAM size of the computer
and try to selects a good compression method, level and chunk size.
The user can define a good memory usage value by using the option
--mem
or setting the environment variable
WIT_MEM.
