prevnext   » WIT: Wiimms ISO Tools » Guides » WIA (Wii ISO Archive)

WIA (Wii ISO Archive)

Contents

1.   Introduction and History

In spring 2010 I have done some experiments with decrypted Wii ISO images. The data of encrypted ISO images looks like random data. Because of this effective compressing (with 7Z, RAR, BZIP2,...) is not possible. The compressors only reduce the size of the holes (=scrubbed data areas). WDF manage such holes with the result of smaller files than the compressed ones. But decrypted images are good for compression. Additional it is possible to ignore the reproducible HASH values (3% of image size).

In August 2010 I have remebered these ideas and I have begun to develop the file format WIA. First it was more a proof of concept to find out a format that reduce the needed disc space and can be handled with my tools directly.

The disc data is stored in peaces (chunks) of a multiple of 2 MiB. Each chunk is compressed (with BZIP2 or LZMA) independent from the other chunks. This allows random access reading. My first choice was BZIP2: It is good because it works already in chunks between 100KB and 900KB. The resulting WIA file is about 10% .. 40% smaller than the WDF file. The original ISO can be restored from WIA completly including HASH errors. The disadvantage of WIA is the multiple time needed for writing and reading.

In a second step I take the LZMA source from the 7zip project and implement the compression methods LZMA and LZMA2. Both produce smaller WIA images than BZIP2.

There are 2 more compression methods: NONE and PURGE. Both do not really compress the data, but they prepare the data for an external packer. This combination is the most effektive packing method, but the best internal mode (BEST = LZMA.7@50) have similar packing quotes.

WIA is the slowest supported data format. If compression is done with mode BEST the images are about 15-50% smaller than the same file in WDF. If copying a WIA file to any other format the new file is a 1:1 copy of the original source. Bad hash values are stored to guarantee the 1:1 copy. For scrubbed discs this is only true for non scrubbed sectors. USB loaders will never use WIA because of the complex structure of WIA and the needed main memory for decompression.

1.1   Some statistics

Here are some compression statistics. The tests have run on a Linux system with a AMD Athlon(tm) II X2 235e Processor, 2.7GHz. The system was almost idle.

The percent value is the compression rate against the WDF format. The size of a compressed plain ISO with an external packer is similar (a little bit larger) to the size of a WDF. The time values are the time needed for the compression (converting from WDF to WIA) and decrompression (converting from WIA to WDF). The source images are scrubbed WDF files and only data partitions are copied.

   size           creation  read   compr.    disc
  in bytes   WDF%     time  time    mode     id6     name of the game
----------------------------------------------------------------------------
  61577490  17.45%    3:14    29  LZMA.7@50  RUUP01  Animal Crossing
   6000758  23.83%      26    10  LZMA.7@50  RTYP01  Wii Chess
 221789150  33.48%    5:18    43  LZMA.7@50  RSPP01  Wii Sports
  54518891  53.62%      56    14  LZMA.7@50  RHAP01  Wii Play
  78482721  55.87%    1:25    19  LZMA.7@50  R64P01  Wii Music
2073756489  62.43%   32:19  4:37  LZMA.7@50  RLRP4F  T.Raider: Anniversary
1710692161  62.96%   25:15  3:48  LZMA.7@50  R3RP8P  S&S All-Stars Racing
 789065334  68.54%   11:30  1:46  LZMA.7@50  RZDP01  Twilight Princess
4853083788  67.06% 1:11:48 10:18  LZMA.7@50  RSBP01  Super Smash Bros. Brawl
5655712840  72.59% 1:33:02 11:36  LZMA.7@50  R3OP01  Metroid: Other M
1095283049  78.96%   16:12  2:20  LZMA.7@50  SB4P01  Super Mario Galaxy 2
2258986334  81.47%   28:33  4:27  LZMA.7@50  RMCP01  Mario Kart Wii
----------------------------------------------------------------------------
Analyzing the rates of the larger games the creation speed is about 1.6 MiB/s and the read speed is about 11 MiB/s. The creation of WIA files is slow because of decryption, hash calculations and compressing. Reading is much faster but slower than reading from a WDF because of encryption and hash calculations.

1.2   Compression rates

The follwong table shows the compression rates and the resulting size for the game Super Mario Galaxy 2 (SB4P01) compressed with different variants. The first part shows the result of the scrubbed image, the second part of the raw image. The percent value is the compression rate against the WDF format. Some image are compressed with an external packer after the conversion. The tests are done on a Linux system with an AMD quad core (2.2GHz) and 7zip used 2 CPUs for compressing.
   size               scrubbed  storing    post     total
  in bytes     WDF%    or raw    mode     compr.     time
-----------------------------------------------------------
 1163400363   73.13%  scrubbed   wia=non  rar -m5  11 min
 1282548222   80.62%  scrubbed   wia=non  7z -mx9  14 min
 1294777300   81.39%  scrubbed  wia=best           20 min
 1298114999   81.60%  scrubbed  wia=best  rar -m5  38 min
 1312530034   82.50%  scrubbed  wia=best  7z -mx9  39 min
 1558919812   97.99%  scrubbed   wia=non            56 sec
 1590904724  100.00%  scrubbed    wdf               34 sec
 1595019594  100.26%  scrubbed    wdf     rar -m5  22 min
 1595231049  100.27%  scrubbed     iso    rar -m5  25 min
 1612414604  101.35%  scrubbed    wdf     7z -mx9  15 min
 1612876679  101.38%  scrubbed     iso    7z -mx9  20 min
 4699979776  295.43%  scrubbed     iso              32 sec
-----------------------------------------------------------
 4127056954   88.92%    raw      wia=non  rar -m5  53 min
 4272920912   92.06%    raw      wia=non  7z -mx9  43 min
 4287339110   92.37%    raw     wia=best           56 min
 4298440747   92.61%    raw     wia=best  rar -m5 115 min
 4347309875   93.66%    raw     wia=best  7z -mx9  96 min
 4499644820   96.95%    raw      wia=non           161 sec
 4641376080  100.00%    raw       wdf              125 sec
 4653392026  100.26%    raw       wdf     rar -m5  67 min
 4653395984  100.26%    raw        iso    rar -m5  65 min
 4699979776  101.26%    raw        iso             112 sec
 4705693014  101.39%    raw        iso    7z -mx9  44 min
 4706027990  101.39%    raw       wdf     7z -mx9  44 min
-----------------------------------------------------------
You can see clearly, that packing an image with RAR/7ZIP doesn't help to compress the image because of the encryption. WDF is here the much faster and better solution.

But if converting to WIA the compression rates are much better. Using WIA/NONE and a post compressor gives the best packing rates, but you must unpack the image before any access. Using WIA/BEST allow you to access some parts of the disc without decompressing the whole image.

1.3   Creating and reading a WIA file

Creating a WIA file is very simple. Use the option --wia to force WIA output. Or use a filename with the extension .wia. See for »Writing a Wii disc image« for details. The options --compression and --mem controll the compression mode.

When reading a WIA file the format is detected automatically. See for »Reading a Wii disc image« for details.

2.   Compression modes

There are 5 compression methods and 5 meta compression modes. A compression mode is a combination of method, level and chunk factor.

2.1   Compression methods

When creating a WIA the data is split into chunks: Data of selected sectors (all if --raw is set, scrubbed else) are stored 1:1. The hash values are removed if they can be calculated from the source.

In a second step theses chunks will be compressed by one of the following methods:

--compression NONE
No compression is done. This format is very good for an external packer. Nearly no management data and hash values for the chunks are stored. Compression levels are not supported and the default chunk factor is 20 (chunk size 40 MiB).
--compression PURGE
Data is purged like WDF: Areas with zeros are detected and skipped. The image is smaller than 'NONE' but contains more management info. There is a SHA1 hash for each data chunk so that data errors can be noticed. Compression levels are not supported and the default chunk factor is 20 (chunk size 40 MiB). This is more an experimental mode.
--compression BZIP2
Each chunk is compressed with BZIP2. BZIP2 has its own data verification code. The default compression level is 5 and the default chunk factor is 20 (chunk size 40 MiB).
--compression LZMA
Each chunk is compressed with LZMA. LZMA has its own data verification code. The default compression level is 5 and the default chunk factor is 20 (chunk size 40 MiB).
--compression LZMA2
Each chunk is compressed with LZMA2 level 7. LZMA2 has its own data verification code. The default compression level is 5 and the default chunk factor is 20 (chunk size 40 MiB). Compression rates and times of LZMA2 are nearly equal to LZMA.

2.2   Compression level

The compression level can be changed by appending a point and a digit at the end of the compression method (e.g. bzip2.3 or lzma.7). The general default is level .5. Compression level .0 means: Use the default level for the selected method.

2.3   Compression chunk factors

The compression is done in chunks. Because of the Wii disc design the minimal chunk size is 2 MiB. The default chunk size for all compression methods is 40 MiB (factor @20). The factor can be changed by appending a '@' and the chunk size factor. Compression factor @0 means: Use the default factor for the selected method.

Examples:

2.4   Meta compression modes

There are some additional meta compressions methods. The assignment to the real modes may change in the future:
--compression FAST (BZIP2.3@10)
This is a fast compression mode.
--compression GOOD (LZMA.5@20)
This is a good compression mode. It is biased towards high compression at expense of speed.
--compression BEST (LZMA.7@50)
This is the best compression mode. The images becomes small like the external 7zip packer, but the compression needs a lot of time and very much system memory (about 800 MiB).

(Compression level 7 is the maximum supported level by LZMA)

--compression MEM
This is a generic compression mode. The tools try to detect the RAM size of the computer and to select a good compression method, level and chunk size. The user can define a good memory usage value by using the option --mem or setting the environment variable WIT_MEM.

The MEM mode can be tested with: »wit compr mem --long --long«

--compression DEFAULT (GOOD = LZMA.5@20)
This is the default compression mode. It will be used if option --compression is not set.
These meta compression modes can also combined with compression level or chunk factor to override the default values. For example: BEST.6 switches the compression level from .7 (default for BEST) to level .6.

Instead of names the user can also enter a meta compressions level between 1 (fast) and 9 (best).

2.5   Memory usage

The LZMA compression needs very much memory when creating a WIA file, much more than the BZIP2 compression. To handle the chunks and support random access the WIT tools need addionally 4.1 MiB for each chunk factor. For the default chunk factor @20 this is 84 MiB.

The following table shows the read and write memory usages for reading and writing WIA files. It is only the memory needed especcially for the (de-)compression:

»wit compr --long --long«

 real           memory usage      input
 mode         reading   writing   mode
-------------------------------------------
 NONE          40 MiB    40 MiB   NONE
 PURGE         40 MiB    40 MiB   PURGE
 BZIP2         42 MiB    44 MiB   BZIP2
 LZMA          56 MiB   225 MiB   LZMA
 LZMA2         56 MiB   225 MiB   LZMA2
-------------------------------------------
 LZMA          56 MiB   225 MiB   DEFAULT
 BZIP2.3@10    21 MiB    23 MiB   FAST
 LZMA          56 MiB   225 MiB   GOOD
 LZMA.7@50    164 MiB   773 MiB   BEST
 LZMA.6@50    132 MiB   469 MiB   MEM
-------------------------------------------
 LZMA.1@5      10 MiB    12 MiB   1
 LZMA.2@10     20 MiB    23 MiB   2
 LZMA.3@30     61 MiB    69 MiB   3
 LZMA.4@30     64 MiB    91 MiB   4
 LZMA.4@50    104 MiB   131 MiB   5
 LZMA.5@20     56 MiB   225 MiB   6
 LZMA.5@50    116 MiB   285 MiB   7
 LZMA.6@50    132 MiB   469 MiB   8
 LZMA.7@50    164 MiB   773 MiB   9
-------------------------------------------
MEM is a generic compression method. The tools try to detect the RAM size of the computer and try to selects a good compression method, level and chunk size. The user can define a good memory usage value by using the option --mem or setting the environment variable WIT_MEM.

3.   Options

3.1   --compression mode

Select one compression method, level and chunk size for new WIA files. The syntax for mode is: [method] [.level] [@factor] 'method' is the name of the method. Possible compressions method are NONE, PURGE, BZIP2, LZMA and LZMA2. There are additional keywords: DEFAULT (=LZMA.5@20), FAST (=BZIP2.3@10), GOOD (=LZMA.5@20) BEST (=LZMA.7@50), and MEM (use best mode in respect to memory limit set by --mem). Additionally the single digit modes 0 (=NONE), 1 (=fast LZMA) .. 9 (=BEST)are defined. These additional keywords may change their meanings if a new compression method is implemented. '.level' is a point followed by one digit. It defines the compression level. The special value .0 means: Use default compression level (=.5). '@factor' is a factor for the chunk size. The base size is 2 MiB. The value @0 is replaced by the default factor @20 (40 MiB). If the factor is not set but option --chunk-size is set, the factor will be calculated by using a rounded value of that option. All three parts are optional. All default values may be changed in the future. --compr is a shortcut for --compression and --wia=mode a shortcut for »--wia --compression mode«. The command »wit COMPR« prints an overview about all compression modes.
With this option the user can override the default compression method. The option expects a compression mode in the following format: method.level@factor

All 3 parts are optional but at least one part must be set and the order is predetermined.

method
This is a name or the index of the compression method. Names can be abbreviated as long as they are unambiguous.
.level
This defines the compression level between .1 and .9. The special compression level .0 means: Use the default level for the selected compression method.
@factor
This defines the compression chunk factor between @1 and @100. The chunk size is factor * 20 MiB. The special compression chunk factor @0 means: Use the default factor for the selected compression method.
--compr is a shortcut for --compression and --wia=mode a shortcut for »--wia --compression mode«.

3.2   --mem size

This option defines a memory usage limit for compressing files (in MiB if no other unit is entered). When compressing a file with method MEM (see --compression) the the compression method, level and chunk size are selected with respect to this limit. If this option is not set or the value is 0, then the environment WIT_MEM is tried to read instead. If this fails, the tool tries to find out the total memory by reading /proc/meminfo. The limit is set to 80% of the total memory minus 50 MiB.
With this option the user can set the memory usage for the compression mode MEM. An alternative is to set the environment variable WIT_MEM.