Date: Sun, 16 May 1993 14:00:10 -0500 From: tonyh@msc.cornell.edu (Tony Huang) Subject: [*] Review of eDisk and TimesTwo This is a review of the only two driver-level compression programs currently available. It also addresses a number of other compression related issues. I'll update this document when new information or new product in this category (such as Stacker for Macintosh) becomes available. I welcome any comment or suggestion, and espeicially your experience with these products. To the Moderators: please replace the file times-two.txt that I previous submitted with this one. Tony Huang tonyh@msc.cornell.edu Review of eDisk and TimesTwo ---------------------------- A little more than six months ago, Golden Triangle introduced TimesTwo, the first driver-level compression program for the Mac. It remained the only product of its kind until now. After a two-month delay, Alysis is finally shipping eDisk (To be fair, TimesTwo was delayed for over three months after the initial announcement). Despite the distinction of being the only two driver-level compression programs currently on the market, the two programs are quite different in a number of respects. The following is a review of these two programs (Comparisons with other file-level compression programs are also made where appropriate). I. Driver-level Compression vs. File-level Compression File-level compression programs for the Mac have been around for a while. They compress (and decompress) data on a file-by-file basis. They basically fall into two categories. In the first category are programs like StuffIt and DiskDoubler. With these programs, the user has to decide which file (or files) to compress and initiate the action. In the second category are programs that compress and decompress files automatically in the background. Programs such as AutoDoubler, More Disk Space, SpaceSaver belong to this second category. To operate in the background, these programs typically require the installation of a control panel in your System folder (More Disk Space patches the System file directly by installing its own resources). Because of this requirement compatibility is a problem. There is always the potential of conflict with another INIT (also known as system Extension or Startup document) or control panel with INIT code, or with the way a particular application works. These conflicts may or may not be the fault of file-level compression but the potential is there and is unavoidable at the file level. Another problem with file-level compression is the lack of total transparency while moving files from disk to disk (especially across a network). Driver-level compression solves all these problems, theoretically. Since your hard disk (or any other rewritable SCSI drive) SCSI driver operates at the lower level than the Mac operating system, compatibility should not be an issue (except with those very few programs that also try to modify the SCSI driver or access the driver directly in an unusual way). For the same reason, driver-level compression is almost totally transparent to the user. One can move the files around from disk to disk or to another Mac on the network without any worry. A disk with data compression driver can be mounted on any Mac and its files accessed without any special program. There is a price to be paid, however, for all this simplicity and elegance. Since driver-level compressor compresses ALL data written to the disk (it cannot distinguish files, by definition), user cannot pick and choose what to compress and what not to. This problem can be solved (at least partially) if the driver-level compression program works with disk partitions. II. SCSI drivers and partitions Most (but not all) SCSI driver programs (also known as disk formatters) support disk partitioning. A disk can be divided into a number of partitions, each of which is then treated by the Mac as a separate volume. For reasons of efficiency, it is generally a good idea to partition a large disk into several smaller volumes. With driver-level compression, there is another reason for disk partitioning. As I mentioned earlier, with driver-level compression one does not have the choice of what not to compress. There are many files you probably do not want compressed. If you use Virtual Memory (VM) you definitely do NOT want to compress the VM swap file (which is invisible). Files that are already compressed by another program generally should not be compressed again. There is little to gain (in disk space -- some highly compressed files may even take more disk space if compressed again by another compressor that does not recognize compressed data) but much to lose (in access time, etc.). I also put System file and Finder in this category since many of their resources are already in compressed form, and these resources are accessed frequently. If the driver-level compression program works with disk partitions, you can install the compressor on some partitions and leave the others alone. I strongly recommend leaving the start-up partition (which contains the System folder) uncompressed if the driver-level compression program supports it. At the moment, only eDisk from Alysis works with disk partitions (the upcoming Stacker for Macintosh will also support disk partitions). Golden Triangle also promised that a future version of TimesTwo would support disk partitioning. TimesTwo is also unique in another respect: it replaces your existing SCSI driver with its own (which is not too surprising considering Golden Triangle also produces the disk formatter DiskMaker). Unlike TimesTwo, eDisk (as well as Stacker) modifies and attaches itself to the existing SCSI driver. I personally prefer the second approach since it gives the user the freedom to choose his or her own SCSI driver. III. Speed and Compression Ratio Besides the program's stability (and perhaps user friendliness), the most important factors in evaluating a compression program are its speed and compression ratio. TimesTwo uses an algorithm licensed from Stac Electronics (which will release its own driver-level compression program Stacker for Macintosh) and is reasonably fast. It performed as well as any file-level compression program. eDisk uses Alysis' own algorithm and is perhaps the fastest compression program thus far (Truly objective benchmarks are nearly impossible with the current crop of SCSI benchmarking programs, however. For example, FWB HDT BenchTest give my hard disk a higher index with eDisk than without it. Decompression is exceptionally fast. On a fast Mac (a Quadra or Centris, for example), applications on a compressed partition launch just as fast as on an uncompressed partition. Compression is also very fast (unlike its sister program More Disk Space), especially when the "Delayed Write" cache is enabled (more on this later). The amount of disk space you can save with either of these two programs, as with any other data compression program, depends the type of files you put on the disk. It is not easy to find out exactly how much a particular file (or a set of files) was compressed by due to the nature of driver-level compression. On a partition full of applications with very few documents (applications are generally not as compressible as most types of documents), eDisk did a very good job and resulted in a roughly 2:1 compression ratio. TimesTwo and eDisk handle differently the amount of available disk space. Because of compression ratios vary for different types of files, TimesTwo uses an invisible file to "adjust" the amount of the space taken up by the files. For example, if a 1 MB file is compressed down to 0.6 MB the invisible file would grow by 0.1 MB (since the 1 MB file is "supposed" to occupy only 0.5 MB on a x2 disk). Instead of using an invisible file, eDisk calculates the amount of space dynamically so that the total "size" (amount in disk + amount available) of the disk (or partition) would fluctuate. This approach is obviously more elegant and more accurate. However, there is a bug in the current version. At one point during testing the reported total "size" of the eDisk (actually a partition) was less than the physical size of the partition! The problem turned out to be not serious and went away after I copied a small file onto the partition forcing eDisk to recalculate. Although both TimesTwo and eDisk achieve good compression ratio, eDisk enjoys a slight advantage here also, partly due to its unique background optimizing feature. Because files are stored differently, regular disk optimizers will not achieve the desired result on a compressed disk (even though they can be used). eDisk has its own built-in optimizing feature. During idle time eDisk optimizes the disk, and whenever possible, squeezes more space out of the disk. This feature can be turned off if you are the type that worries about excessive disk access (you should at least turn it on once in a while or use the repair function to run the optimizer in the foreground). IV. Installation and Setup Installation of either TimesTwo or eDisk is not difficult but time consuming. TimesTwo offers non-destructive installation but it only works under limited circumstances. For non-destructive installation to work, the disk must not have been partitioned since TimesTwo does not currently support disk partitioning. I have found with some SCSI drivers TimesTwo installer has trouble recognizing a non-partitioned disk as non-partitioned. As a result, the installer would insist on erasing all data on the disk (You are warned and allowed to back out the installation process, however). Non-destructive installation also does not work if the disk does not have sufficient space for the installer to work with. In any event, you are encouraged to backup your data before installation. eDisk installation is not non-destructive so you have to do a backup. On the positive side, only the partition you intend to install eDisk on is affected and the installation process is somewhat faster than TimesTwo's probably due to the fact that the installer does not have to move files around. Before the installer modifies your disk you have to choose an expansion factor (either 2, 3, or 4) for the disk. The expansion factor has no effect on how tightly data are compressed and is used only in available space calculation (see last section). To me, it is just an advertising gimmick. Once eDisk is installed you should change some of the eDisk settings (using the default settings is not recommended). First, you should disable the "Smart" eDisk feature (It does not work very well). Then set the compression speed to either fast (if your Mac is reasonably fast) or faster (if you have a slow Mac such as an SE or Classic). The most interesting feature is the "Delayed Write" cache. I recommend turning it on (I set aside 1 MB for the cache but you set it to any value between 64K to 10MB depending on how much RAM you have). It improves compression speed significantly. On the down side, if the system crashes you risk losing data still in the cache. The cache is flushed during idle time or when full. V. Driver-lever Compression and Removable Drives Because of the existence of numerous SCSI drivers and the lack of standard, auto-mounting of removable cartridges (or discs) is a challenge for a driver-level compression program. With removable media there is always the possibility of some cartridges (Here I use the term "cartridge" to mean any removable medium including MO disc) formatted with incompatible drivers. If you routinely use cartridges formatted with different drivers, the standard solution is to use SCSIProbe to manually mount the cartridge (driver loading) and close the driver on ejection (driver purging). SyQuest has been promoting this solution. However, very few SCSI drivers support this scheme at this time. As a result, auto-mounting INITs that are supplied with many SCSI formatters do not work with SCSIProbe. More specifically, these auto-mounting INITs will not reload the SCSI driver from a new cartridge if the old SCSI driver was purged, or they will use the old driver on the new cartridge if it was not (which will likely lead to a crash, at some point if not immediately, or the dreaded message saying "This is not a Macintosh disk. Do you want to initialize it"). The problem is further complicated by driver-level compression. You may want to install drive-level compression program on some cartridges but not on others. Unfortunately, neither TimesTwo nor eDisk handles the problem very well. With TimesTwo, users of removable cartridge drive would have to give up on auto-mounting of cartridges and use SCSIProbe or similar utility to mount cartridges manually. The situation is not much better with eDisk. Alysis advises against using auto-mounting INIT. Instead, a "special" version of SCSIProbe is included with eDisk (As it turns out, there is nothing special about this "special" version. It is just the latest version 3.5 of SCSIProbe altered to display the Alysis logo). However, this combination does not work very well (at least for me). Worse yet, there is apparently a bug in eDisk. If you boot your Mac without any expanded cartridge in the removable drive and later want to mount an "expanded" cartridge (i.e., one with eDisk installed) with SCSIProbe, you will get an error message saying there is insufficient memory to load the driver (I am not sure at this point if this bug affects all SCSI drivers or just some of them). There is no problem mounting cartridges this way if you had any expanded cartridge in the drive when the Mac starts up. After some experimentation, I found there is an alternative provided that you formatted all your cartridges with the same SCSI formatter (and you do not routinely use cartridges formatted by other people). With this method you can continue to use your favorite auto-mounting INIT (most likely the one bundled with the SCSI formatter). Just power on (or restart) your Mac with an expanded cartridge in the drive (you have to do this regardless which method you use if you want to use an expanded cartridge during the session). Once the eDisk driver is loaded you can swap cartridges (whether expanded or not) without any problem (assuming all cartridge are prepared with the same SCSI formatter, of course) and the cartridges will auto-mount. VI. Compatibility Issues As I mentioned earlier, incompatibility is not a big issue with well-written driver-level compression programs. The only class of programs that are potentially incompatible are other driver-level programs. Many disk security programs (such as DiskLock) belong to this category. The password feature of many SCSI drivers may not work, either. You should still be able to use the password feature on an unexpanded disk or partition, however. Alysis also warned using Public Utilities or AutoDoubler (both from Fifth Generation Systems) with eDisk. Alysis has also admitted that they have not thoroughly tested these two programs. I did not test them either because I do not have Public Utilities and there is no reason to use AutoDoubler with eDisk. Nonetheless, I suspect this is just a continuing feud (which started with AutoDoubler and More Disk Space) between the two companies. VII. Summary Driver-level compression programs are maturing and they compete favorably with any file-level compression program. The following comparison chart should help you decide which program is best for you. (The rating is on a scale of 1 through 5: 1 = poor, 5 = excellent). +---------------+---------------+--------------+--------+-------------+ | | Compatibility | Ease of use/ | Speed* | Compression | | | | Transparency | | ratio | +---------------+---------------+--------------+--------+-------------+ |AutoDoubler | 3 | 3/2 | 3 | 4 | +---------------+---------------+--------------+--------+-------------+ |eDisk | 4 | 4/5 | 5 | 4 | +---------------+---------------+--------------+--------+-------------+ |More Disk Space| 2 | 2/2 | 4** | 3 | +---------------+---------------+--------------+--------+-------------+ |SpaceSaver | 4 | 4/2 | 2 | 4 | +---------------+---------------+--------------+--------+-------------+ |TimesTwo | 4 | 4/4 | 3 | 3 | +---------------+---------------+--------------+--------+-------------+ * The numbers are weighted heavily in favor of decompression speed, since compression is usually performed in idle time and does not affect overall performance very much. This applies to all file-level compressors and eDisk with "Delayed Write" cache turned on. Only TimesTwo's overall performance is significantly affected by compression speed. ** More Disk Space has the slowest compression speed. ----------------------------------------------------------------------- Some Final Thoughts This document will be updated when new information or new product in this category (such as Stacker for Macintosh) becomes available. I welcome any comment or suggestion, and espeicially your experience with these products. Tony Huang Revision: 1.0 tonyh@msc.cornell.edu Date: 5/14/93