Subj : file size issues? To : Mike Tripp From : Roy J. Tellason Date : Mon Aug 02 2004 05:07 am Mike Tripp wrote in a message to Roy J. Tellason: MT> Hello Roy! MT> 01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp: RT> I'm seeing the same numbers in both cases there because I have no RT> settings in the area. MT> Lack of settings alone won't keep the numbers static. If you MT> manually kill one message and SQPACK again, Old should be larger MT> than New. If you manually edit one existing message to remove a few MT> lines from the message, Old should be larger than New. Sure. RT> Right. But to get to that point I had to *move* the remaining RT> messages I wanted to keep from one area to another. MT> Gotcha...and obviously the stats you're posting are for the new MT> version of the area and not the original hosed version. Right. That one's gone. Old habit, getting rid of stuff like that, from when disk space was a scarcer resource. MT> It is the hosed version that should've been shrinking as messages MT> were moved out of it. No, the hosed version wouldn't have been shrinking as sqpack wouldn't touch it... Too bad my "pack.log" gets rewritten every day, or I'd pull a line out of that one to show it. MT> The new version was only growing to swallow the new messages being MT> appended, so there's no reason for them to be occupying more than MT> the actual number of bytes required to do so. Yup. And the way things looked it wasn't ever gonna get any smaller. I'm looking at something like 38M currently on that drive, not so much that I need to worry about an 8M file eating too much space but still... RT> All with today's date, and a timestamp within the past hour when I RT> was reading it last. What's that sqi file used for anyhow? MT> Those that've been into the source up to their elbows can probably MT> find plenty of holes with my analogy, but if you understand the MT> basics of how DOS manages files and space allocated on a FAT hard MT> drive: the .SQI is the FAT table for the files (messages) within MT> Squish's hard drive (.SQD). Ok. MT> The .SQI basically has the pointers to where the header for each MT> message starts and how many blocks it occupies. If you delete a MT> message, Squish just updates the .SQI file that references the MT> deleted data to remove the pointer to its header and updates it's MT> usage map to show that space as available for writing new data MT> again. The size of the .SQD is not necessarily representative of MT> the actual number of bytes required by the currently retained MT> messages. Squish will expand the .SQD if more space is needed to MT> accomodate a new retention peak, but it will not shrink it just MT> because fewer are required today than yesterday. Makes sense. This also explains some why TimED's undelete function seems to grab *all* deleted messages, I guess that was just simpler to code, or something. MT> Just as bad things can happen to a FAT if operations fail or are MT> interrupted by a system lockup or power failure, the same is true MT> for Squish and .SQI/.SQD manipulations. SQFIX does for the MT> Squishbase what CHKDSK and/or Disk Doctor does for the integrity of MT> the FAT. It looks to make sure that a message header actually MT> starts where each .SQI entry says it does and looks for space that MT> the .SQI says should be in use by a retained message but actually MT> isn't (akin to lost clusters and cross-linked files) and attempts MT> to either fix or remove .SQI entries that don't seem to match MT> reality found in the .SQD. MT> SQPACK is analagous to DEFRAG. If you have specified parameters, MT> it first marks the messages that fail those parameters as deleted MT> and then "compresses" the .SQD data to the beginning of the .SQD MT> file and shrinks the .SQD file down to the actual size required to MT> store the actual bytes associated with those messages that are MT> still retained, rather than the size being some former worst-case MT> maximum that was required to store messages that met the SQSET MT> parameters at some point in the past since you last SQPACKed. It also seems to write the results out to a new file, so for example if a single message area is larger than the available space on disk, it will *never* get packed down. I have one such area at present, the .sqd file is something around 66 megs. This was from an internet mailing list I was carrying for a while, back when I had that capability on the system, and I figured that I'd just delete stuff as I read 'em. I somehow haven't found the time to mess with that one lately, and if I don't eventually I may end up just deleting it. MT> So if you use no purge parameters, never manually delete/edit a MT> message in an area, and never have any problem that causes the .SQI MT> to lose synch with the actual contents of the .SQD, it will appear MT> that SQPACK never does anything...just as running DEFRAG wouldn't MT> accomplish anything for your hard drive if you only appended new MT> files one at a time and never deleted or modified an existing file MT> since your last DEFRAG. If, however, you make any of those changes MT> or SQFIX frees a pointer to an existing message because of a data MT> integrity issue, it is possible/probable that there will be some MT> delta between the old/new bytes when you SQPACK next, even if you MT> haven't assigned specific purge criteria to the area. There are only a few areas like that here, most are set for specified time frames, typically 60 days. And there do seem to be times when random errors creep in, I don't know if it's memory glitches, power surges, or what, though for the most part things are pretty quiet around here in the small hours when maintenance is done. Every so often I'll scan my pack.log file for "err" and find some, like that one mentioned above, but mostly it won't find it, and I'll figure things are okay at that point as there's plenty of areas and they all get packed down each night. --- * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)