Synopsis: My experiences with compression of in-memory MUD data. June 98 From erwin@andreasen.com Tue Jun 30 21:16:46 1998 Date: Tue, 30 Jun 1998 21:16:41 +0200 (MET DST) From: "Erwin S. Andreasen" X-Sender: erwin@shire.shire.dk Reply-To: "Erwin S. Andreasen" To: Zimmer cc: Rom mailing list Subject: Re: just an idea.. In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Status: O X-Status: On Wed, 1 Jul 1998, Zimmer wrote: > Was toying with the idea of changing area files to save (at least) room > descriptions in a compressed format and uncompress them when someone > actually wants to read them (tends to be rare imo). Just seems that muds > seem to grow to a disgusting size memory wise in a short time, with or > without players, and this would be the first place I'd like to cut it > down. I don't want to remove the option for people to read them tho, for a > start all the builders (myself included) would revolt. I've tried it, and it was not very sucessfully. Using LZO, a very fast and light compressor, I think I gained something like 10-20% - which was just about .5mb - that's 1-2% which was not worth it. LZO is very fast however - I think the light compression algorithm decompresses at half the speed of a straight copy of data. zlib is more promising - you can tell it to use a preset-dictionary. I wrote some perl scripts to extract all room descriptions and generate what words were used the most. That gave 50% I think. It could have given more, since I used only words which was a mistake - I should have searched for strings (e.g. 'The ' - with space! is probably more common than 'the'). However, the problem with that was that I could not find a way to keep a preset dictionary from session to session - so each time something was compressed or uncompressed, it had to rebuild the dictionary. You might figure it out if you dwelve more into the source. You can find both Markus Obenheimer's LZO and zlib (the core of gzip) with any net search. The approach I took was BTW, to mark compressed descriptions as starting with a magic character - and whenever showing the room description, basically something like decompress(&room->description) was called which would decompress and replace the compressed description with a decompressed one (so frequently visited areas would not require constant decompression). Both libs are pretty easy to use - the code for direct integration was 40-50 lines of code AFAIR. RAM is cheap (unless you are renting it; I think most mud hosting companies running on x86 machines charge typically something like half the price it would cost to buy the RAM each month :) 64mb EDO costs what, $60-70? ============================================================================== Herlev, Denmark UNIX System Programmer <*> (not speaking for) DDE ==============================================================================