Developer's Blog

On Saturday, October 4, 2008, just as the fall leaves start to turn (and a mere three days from today), the lovely Miss Robin and I will be married near sunset in a little white Methodist church in the even littler town of New Milford, Pennsylvania, a pretty picture that's right out of a Norman Rockwell painting. Say hello to the soon-to-be Mr. and Mrs. Werkema :)

We have about 120 guests who will be coming from all over the world (literally!). There'll be the ceremony itself (very traditional, too!), the reception, music, a sit-down dinner, dancing, and an open bar, so it's gonna be one heckuva party.

On the night of the reception, she and I will disappear around ten o'clock, to hop into our limousine to the Philadelphia International Airport to catch our early-morning flight to the sunny warm aqua waters and fresh winds of the Aloha State.

Me, I'm incredibly happy and terrified out of my knickers at the same time. I've never been good in front of large crowds, and I'm a lot more comfortable behind the scenes than being the center of attention. But I really, really want to be married to Robin; she's the best thing that's ever happened to me, and if that means I have to spend a day scared [bleep]less, I'll do it in a heartbeat. Robin, I love you, and I can't wait to see you walking down the aisle on Saturday.

I thought it might be beneficial to provide a list of some of the free software I use. (I use a lot of commercial software too, and I'll go into that list in next week's posting; and after that, I'll cover free web services and scripts.) Go download this stuff, folks; it's great and won't cost you a dime!

• Image viewing: IrfanView. This thing is the jack-of-all-trades swiss-army-knife of image viewers, and if you're using anything else to view images on your computer, you don't know what you're missing.
• Power tools: Cygwin. Windows sucks. Un*x sucks less. Cygwin brings the full suite of GNU and Un*x tools to the Windows command-line, and is the only reason I don't pull my hair out daily when dealing with Windows. If you're a serious power user or serious programmer, get Cygwin and learn how to use it.
• Web browsing: Firefox. IE is old and moldy and getting moldier, while Firefox just keeps getting better. It's fast; it's stable; it's reliable; it's extensible; and it has infinitely fewer security holes. While I do like Opera, Firefox wins out for its addon capabilities.
• E-mail: Thunderbird. If you're still using Outlook Express, ditch it in favor of a good e-mail client Thunderbird handles most of my important mail; and while I also have a Yahoo! Mail account for less important stuff and my fiancée has a GMail account for her stuff, I still use Thunderbird for the bulk of my business e-mail.
• Music playing: WinAmp v5. It's getting a little long in the tooth these days, but it's still fairly lightweight and very reliable.
• Telnet and SSH: PuTTY. Don't bother with the built-in "telnet" command in Windows; use a real telnet client. PuTTY does SSH, too, and is nicely configurable.
• File transfers: WinSCP. Forget FTP; this uses a secure connection to ensure that your data doesn't get snooped while you're sending it, and it's a lot more flexible too.
• Password storage: Password Safe. Cryptography expert Bruce Schneier wrote the original code behind this, so you know that when you put your passwords into it, not even the military can get them back out again without your master password.
• File safety: TrueCrypt. This creates encrypted "folders" on your computer that are safe from prying eyes. It's open source, and uses the AES algorithm, which means that any files you put in there are for your eyes only. Great for storing sensitive data (like company documents).
• Instant messaging: Trillian. Trillian isn't open source like a lot of the rest of these, but it's fast and flexible, and I've been using it for years. If you're still suffering through AIM and its noise and ads, get Trillian: You'll never go back.
• BitTorrent: Azureus. There are plenty of legitimate torrent files out there (like Linux distros!), and if you're gonna download them, you should use a decent BitTorrent client to do it. I've used several BT clients, but none have worked as nicely and reliably as this little blue frog has.
• Remote file sharing: Hamachi. If you find yourself regularly needing to share data between two computers on opposite sides of the planet, you're probably a good candidate for a Virtual Private Network (VPN). Hamachi provides VPNs for small-scale use for free, and it's so numbingly easy to use that you'll wonder why it doesn't ask you more questions.
• Remote computer use: TightVNC. For a lot of people, Windows XP's built-in "Remote Desktop" feature works very nicely, but there are cases where it doesn't work so well: For example, if you need remote access to an XP Home computer, or you're mixing and matching Windows, MacOS, and Linux machines for the server and client. For that, VNC is the best way to go, and I haven't found a better VNC than TightVNC, with its high-speed transfers and neat multi-connect features.
• Office software: OpenOffice.org. Microsoft Office or WordPerfect Office might be decent enough packages, but if you want a good word processor and spreadsheet and don't feel like shelling out hundreds of bucks, OpenOffice is hard to beat. It has about 90% of what you'd expect from Microsoft Office, and it's free. Is that last 10% worth the other $400? Not for me, it isn't.
• Spell checking: tinySpell. It lives in your system tray and tells you if you misspell things while typing. Simple, sweet, unobtrusive, and elegant, the best kind of software.
• Hex editor: frhed. Everybody should have a hex editor on their computer somewhere (don't ask why; just do it), and frhed is one of the best I've found. I've used it far more often than I'd like to admit.
• Graphing and charting: Graphviz. This one's for the geek in all of us: It takes conceptual descriptions of graphs, trees, DAGs, and similar constructs and produces nice, neat, graphical renderings of them. I don't use it often, but I use it every chance I get :-)
• Installers: Inno Setup. I once used NSIS, but despite the claims that it didn't suck, it sucked. Microsoft's installer is huge and doesn't work on old OSes. Others cost tons of money. Inno Setup is a decent package, nicely scriptable, and very configurable: It's what I finally used for the release builds of SpaceMonger, and I've been glad I switched to it.
• Anti-spyware: Ad-Aware SE and SpyBot: Search and Destroy. These are the best way to keep yourself free from spyware, adware, and other uglies that you don't want in your system. Don't pay for spyware removal --- get the good free software and do it for nothing!
• CD ripping: CDex. There are lots of packages to rip CDs into MP3 files, but none do it as nicely as this open-source one does. It's fast, it's flexible, it has great error-correction code, and it's ripped every CD that I own :-)
• Movie playing: Zoom Player Standard. I've been using Zoom Player for a while now for playing AVI and MPEG files, and it beats the pants off Windows Media Player in more ways than I can count. (For one thing, it correctly supports multi-monitor computers, which WMP doesn't.) Zoom Player is light, it's fast, it's nice, and it doesn't get in the way of enjoying your movies.
• File compression: 7zip. If you're gonna do point-and-click file compression, this is the best way to do it. 7zip is powerful, it's free, and it handles just about every file format on the planet. Forget WinZip and WinRAR and the rest: 7zip's the one you want.
• Tweaking: TweakUI. Microsoft's own toy for changing all the settings of your system. It's dangerous, but it's powerful, and it should've been installed on every computer running Windows, but since Microsoft didn't include it, you need to download it. Don't ask. Get it.
• Add/Remove Programs: Safarp. The standard Add/Remove Programs thingy in the Windows Control Panel is slow and irritating and glitchy, and that's on a good day. "Safarp" stands for "Simple And Fast Add/Remove Programs" and it lives up to its name: It's simple, it's fast, and it works. With this, you can be done uninstalling an irritating program long before the standard Add/Remove Programs window has even finished "Populating." Get it; you won't be sorry you did.
• File synchronization: TreeComp. If you regularly carry data around between data sticks and laptops and a desktop computer or three, you may often find yourself trying to guess which one has the newest version of something. TreeComp can help: It can keep any two sources synchronized with the newest versions of files, and it lets you make the final decision over which file to keep. It's simple and fast and does the job well.

There's other good freeware out there, and even other freeware on my computer, but this is the stuff I use regularly. What do you use? Feel free to comment.

It's been a long time since I last updated this blog, and there hasn't been all that much to say. Sales of SpaceMonger have been good; certainly not enough to retire on, but enough that I can call this my day job (mostly; a little side consulting sure hasn't hurt).

Those of you who've been following this have probably been wondering what's coming with SpaceMonger. The official answer is that version 2.2 is going to be the next one out the door, and betas of it will be showing up as soon as I can finish the code of its new scanner. I'm replacing a lot of the code --- and a lot of the general model behind the code --- in v2.2's scanning engine, mainly because the old engine is slow, unwieldy, and identifiably the cause of nearly every crash you might have ever experienced in SM. I haven't been paying too close attention to the bug reports over the last few months, because they're all pointing at the same code, and it's code that's going bye-bye (actually, it's already gone).

I've prepared a presentation describing exactly what's changing in SM for those of you who are technically inclined. If you want to read it, it's in the longer version of this post, and covers the history of SM's scanner in detail as well as explaining what the new design is. That said, it's not for the programming-illiterate, so if you don't grok threads and databases and message-passing models, you might want to skip it.

I'm not exactly sure when v2.2 is going to be finished, simply because of all that's changing in the scanner. Right now, the code is in enough disarray that attempting to scan something doesn't fail, but it doesn't display anything either --- not terribly useful.

That said, the rest of the new features for v2.2 are implemented already (and were before I got going on the scanner), and a short list of them is added below for those who haven't been following the forums or testing the betas. They include, among others:

• Image previews! When you select an image file (JPEG, PNG, GIF, BMP, or Targa), you can see it in the left-side panel.
• More configuration options, such as the ability to reposition the scrollbar arrows.
• Full Vista compatibility, including with Vista's soft-links (symbolic links).
• Scanfiles: The ability to save a scan and reload it, or even to load scanned data from an external source!
• And, of course, lots of bug fixes, and hopefully some performance improvements too.

It's been nearly two months since the last update, and people have been wondering (again) what's going on with the dev work.

I've been doing my best on v2.2 to get the new engine where it should be, and, honestly, that's about it. The new engine uses a radically different communication and storage model than the old one did. Unlike during the development of SM v2.1, SM v2.2 has always been in a "compilable and running" stage, but until all the bits and pieces of the new engine are put together the way they should be, you, the users, won't really see any measurable difference despite all the work involved. When all the necessary new bits and pieces are together, the thing's gonna run quite a bit faster and more efficiently, but that doesn't happen without the internals reaching a "critical mass" of new additions, and it isn't there yet. The new storage model works beautifully, I'm happy to say, and is a little faster and cleaner than the old one, but it hasn't shown the massive improvement I expect when all the parts finally go together.

I realize that there are features in v2.1.2-pre2 that really need to make it out to the light of day soon. Vista's out, and the fixes that allow SM to run cleanly on Vista need to be formally released; likewise, things like image previews and better config controls are overdue for getting out there in a formal release too. Those of you running a beta shouldn't be running a beta forever, right?

But, still, it's hard to want to do a big release when one of the most critical new features --- that engine --- isn't ready yet. I'm going to continue working on it, and if we get down to like March-ish and I'm still working on it, then I'll seriously look at releasing v2.2 without the performance gains. v2.2 will still have a lot of the new code and all the new features; it just won't be running significantly faster than v2.1.1 because of the lack of the "critical mass" in the internals. (On the plus side, it is likely to be a bit more stable than v2.1.x was, since the new storage model should eliminate a lot of the crash problems associated with the old array storage model.)

So, well, there ya have it, I guess. I'd like to say more, but I don't have that much more to say. :-/

--- and she said yes.

Best. Christmas. Ever.

And it's October 2008, for those who are curious.

:-D

Some have wondered (and rightly so) what's been going on with SpaceMonger over the past month, and now that the hectic craziness of Robin's graduation is done (congratulations again!), I finally have some time to talk about it.

I have to confess that I didn't do that much with the code over November; I was mainly in maintenance mode, fixing a few bugs here and there, and spending a lot of time fielding e-mails from customers and potential customers. As important as upgrades and bug fixes are, customer service is a higher priority for me; I don't want it to ever be said that I wasn't paying attention to the people who are responsible for my income. (And if you were expecting a reply from me and haven't gotten one, please let me know that you didn't get it so that I can give you the attention you deserve.)

But lest you think otherwise, there has been quite a bit of work done on SM since its v2.1.1 release back in September, so if you want to see what's new and haven't yet, go try one of the 2.1.2 betas.

Since the start of December, however, I've been working on rewriting the internal storage engine. This is a project I've been turning over in my head since around March, but I wasn't able to make any decent inroads into the problem until a recent spurt of inspiration (And whoever says that programming is purely engineering should be shot: This field is more an art form than it is a science.) Since then, I've been rebuilding the core storage engine, and it's about two-thirds complete now. The new engine is noticeably faster than the old one, and it uses much simpler code but much more complicated data structures. Overall, it's a far more elegant solution to the problem.

(What is the storage engine, you ask? SpaceMonger's core is broken down into four major pieces: The scanner, the storage engine, the treemapper, and the hupper. The scanner is responsible for scanning your disk and collecting the data it finds there into RAM, as quickly as possible. The storage engine takes the results from the scanner and builds it up into a nice in-memory database with a variety of aggregate statistics attached, like what percentage of files are between 2K and 4K; the storage engine also maintains the current file selection. The treemapper takes the data in the storage engine and generates treemaps from it, and renders those treemaps to the screen. The hupper watches your disk for changes, and tells either the scanner or the storage engine to update things when changes happen.)

So why a new engine? The answer is that the old storage engine was never all that good. It used a set of data structures that were simple but inefficient, and that couldn't handle certain edge cases. The storage engine's tree-of-arrays design has been conceptually consistent since SpaceMonger v1.0 --- but it hasn't scaled well. The new demands of the software and the users have made it increasingly brittle. This is why a handful of you have experienced crashes when doing very large deletions, or slowdowns when you try to install programs or copy things while SM is running. It wasn't a great design, but it was a working design, and so I've been loathe to change it since I built it, despite the bugs.

Well, loathe to change a working design, true, but it's also one of the innermost components of the software, and changing it is akin to trying to change a car's engine from a V4 to a V8 while the car is doing 70 MPH down the interstate. It's not a task you want to attempt lightly.

But I'm pleased to say that in a burst of inspiration, I figured out a way to do it, and the new engine is about two-thirds complete now. It's already faster than the old one, and if nothing else, it theoretically should be able to scale --- whereas the old design theoretically shouldn't have been able to do as much as it has. I probably wouldn't ever make the old design 100% stable, but this one at least has a good chance of being that eventually. It comes at a small cost in memory (~20 bytes per file), but it runs a lot faster, updates faster, and is generally elegant instead of the brute-force methods I used before.

The plan is that I'm going to try to get the whole of it finished off around the turn of the year, and release that as another beta. Assuming it doesn't blow up or self-destruct when you folks start throwing multi-terabyte disks at it, I'll ask the translators to produce the updated foreign-language translations, and then release the complete set of it as SpaceMonger v2.2. Yes, we're going up to v2.2 and not v2.1.2 because enough has changed since v2.1.1 that the software warrants a new version number. There are a lot of new features, and some major improvements to the internals, so it's worth it.

So there ya have it. v2.2 coming soon, and I think you're all gonna like the result.