I put up a post last week about IBM's new Lotus Notes Symphony office software suite, saying that based on an article in PC World, it seemed to be sloow in loading and a significant consumer of system resources. In short, the free software had some hidden costs. Shazaam, I got a ping from IBM Analyst Relations along the lines of, mmm, a few facts are not correct and how about a briefing on the product?
Fair enough, I thought. I'm still waiting for that to occur. But in the meantime, I figured I'd download the software and try it out myself, so I could ask some intelligent questions during the briefing. At a summary level, here's what I found, when running the software on a Pentium 4 with 2 GB of memory:
- On average, an IBM Lotus Notes Symphony app (Beta 1) takes three to four times as long to load as the comparable Microsoft Office 2003 product (with some significant outliers: e.g., 15 and 33 times as long).
- An IBM Lotus Notes Symphony app (Beta 1) consumes more CPU at load time than the comparable Microsoft Office 2003 product.
- An IBM Lotus Notes Symphony app (Beta 1) consumes three to five times more memory than the comparable Microsoft Office 2003 product.
Note: I tested Symphony against Office 2003 because (1) that's what I have loaded on my desktop PC and (2) that's what the majority of information workers are currently using. Here are the specifics.
These tests were conducted on my desktop PC. It's a Dell 9100 running Microsoft Windows XP, Home Edition, Service Pack 2, with all of the latest patches applied. It's a Pentium 4 (Prescott core supporting hyperthreading) running at 3.00 GHz with 2 GB of RAM installed. Drive C (where both Symphony and Office reside) is a freshly defragged Maxtor 6Y160M0 SATA hard drive: 160 GB capacity, average seek time of 9.2 MS, 8MB of onboard cache, spinning at 7200 RPM, and with an external transfer rate of 150 MB/second. The Microsoft Office suite is Office 2003, with all the latest patches applied. The IBM Lotus Symphony suite is a beta version, rev. 20070913.1045. Timings were done with a stopwatch; I used Windows Task Manager to monitor CPU and memory consumption.
The IBM Lotus Notes Symphony installation utility said that the install would require 315 MB of disk space. It actually requires 10% more: 349 MB, which on the disk translates into 365 MB, due to the inefficiencies of block size.
The test I ran was testing the time that elapsed between (1) double-clicking on the program icon on my Windows Desktop and (2) seeing the screen quiesce with a blank document/spreadsheet/presentation open. To make sure I got a representative set of numbers I ran each test at least five times, and varied the environment: after doing some work on the PC for awhile, as well as immediately after a reboot.
Document
- Symphony Documents: Most of the time, it took an average of 12 seconds to load, with times ranging from 10 to 47 seconds. However, immediately after rebooting, it took 1 minute and 40 seconds to load; the next time, 11 seconds. This huge initial startup time must be the result of initially loading some base code that then stays resident (Symphony is built with Eclipse). At startup, CPU consumption goes into the 70-80% range, with CPU consumption peaking at around 95% for a short while. When running, Documents (Expeditorw.exe) consumes 68.2K of memory.
- Office Word: On average, taking 3 seconds to load, with times ranging from 1 to 4 seconds. At startup, CPU consumption goes to 65% and then quickly drops. When running, Word (Winword.exe) consumes 21.9K of memory.
Spreadsheet
- Symphony Spreadsheets: On average, taking 12 seconds to load, with times ranging from 8 seconds to 18 seconds. At startup, CPU consumption goes into the 75-85% range, with CPU consumption peaking at around 95% for a short while. When running, Spreadsheets (Expeditorw.exe) consumes 68.9K of memory.
- Office Excel: On average, taking 3 seconds to load, with times ranging from 2 to 5 seconds. At startup, CPU consumption goes to 75%, then to 55%, and then quickly drops. When running, Excel (Excel.exe) consumes 14.9K of memory.
Presentation
- Symphony Presentations: On average, taking 13 seconds to load, with times ranging from 10 to 15 seconds. However, immediately after rebooting, it took 37 seconds to load; the next time, 11 seconds. At startup, CPU consumption goes into the 80-90% range, with CPU consumption pinning at 100% for a short while. When running, Presentations (Expeditorw.exe) consumes 77.3K of memory.
- Office PowerPoint: On average, taking 4 seconds to load, with times ranging from 2 to 5 seconds. At startup, CPU consumption goes to 65%, then to 55%, and then quickly drops. When running, PowerPoint (Powerpnt.exe) consumes 16.3K of memory.
When the Symphony apps started up, two Eclipse tasks were spawned. Once the app was up and running, one of them went away and the remaining task (Eclipse.exe) consumed 2.8K of memory. This Eclipse task went away when the Symphony app was closed.
In all cases, Page File Usage hovered at around 1 GB and Available Physical Memory was 1.13 GB: when the system wasn't running these Symphony and Office apps, when they were starting up, and when they were running.
I have two quibbles here. First, IBM didn't set expectations correctly in the beginning. Yes, it is beta code, and IBM does say on its download page that it's working to improve the system's performance in Beta 2. However, this "hey, we're working on it" message is a backpedal from the initial press release, where the terms "beta" and "still undergoing testing" weren't even mentioned. The sentence, "Beginning today at www.ibm.com/software/lotus/symphony, business, academic, governmental and consumer users alike can download this enterprise-grade office software, which is the same tool inside some of IBM's most popular collaboration products, such as the recently released Lotus Notes 8," certainly gives me the impression that this is fully tested, generally available software.
Second, beta code is supposed to be pretty close to final--at least it was in the days when I did software development. Google, the king of the perpetual beta, has certainly cheapened the concept, but the term still implies that internal testing is coming to a close, internally generated bug counts are rapidly declining, and now it's time to test the software amongst a broader set of users and systems as a way to find those elusive bugs and quirks that didn't surface in internal testing. (Beta software is called beta because it comes after alpha testing [QA=unit and integration testing done by Quality Assurance via test suites, alpha=real world testing performed by internal employees, beta=real world testing performed by customers]). I would contend that IBM employees (whether in QA or alpha testers) must have seen the sluggish load behavior (20+ second load times are hard to miss) and should have demanded that Development speed up the software before allowing the beta. In my view, IBM is having its customers do testing that it should have done on its own.
IBM Lotus has had a long and proud history in office applications. Lotus 1-2-3 and Lotus Notes wowed people and enterprises--and rightfully so--when they were released. Unfortunately, I don't see the current version of IBM Lotus Symphony as continuing that proud tradition: rather than easing my work, it wastes my time, over consumes system resources, and thereby gets in my way. Still, it is a nice looking UI. Let's hope that once IBM wrings the bugs out that it will become a viable option for folks looking for a free, lean and mean office suite.
Comments