Scaling Notes E-mail as a Competitive Sell

Tripp Black  July 26 2010 10:56:00 PM

Today, Ed Brill posted a blog entry about how an IBM Power mega server can save a bunch of money.
It's an analysis piece. In other words, it's not a real case here's how we did it and proved it. I may not have thought much about it until I glanced at the first couple comments.

Already, both of the scalability and what's best camps are played -- It's the Lotus paradigm's version of "Why you want a server farm" and "Why you don't want a server farm."

#1 Commentator said basically:
The numbers look right, but one massive box doesn't give you as much flexibility. Server farm is better because you have flexibility and one big box doesn't give it to you.

#2 Commentator said basically:
Yeah, but by the time you add a redundant cluster member for each mail server you've doubled your servers, traffic heat and energy and Power makes more sense. You're limited to 750-800 users per box. (Which I think is low with modern servers, even if these are "good" admins who give their users big mail mails and full-text indexes for searching.

It made me recall what some of my larger customers taught me.
You CAN have your cake and eat it too.

For a duel core machine, a good general rule-of-thumb for decent user experience is about:
1000 concurrent users on a dual core server with 3GB of memory.  
You can do 2 GB of memory if on Linux w/o a graphical interface (GUI). Same for VM if you give the Domino server a processor reservation so there isn't that short wake-up delay during slow times. Obviously what I consider 1000 concurrent users might not be exactly what someone sees as one. I'm talking 1000 users hitting the box within say the same 15-30 seconds bring up mostly Notes-based mail with some doing web-mail and some doing replication to their local mail replicas. If you are talking about a constant bombardment each second of 1000 full hits, then yeah it might drop to 800 users assuming the server is also doing other things like indexing, replicating, serving up custom apps, etc.

Back to the main topic. . .
As comment @2 said, we don't necessarily want "mirrored" clusters. I do personally like the 2 server "mirror" clustering because it's simple - I like KISS. Any new admin or consultant can walk up to a mirrored cluster and immediately "get it".. But as I was shown by a few of my larger clients over the years, clustering with 3-5 servers makes more sense when ROI and being "green" are more important. Although you still have users only on 2 of the 5 servers, it begins to act more like RAID 5 than RAID 1 by "scattering". The technique is actually simple to do. The users have one primary server and their secondaries are "scattered" on the others. Typically, the admins use policies and OU units to "scatter" the second additional/fail-over mail server. The fail-over load on each remaining cluster member isn't 50% anymore but something like 15-20%. Hopefully, all us admins have scaled our boxes to have at least 10-15% free during "rush hour" and we can handle one server coming down for maintenance or the unexpected issue. So you no longer have a server with only a 50% load -- or realistically 40% since you still need that extra 10% - wasting 60%. Add VMs with CPU reservations and you're basically doing the same thing as the Power example but for far less of a cash outlay. VMware (my preferred flavor) takes care of absorbing the rest of that wasted CPU and memory. If fact, I have a couple clients, who don't worry about that wasted 70-80% per server because they let VMware do it all. As I am more than just a VM administrator, I still want to eat cake if my primary mail server goes down though.

Assuming you still have at least 2 pieces of server hardware for the VMs, you'll still save a lot of heat and electricity on what would be lost cycles. This a the way to come close, meet. or possibly beat the Power example above except with our cheap duel and quad core IBM x eServers instead.

If the 800-1200 users are not "concurrent" users, then obviously the whole argument doesn't matter, anyway. However, in my opinion, you should still have at least two Domino servers clustered so you can "mirror" them.  But with "low" load Domino servers, you can put as many users on one box as the NIF (Notes Indexing Facility) can keep up with on the box. Realistically, on a quad core system, which is typically what I see in a non-VM environment, that's anywhere from 5000-15000 users on a box as only still about 750 - 1000 are really "concurrent" sessions. This huge range is for the differences in e-mail and activity overall. How much mail is still being delivered and sent to those other 5k to 15k users. Even if they're not an active session, their mail files are still getting copied those thousands of e-mails and being indexed (hopefully) so they can find something.

Of course we could reduce scenario quite significantly per server with DAOS. We'll see storage savings of 50-75% pretty easily with this kind of user base and can probably increase that load another 10%. (We still have the NIF.) But that leads us to another post some other day . . .

• Comments [0]