Scaling Notes E-mail as a Competitive Sell

Tripp Black  July 26 2010 10:56:00 PM

Today, Ed Brill posted a blog entry about how an IBM Power mega server can save a bunch of money.
It's an analysis piece. In other words, it's not a real case here's how we did it and proved it. I may not have thought much about it until I glanced at the first couple comments.

Already, both of the scalability and what's best camps are played -- It's the Lotus paradigm's version of "Why you want a server farm" and "Why you don't want a server farm."

#1 Commentator said basically:
The numbers look right, but one massive box doesn't give you as much flexibility. Server farm is better because you have flexibility and one big box doesn't give it to you.

#2 Commentator said basically:
Yeah, but by the time you add a redundant cluster member for each mail server you've doubled your servers, traffic heat and energy and Power makes more sense. You're limited to 750-800 users per box. (Which I think is low with modern servers, even if these are "good" admins who give their users big mail mails and full-text indexes for searching.

It made me recall what some of my larger customers taught me.
You CAN have your cake and eat it too.

For a duel core machine, a good general rule-of-thumb for decent user experience is about:
1000 concurrent users on a dual core server with 3GB of memory.  
You can do 2 GB of memory if on Linux w/o a graphical interface (GUI). Same for VM if you give the Domino server a processor reservation so there isn't that short wake-up delay during slow times. Obviously what I consider 1000 concurrent users might not be exactly what someone sees as one. I'm talking 1000 users hitting the box within say the same 15-30 seconds bring up mostly Notes-based mail with some doing web-mail and some doing replication to their local mail replicas. If you are talking about a constant bombardment each second of 1000 full hits, then yeah it might drop to 800 users assuming the server is also doing other things like indexing, replicating, serving up custom apps, etc.

Back to the main topic. . .
As comment @2 said, we don't necessarily want "mirrored" clusters. I do personally like the 2 server "mirror" clustering because it's simple - I like KISS. Any new admin or consultant can walk up to a mirrored cluster and immediately "get it".. But as I was shown by a few of my larger clients over the years, clustering with 3-5 servers makes more sense when ROI and being "green" are more important. Although you still have users only on 2 of the 5 servers, it begins to act more like RAID 5 than RAID 1 by "scattering". The technique is actually simple to do. The users have one primary server and their secondaries are "scattered" on the others. Typically, the admins use policies and OU units to "scatter" the second additional/fail-over mail server. The fail-over load on each remaining cluster member isn't 50% anymore but something like 15-20%. Hopefully, all us admins have scaled our boxes to have at least 10-15% free during "rush hour" and we can handle one server coming down for maintenance or the unexpected issue. So you no longer have a server with only a 50% load -- or realistically 40% since you still need that extra 10% - wasting 60%. Add VMs with CPU reservations and you're basically doing the same thing as the Power example but for far less of a cash outlay. VMware (my preferred flavor) takes care of absorbing the rest of that wasted CPU and memory. If fact, I have a couple clients, who don't worry about that wasted 70-80% per server because they let VMware do it all. As I am more than just a VM administrator, I still want to eat cake if my primary mail server goes down though.

Assuming you still have at least 2 pieces of server hardware for the VMs, you'll still save a lot of heat and electricity on what would be lost cycles. This a the way to come close, meet. or possibly beat the Power example above except with our cheap duel and quad core IBM x eServers instead.

If the 800-1200 users are not "concurrent" users, then obviously the whole argument doesn't matter, anyway. However, in my opinion, you should still have at least two Domino servers clustered so you can "mirror" them.  But with "low" load Domino servers, you can put as many users on one box as the NIF (Notes Indexing Facility) can keep up with on the box. Realistically, on a quad core system, which is typically what I see in a non-VM environment, that's anywhere from 5000-15000 users on a box as only still about 750 - 1000 are really "concurrent" sessions. This huge range is for the differences in e-mail and activity overall. How much mail is still being delivered and sent to those other 5k to 15k users. Even if they're not an active session, their mail files are still getting copied those thousands of e-mails and being indexed (hopefully) so they can find something.

Of course we could reduce scenario quite significantly per server with DAOS. We'll see storage savings of 50-75% pretty easily with this kind of user base and can probably increase that load another 10%. (We still have the NIF.) But that leads us to another post some other day . . .

• Comments [0]

UC in the Notes Sidebar with trixbox (FreePBX)

Tripp Black  July 5 2010 07:53:31 PM

Mindwatering has used a trixbox installation for the last few years. Included with trixbox is the ARI (Asterisk Recording Interface) . It is a interface which allows a phone user to work with Voicemail, view history and recorded phone calls, toggle phone features, update the VMx Follow Me, and view system codes.

We customized the ARI to make it Notes 8.x friendly. We updated the colors and layout for a thinner more compact format. As the phone system is an open-source solution, it only seems appropriate to return the favor and publish Mindwatering's customized ARI back out to the community...

• Comments [0]

SIP Trunks with trixbox (FreePBX)

Tripp Black  June 30 2010 11:08:08 PM

One of my "side projects" that has consumed a couple evenings the last couple days is putting back together our old phone system.

Why? We had unhooked it to participate in a Beta of another phone system that had some really awesome Lotus Notes and UC integration out-of-the-box, but it hit a snag about a month ago. Unfortunately, we've have had issues with this new product and have been mostly without a working phone system since. So, with our clients complaining when we did try to use it, I was finally mandated to get something done this past weekend.  (The times we "up" were really just our main number forwarded to my cell phone - not good.)

So I decided not to turn back on my old 5 year old server that served us nicely as it's second life as a phone server over the last couple years. Instead, I decided to download and check out the new trixbox and start clean.

So Sunday night:
1.  Installed Asterisk / trixbox in a new VM. (0.5 hours)
(Yes, I know, bad, bad, but we only have 6 phones and I've dedicated 500Mhz of processor to the VM which seems to be more than trixbox needs. I also just love having all that hardware abstracted. What I lose I gain in other ways.)
2. Set up the Extensions and Ring Groups and IVR (except actually recording it). (1 hour)

Monday evening:
3. Did the Extension Manager part and issued the setup command for our phone types. (0.25 hours)
4. Pulled our old SIP phones from their boxes, dusted them, and populated the TFTP site with the firmware files from our CD backup (0.50 hours)
5. As I was missing a few files, wasted a bunch of time searching our CD archives for them, and subsequently renewing a support/access contract we let expire on the Phones. (We haven't got that yet.) (4+ hours wasted on nothing productive.)
6. Just ran with the existing old phone firmware. (They work great except our Bluetooth headsets don't work even though they're enabled in the Device configuration.) (0 hours, which is what I should have done #5 above)

This evening (Tuesday):
7. Set up the SIP trunk with the same company that was doing it with the Beta product we were helping test and evaluate. This was a 2 step process:
a. Give external IP for their SIP service, get their information for us to access, and set up the firewall forwards for our end and testing. (0.5 hours)
b. Setup the trunk in trixbox. This was an ordeal, as it was evidently "very simple". However, I've never done a SIP trunk before. So what should have taken about a half hour to an hour at most, took almost 5 in two chunks. (4.75 hours)
As we tend to document these kinds of things, I'd like to give other folks the ability to do step 7b in only 30 minutes or less  . . .

So for anyone else who wish it was Google'd here you go: How to Setup your SIP Trunk with trixbox / FreePBX / Asterisk with EtherSpeak.

I've sacrificed my cool Notes Client sidebar apps that can listen/manage voice-mails and calls, and the aforementioned wireless headset. But...
No headaches with the new old phone system.
No telling clients sorry that our phone system sucks.
And shortly, no main phone number forwarded to my cell.

Time for bed....

• Comments [0]

Notes 8.5 DominoBlog.ntf - The Missing Using Document

Tripp Black  June 22 2010 01:15:32 PM

In the last few days, we implemented the Domino Blog that comes with Notes/Domino 7, 8, and 8.5. This is the first time, because we already had blogs with and without comments for years. We just called them discussion rooms and journals in our applications. For any new blog for clients, we used the Blogsphere template on OpenNTF. However, this past week, I finally made time to create my (this blog) and Nicki started wearing a new hat and and needed another separate blog/journal. We decided to try the out-of-the-box Blog template that came with Lotus Domino.

For the first 5 minutes it was great, we pasted in a blog posting. We set up some links. Added categories.

Then went to the web to look for them. We found no categories, no links, and only tags with the standard yellow bar header at the top. We looked in the Using document (nothing there), we did a quick Wiki search which returned nothing - we didn't yet know we needed to search for specific phrases. So I proceeded to reverse engineer ...

Here we'd like to document our mistakes due to poor documentation:
I. Customizing the "Look" and header.
I found that global.css was appended to the default.css in the stylesheets folder a secondary css, Seeing that the global.css was a document, and we have the 8.5 Designer, we decided to hack/update the default.css with our customizations to make both blogs look like us.

Result:
What we should have done for what the original authors intended was to simply override in global.css. I'm not much on this as a purist, but it would have made some other things later easier had we known this was the paradigm.

II. Updating the Sidebars.
Luckily we figured out the Block templates make up a page and correctly assumed to properly adjust the layout you should use the existing block templates templates. Specifically, there are two for the left and right templates. (They are under the configuration/advanced options for the blog.)

We clicked the option in the configuration document to do categories over tags. That did not display categories instead - this is what we wanted for Nicki's blog. In fact it did nothing useful.

Mistake:
We ended up creating a manual list of her categories and using the URL to restrict to a category.

Result:
However, as new blog categories change often, this quickly became too painful for those used to Lotus Notes automation. That's when I was doing a search on tags and categories and stumbled on the Domino Wiki post on DX Tags. That lead me to Steve's blogs and the light bulb came on just I was about to just say we go back to Blogsphere or just update our own applications journal component to do comments, and bring it up-to-date to the current re-inventing of the wheel.

III. The List of what we should have done...

1. Make all overrides in "look" formatting in global.css.
a. Option 1: Do it as a document. Even though you lose the ability to edit in-line. It's a nice clean way to do an override and you don't have to remember which lines of the existing default.css are the mods.
b. If you like the color coding and formatting in Designer, ignore the document and create a stylesheet with the same name and hack the template to point there instead.
(If  I was starting over, I'd do this route.)

2. Before you starting customizing the layout of the side tables, you need to learn the paradigm.
Start with these two posts:

Steve's blog:
Steve appears to be one of the original authors. Here he defends that his template is not difficult to use, but doesn't quite get that is very difficult because you just "get" his paradigm from looking at a cluttered configuration document and basic user help on creating a post in the Notes help file.
Get past the text and the links at the bottom are VERY helpful.
a. To get started, check the link to the Wiki article on themes and tags.
The top part of the page is useful if you don't already understood the paradigm.  Since you don't:

• First read only the Default CSS paragraph. The CSS in there is used by the Page templates (which are two global for light customization) and the Blog Templates.
• Next, in your new blog application, go into the Advanced area and locate the HTML Block templates. Edit one and reformat the big blob adding so you can see the structure easily for all the DIVs in the blocks. Go back and forth between the default.css and particularly the left and right blocks to see how the CSS formats the Blocks.
• Next, go back and re-read the Wiki article and don't worry about the parts that go over your head right now,
• Did you notice those DX Tags in the left and right Block Templates. This is what sold me on the usefulness of the template (a.k.a. saved me from dumping it). Take a look at this list of the DX tags as a view and as a document in the Wiki article hidden under Attachments. Yes, a bunch are poorly documented that aren't completely obvious and they aren't document if they are used with various fields on the configuration document (some are, we've found). So the best thing to do is add them in to one of the Blocks, open the browser, and see what displays.

b. The best way to describe this is by example.

Example 1:
Here is what we did to Nicki's blog to get the Categories working including a link to each as a RSS feed using the RSS feed image:

We created a new DIV in the "LeftSideBar" block template (Advanced --> HTML Templates --> Block Templates). After we reformatted the code with carriage returns and indents to easily see what's there, we tried one of DX Tags and added this code:

<div class="list">
 <h3>Categories</h3>
 <ul><$DXCategories$></ul>
</div>

However, just that puts a weird separate RSS text link as a second link after each category. That's cool, but not what the user expects. However, to fix it ended up being a piece of cake.
Open the Configuration document (Advanced --> Configuration --> Open the document). Now choose the tabs: Formatting --> Categories --> Basics.
In the RSS Link Text/HTML field, we entered:

<img src="/NICKI/nickiblog.nsf/feed.gif" alt="RSS">

It appears we could have even abstracted the hard-coding of Nicki's filepath using the DXURL tag. But we didn't even try.

Example 2:
Here is how we got the links to work:

We created a new DIV in the "RightSideBar" block template (Advanced --> HTML Templates --> Block Templates). After we reformatted the code to see, we added the following new DIV:

<div class="list">
<h3>Links</h3>
<ul id="list">
 <li class="page_item"><$DXLinks$></li>
</ul>
</div>

After that, the links, we had already created, just displayed.

Example 3:
After we saved this blog post, we noticed our HTML rendered in the browser instead of displaying as code. We tried < code > and < blockquote > (spaces removed), but they didn't work. Another quick Google search back onto Steve's blog found the little tag:

< htmlblock >
code goes here
< /htmlblock >
(spaces removed).  

We also could have used the insert action button.


c. After we got our layout, we just went through the CSS file and tweaking it for text colors, size, spacing, and other attributes of the various sections.

To figure out what to edit, you can scan through the source and basically play a headache game of concentration. Don't do that to yourself. It will take hours.
To quickly do this, use Firefox with the Web Developer and Firebug plugins in Firefox.  It makes it so simple:
Open your page in Firefox.
Right click on the text/object to update. Then choose "Inspect This Element". This will cause a pop-up window to come up from the bottom of the Firefox window.
The right side has the CSS. Any CSS overwritten has a strikethrough line.
So basically you just look for what's left on top, note the line number and either hack default.css or better yet add the same tag in global.css as an override tag with your new values.

IV. Summary:
We made it! As you can see, the DominoBlog.ntf template's real coolness/power are the block templates and the DXTags . If we had this page of documentation to start with, we would have saved hours of reverse engineering and trial-and-error.

Since the DXTag library's PDF document is lacking in examples, you still have trial and error figuring out how to use an individual tag, but our hope is that by passing along this "loose documentation", it will allow those that follow us to not give up and be able to more readily use one the templates that comes with the product.

Also there are other great templates. We have some on our free apps page. We also have custom ones which we can customize and sell as a consulting solution. In addition, there also great templates out there on OpenNTF including Blogsphere which I mentioned earlier, which we've deployed for several clients.

 

• Comments [0]

Ubuntu Made It Almost Easy Enough to Upgrade?

Tripp Black  June 21 2010 07:25:00 PM

Today is the first time after using Ubuntu for several years that I tried to update using the Update Manager. Our scanning workstation (it's original use) has been on Hardy back when Hardy was new. We upgraded it to Lucid (10.04), and it was almost too simple.

There were only two parts of it that even required you to think:
Part I:
Starting it. To upgrade instead of update Ubuntu, you have to add a flag to the Update Manager and run from Terminal. A button on the Update Manager would be a nice improvement.
I had to:
1. Start Terminal.
2. At the prompt issue the command: update-manager --devel-release

Part II:
The next section where you had to think was the Installing Packages. Several times it stopped installing to ask what do do about all my manually updated configuration docs.
So basically, I choose the side by side comparison, I looked for the lines with the > and see what was different, and I choose to keep, merge, or overwrite my old/current configurations.

I chose to keep my SMB and NFS configs and throw away/replace my printer (CUPS) and several others that really were not so "manually" configured.

Since the upgrade obviously updated the kernel, it asked for a reboot this time. First time since I installed it. Not bad.

My only other gotcha was that we all share and access this workstation remotely over VNC. Since I only kept the SMB and NFS configs, I had to setup the remote control preferences again... oh well.

• Comments [0]

Where Has the Day Gone...

Tripp Black  June 16 2010 12:40:25 AM

Today was work all day, but not get a lot of "real" work done. More, by the time I save this, it will be tomorrow.

Part of my day was spent on Ed Brill's blog today. Once again, Ed put himself out on a limb and practically begged to be attacked by all. The post subject is what caught my eye: "I've never been a developer." Although I'm an Administrator and Instructor, it's the development I enjoy most.

I posted twice. However, my second post, is where something I already knew became clarified all the more. Most of us griping, were complaining about marketing and application platform positioning. Lotus is far from dead; it's continually evolving. Where we are seeing pains are not all growth pains but one is an older paradigm that's been partially lost. It's about Enterprise verses Workgroup collaboration, but more core, more soul.

Collaborative applications are where Notes really shines. So it's also where hurt is going to be the "most" felt.

,,, apps is where IBM, BPs, and Lotus admins have all failed us as a community. (I'm a BP, a customer, and an admin.)

How long has it been, since an admin allowed all users to create new applications (aka security field blank) on the server? Where's the start my blog or podcast on the Notes Welcome page that auto kicks off the New App command, brings up a personalization wizard screen and allows me as the user to see "it just works"? That afterwards auto updates to let me add to my new blog or podcast?

It's that shift from groupware to enterprise software. Lotus is so far down the enterprise path that we dole out breadcrumbs to the users who are just barely able to do their jobs (maybe not even). Great, I can scale to 10,000 concurrent users on a 4 procesor box with 8GB of RAM - but they're misserable. We've lost that "SuperHuman" concept which is where Apple markets well, but they still don't even have way of doing a meeting with invitees. iMail and iCal are even separate apps! As for the other goliath in Redmond, you get past a form, a view/table, and a document repository and it's done, you need a real app anyway. Who cares if it had great "themes" to pick from. Let's learn and acquire those attributes back that are lacking now.

Again, a lot of our pain is self inflicted - it's a side effect of our growth and success as "enterprise software". As an admin, I don't want my users messing up the place because they are inherently messy. SharePoint and Quickr are great for admins because they limit the sandbox mess to nice little controlled containers. How many of us really want to get dirty? As an admin, I know I don't want or have time to fix my environment messed up by users. Users hate Notes because we admins make Notes suck - most admins don't even turn on a full-text index in mail files so users can find documents. IBM is partly to blame as they've happily given us what we asked for, but some of the blame is ours. The pendulum has swung too far.

We need to keep what we've gained in scalability, performance, and Notes/Domino's great "guts". What we need to get back to is empowering the user to get work done and get out of their way. That is what we believe we need to demand, and what will get the users on "our side". Because what users want is anyone on their side.

In my opinion, we need to keep the Enterprise global focus but develop and target the actual user as an empowered individual user or small group of users. We need to do it as part of the core.

• Comments [0]

A Lotus How To on Directory Assistance

Tripp Black  June 15 2010 02:38:23 AM

Our Lotus Notes/Domino clients have the same needs.

Yes, I know, of course they do. In a general sense they need what we all need, too: a transfer of knowledge, experience, in a collaborative format.
It's what we do, it all comes down to one word, training. This is Mindwatering's core offering for my side of the "house". But in this case, this is not exactly what I mean by that...

Without knowing each other, several of our clients will ask for the same thing at the same time. For example:

• Recently, it was data "replication" to or from Lotus Notes and a relational database from several clients. For a couple, we used a product we created called the "Data Replicator"; the rest were done with simple Notes Connector LSX agents.
• For the past month, we're talking reporting and exports, but that one is not that unusual, we are consistently doing reporting add-ons to Notes/Domino apps. It's just heavier right now.
• This past week was something basic and core - Directory Assistance. This is a basic Lotus Domino integration point.
• This week topic may be SSO as I've had clients asking for it. This week it's not web SSO, but Notes Client SSO on Windows.

This interesting timing of my clients brings me back to the big picture of Training. Both of the most recent needs are something I've been wanting to get started since January - quick "How To" videos on common, but core, Lotus Domino administration topics. Videos that we can give to the Lotus community at large to jump start the Domino Admins just getting started.

Directory Access was not the first thing on my list, but it was on my list. So today Nicki and I were somewhat inspired. So, with the kid's in bed and only the noise of my servers to cause distraction in the video, I set out to put the movie together. After 30 minutes of prep, recording and QuickTime conversion, and another hour or so re-learning iMovie, we present what we intend to be a regular posting on this blog and the Mindwatering Support & Reference Site:

Full Article: Directory Assistance How To
Movie "HD" MOV/QT / "SD" MP4/M4V

• Comments [0]