Do I need that other Jazz server?

I have been working with a team trying to determine if they need a second Change and Configuration Management (CCM) server to support their Rational Team Concert (RTC) deployment.  The performance of the existing CCM isn’t bad.  While the CPU utilization is low, the memory consumption is peaking a little high so some monitoring would be helpful to understand the usage patterns more.

The main motivation for considering a second CCM server is the expected growth to occur this year.  They know their current average and peak license usage, current count of licensed/registered users and how many new registered users they expect to add.  They also know the recommended range of users for a CCM based on IBM’s testing and analysis as shown in CLM Sizing Strategy.

The slight disconnect is that the user ranges we recommend are based on concurrent users not how many registered users there are or how many on average have a license in use (checked out).  Unfortunately, as yet, we don’t have a good way to measure the actual number of concurrent users.

So what do we do?  I see two ways to make this estimate.

First, the CLM Sizing Strategy states:

The number of concurrent users can often be estimated from the number of registered users. In an organization located on one continent or spanning just a few time zones, the number of concurrent users can be estimated to be between 20% and 25% of the total number of registered users. In an organization which spans the globe where registered users work in many different time zones, the number of concurrent users can be estimated to be between 10% and 15% of the total number of registered users.

What you can then do is take a percentage of the current plus expected registered users, e.g. 20-25% above when spanning few time zones, and compare that to the RTC ranges based on workload.  That will help you determine if another CCM (or more) is needed.  The percentage you apply may be different based on your experience.  In fact, in my customer case, a large number of users were registered and licensed in anticipation of them being migrated to RTC.  Applying that percentage today would give them a higher number of estimated concurrent users than was being experienced.

Another method would be to look at the average/peak number of licenses in use for RTC vs those that are licensed.  From this ratio, assuming linear growth, you can project how many licenses would be in use in the future.  Again, you would then compare that to the recommended ranges.  This method should be a little more accurate than the anecdotal, gut feel percentage of registered users in the first method.  We at least know that the number of concurrent users isn’t any more than the number of licenses in use.  Based on what you know of the users  to be added, linear growth may not be quite right and you may need to adjust the projection (e.g. the users being added are expected to be heavier users of RTC).

This latter method is what we used.  It seemed more accurate plus, as mentioned, the number of licensed users included many who weren’t onboarded yet.

Now the CLM Sizing Strategy is based on the workloads we generated on a given set of hardware.  Your workload and hardware may be such that you can handle more users (or maybe less).  I’ve had customers report that they were able to support more users than our recommended ranges.

This then shows that all the numbers and percentages need to be balanced against real experience with the environment, not only the license usage data but other things like CPU utilization, JVM size, database size, etc.  Monitor your current environment and perform trend analysis to assess if the server is nearing capacity.  Look at Monitoring: Where to Start? and Monitoring and Troubleshooting Guide for IBM Rational Collaborative Lifecycle Management (CLM).

Note that using the built in Jazz server license usage reports or the Rational License Key Server Administration and Reporting Tool, one can get an idea of license usage trends. For example, the reporting tool will show license usage by product. The recent release now includes support for authorized licenses.  Support for showing reports by role is being considered for a future release.

peakReport.jpg

As mentioned in the beginning, in this customer’s case, they weren’t experiencing any performance issues with the server.  Their CPU utilization looked fine.  There were memory usage peaks hitting 80% – 90% of the total heap so I suggested monitoring this and increase the heap as needed.

In the end, though the straight numbers and percentages suggested additional servers would be needed, this customer didn’t have good monitoring data, wasn’t experiencing performance issues and seemed to have capacity to support more users.  My recommendation was to hold off on a topology decision, put some monitoring in place, adjust the server configuration as needed/able (memory, JVM, CPU (number, speed)), assess performance/capacity of server vs expected users yet to add, if capacity/performance can’t be adjusted by manipulating the server configuration and it looks like the growth will be more than the server can handle, then add the additional CCM.

We didn’t want to take the addition of the second CCM lightly because doing so would incur another set of issues, that being the behavioral differences between a single CCM environment and a multiple CCM environment that I’ve blogged about previously and documented on the deployment wiki.  See Planning for Multiple Jazz Application Server Instances.

When assessing whether a multiple CCM topology was needed, one consideration we made was whether to mitigate some of the primary behavioral differences of concern to the customer by making one CCM be the hub for all work items and planning and the other CCMs for source control and build only.  There are tradeoffs for sure but that’s another blog discussion for another time.