Detecting mixed use of RTC SCM clients

The IBM Continuous Engineering (CE) solution supports the notion of N-1 client compatibility. This means a v5.0.x client such as Rational Team Concert (RTC) for Eclipse IDE can connect to a v6.0.x RTC server. Customer deployments may have 100s if not 1000s of user deployments with various clients. It is not typically feasible to have them all upgraded at the same time as when a server upgrade occurs. In some cases, to upgrade the client requires a corresponding upgrade in development tooling, e.g. version of Eclipse, but doing so is not possible yet for other reasons. 

In an environment when there is a mix of RTC clients (inclusive of build clients), a data format conversion on the server is required to ensure that the data from the older clients is made compatible with the format of the newer version managed by the RTC server. Reducing these conversions is one less thing for the server to do, especially when multiplied by 100s or 1000s of clients.  Additionally, new capabilities from more recent versions often can’t be fully realized until older clients are upgraded.

There are two ways to determine the extent of how much N-1 activity is ongoing in your environment.

ICounterContentService

Enable the RepoDebug service and go to https://<host:port/context>/service/com.ibm.team.repository.service.internal.counters.ICounterContentService. From there, scroll down to the Compatible client logins table.

The sample table above was taken from a 6.0.3 RTC server. It shows that out of 75685 client logins since the server was last started, only 782 of them were from the most recent client. 70% are from 5.x clients. Ideally, these would be upgraded to the latest client.

Compatible client logins MBean

If using an enterprise monitoring application integrated with our JMX MBeans, starting in 6.0.5, the Client Compatible Logins MBean can be used to get similar details as shown in the table from ICounterContentService. The advantage of the MBean is that the login values can be tracked over time so you can assess whether efforts to reduce use of old clients are succeeding

Detecting which users (or builds) are using old clients

In order to determine which users, build servers or other integrations are using older clients, you could parse the access log for your reverse proxy. In particular, find all occurrences of versionCompatibility?clientVersion in the log.

As you can see from the sample access log, there are several instances of calls with a 5.0.2 client. The log entries have the IP address of the end client sourcing the call. If this can be associated to a particular user (or build user), you can then have someone to talk to regarding upgrading their client version.

Advertisements

Detecting RTC SCM access not using Content Caching Proxy

One of our best practices for improved Rational Team Concert (RTC) Software Configuration Management (SCM) response times and reduced load on the RTC server is to use a content caching proxy server. These are located near users at servers in remote locations where the WAN performance is poor (high latency to RTC server). What is often missed, is that we also recommend they be placed near build servers, especially with significant continuous integration volume, to improve the repeated loading of source content for building.

This practice is not enforceable. That is, SCM and build client configurations must be manually setup to use the caching proxy. This is particularly troublesome for large remote user populations or where large numbers of build servers exist, especially when not centrally managed.

The question that naturally comes is how can one detect that a caching proxy is not being used when it should? One way is to look at active services and find service calls beginning with com.ibm.team.scm and com.ibm.team.filesystem for RTC SCM operations or com.ibm.team.build and com.ibm.rational.hudson for RTC build operations.

Since the IP address of the available caching proxies are static and known, you can find any entries on the Active Services page with a Service Name of any SCM or build related service calls that are not coming from an IP address (Scenario Id) belonging to a caching proxy. Since the active services entry captures the requesting user ID (Requested By), you can then check with the offending user to understand why the proxy wasn’t used and encourage them to correct their usage.

Active services detail is also available via the Active Services JMX MBean. If an an enterprise monitoring application is being used and integrated with our JMX MBeans, then it can be configured to capture this detail, parse it and generate appropriate alerts or lists to identify when a proxy is not being used.

One other option is to parse the access log for your reverse proxy.  Shown below is sample output from an IBM HTTP Server (IHS) access log.

The access log does not have user ID information but it does have the service calls and the IP address they are coming from.  You would need to have a way to determine associate an IP address with a user machine (for those entries not coming from a caching proxy).  Note that if a load balancer is used, the IP address recorded in the access log may not be the true IP address that originated the request.  For this reason and since the user ID information is not directly available, the Active Services method may be better.

Tips for improved monitoring of your DNG environment

Let’s assume that you are convinced that application monitoring of your IBM CE/CLM environment is a good best practice. There are many JMX MBeans defined in the MBeans Reference List that your enterprise monitoring application can collect and manage. If just getting started, focus on the set of MBeans described in CLM Monitoring Primer. Once you’ve implemented the base set, you can expand from there.

Proactively monitor those MBeans by setting recommended thresholds with corresponding alert notifications to prompt further investigation. Thresholds may need to be adjusted over time based on experience. Monitor normal operations to establish appropriate baselines and adjust thresholds accordingly to reduce false negative alerts. Note that some monitoring tools have the ability to use machine learning and statistical analysis to adapt thresholds.

Based on some of my recent customer experiences, for DNG in particular, there are two key items I recommend you monitor beyond what’s called out in the primer or even currently available via an MBean. These will help you optimize the performance of your deployment.

JFS indexBacklog
Jena index updates occur when a write is being processed by DNG. The status of the index can be monitored through an indexing page (https://<server&gt;:<port>/rm/indexing). A backlog indicates there are updates yet to be passed on to the Jena indexer for processing (e.g. after a large import). Idealy the backlog of the indexer should be low on a well performing system. When high, system performance may suffer temporarily until the indexer catches up. Symptoms of heavy indexing can be slow performance, or users not seeing data immediately after creation. See technote 1662167.

Even better, for those clients using an enterprise application monitoring tool and gathering our pubished MBeans, there is one that tracks the index backlog. The JFS Index Information MBean is available as of 6.0.5 and collected by the IndexDataCollectorTask. It can be used to gather not only the size of the index but the backlog of items waiting to be indexed. By default, data is collected every 60 mins. Alerts can be set so that if the backlog gets high, e.g. over 1000, admins may choose to warn users and slow down system activity so the indexer can catch up.

DNG write journal file
Once the DNG indexer completes indexing of changes to artifacts, if there are no read operations in progress, the update is committed to the main index, otherwise, it is written to a journal file. Once the last active read operation completes, changes in the journal file are written to the main index. This approach allows in-progress read queries to maintain a consistent view of the index, while allowing new read queries to see the latest data.

The DNG write journal file (journal.jrnl) is located in server/conf/rm/indices/<id>/jfs-rdfindex. The size of the journal file can be monitored through standard OS commands and scripts or through OS integrations typically available with enterprise monitoring applications. This file will grow over time but should regularly go back to zero. In the unlikely event that it does not, it’s a sign of a bottleneck where read activity is blocking write activity. System performance may be impacted at this point. When this happens, it’s best for DNG users to pause their DNG activity while the system catches up. One customer does this by notifying their users, removing DNG from the Reverse Proxy configuration (commenting out its entry), monitoring the journal file size until it returns to zero, then adding DNG back into the proxy configuration and informing the users.

As a teaser to future content, check out 124663: As an administrator, I should be able to monitor my server using MBeans, which will provide new DNG application MBeans to further aid administrators in proactively monitoring and managing a DNG system.

More guidance on using IBM IoT CE solution with configuration management

Last year I announced several new guidance articles on use of our configuration management capability in Guidance on adopting IBM CLM configuration management across the lifecycle. Our workgroup continued and has several guidance articles in progress; we’ve just published the first in a series we expect to complete by end of year.

As the articles are published, I’ll update the list above with the proper links.  All of our guidance articles can be accessed here.

How many artifacts do I have in my Jazz application repository?

One element of sizing the servers for the IBM Continuous Engineering (CE) solution is the current and projected data scale (along with data shape, user scale and workload). There are also recommended artifacts limits to keep an application performing well, such as 200K artifacts per DNG project area (as of v6.0.5 and noted here).

Whether you are trying to project future growth based on current sizing or ensure you are staying withing recommended limits, it is useful to know how many artifacts currently exist in a repository (or other “container” such as a project area). Each application provides different means of getting this information.

DOORS Next Generation (DNG)

Vaughn Rokosz has written a very good article on the impact of data shape on DNG performance. He provides several SQL and SPARQL queries to monitor artifact counts.  I won’t repeat them here but go to the link to minimally get the queries for total number of artifacts and versions in the repository and artifacts in the project areas.

Rational Team Concert (RTC), Rational Quality Manager (RQM) and Rational Model Manager (RMM)

Since these applications share a common storage service, they have similar means to get to the artifact counts. As a Jazz Admin you can run a repotools command or a web service.

Option 1: use repotools from command line
repotools-<context>.bat -listItemStats adminUserId=<jazz admin ID> adminPassword=<jazz admin password> repositoryURL=https://<server:port>/<context>logFile=<filename>

Option 2: use web service from browser

https://<server:port>/<context>/service/com.ibm.team.repository.migration.internal.stats.IDBTableSizeHttpService/

for <context>, use ccm, qm or am for Change Configuration Management, Quality Management or Architecture Management applications.

Note that both of these options can take some time to execute so be aware of possible load put on the server. I suggest running them during lighter load times. You can first run in a test environment with production like data to get a sense of timing and load.

Sample CCM artifact counts output

 

Sample QM artifact counts output

Starting with v6.0.3, administrators can monitor Jazz application metrics through the use of JMX MBeans. One of the MBeans is Item Count Details which contains similar information as provided by the listItemStats repotools command and IDBTableSizeHttpService web service. The Item Count Details MBean, once enabled can be viewed from RepoDebug or an enterprise monitoring tool capable of receiving published JMX inputs. This is the preferred method as you can capture that data over time, see trends, set alerts and thresholds and correlate with other monitored data.

CCMMBeans

Item Count Details MBean Objects

Attachment

Attachment Item details

Monitoring Jazz Applications using JMX MBeans

I recently published a blog post on jazz.net regarding our serviceability strategy and use of JMX MBeans to monitor Jazz Applications. If you’ve heard me speak on this topic, you know that I believe that having an monitoring strategy is a best practice and essentially imperative for any deployment involving our global configuration management capability. I would even extend that to deployment of RTC clustering as well.

Have a look at the blog post here:
Monitoring Jazz Applications using JMX MBeans

Guidance on adopting IBM CLM configuration management across the lifecycle

Earlier this year, Kathryn Fryer and I formed a workgroup to share experiences we’ve had with customers adopting the IBM CLM configuration management solution. The purpose being to elicit our shared recommended practices and guidance.  The workgroup consisted of representatives from Services, Support, Development, Test, Enablement and Product Management.

When forming your Configuration Management plan we recommend it consider the following key topics:

  • End to End Process Flow (context for use of configuration management)
  • Component Strategy
  • Stream Strategy
  • Baseline Strategy
  • Change Management (including DNG change sets)
  • Cross-stream Updates
  • Reviews and Approvals
  • Traceability, Link Validity and impact analysis
  • Naming, Tagging, Custom Attribute Conventions
  • Roles and Permissions
  • Configuration aware Reporting
  • Integrations (including non-CLM)
  • Communication Plan

Our aim is to create guidance that encompasses each of these topic areas to aid you in creation of your configuration management plan.

The first such guidance is now available on jazz.net.

Our focus has been on general adoption guidance along with component and stream strategies (both are critical at the outset and go hand in hand). Next focus areas are yet to be determined but could include finishing out the stream strategies (a couple more patterns remain) baselining strategy and change management.

We value your feedback on the guidance to date and input on areas to focus on next.

If you are at the IBM Watson IoT: Continuous Engineering Summit 2017 this week in New Orleans, be sure to say hello. Check out the talk I have with Ian Zimmerman on Friday at 2:30pm (Azalea 1): CM05 – Adopting Configuration Management: What You Need to Know!