Getting to a right-sized Engineering Lifecycle Management environment – updated

The following is a refresh of a previous post in 2016.

You’ve just made the decision to adopt one of the Jazz solutions from IBM.  Of course, being the conscientious and proactive IT professional that you are, you want to ensure that you deploy the solution to an environment that is performant and scalable.  Undoubtedly you begin scouring the IBM Knowledge Center and the latest System Requirements.  You’ll find some help and guidance on Deployment and installation planning and even a reference to advanced information on the Deployment wiki.  Unlike the incongruous electric vehicle charging station in a no parking zone, you are looking for definitive guidance but come away scratching your head still unsure of how many servers are needed and how big they should be.

This is a common question I am often asked, especially lately.  I’ve been advising customers in this regard for several years now and thought it would be good to start capturing some of my thoughts.  As much as we’d like it to be a cut and dried process, it’s not.  This is an art not a science.

My aim here is to capture my thought process and some of the questions I ask and references I use to arrive at a recommendation.  Additionally, I’ll add in some useful tips and best practices.

I find that the topology and sizing recommendations are similar regardless of whether the server is to be physical or virtual, on-prem or in the cloud, managed or otherwise.  These impact other aspects of your deployment architecture to be sure, but generally not the number of servers to include in your deployment or their size. One exception is that managed cloud environments often start lower than the recommended target since those managing understand how to monitor the environment, look for indicators that more resources are needed and can quickly respond to increasing demands.

From the outset, let me say that no matter what recommendation I or one of my colleagues gives you, it’s only a point in time recommendation based on the limited information given, the fidelity of which will increase over time.  You must monitor your Jazz solution environment.  In this way you can watch for trends to know when a given server is at capacity and needs to scale by increasing system resources, changing the distribution of applications in the topology and/or adding a new server.  See Deployment Monitoring for some initial guidance.  Since 6.0.3, we have added capabilities to monitor Jazz applications using JMX MBeans. Enterprise monitoring is a critical practice to include in your deployment strategy.

Before we even talk about how many servers and their size, the other standard recommendation is to ensure you have a strategy for keeping the Public URI stable which maximizes your flexibility in changing your topology.  We’ve also spent a lot of time deriving standard topologies based on our knowledge of the solution, functional and performance testing, and our experience with customers.  Those topologies show a range in number of servers included.  The departmental topology is useful for a small proof of concept or sandbox environment for developing your processes and procedures and required configuration and customization.  For most production environments, a distributed enterprise topology is needed.

The tricky part is that the enterprise topology specifies a minimum of 8 servers to host just the Jazz-based applications, not counting the Reverse Proxy Server, Database Server, License Server, Directory Server or any of the servers required for non-Jazz applications (IBM or 3rd Party).  For ‘large’ deployments of 1000 users or more that seems reasonable.  What about smaller deployments of 100, 200, 300, etc. users?  Clearly 8+ servers is overkill and will be a deterrent to standing up an environment.  This is where some of the ‘art’ comes in.  I find more often than not, I am recommending a topology that is some where between the department and enterprise topologies.  In some cases, a federated topology is needed when a deployment has separate and independent Jazz instances but needs to provide a common view from a reporting perspective and/or for global configurations, in case of a product line strategy. The driving need for separate instances could be isolation, sizing, reduced exposure to failures, organizational boundaries, merger/acquisition, customer/supplier separation, etc.

The other part of the ‘art’ is recommending the sizing for a given server.  Here I make extensive use of all the performance testing that has been done, including the following.

• CLM Sizing Strategy for the 6.0 release
• Performance test results for Rational DOORS Next Generation 6.0.6 ifix3 with Components
• Rational DOORS Next Generation: Organizing requirements for best performance
• Collaborative Lifecycle Management performance report: Rational Quality Manager 6.0.6.1
• CLM 6.0 Performance Report for Global Configuration and link indexing services
• Rational DOORS 9 to Rational DOORS Next Generation migration sizing guide
• Deployment Topologies forJazz Reporting Service
• Performance summary and guidance for the Data Collection Component in Rational Reporting for Development Intelligence
• Best Practices for Configuring LQE For Performance and Scalability
• Performance and Data Scalability of Rational Rhapsody Design Manager

The CLM Sizing Strategy provides a comfortable range of concurrent users that a given Jazz application can support on a given sized server for a given workload.  Should your range of users be higher or lower, your server be bigger or smaller or your workload be more or less demanding, then you can expect your range to be different or to need a different sizing.  In other words, judge your sizing or expected range of users up or down based on how closely you match the test environment and workload used to produce the CLM Sizing Strategy.  Concurrent use can come from direct use by the Jazz users but also 3rd party integrations as well as build systems and scripts.  All such usage drives load so be sure to factor that into the sizing.  There are other factors such as isolating one group of users and projects from another, that would motivate you to have separate servers even if all those users could be supported on a single server.

Should your expected number of concurrent users be beyond the range for a given application, you’ll likely need an additional application server of that type.  For example, the CLM Sizing Strategy indicates a comfortable range of 400-600 concurrent users on a CCM (Engineering Workflow Management) server if just being used for work items (tracking and planning functions).  If you expect to have 900 concurrent users, it’s a reasonable assumption that you’ll need two CCM servers.  Scaling a Jazz application to support higher loads involves adding an additional server, which the Jazz architecture easily supports through multi-server or clustering topology patterns.  Be aware though that there are some behavioral differences and limitations when working with the multi-server (not clustered) pattern.  See Planning for multiple Jazz application server instances and its related topic links to get a sense of considerations to be aware of up front as you define your topology and supporting usage models.  As of 6.0.6.1, application clustering is only available with the CCM application.

This post doesn’t address other servers likely needed in your topology such as a Reverse Proxy, Jazz Authorization Server (which can be clustered), Content Caching Proxy and License Key Server Administration and Reporting tool.  Be sure to read up on those so you understand when/how they should be incorporated into your topology.  Additionally, many of the performance and sizing references I listed earlier include recommendations for various JVM and application settings.  Review those and others included in the complete set of Performance Datasheets and Sizing Guidelines.  It isn’t just critical to get the server sizing right but the JVM and application properly tuned.

To get to the crux of the primary question of number of servers and their size, I ask a number of questions.  Here’s a quick checklist of them.

1. What Jazz applications are you deploying?
2. What other IBM or 3rd party tools are you integrating with your Jazz applications?
3. How many total and concurrent users by role and geography are you targeting and expect to have initially?  What is the projected adoption rate?
4. What is the average latency from each of the remote locations?
5. How much data (number of artifacts by domain) are you migrating into the environment? What is the projected growth rate?
6. If adopting IBM Engineering Workflow Management, which capabilities will you be using (tracking and planning, SCM, build)?
7. What is your build strategy? frequency/volume?
8. Do you have any hard boundaries needed between groups of users, e.g. organizational, customer/supplier, etc. such that these groups should be separated onto distinct servers?
9. Do you anticipate adopting the global or local configuration management capability?
10. Do you make significant use of resource-intensive scenarios?
11. What are your reporting needs? document generation vs. ad hoc? frequency? volume/size?

Most of these questions primarily allow me to get a sense of what applications are needed and what could contribute to load on the servers.  This helps me determine whether the sizing guidance from the previously mentioned performance reports need to be judged higher or lower and how many servers to recommend.  Other uses are to determine if some optimization strategies are needed.

As you answer these questions, document them and revisit them periodically to determine if the original assumptions, that led to a given recommended topology and size, have changed and thus necessitate a change in the deployment architecture.  Validate them too with a cohesive monitoring strategy to determine if the environment usage is growing slower/faster than expected or detect if a server is nearing capacity.  Another good best practice is to create a suite of tests to establish a baseline of response times for common day to day scenarios from each primary location.  As you make changes in the environment, e.g. server hardware, memory or cores, software versions, network optimizations, etc., rerun the tests to check the effect of the changes.  How you construct the tests can be as simple as a manual run of a scenario and a tool to monitor and measure network activity (e.g. Firebug).  Alternatively, you can automate the tests using a performance testing tool.  Our performance testing team has begun to capture their practices and strategies in a series of articles starting with Creating a performance simulation for Rational Team Concert using Rational Performance Tester.

In closing, the kind of guidance I’ve talked about often comes out in the context of a larger discussion which looks at the technical deployment architecture in a more wholistic perspective, taking into account several of the non-functional requirements for a deployment.  This discussion is typically in the form of a Deployment Workshop and covers many of the deployment best practices captured on the Deployment wiki.  These non-functional requirements can impact your topology and deployment strategy.  Take advantage of the resources on the wiki or engage IBM to conduct one of these workshops.

Getting to a right-sized Jazz environment

You’ve just made the decision to adopt one of the Jazz solutions from IBM.  Of course, being the conscientious and proactive IT professional that you are, you want to ensure that you deploy the solution to an environment that is performant and scalable.  Undoubtedly you begin scouring the IBM Knowledge Center and the latest System Requirements.  You’ll find some help and guidance on Deployment and Installation and even a reference to advanced information on the Deployment wiki.  Unlike the incongruous electric vehicle charging station in a no parking zone, you are looking for definitive guidance but come away scratching your head still unsure of how many servers are needed and how big they should be.

This is a common question I am often asked, especially lately.  I’ve been advising customers in this regard for several years now and thought it would be good to start capturing some of my thoughts.  As much as we’d like it to be a cut and dried process, it’s not.  This is an art not a science.

My aim here is to capture my thought process and some of the questions I ask and references I use to arrive at a recommendation.  Additionally, I’ll add in some useful tips and best practices.  If this proves useful, it will eventually move over to the Deployment wiki.

I find that the topology and sizing recommendations are similar regardless of whether the server is to be physical or virtual, on-prem or in the cloud, managed or otherwise.  These impact other aspects of your deployment architecture to be sure, but generally not the number of servers to include in your deployment or their size.

From the outset, let me say that no matter what recommendation I or one of my colleagues gives you, it’s only a point in time recommendation based on the limited information given, the fidelity of which will increase over time.  You must monitor your Jazz solution environment.  In this way you can watch for trends to know when a given server is at capacity and needs to scale by increasing system resources, changing the distribution of applications in the topology and/or adding a new server.  See Monitoring: Where to Start? for some initial guidance.  There’s a lot going on in the monitoring area ranging from publishing additional information to existing monitor solutions or providing a lightweight appliance with some monitoring capabilities.  Keep an eye on work items 386672 and 390245.

Before we even talk about how many servers and their size, the other standard recommendation is to ensure you have a strategy for keeping the Public URI stable which maximizes your flexibility in changing your topology.  We’ve also spent a lot of time deriving standard topologies based on our knowledge of the solution, functional and performance testing, and our experience with customers.  Those topologies show a range in number of servers included.  The evaluation topology is really only useful for demonstrations.  The departmental topology is useful for a small proof of concept or sandbox environment for developing your processes and procedures and required configuration and customization.  For most production environments, a distributed enterprise topology is needed.

The tricky part is that the enterprise topology specifies a minimum of 8 servers to host just the Jazz-based applications, not counting the Reverse Proxy Server, Database Server, License Server, Directory Server or any of the servers required for non-Jazz applications (IBM or 3rd Party).  For ‘large’ deployments of 1000 users or more that seems reasonable.  What about smaller deployments of 100, 200, 300, etc. users?  Clearly 8+ servers is overkill and will be a deterrent to standing up an environment.  This is where some of the ‘art’ comes in.  I find more often than not, I am recommending a topology that is some where between the department and enterprise topologies.  In some cases, a federated topology is needed when a deployment has separate and independent Jazz instances but needs to provide a common view from a reporting perspective and/or for global configurations, in case of a product line strategy. The driving need for separate instances could be isolation, sizing, reduce exposure to failures, organizational boundaries, merger/acquisition, customer/supplier separation, etc.

The other part of the ‘art’ is recommending the sizing for a given server.  Here I make extensive use of all the performance testing that has been done.

• CLM Sizing Strategy for the 6.0 release
• CLM 6.0 Performance Report for Global Configuration and link indexing services
• Sizing and tuning guide for Rational DOORS Next Generation 6.0: configuration management
• Rational DOORS 9 to Rational DOORS Next Generation migration sizing guide
• Deployment Topologies for Jazz Reporting Service
• Performance summary and guidance for the Data Collection Component in Rational Reporting for Development Intelligence
• Best Practices for Configuring LQE For Performance and Scalability
• Performance and Data Scalability of Rational Rhapsody Design Manager

The CLM Sizing Strategy provides a comfortable range of concurrent users that a given Jazz application can support on a given sized server for a given workload.  Should your range of users be higher or lower, your server be bigger or smaller or your workload be more or less demanding, then you can expect your range to be different or to need a different sizing.  In other words, judge your sizing or expected range of users up or down based on how closely you match the test environment and workload used to produce the CLM Sizing Strategy.  Concurrent use can come from direct use by the Jazz users but also 3rd party integrations as well as build systems and scripts.  All such usage drives load so be sure to factor that into the sizing.  There are other factors such as isolating one group of users and projects from another, that would motivate you to have separate servers even if all those users could be supported on a single server.

Should your expected number of concurrent users be beyond the range for a given application, you’ll likely need an additional application server of that type.  For example, the CLM Sizing Strategy indicates a comfortable range of 400-600 concurrent users on a CCM (RTC) server if just being used for work items (tracking and planning functions).  If you expect to have 900 concurrent users, it’s a reasonable assumption that you’ll need two CCM servers.  As of v6.0.2, scaling a Jazz application to support higher loads involves adding an additional server, which the Jazz architecture easily supports.  Be aware though that there are some behavioral differences and limitations when working with multiple applications (of same type) in a given Jazz instance.  See Planning for multiple Jazz application server instances and its related topic links to get a sense of considerations to be aware of up front as you define your topology and supporting usage models.  Note that we are currently investigating a scalable and highly available clustered solution which would, in most cases, remove the need for distributing projects and users across multiple application servers and thus avoid the behavioral differences mentioned.  Follow this investigation in work item 381515.

This post doesn’t address other servers likely needed in your topology such as a Reverse Proxy, Jazz Authorization Server (which can be clustered), Content Caching Proxy and License Key Server Administration and Reporting tool.  Be sure to read up on those so you understand when/how they should be incorporated into your topology.  Additionally, many of the performance and sizing references I listed earlier include recommendations for various JVM settings.  Review those and others included in the complete set of Performance Datasheets and Sizing Guidelines.  It isn’t just critical to get the server sizing right but the JVM properly tuned for a given application.

To get to the crux of the primary question of number of servers and their size, I ask a number of questions.  Here’s a quick checklist of them.

1. What Jazz applications are you deploying?
2. What other IBM or 3rd party tools are you integrating with your Jazz applications?
3. How many total and concurrent users by role and geography are you targeting and expect to have initially?  What is the projected adoption rate?
4. What is the average latency from each of the remote locations?
5. How much data (number of artifacts by domain) are you migrating into the environment? What is the projected growth rate?
6. If adopting Rational Team Concert, which capabilities will you be using (tracking and planning, SCM, build)?
7. What is your build strategy? frequency/volume?
8. Do you have any hard boundaries needed between groups of users, e.g. organizational, customer/supplier, etc. such that these groups should be separated onto distinct servers?
9. Do you anticipate adopting the global or local configuration management capability (released in v6.0)?
10. What are your reporting needs? document generation vs. ad hoc? frequency? volume/size?

Most of these questions primarily allow me to get a sense of what applications are needed and what could contribute to load on the servers.  This helps me determine whether the sizing guidance from the previously mentioned performance reports need to be judged higher or lower and how many servers to recommend.  Other uses are to determine if some optimization strategies are needed (questions 4 and 7).

As you answer these questions, document them and revisit them periodically to determine if the original assumptions, that led to a given recommended topology and size, have changed and thus necessitate a change in the deployment architecture.  Validate them too with a cohesive monitoring strategy to determine if the environment usage is growing slower/faster than expected or detect if a server is nearing capacity.  Another good best practice is to create a suite of tests to establish a baseline of response times for common day to day scenarios from each primary location.  As you make changes in the environment, e.g. server hardware, memory or cores, software versions, network optimizations, etc., rerun the tests to check the effect of the changes.  How you construct the tests can be as simple as a manual run of a scenario and a tool to monitor and measure network activity (e.g. Firebug).  Alternatively, you can automate the tests using a performance testing tool.  Our performance testing team has begun to capture their practices and strategies in a series of articles starting with Creating a performance simulation for Rational Team Concert using Rational Performance Tester.

In closing, the kind of guidance I’ve talked about often comes out in the context of a larger discussion which looks at the technical deployment architecture in a more wholistic perspective, taking into account several of the non-functional requirements for a deployment.  This discussion is typically in the form of a Deployment Workshop and covers many of the deployment best practices captured on the Deployment wiki.  These non-functional requirements can impact your topology and deployment strategy.  Take advantage of the resources on the wiki or engage IBM to conduct one of these workshops.

Finding suspect links and requirements to reconcile in CLM 6.0 configuration management enabled projects

The newly released CLM 6.0 has some great configuration management across the lifecycle capabilities.  As described in Enabling configuration management in CLM 6.0 applications there are some considerations to be made before enabling configuration management.  One of these is

Do you rely on suspect links for requirements in RM or for requirement reconciliation in QM?

The reason being is that in this release, automatic detection of suspect links (artifacts that need to be evaluated and possibly modified because a linked artifact has changed) is not working in configuration management enabled DOORS Next Generation (DNG) projects. Further, the requirements reconciliation process in Rational Quality Manager (RQM) to find any changed/deleted requirements impacting associated test cases is also not supported in configuration management enabled RQM projects.  Note that a new mechanism for determining link suspicion is intended for a future release, in the interim though, teams must use a workaround.  I’ll explore some of those in this blog.

Suspect Links Workaround

To determine what requirements have changed and which impacted test cases may need updating, we’ll need to look at a test coverage view of requirements filtered by a some modified date.  It’s likely that you’ll want to know which requirements have changed since the last release baseline.

Let’s assume we are working in the AMR Mobile US global configuration in DNG.  Open the Configuration Context Banner Button (CCBB) and navigate to the global configuration.

Show the baselines for this global configuration.

Observe the most recent baseline is AMR Mobile 3.0 US GA.  Drill down into it and see the baseline was created on May 27, 2015, 12:35:23 AM.

Note at this time, baselines for the each of the contributors to the global configuration must be created first before the baseline for the global configuration can be committed/finalized.  This means there likely be some time disparity between the global configuration baseline and the baselines for contributing local configurations (they could be earlier or later than the global configuration creation date).  For this reason, it’d be more accurate to use the local configuration baseline creation time for the filtering.  While looking at the global configuration baseline, select the row with the desired local configuration and observe its creation time on the right.

While viewing requirements of interest, likely on a per module by module basis, e.g. AMR System Requirements, open a Test Coverage view and add a Filter by Attribute for Modified on set to be after the baseline date.

Here we see there have been four requirements altered, two which have no linked test case, but may need one if they are new, and two that have linked test cases, but may need to be updated.

Now you could look at each individual requirement one by one to understand if they are new or modified.  In this example, opening requirement 1830 shows it has been created since the baseline date.

You could also add the Created On attribute to the view columns displayed.

This requirement doesn’t have a related test case so now you would evaluate whether one should be created.

Looking at requirement 517, you observe that it was created before the baseline and modified since.  There is a related test case but you need to understand what the change was to better evaluate if it necessitates a change in the test case.

Open the requirement history and get a sense of the changes.

Should the changes be such that a reevaluation of the test case is warranted, follow the Validated By links to navigate to the test case(s) and check if they need updating.

To track the potential impact of those substantive requirement changes, you could tag the requirements and create a traceability view to look only at those.

Alternatively, create a Tracked By link from the suspect requirement to a new RTC work item task that would be assigned to the test team to evaluate whether any linked test cases should be updated.

Now rather than going through each requirement individually, an alternative is to make use of the Compare Configuration capability to compare the current local DNG configuration/stream to the DNG baseline included in the AMR Mobile 3.0 US GA baseline.

The compare results will show additions, changes and deletions to project properties, folders and artifacts.  With this information, the analyst can make a reasonable determination of how substantive each change was.  Armed with this information, they would need to return to the Test Coverage view(s) and tag the appropriate requirements as suspect and/or create RTC work items to analyze the linked test cases.  Note that previously we were looking at requirements changes on a per module basis (if modules were being used) but the Compare Configuration will look at all changes to all artifacts across the stream, that is, consolidating all change and not giving a module perspective.

Now if you were paying attention to my scenario,  you’ll notice that the last screen shot above, showing the results of the compare, doesn’t line up with the Test Coverage view shown earlier as the system requirements that were shown to have changed since the baseline are not shown in the compare results.  No, this isn’t a bug in the software.  I was using a shared internal sandbox whose configuration was changed in between the time when I started writing this blog and the time I tried to capture the compare results.  Rather than trying recreate the scenario, I left things as they were as I think I still get the concept across (though the anal side of me really has a problem with it all not lining up!).

Requirements Reconciliation Workaround

Requirements reconciliation is a capability in RQM that looks at the requirements linked to a test case or the requirement collection linked to a test plan and determine if there are new, updated or deleted requirements that necessitate changes in the test case(s).  In CLM 6.0, requirements reconciliation is not supported in projects that have configuration management enabled.

While you can query for test cases changed since a certain baseline date, this doesn’t really help determine if there are test cases to be updated due to requirements changes.  It’s not possible from RQM to query on test cases and filter based on attributes of the linked requirements.

Thus, the reconciliation process would need to be driven by DNG such that the tester would need to use the same technique used by the analyst for the suspect links workaround.  That is, the tester would look at a test coverage view of requirements in DNG, filtered to show requirements updates since the baseline date and evaluate if a test case addition, deletion or update was warranted.  This process would be further helped if the analysts used tagging as previously described so that that tester wouldn’t need to sift through all the requirements to find only those with substantive changes.  Use of impact analysis tasks in RTC would as well.

Use of these test coverage views would only identify requirements added or changed since the baseline.  It would not list requirements removed.  So for a comprehensive view of requirements changes that need to be reconciled with test cases, RM stream needs to be compared against the baseline to see any requirements that have been deleted.

Conclusion

While it is unfortunate that this initial release of the global configuration management capability doesn’t include support for suspect links and requirements reconciliation, there are some manual workarounds available, while not ideal, can help mitigate the gap until such time such time a replacement is available.  For some customers, such a manual process may be untenable due to the volume of changes between baselines.  Rather than performing the analysis after the fact, perhaps being more proactive about flagging potential impacts from the beginning of and throughout the release is more appropriate.  As requirements get changed, assess then whether there are test case updates needed by tagging the requirement and/or creating an impact analysis task in RTC.  These can be reviewed/refined at different milestones throughout the release.  Again not ideal but does distribute the analysis burden.

Help! My RTC Database is getting big!

Many customers who have been using RTC for some time are seeing their database size grow, in some cases, to 100s of GBs if not TBs of data.  While growth is to be expected and is of no issue to modern DBMSes, proactive administrators want to understand the growth and how it can be mitigated especially in light of anticipated user growth in their environments.  Naturally, they come to us and ask what can be done.  While at this time we don’t have the solutions many customers are asking for (project copy, move, archive, delete, etc.), that isn’t to say we don’t have approaches that may be of value in some situations.

Working with our own self-hosted RTC environment as well as those of our enterprise customers, we generally find that the largest contributors to database size our build results, work item attachments and versioned content.  How would you know that?  Fortunately, we have a couple of out of the box reports you can run: Latest Repository Metrics and Latest Repository Metrics by Namespace.  Below are some samples showing a subset of the available namespaces and item types.

Looking at all the namespaces and item types begs the question…what do they all mean?  Yeah, they aren’t all obvious to me either.  Luckily, I have access  to the smart developers who wrote this stuff and can tell me.  If you find one you don’t know, send me a note/comment or post it on the jazz.net forums. 

Once you find the larger contributors to size, the next questions asked are can they be deleted and who (that is, which project) is producing them.  In keeping with my team’s No Nonsense Tech Talk theme, I’ll be honest, there’s not much we can delete/archive and we certainly can’t do it at a project level, which would be of greater value, nor can we easily tell who produced it all.  It’s not all doom and gloom because there are some things we can do.

As mentioned earlier, we can delete build results, which often are a huge contributor to size growth.  We can delete work items, even attachments from work items.  Versioned content can be deleted, though you don’t usually want to do that, except for security reasons or to remove binaries versioned by mistake.  Then there are plans, streams, workspaces, etc, that can be deleted, but these don’t tend to take up much space.

So what happens when something is deleted? Well, in some cases, it’s not really removed from the database, only reference to it is removed or made less accessible.  For example, work item attachments don’t really go away when removed from a work item.  Try this.  Add an attachment to a work item then save the work item.  Hover over the attachment link and copy it.  Now remove the attachment from the work item then save the work item.  In the browser, paste the copied attachment URL and it will be found.  Similarly, if you delete a work item that has attachments, the attachment links still remain valid.  However, if you delete (not remove) a work item from the Eclipse client, the work item is actually deleted.

If you find that you’ve removed but not deleted an attachment, it is possible to go back and have it truly deleted.  To do so, using the Eclipse client, paste the URL to the attachment (which should be visible in a discussion comment from when it was first attached) somewhere into the work item (into a comment or the description), right click over that link and select “add to favorites”. Once it is in the favorites, you can drag it from Favorites and drop onto the Attachments section, which re-attaches it to the work item, at which point you can then delete it in the normal way.

Now some things like build results and versioned content once deleted can truly be deleted from the database.

At the repository level there are two key background tasks that are used to mark content as deletable then later deletes it.

• an “Item Cleanup Task” background task at repository level marks newly orphaned content blobs as needing deletion (runs every 17 mins by default)
• a “Deleted Items Scrub Task” background task at repository level deletes any content blobs that have been orphaned for more than ~2 hours (runs every 24 hours by default)

Once these both run, any content blobs that were deleted more than 2 hours ago should be fully deleted from the database.

However, DBMSes (particularly those in production) don’t generally release storage allocated for their tables immediately.  A compaction task usually needs to be run to reclaim the disk space.  The DBMS should have tools to indicate in advance how much space can be reclaimed by compaction.  Typical utilities to do this are shown below.  Details for using them should be left to a qualified DBA.

• Oracle – ALTER TABLE … SHRINK SPACE
• DB2 – REORG
• SQL Server – SHRINKDATABASE

My teammate Takehiko Amano has done a very nice job of showing how deleting versioned content and later running DB2 commands reduces database size.  See his article Reducing the size of the Rational Team Concert repository database.

We find the build results often take up a good amount of the RTC database size.  These results often include all the log files from compilations, tests and other activities performed during the build.  Some times they will contain downloadable outputs, e.g. application archives and executables.  What happens is these results are often kept around and never deleted.  In some cases, especially for release milestones, they should be, but all those personal or interim continuous builds don’t need to be kept.  Build results can be deleted and will result in their database content being orphaned and subsequently deleted per the aforementioned process.  Rather than manually deleting results, consider setting up a pruning policy to automatically delete old results.  For those results you want to keep around and not be pruned, just mark them as not deletable.

In cases you know your build results are taking up a lot of space, the natural follow on question is which builds and who owns them.  Our development team recently had cause to address that question which resulted in a very useful script written by Nick Edgar, the RTC Build Component Lead.

Nick created a Groovy script that collects into a CSV file the pruning policy settings and footprint data for each build in all projects across a repository.

He further creates an interactive HTML report that parses the CSV file for display in a more visual form. 

With this information you can find out which builds from which projects are taking up the most space and whether they have a pruning policy in place.  Armed with this information, an administrator could go to the appropriate release teams and take action.  Imagine running it weekly and posting to a dashboard or emailing it weekly to release teams.  The Groovy script to collect the data and index.html to render the report are attached to task work item 330478.

For gathering the CSV data you’ll need to 1) install Groovy, 2) install a build system toolkit that’s compatible (ideally at same version) as RTC server, 3) set environment variables (see top of Groovy script), and 4) run the script with: groovy -cp “$BUILD_TOOLKIT/*” <groovy file name> <any arguments needed by script>.

For the chart, just put the chart index.html file and CSV in the same directory and open the HTML file. Some browsers will require these to be served up by a web server to allow the HTML file to read the CSV file. For my testing, I used Python’s simple server support for this: python -m SimpleHTTPServer.

Given I am referencing code samples, I’ll keep our lawyers happy by stating that any code referenced is derived from examples on Jazz.net as well as the RTC SDK. The usage of such code is governed by this license. Please also remember, as stated in the disclaimer, that this code comes with the usual lack of promise or guarantee. Enjoy!

Being able to monitor the size and growth of your data, getting granular and actionable information about it and ultimately, being able to do something positive about that growth, is a key concern for IBM Rational and something we are exploring as part of our Platinum Initiative.  I welcome your input in this area.  Perhaps we can interact at IBM InterConnect 2015.

Automating changes to a build workspace configuration

Here’s the scenario.  A build engineer gets a request to establish a build workspace and corresponding build workspace.  He creates the build workspace, gets the desired components added to it, sets the flow targets then changes ownership of the workspace to the build user’s functional ID.  After that, he creates the build definition and associates it with the newly created workspace.  All is good and developers can begin to use the new build definition.  After a time, a request comes in to change the configuration of the build workspace, specifically the components to use.  Since the build workspace is owned by the build user, any changes to it must be made by that user.  Many organizations eschew the use of functional IDs or at least minimize who knows their credentials and are concerned about extra maintenance brought on by password expiration rules.  What to do?

This specific scenario came up recently with a customer of mine.  In particular, their build workspaces have three flow targets.  The current/default is the stream with the application source and it is scoped to include a subset of the components in the stream.  The other two streams are build script and configuration related.  At times, the release team needs to change the components included in the scope of the application source stream.  They do so currently by logging in as the build user, something they detest doing.

What they much prefer is support for team ownership of repository workspaces but that isn’t currently possible (though requested via 271760).  We instead proposed a solution that put the workspace configuration change in the build script which is already performed as the build user and owner of the workspace.

As of 4.0.1, the SCM CLI includes the capability to add/change workspace flow targets.  It was later refactored in 4.0.6 to the current verb-oriented form.

${scmExe} –non-interactive set flowtarget -r ${repositoryAddress} -u ${userId} -P ${password} ${team.scm.workspaceUUID} ${targetStreamUUID} –flow-components ${componentsToLoad}

Where:

• scmExe – path to scm CLI executable
• repositoryAddress – URL of CCM server
• userId – build user ID
• password – password of build user
• team.scm.workspaceUUID – build repository workspace UUID
• targetStreamUUID – stream flow target to set component scope
• componentsToLoad – space delimited list of components

Unfortunately the SCM CLI does not understand the password file format used by the Ant tasks. You either need to give it as plain text using –P shown above or login outside the build system with the option to remain logged in (on each build machine).

A simple way to get the UUID of a workspace or stream is to open it up in the Eclipse editor and select Copy URL workspace editor menu in the view header or browse to it in the Web UI to get the URL.  The end portion of the URL is the UUID.  For example, the UUID for the repository workspace URL shown below is _GVqXYLRpEeOdavKqgVc36Q

https://clm.jkebanking.net:9443/ccm/resource/itemOid/com.ibm.team.scm.Workspace/_GVqXYLRpEeOdavKqgVc36Q

For components with spaces in their name, care must be taken to offset them by appropriate quotes.  For my tests, single quotes worked on Linux and double on Windows.

The command will set the scope of the specified flow target (e.g. application development stream) for the specified workspace (e.g. build workspace) to include only those components in the specified list.  The current configuration of the component(s) in the target stream is used.

Note if multiple flow targets exist for the workspace and a component listed is included in multiple flow targets, then scoped flows need to be used to avoid conflicts.  See How should my source code be loaded from Jazz SCM?

Let’s take a look at an example.  Below, the build.brm.continuous repository workspace, owned by the build user, has six components and three flow targets in its configuration. Banking Logic, Database, Java UI and Prerequisites are from the BRM Stream.  Build comes from Build Scripts and Build Config comes from the stream of the same name.

The BRM Stream is scoped to only include only a subset of the components.

Assume that we wanted to add the Database component to the scope.  Using SCM CLI command similar to that shown in the screenshot below, the scope can be changed to add it in.

This results in the flow target scope being changed.

Now to add this to the build script so it can be automated.  Observe in the editor screenshot below that the command has been added to an exec statement in an Ant build script. 

The targetStreamUUID and componentsToLoad values need to be passed to the build script.  Add these as properties of the build definitions.  For example:

When the build is requested, the componentsToLoad value can be changed.  In our original example, the Database component can be added in by editing the componentsToLoad build property at the time of the build request.

Should you be concerned that adding the ‘set flowtarget’ command to the build script will add unnecessary overhead, albeit very minimal, to every build execution even when component list isn’t changed, you can create a build script and definition that only performs the ‘set flowtarget’ command and run it when needed.

The example shown was for changing the flow target component scope.  The technique used can be applied for other build workspace manipulations needed and supported by the SCM CLI.

Thanks to Nick Edgar, RTC Build component lead, for making me aware of the ‘set flowtarget’ command and suggesting its application to the problem described.

Do I need that other Jazz server?

I have been working with a team trying to determine if they need a second Change and Configuration Management (CCM) server to support their Rational Team Concert (RTC) deployment.  The performance of the existing CCM isn’t bad.  While the CPU utilization is low, the memory consumption is peaking a little high so some monitoring would be helpful to understand the usage patterns more.

The main motivation for considering a second CCM server is the expected growth to occur this year.  They know their current average and peak license usage, current count of licensed/registered users and how many new registered users they expect to add.  They also know the recommended range of users for a CCM based on IBM’s testing and analysis as shown in CLM Sizing Strategy.

The slight disconnect is that the user ranges we recommend are based on concurrent users not how many registered users there are or how many on average have a license in use (checked out).  Unfortunately, as yet, we don’t have a good way to measure the actual number of concurrent users.

So what do we do?  I see two ways to make this estimate.

First, the CLM Sizing Strategy states:

The number of concurrent users can often be estimated from the number of registered users. In an organization located on one continent or spanning just a few time zones, the number of concurrent users can be estimated to be between 20% and 25% of the total number of registered users. In an organization which spans the globe where registered users work in many different time zones, the number of concurrent users can be estimated to be between 10% and 15% of the total number of registered users.

What you can then do is take a percentage of the current plus expected registered users, e.g. 20-25% above when spanning few time zones, and compare that to the RTC ranges based on workload.  That will help you determine if another CCM (or more) is needed.  The percentage you apply may be different based on your experience.  In fact, in my customer case, a large number of users were registered and licensed in anticipation of them being migrated to RTC.  Applying that percentage today would give them a higher number of estimated concurrent users than was being experienced.

Another method would be to look at the average/peak number of licenses in use for RTC vs those that are licensed.  From this ratio, assuming linear growth, you can project how many licenses would be in use in the future.  Again, you would then compare that to the recommended ranges.  This method should be a little more accurate than the anecdotal, gut feel percentage of registered users in the first method.  We at least know that the number of concurrent users isn’t any more than the number of licenses in use.  Based on what you know of the users  to be added, linear growth may not be quite right and you may need to adjust the projection (e.g. the users being added are expected to be heavier users of RTC).

This latter method is what we used.  It seemed more accurate plus, as mentioned, the number of licensed users included many who weren’t onboarded yet.

Now the CLM Sizing Strategy is based on the workloads we generated on a given set of hardware.  Your workload and hardware may be such that you can handle more users (or maybe less).  I’ve had customers report that they were able to support more users than our recommended ranges.

This then shows that all the numbers and percentages need to be balanced against real experience with the environment, not only the license usage data but other things like CPU utilization, JVM size, database size, etc.  Monitor your current environment and perform trend analysis to assess if the server is nearing capacity.  Look at Monitoring: Where to Start? and Monitoring and Troubleshooting Guide for IBM Rational Collaborative Lifecycle Management (CLM).

Note that using the built in Jazz server license usage reports or the Rational License Key Server Administration and Reporting Tool, one can get an idea of license usage trends. For example, the reporting tool will show license usage by product. The recent release now includes support for authorized licenses.  Support for showing reports by role is being considered for a future release.

As mentioned in the beginning, in this customer’s case, they weren’t experiencing any performance issues with the server.  Their CPU utilization looked fine.  There were memory usage peaks hitting 80% – 90% of the total heap so I suggested monitoring this and increase the heap as needed.

In the end, though the straight numbers and percentages suggested additional servers would be needed, this customer didn’t have good monitoring data, wasn’t experiencing performance issues and seemed to have capacity to support more users.  My recommendation was to hold off on a topology decision, put some monitoring in place, adjust the server configuration as needed/able (memory, JVM, CPU (number, speed)), assess performance/capacity of server vs expected users yet to add, if capacity/performance can’t be adjusted by manipulating the server configuration and it looks like the growth will be more than the server can handle, then add the additional CCM.

We didn’t want to take the addition of the second CCM lightly because doing so would incur another set of issues, that being the behavioral differences between a single CCM environment and a multiple CCM environment that I’ve blogged about previously and documented on the deployment wiki.  See Planning for Multiple Jazz Application Server Instances.

When assessing whether a multiple CCM topology was needed, one consideration we made was whether to mitigate some of the primary behavioral differences of concern to the customer by making one CCM be the hub for all work items and planning and the other CCMs for source control and build only.  There are tradeoffs for sure but that’s another blog discussion for another time.

Considerations when deploying multiple instances of a Jazz application

As you have seen from previous posts I’ve made on the impact of deploying multiple QM or CCM applications in your CLM environment, which were then turned into an Innovate 2014 presentation, I’ve been very interested in getting the word out on the subject.  We tend to focus on getting a good infrastructure in place, as we should, but we don’t always factor in what might be different in the usage of the Jazz applications when we have multiples of them.  I’ve been on a crusade of sorts to gather what is known in this area, consulting other SMEs like my colleague (and co-author/presenter) Ralph Schoon, architects in Rational development for the different Jazz applications and several customers.

My goal is that there would be no surprises when multiple instances of an application exist and that the behavior differences are factored in up front during usage model design.  Where differences exist and where appropriate, I’m pursuing product changes to remove them.

Following a good discussion during our session at Innovate, incorporating feedback received, Ralph and I have converted our presentation to a page on the Jazz.net deployment wiki (see Planning for multiple Jazz application server instances).  Have a look and if you have any feedback, post a comment here or better yet, post it on the wiki (see How can customers make comments on the wiki pages?).

I did a simple straw poll at Innovate to get a sense of the ranges for the number of instances of a particular Jazz application in different customer environments.  Of the 30-40 attendees, most only had one JTS but there were 3-5 that had 2-3 JTS. Several had 3-5 CCMs. Only one had multiple QMs (3). None were planning multiple RMs yet.  I’d be very interested in getting other samplings.  If you have multiple instances of any one of the Jazz applications (registered to the same JTS) or multiple JTS instances, I’d like to hear how many and understand a little more about the environment.  You can post a reply to this blog or contact me directly (tfeeney@us.ibm.com).