Guidance on adopting IBM CLM configuration management across the lifecycle

Earlier this year, Kathryn Fryer and I formed a workgroup to share experiences we’ve had with customers adopting the IBM CLM configuration management solution. The purpose being to elicit our shared recommended practices and guidance.  The workgroup consisted of representatives from Services, Support, Development, Test, Enablement and Product Management.

When forming your Configuration Management plan we recommend it consider the following key topics:

  • End to End Process Flow (context for use of configuration management)
  • Component Strategy
  • Stream Strategy
  • Baseline Strategy
  • Change Management (including DNG change sets)
  • Cross-stream Updates
  • Reviews and Approvals
  • Traceability, Link Validity and impact analysis
  • Naming, Tagging, Custom Attribute Conventions
  • Roles and Permissions
  • Configuration aware Reporting
  • Integrations (including non-CLM)
  • Communication Plan

Our aim is to create guidance that encompasses each of these topic areas to aid you in creation of your configuration management plan.

The first such guidance is now available on jazz.net.

Our focus has been on general adoption guidance along with component and stream strategies (both are critical at the outset and go hand in hand). Next focus areas are yet to be determined but could include finishing out the stream strategies (a couple more patterns remain) baselining strategy and change management.

We value your feedback on the guidance to date and input on areas to focus on next.

If you are at the IBM Watson IoT: Continuous Engineering Summit 2017 this week in New Orleans, be sure to say hello. Check out the talk I have with Ian Zimmerman on Friday at 2:30pm (Azalea 1): CM05 – Adopting Configuration Management: What You Need to Know!

Advertisements

Resource-intensive scenarios that can degrade CLM application performance

About a year ago, I was asked to begin considering what scenarios could drive load on a Collaborative Lifecycle Management (CLM) application server that could lead to outages or overall diminish the end user’s quality of service.  These aren’t necessarily long running scenarios but those that are known to use large amounts of system resources (e.g. high CPU, memory, heap usage).  As such, they have been known at times to degrade server performance and negatively impact user experience.

After reviewing a number of problem reports and escalations plus several discussions with Support, Services and Development resources, I identified scenarios for several of the applications.  We coined the term ‘expensive scenarios’ though our User Assistance team recently indicated that it could be misconstrued and a more apt name would be ‘resource-intensive’.

The first set of scenarios were published in v6.0.3 and documented as Known Expensive Scenarios.  The title will be changed in the next release to be Known Resource-intensive Scenarios.

For each of the identified scenarios, there is a description of what it is and under what conditions it could become resource-intensive.  Further, if there are any known best practices to avoid or mitigate the scenario from becoming resource-intensive, these too are captured.  These practices could include adjusting some application advanced properties that tunes the scenario behavior some or a change in work practices for when and how the scenario is invoked.

For example, importing a large number of requirements into DOORS Next Generation (DNG) can consume high resources as subsequent to the import, indexing of the newly imported artifacts occurs, which can block other user activity.  When the volume of imported data is high and/or several occur at once, system performance could degrade.  The wiki describes this scenario, identifies that there are some advanced properties that limit the number of concurrent ReqIF imports as well as the recommendation that these imports be kept under 10K requirements or be performed when the system is lightly loaded.

Knowing these scenarios help in a couple of ways.  First, as your process and tools teams define usage models for one of these applications, knowing that a particular usage pattern can potentially drive load on the server leading to degraded performance allows that usage model to be adjusted to avoid or reduce the likelihood of that occurring. Second, in situations of poor performance or worse, knowing if these scenarios are occurring could help identify root cause.

This latter case is helped by the logging of start and stop markers when a resource-intensive scenario occurs.  Each marker includes the Scenario ID (from Table 1) and a unique instance ID.

ScenarioStartStopTo get additional details when the scenario occurs and to aid in understanding its characteristics, advanced (verbose) logging can be enabled.  This can be done from the Serviceability page of an application’s admin UI.  Note the enabling verbose logging does not require a server restart.

ScenarioEnableAdvLogging

Now when a performance or system anomaly occurs and the application logs are reviewed, should it have occurred during a resource-intensive scenario, you may have a clue as to cause.  The additional logging should at a minimum include the data specified in Table 2.

ScenarioAdvLogging

As part of our serviceability improvements in v6.0.3, the CLM applications publish various JMX MBeans that may be collected and trended by enterprise monitoring tools such as IBM Monitoring, Splunk, LogicMonitor and others.  MBeans exist for several application metrics including counts/occurrences of resource-intensive scenarios.

Each MBean to be published must first be enabled from an application’s admin UI advanced properties page.

MBeansEnable

After doing so, the monitoring application can be configured to capture that data and displayed on a dashboard.

MBeansStats

Having a comprehensive enterprise monitoring strategy is essential for a well-managed CLM environment.  Tracking occurrences of these scenarios and correlating them against other environment measurements give administrators (and IBM Support) insight when troubleshooting anomalies or proactively evaluating environment performance.  In a subsequent post, I will talk further about what to monitor.

 

Adopting the IBM Continuous Engineering (CE) solution Configuration Management Capability

Adopting the IBM Continuous Engineering (CE) solution Configuration Management Capability is the title of a webinar that Kathryn Fryer and I recently presented.  We’ve been working with ‘new’ configuration management capability since it was in development prior to its launch in v6.0.  Adopting it takes careful consideration in order to successfully realize its benefits.

Objectives of the presentation

In version 6, the IBM CE solution added exciting new configuration management capabilities across the lifecycle, better enabling parallel development and strategic reuse. Simply enabling these capabilities won’t help you realize their potential; you must consider changes to your process and usage model to achieve results. This presentation describes current considerations, limitations and strategies for adopting configuration management.

  • Configuration management overview
  • Trade-offs and considerations – as of current release (6.0.2)
    • Primary factors
    • Reporting
    • OSLC integrations
    • Linking
    • QM utilities
    • Additional considerations
  • Enabling configuration management
  • Upgrade and migration
  • Adoption path and additional resources

If you are interested in this presentation, you can find the replay of the webinar here in the DOORS Enlightenment Series.

The slides are shared here.

Additional Reading

Scaling the Collaborative Lifecycle Management (CLM) solution across an enterprise

It’s critical that clients understand the implications when scaling your Jazz topology by adding additional servers for a given application. Ralph and I have been documenting these considerations and discussing them with customers over the last few years. We recently gave a webinar on this topic. See the following blog post for details.

rsjazz

Scaling the Collaborative Lifecycle Management (CLM) solution across an enterprise, is the title of the webinar Tim and I presented recently. Tim and I have worked on this content for a while and consider it important. Please also see  Tim’s related post Getting to a right-sized Jazz environment. There is additional reading below.

Objectives of the presentation

Scaling the v6.0.x Collaborative Lifecycle Management (CLM) solution across an enterprise often includes multiple instances of a given Jazz application. What multi-Jazz application options are available and what are the considerations?

  • What topologies and general multi-Jazz application options are available
  • How Jazz applications such as Change Configuration Management (CCM), Quality Management (QM) and Requirements Management (RM) relate to a Jazz Team Server (JTS)
  • The impact, advantages and disadvantages of multiple CLM Applications
  • What to consider when scaling and developing usage models adopting one of these deployment options to avoid surprises

If you…

View original post 73 more words

Getting to a right-sized Jazz environment

2015-05-24 14.05.27You’ve just made the decision to adopt one of the Jazz solutions from IBM.  Of course, being the conscientious and proactive IT professional that you are, you want to ensure that you deploy the solution to an environment that is performant and scalable.  Undoubtedly you begin scouring the IBM Knowledge Center and the latest System Requirements.  You’ll find some help and guidance on Deployment and Installation and even a reference to advanced information on the Deployment wiki.  Unlike the incongruous electric vehicle charging station in a no parking zone, you are looking for definitive guidance but come away scratching your head still unsure of how many servers are needed and how big they should be.

This is a common question I am often asked, especially lately.  I’ve been advising customers in this regard for several years now and thought it would be good to start capturing some of my thoughts.  As much as we’d like it to be a cut and dried process, it’s not.  This is an art not a science.

My aim here is to capture my thought process and some of the questions I ask and references I use to arrive at a recommendation.  Additionally, I’ll add in some useful tips and best practices.  If this proves useful, it will eventually move over to the Deployment wiki.

I find that the topology and sizing recommendations are similar regardless of whether the server is to be physical or virtual, on-prem or in the cloud, managed or otherwise.  These impact other aspects of your deployment architecture to be sure, but generally not the number of servers to include in your deployment or their size.

BUS30093From the outset, let me say that no matter what recommendation I or one of my colleagues gives you, it’s only a point in time recommendation based on the limited information given, the fidelity of which will increase over time.  You must monitor your Jazz solution environment.  In this way you can watch for trends to know when a given server is at capacity and needs to scale by increasing system resources, changing the distribution of applications in the topology and/or adding a new server.  See Monitoring: Where to Start? for some initial guidance.  There’s a lot going on in the monitoring area ranging from publishing additional information to existing monitor solutions or providing a lightweight appliance with some monitoring capabilities.  Keep an eye on work items 386672 and 390245.

enterpriseBefore we even talk about how many servers and their size, the other standard recommendation is to ensure you have a strategy for keeping the Public URI stable which maximizes your flexibility in changing your topology.  We’ve also spent a lot of time deriving standard topologies based on our knowledge of the solution, functional and performance testing, and our experience with customers.  Those topologies show a range in number of servers included.  The evaluation topology is really only useful for demonstrations.  The departmental topology is useful for a small proof of concept or sandbox environment for developing your processes and procedures and required configuration and customization.  For most production environments, a distributed enterprise topology is needed.

The tricky part is that the enterprise topology specifies a minimum of 8 servers to host just the Jazz-based applications, not counting the Reverse Proxy Server, Database Server, License Server, Directory Server or any of the servers required for non-Jazz applications (IBM or 3rd Party).  For ‘large’ deployments of 1000 users or more that seems reasonable.  What about smaller deployments of 100, 200, 300, etc. users?  Clearly 8+ servers is overkill and will be a deterrent to standing up an environment.  This is where some of the ‘art’ comes in.  I find more often than not, I am recommending a topology that is some where between the department and enterprise topologies.  In some cases, a federated topology is needed when a deployment has separate and independent Jazz instances but needs to provide a common view from a reporting perspective and/or for global configurations, in case of a product line strategy. The driving need for separate instances could be isolation, sizing, reduce exposure to failures, organizational boundaries, merger/acquisition, customer/supplier separation, etc.

The other part of the ‘art’ is recommending the sizing for a given server.  Here I make extensive use of all the performance testing that has been done.

4.1.1The CLM Sizing Strategy provides a comfortable range of concurrent users that a given Jazz application can support on a given sized server for a given workload.  Should your range of users be higher or lower, your server be bigger or smaller or your workload be more or less demanding, then you can expect your range to be different or to need a different sizing.  In other words, judge your sizing or expected range of users up or down based on how closely you match the test environment and workload used to produce the CLM Sizing Strategy.  Concurrent use can come from direct use by the Jazz users but also 3rd party integrations as well as build systems and scripts.  All such usage drives load so be sure to factor that into the sizing.  There are other factors such as isolating one group of users and projects from another, that would motivate you to have separate servers even if all those users could be supported on a single server.

Should your expected number of concurrent users be beyond the range for a given application, you’ll likely need an additional application server of that type.  For example, the CLM Sizing Strategy indicates a comfortable range of 400-600 concurrent users on a CCM (RTC) server if just being used for work items (tracking and planning functions).  If you expect to have 900 concurrent users, it’s a reasonable assumption that you’ll need two CCM servers.  As of v6.0.2, scaling a Jazz application to support higher loads involves adding an additional server, which the Jazz architecture easily supports.  Be aware though that there are some behavioral differences and limitations when working with multiple applications (of same type) in a given Jazz instance.  See Planning for multiple Jazz application server instances and its related topic links to get a sense of considerations to be aware of up front as you define your topology and supporting usage models.  Note that we are currently investigating a scalable and highly available clustered solution which would, in most cases, remove the need for distributing projects and users across multiple application servers and thus avoid the behavioral differences mentioned.  Follow this investigation in work item 381515.

This post doesn’t address other servers likely needed in your topology such as a Reverse Proxy, Jazz Authorization Server (which can be clustered), Content Caching Proxy and License Key Server Administration and Reporting tool.  Be sure to read up on those so you understand when/how they should be incorporated into your topology.  Additionally, many of the performance and sizing references I listed earlier include recommendations for various JVM settings.  Review those and others included in the complete set of Performance Datasheets and Sizing Guidelines.  It isn’t just critical to get the server sizing right but the JVM properly tuned for a given application.

To get to the crux of the primary question of number of servers and their size, I ask a number of questions.  Here’s a quick checklist of them.

  1. What Jazz applications are you deploying?
  2. What other IBM or 3rd party tools are you integrating with your Jazz applications?
  3. How many total and concurrent users by role and geography are you targeting and expect to have initially?  What is the projected adoption rate?
  4. What is the average latency from each of the remote locations?
  5. How much data (number of artifacts by domain) are you migrating into the environment? What is the projected growth rate?
  6. If adopting Rational Team Concert, which capabilities will you be using (tracking and planning, SCM, build)?
  7. What is your build strategy? frequency/volume?
  8. Do you have any hard boundaries needed between groups of users, e.g. organizational, customer/supplier, etc. such that these groups should be separated onto distinct servers?
  9. Do you anticipate adopting the global or local configuration management capability (released in v6.0)?
  10. What are your reporting needs? document generation vs. ad hoc? frequency? volume/size?

Most of these questions primarily allow me to get a sense of what applications are needed and what could contribute to load on the servers.  This helps me determine whether the sizing guidance from the previously mentioned performance reports need to be judged higher or lower and how many servers to recommend.  Other uses are to determine if some optimization strategies are needed (questions 4 and 7).

4.1.1As you answer these questions, document them and revisit them periodically to determine if the original assumptions, that led to a given recommended topology and size, have changed and thus necessitate a change in the deployment architecture.  Validate them too with a cohesive monitoring strategy to determine if the environment usage is growing slower/faster than expected or detect if a server is nearing capacity.  Another good best practice is to create a suite of tests to establish a baseline of response times for common day to day scenarios from each primary location.  As you make changes in the environment, e.g. server hardware, memory or cores, software versions, network optimizations, etc., rerun the tests to check the effect of the changes.  How you construct the tests can be as simple as a manual run of a scenario and a tool to monitor and measure network activity (e.g. Firebug).  Alternatively, you can automate the tests using a performance testing tool.  Our performance testing team has begun to capture their practices and strategies in a series of articles starting with Creating a performance simulation for Rational Team Concert using Rational Performance Tester.

In closing, the kind of guidance I’ve talked about often comes out in the context of a larger discussion which looks at the technical deployment architecture in a more wholistic perspective, taking into account several of the non-functional requirements for a deployment.  This discussion is typically in the form of a Deployment Workshop and covers many of the deployment best practices captured on the Deployment wiki.  These non-functional requirements can impact your topology and deployment strategy.  Take advantage of the resources on the wiki or engage IBM to conduct one of these workshops.

My first foray into IBM Internet of Things Foundation (part 2)

In part 1 of this series, I setup an environment where a Python application running on a Raspberry Pi, collected temperature, humidity and soil moisture data from connected barometer and moisture sensors.  This data was published to an MQTT message broker then retrieved by an application hosted on IBM Bluemix.  This application stored the data in a Cloudant database for later visualization via an HTML5 application. 

There are many different types of MQTT brokers available given it is an open source protocol.  These brokers vary by feature sets provided, some of which are in addition to the standard MQTT functionality.  IBM’s IoT Foundation, now known as Watson IoT Platform, serves as an managed and secure MQTT broker and much more.  With the Watson IoT Platform, you have a fully managed IoT platform allowing secure connection of devices with further access to a scalable platform as a service environment and a plethora of other services such as storage and analytics.

The diagram below provides a simple depiction of the interactions between a ‘Device’ and ‘Application’ through an MQTT Broker.  In part 1, I used an open source MQTT Broker.  In part 2, I will be using IBM Watson IoT platform as the MQTT Broker.  In my environment, I am only publishing ‘Events’ to the MQTT Broker for action by the Node-Red app.  I am not currently publishing ‘Commands’ back to the Raspberry Pi device.

image

My goal then for this next step in the series is to modify the Poseidon Client (Python application running on Raspberry Pi) to publish events to Watson IoT Platform and change the application running on Bluemix to receive that published data from the new source.  In this entry, I will outline the changes to each of these applications.

Before making any application modifications, I first need to configure the Pi to connect to the Watson IoT Platform using this recipe. At a high level, this recipe had me do the following:

  1. Download, install and start the IoT service which would manage the connection and publish events to the IBM IoT Foundation Quickstart.
  2. Register my Raspberry Pi device in Watson IoT Platform following another recipe.
  3. Use the credentials provided by the registration process to connect the device to Watson IoT.

When complete, I had a device registered with the IBM Watson IoT Platform as shown below.  In the process I was provided a device ID and authentication token, to be used later when connecting the device to the Watson IoT Platform service.

image

Now I am set to modify the application code to publish events directly to Watson IoT Platform using the IoT service now running on my Raspberry Pi.  I will keep the existing code that is using the paho MQTT client libraries to publish to an open source MQTT broker.  This code is gated by a configuration parameter that I will just turn off so it no longer executes.  I will add new code and configuration parameters to use the Watson IoT Platform Device API to publish events to the Watson IoT Platform.

The config.py file contains a number of configuration parameters for the Poseidon Client.  Set ‘sendToCloud’ to ‘False’ in order to turn off sending data through the open source MQTT broker.  Add a new ‘sendToIoTF’ parameter and set it to ‘True’ to publish via the Watson IoT Platform.

image

In the config.py file add a parameter to capture the credentials used to connect to the Raspberry Pi device I registered earlier.

image

In the PoseidonClient.py code add the requisite import statement to utilize the Watson IoT Platform Device API.

image

Skip ahead in PoseidonClient.py to the section of code that initializes the sensors and device interfaces.  The new code below creates a device client instance using the credentials added  to confg.py then connects to it.

image

Go back in PoseidonClient.py to the processData code and a publishEvent call (with appropriate callback) to send the sensor data to the Waston IoT Platform to be read later by the application running in Bluemix.

image

At the end of the main processing loop, add a call to disconnect from the device client instance.

image

Now we move on to the application in Bluemix.  Here is how it looked at the end of part 1.

image

Use the Node-Red editor to add the following nodes and connections.

  1. Add an ibmiot input node and configure it to read events published by the Raspberry Pi device (see subsequent screen capture with showing the parameters for the node)
  2. Add a debug node to capture the data received from the device
  3. Connect the ibmiot input node to the debug node
  4. Connect the ibmiot input node to the existing Cloudant output storage node
  5. Connect the ibmiot input node to the existing json function node to transform the sensor data payload from json format to Javascript object.

image

This screen capture illustrates the parameter settings for the new ibmiot input node.

image

After making the above changes, running the PoseideonClient on the device and the Node-Red app in Bluemix caused the following debug output in the Node-Red debug console.

image

The first debug output is the sensor data coming in from the Watson IoT Platform.  The next debug output is the formatted message being sent to Twitter.

Nothing changed in the Cloudant queries and views nor the HTML5 app.  From part 1 of this series, these will produce a visualization similar to the following:

image

I have successfully altered the device to publish events to the managed IBM Watson IoT platform MQTT Broker and the Node-Red app in Bluemix to receive and process those events.  Next up for me is to investigate possible inclusion of the IBM IoT Real-Time Insights service on Bluemix or perhaps Watson Analytics.  Another area of interest is the currently experimental IBM IoT Workbench.

My first foray into IBM Internet of Things Foundation

For the last year, I have been a part of what is now the IBM Watson Internet of Things (IoT) business unit.  I’m still in the same role but have just been working with a different set of customers and interested in a solution that addressed their Continuous Engineering (CE) needs.  This solution is only one aspect of the overall IBM Internet of Things solution.  When you consider the broader IoT solution, it involves not just the designing and engineering of ‘things’ (using our CE solution) but the operation/management of them, the collection/control of data from them and the analysis/optimization of that data with all of that being cloud enabled and done in a secure manner.

Since I like to learn new things and since my solution focus is a part of a broader solution I was less familiar with, I decided to begin exploring one foundational element of it, namely the IoT Foundation.

image

“The IBM IoT Foundation platform allows organizations to securely and easily connect devices, from chips to intelligent appliances to applications and industry solutions. Scaling through cloud-based services and using rich analytics, IoT Foundation platform provides organizations with new insight for innovation and transformation.”

 

 
The next question was how best to get that experience.  My colleague Dennis Schultz, who has written a nice series of blogs on My Internet of Things and MobileFirst adventure advised that I decide on a project that was practical, real and of interest to me.  At first I thought of something related to a weather station but then I thought it would be cool to do something that helped me figure out when to water my yard.  You see, I live in Texas and as I was considering this, it was in the middle of summer and our typical water restrictions were in effect such that I couldn’t always water when it was needed.  I also don’t want to over water so on my day of the week to water, I would typically stick my finger in the soil to see how wet the soil was and decide whether to turn the sprinklers on or not.  Wouldn’t it be nice if I could some how do that with a sensor that would let me know that through some sort of email, text or other mobile alert?

As I searched around, I found that some IBMers have already done some investigation along those lines.  The Poseidon Project, led by a team in Europe, “is a voluntary initiative that aims to reduce water usage in the world”.  They created a great three part tutorial series that connects a Raspberry Pi with sensors to monitor temperature and soil moisture, have that data published to an MQTT Message Broker, then input it to an application hosted on IBM Bluemix built using a Node-Red editor.  This application tweets the sensor data and also stores it in a Cloudant database for later visualization through an HTML5 app.  This appeared to have much of what I needed and more and would provide a nice launching point for my project.  Take a look at the solution architecture below.

Diagram of the solution design

I didn’t want to just jump in and follow the tutorial.  I first wanted to do some background reading and get some basic experience with the IBM IoT Foundation on Bluemix.

As you could imagine, there is a plethora of background information online.  Here are some links I found useful:

To get started with IBM IoT Foundation on Bluemix, I found there is a growing number of recipes on IBM developerWorks.  These are step by step tutorials that walk you through some aspect of using the technology of interest to you.  In particular, I followed the recipe that used a simulated device to provide temperature and humidity data that I could visualize using a simple quickstart web app (which can take inputs from real devices too).  From there I tried the IoT starter application which allowed me to feed the simulated sensor data into IoT Foundation application using a Node-Red editor in Bluemix.  I found this all to be quite fun and informative.

Thus armed and dangerous with some background information and basic experience I charged ahead into the Poseidon Project tutorial. My setup is depicted below.

$7B01FE0F3C44879

The specific hardware I purchased included:

The tutorial is well written and I pretty well followed it verbatim. To keep it impervious to changes in the Bluemix user interface, some of the steps were written more general with fewer specific screenshots or navigation steps making some harder, but only minimally so, for new users like me to follow. I did find that several steps were missing in Part 3, Step 2 relating to creating the APIs to be used by the HTML5 visualization app.  The authors intend to submit an update with the missing steps.  Should you decide to follow the tutorial, if it doesn’t show a revision date since it’s original publication of 16-December-2014, check back with me for the missing steps.

Here’s one output from the visualization app.

image

My experiment isn’t ready to be taken out of doors and used to monitor the soil moisture of my yard, I’d have to weatherproof the setup and find a different moisture sensor as I subsequently found that the one I am using isn’t meant to be used outdoors nor prolonged time in soil.  Still, I enjoyed the experience and believe with a little more work it could be applied as I intended.

It also gave me broad exposure to a number of different technologies enabling rapid development of IoT applications.

  • IBM Bluemix
  • Node.js
  • Node-Red
  • MQTT
  • IBM IoT Foundation
  • Cloudant DB
  • IBM DevOps Services
  • Git
  • Twitter
  • Data Drive Documents
  • HTML5

There are a couple of extensions to the tutorial I’d like to do.  First, it was written so that the sensor data is published to an MQTT Message Broker then received by the IoT Foundation app on Bluemix.  I want to instead publish the data directly to the IBM IoT Foundation using the ibmiotf client library.  I am currently working on that.  Second, I’d like to explore making use of the IBM IoT Real-Time Insights service on Bluemix.  I’ll publish any work on these in a subsequent post.

Thinking beyond this project, another idea I’m thinking of is making use of the TI SensorTag and Estimote Beacons to monitor temperatures in my home at the thermostat and in areas seemingly hotter/colder, compare these to weather outdoors and visualize on a mobile device with an app built using the IBM MobileFirst Platform or perhaps use the MyWeatherCenter app.

Finally, I will close with a thanks to Bram Havers and Davy Vanherbergen who helped me with any issues I came across in the Poseidon Client tutorial.