“Surely the ADDM team wouldn’t have just forgotten to give us a place to start?”, you think to yourself.

Well, you’re correct – one of the things we’re quite keen on is giving people clear and simple jumping off points for any new features. We like putting new features and components in the UI in places where you’ll already be looking. So that when you’re trying to do something, you’ll easily find the new user interface components that can help you. The natural instinct to think “hang on, that wasn’t there before” is a good thing when it comes to trying to encourage our users to explore and experiment with new features. It makes people curious, so they investigate and mess about with things until they understand what they do.

But sometimes, the new stuff just doesn’t show up for some people. They hear about it, and see it in the new videos, but it all remains curiously absent from their appliance’s UI. If you’re one of those people, the chances are that you’ve got some customization in place that’s stopping the new stuff from appearing. At some point, somebody has written some custom report definitions that override the normal dashboard channels. Because we respect those customizations across the upgrade process, It’s possible that new things might not appear because doing so would overrule your customization.

This recently came up as a topic in the forum, and it looks like an issue which is only going to get more common as the product advances. So I thought it was something worth talking about a little, just to make sure you don’t lose out on the new stuff.

Now, the best advice from the forum thread doesn’t come from me, but from Mark Wilkinson, so I’m just going to quote him here on how to rectify this:

Hi folks,

If you are not seeing the new Application Mapping channel, I suspect that you are on an upgraded appliance, and at some point a customised definition of the Applications page has been created, which is therefore overriding the current page layout.

Take a look in /usr/tideway/data/custom/reports and look for a definition in one or more of the reports definition files with a .xml suffix. It might look something like this:

<page name=“ApplicationHome”>
    <channel>Applications.Channel.Summary</channel>
    <channel>Impact.Channel.ApplicationImpact</channel>
</page>

If there is something like this, it is overriding the defined UI and hiding any product updates. There may be other page definitions too, doing the same thing.

The content of data/custom/reports is all customisation; if you don’t know what it’s customising, you could start by moving all the xml files somewhere else and restart the services, which will return everything to default, and therefore reveal the current UI. Then if necessary, you can selectively re-apply any required customisations.

We’ll investigate how we can at least detect and highlight any potential overrides like this during upgrade, though it probably won’t make the next release.

To close off, I’d like to reiterate Mark’s parting comment. It might not happen straight away, and I can’t comment on when you’ll see any changes, but we’re going to look into how customizations like this interact with upgrades. If you have any thoughts on that, feel free to drop by the forums to let us know.

So we dedicated time in our last two development sprints to focus on the Reasoning slowdown. We set out with two questions in mind: Can we reproduce this slowdown in our labs in a controlled environment, and will restarting the Reasoning service reliably speed things back up?

We’re not through with this investigation yet, and I’ll explain the road we’ve been down in a second, but the summary so far is that we’ve found two interesting results:

1. (At least in tests in our labs) the size of the ADDM Datastore’s cache contributes more than we previously believed to the performance of discovery runs.
2. Once the Datastore cache is full, the Reasoning service CAN start slowing down, which will increase the duration of your discovery runs. A restart of just the Reasoning service DOES improves performance (although it won’t bring run durations back to initial levels).

So how did we come to these findings?

Several Weeks Earlier

For performance investigations, if possible, we tend to use Record data and run ADDM in playback mode in order to remove the timing variance introduced by discovering real targets on a network. We already have sets of test Record data and we picked one of these – a range of 3,700 IP addresses containing 2,800 Hosts and 180 Network Devices. We created a single discovery run to be executed on a stand-alone ADDM appliance (no Consolidation) running ADDM 8.3 and configured to use 3 ECA Engines.

The first aim was to reproduce a slowdown on successive scans. Starting from an empty Datastore we executed our test discovery run multiple times, consecutively. With these sort of tests you have to discount the first scan as ADDM will have to do a lot more work creating new Hosts and Network Devices in an empty Datastore, compared to validating them on subsequent scans. What we saw was the initial scan took 3hr20m, and the first two rescans took 4hr and then 5hr20m. Great, so we’re seeing a slowdown already – time to restart Reasoning and see if this helps. We did, and it did – after a restart we were back down to 3hr45m.

We repeated these tests a few times and were convinced that there WAS a slowdown, and that restarting (just) Reasoning DID speed things back up. So far, so good – we have reproduced what our community has been seeing in a controlled environment. Unfortunately we’re only just getting started – we need to know WHY the system is slowing down, and more importantly what we can do to address this.

The next few days were spent eliminating possible causes of the slowdown. Some of the more interesting things we proved were NOT causing the slowdown were:

• The TKU patterns: We removed these from our test system with the effect that our tests now ran faster as no patterns were being triggered, but the relative slowdown remained.
• The use of persistence files in Reasoning: This feature allows Reasoning to pick up where it left off when restarted in the middle of a scan – we disabled this with no improvement to performance.
• The cache of Discovered Processes which Reasoning holds in memory: We have seen in the past that this is essential to get good performance from our patterns, but it does use a fair bit of memory. However, once again, disabling it gave us no performance improvements.

We rolled back all these changes – apart from disabling the TKU patterns. As the patterns made no difference we left them disabled to speed up our tests. A new run of our test produced the results in figure 1.

Around this time we also started adding lots of extra instrumentation to our code to help us drill into exactly where the extra time was going. Analysing the output from this instrumentation yielded our first significant result – comparing the fast and slow scans we determined that the extra time was being spent inside the Reasoning service, waiting for transactions to commit in the Datastore service.

A small bit of background here – during a discovery run, Reasoning essentially responds to discovered data by executing rules and patterns. Think of rules as just built-in patterns, and since we’d already disabled the TKU patterns the only relevant bit of work being done by Reasoning was in those rules. As part of rule execution, Reasoning will sometimes request more discovered data and will certainly interact a lot with the Datastore – reading and updating existing nodes, and creating new nodes. The updates and creations are batched up and sent to the Datastore inside transactions – and the time these transactions took to complete was increasing as our tests progressed.

To give you an idea of the numbers we’re talking about, during the execution of one of our test discovery runs that discovers 2800 Hosts, Reasoning asks the Datastore to create 600,000 new nodes spread across 45,000 transactions. On average that’s over 200 new nodes for every Host that we discover. This figure is explained by node kinds such as Discovered Processes, Packages and Patches – there can be an awful lot of these, depending on the Host being scanned.

Anyway, our instrumentation was showing that the number of node creations requests and spread of transactions was the same between the slow and fast runs, but the time the Datastore took to service those requests was much larger in the slower requests. Crucially this extra time that Reasoning was waiting for the Datastore could explain the whole of the increase in the overall discovery run duration. And for some reason, a restart of Reasoning was bringing this time down again (although never back to the original discovery run durations). This was very, very odd – we don’t hold state inside the Reasoning service across discovery runs so what was happening inside it to cause another service, the Datastore, to slowdown?

Our second interesting result followed shortly afterwards, as we experimented with changing the number of ECA Engines that Reasoning was running in parallel. We found that when we ran with a single engine we did still experience a slowdown in those transaction commit times, but a restart of Reasoning did NOT help recover any time. So it now looked like we were dealing with two issues – an unexpectedly large slowdown of the Datastore as it fills up, and some sort of interaction between multiple Reasoning engines which exacerbates the situation.

Just The Datastore

In order to isolate the Datastore issue we started coding and knocked up a test harness that performed exectly the same 600,000 node creation requests that Reasoning was doing during one of our discovery runs (and across the same 45,000 transactions). We emptied our Datastore and then left this test harness running in a loop. The first iteration of the harness completed in around a quarter of an hour, but by the 15th iteration it was taking one and a quarter hours – so the rate of node creation we were seeing had degraded from 600/sec to 120/sec. See figure 2 for the full results.

So we could reproduce the Datastore slowdown in a simple test harness – that’s very useful because it makes it easier to drill into the problem. A further round of instrumentation led us in fairly short order to the code that creates indexes for the new nodes in the Datastore.

A bit more background: our Datastore indexes all attributes on all nodes automatically so that searching them later on is really fast, whatever query is submitted (by the system or a user). We use Berkeley DB (BDB) as our underlying data storage solution, and store both raw nodes and index information in this. We index on a per-word basis too, so we can search quickly for single words in our data – and this requires many index records to be created in BDB for each higher-level node that we create in ADDM. Especially for some of the DDD node types like Discovered Processes, that can be a lot of index records. It seemed that this index creation process was slowing down much faster than we’d expect as the Datastore filled up. Also important to note is that BDB contains an in-memory cache – the size of which is configurable via the ADDM Administration/Model Maintenance page.

We put our heads together, thinking through how we model node indexes deep down in BDB and decided we needed to see just how effective that BDB cache was proving for our index data. It’s fairly easy to capture statistics from BDB about the cache usage, and when we did this we saw that the number of cache misses was rising over time, as the number of records we had inserted into BDB increased. So the cache wasn’t big enough for our test – we can change that. From the Model Maintenance page we upped the cache size from (the default) 1Gb to 4Gb and re-ran our test harness. The results? No change in performance between iteration 1 and 15 – the slowdown has gone away, as can be seen in figure 3.

Add Reasoning back into the mix

That’s great, but putting our test harness to one side does the cache explain the behaviour we’d been seeing when running the full system? Well, capturing the number of cache misses per test discovery run and overlaying them on our original full results from the discovery runs, we see a fairly striking correlation (figure 4).

What we couldn’t explain yet is why restarting Reasoning temporarily reduces the number of cache misses, but holding that thought for a second, we increased the cache size in these tests to 4Gb as well and then re-ran. The results were good: once again the slowdown went away – see figure 5.

(Note we didn’t clear out our Datastore before running this last test – hence no inital spike for the first full scan. But the fact that we started from a populated Datastore shows that the cache really helps us cope with larger amounts of data).

So where are we? It looks like as long as we don’t run out of Datastore cache space, the performance of the Datastore and of Reasoning stays good. But once the cache fills up, the Datastore can slow down, and Reasoning makes the situation worse – which is at least partially addressed with a restart.

We’re still committed to find the cause of the Reasoning part of the slowdown, but now we understand more about the importance of the cache in controlling our test discovery run durations, we want to see if this translates to real customer deployments. It’s too big a jump from tweaking a setting in a controlled experiment in the lab that uses Record data, to stating that the Datastore cache is the cause of performance problems in the field. Every customer uses ADDM in a slightly different way, and some of you have already tried changing your cache sizes – so we’re in the process of modifying ADDM to capture Datastore cache statistics so we can help our customers see if they are suffering from this problem and advise them accordingly.

And we should also mention we’re fully aware that finding a solution to this issue won’t be the end of performance improvements in the product. At the same time this investigation has been taking place we’ve also been rewriting key parts of our SNMP discovery code – which is mainly used when ADDM discovers Network Devices. This code was originally inherited from a legacy BMC product and we’ve been wanting to make some important changes to it for a while now – to improve stability and performance. This work is going well, with calls to our SNMP code now less monolithic – meaning SNMP discovery requires less memory and ADDM can optimise so that devices are not repeatedly queried in the same discovery run. Expect these changes in ADDM 8.3.2.

At a high level in terms of scale, we’ve also begun an exciting project to turn ADDM into a farm-based tool that can be scaled out by adding more compute nodes. This is going to be our largest area of investment in the next year. And two more items we know we need to look at in terms of performance are DDD aging and Consolidation. Both of these are having development time allocated to them in upcoming sprints, just as this issue has.

The latter of those is still an ongoing process, and we’d like to apologise for any links you might find that lead to restricted pages – those will be either fixed or tidied away in the very near future. We’ve got a lot of content that’s been moved to a locked-down “holding pen” whilst we determine where it needs to go and what patching up is needed. If the pages only need a lick of paint and a bit of a polish then they’ll be back shortly. If they’re beyond saving then we’ll patch up the holes they’ve left.

You should expect to see more changes to discovery.bmc.com in the coming months as we focus in on ways of getting our web content and community tools to be as useful and intuitive as possible. We’re also continually looking into ways to get them to work with other tools within BMC so you don’t have to waste time checking around different places quite so often.

Last time I posted, I mentioned our “Migration Masterplan”. As you might expect, that’s undergone a few revisions in the past year as the landscape has changed around us. But we’ve not been standing still. Our approach to documentation is starting to spread within BMC, and Vitaly Burlai (discovery.bmc.com’s fearless developer) is heavily involved in that process. Some other BMC products are now producing their documentation on the Confluence platform, and we plan to eventually bring our own product documentation into that. Our approach to community is also gaining some traction, although that’s a slower process… After all, a community is a group of people, not a piece of software. It takes a bit more time… you can’t just upgrade to a new version!

In line with what I said in my previous post, our focus is on doing the right things and working with our community. We plan to keep moving forward, rather than just migrating to existing platforms.

About a year ago, Tideway was acquired by BMC, and Tideway Foundation became BMC Atrium Discovery & Dependency mapping. As you might expect, there will be changes to tideway.com and the Tideway Community as a result. If you’re reading this, you’ve probably noticed that Tideway.com has a new look – one that is much more BMC-ish. This is the first of a series of significant changes that you’ll see over the coming months.

We have a fairly involved plan for the gradual migration of this website into various different places within BMC, but we (being a mix of BMC and ex-Tideway folks) realise that we can’t just lift things up from one platform and drop them onto another. It’d be like trying to mix two drinks by putting them in different cups, and then banging the cups together. In other words, it’d be both counter-productive and messy. Instead, we’re taking our time and migrating our web content a piece at a time. Since this is going to take a while, we’ve redesigned this site to bring it in line with other BMC web properties, and soon you’ll be able to reach it via BMC web addresses as well as Tideway ones.

The first thing that you’ll notice is that the “brochureware” side of the site isn’t up there in the tabs anymore. If you’re here instead of on bmc.com, you’re probably one of our users already and you don’t need all of that. We’re filling that need by building up the product pages on the BMC website instead.

This means that the focus of this site is now squarely on the community. This isn’t an accident – our community of users is important to us and we’re not going to take this place away until we’re certain that there’s a good home ready for us in the BMC Developer Network.

So what is the plan?

We’re working to something I’ve been calling The Tideway Migration Masterplan. It’s all very involved, and contains a lot of interdependencies, so I’m not going to go into it in very fine detail.

It’s got six steps, and we’ve just completed step 3 – migration of marketing material. We’re also well underway with step 4, which will be the major step of bringing everything under a bmc.com domain name and branding. Why are we doing this? So it’s less jarring for folks who’ve only ever dealt with BMC and have never heard of Tideway! It will also make it easier for us to gradually tie into other systems within the BMC web presence, and gradually pull the two communities together in a way that makes sense.

Once that’s done, you should expect to see our documentation start to integrate with the rest of BMCs product documentation. This isn’t going to happen right away as there are changes afoot in how BMC handles and presents its documentation. We’re heavily involved with those changes, but we’re waiting for them to fully bed in before moving our documentation into BMC’s new (yet curiously familiar) platform.

Lastly, you’ll see our forums and other community features merge in with the BMC communities. As with documentation, there is work taking place within BMC to evolve their website’s community features, and we’re not going to bring the two together until we’re sure it’s a good fit.

When should we expect to see all this?

We’re not working to a strict timeline at the moment. This might seem odd, but doing it right is far more important to us than meeting arbitrary deadlines. We’ve always worked with the idea that a community is a group of people with common interests, not just some software on a website. We support that community by trying to keep the tools we all use to communicate from getting in the way, and by trying to make our communication as seamless and easy as possible. We want to make sure that when we change the tools around, we don’t damage the community that they’re meant to help. We want to make sure that the move from one platform to another doesn’t get in the way of the community, and that we can maintain the level of communication and interaction that you’re all used to.

http://communities.bmc.com/communities/community/bmcdn/bmc_atrium_and_foundation_technologies/discovery

The January release includes 6 new products taking the total to 369 software products and over 29,000 software product configurations discoverable.

The release focuses on extending discovery coverage of BMC products and includes patterns for:

• BMC Atrium CMDB
• BMC BladeLogic Client Automation
• BMC CONTROL-M/CM for Advanced File Transfer
• BMC Patrol Enterprise Manager
• BMC ProactiveNet Analytics
• Remedy AR System

You can visit the In the Spotlight page on Configipedia to find out more about the release so you can confident in the discovered results returned.

The December release includes 8 new products taking the total to 364 software products and over 29,000 software product configurations discoverable.

The release focuses on extending discovery coverage of BMC products , particularly the CONTROL-M family.

You can visit the In the Spotlight page on Configipedia to find out more about the release so you can confident in the discovered results returned.

The November release includes 13 new products taking the total to 356 software products and over 29,000 software product configurations discoverable.

The release focuses on extending discovery coverage of IBM products specifically to support license audit initiatives and provides broad coverage across the all the major IBM software product brands such as:

• Cognos
• DB2
• FileNet
• Infosphere
• Rational
• Tivoli
• Websphere

To ensure quality, pattern developers leverage our relationship with IBM by either accessing product expertise or accessing products in IBM Innovation Center labs.

You can visit the In the Spotlight page on Configipedia to find out more about the release so you can confident in the discovered results returned.

Why is it important to track software going end of life? Here’s a few reasons Tideway customers tell us why they are using Foundation to track software nearing end of life.

• Identify operating systems and software that are exposed to security / stability risks as they are no longer supported by patch updates
• Identify operating systems and software that we may be paying a premium maintenance subscription for
• Identify operating systems and software that are no longer shipping may lead to increased costs and risks associated with variance

Products Reaching End Of Life In November 09

• HP Client Automation 4.2, 5.0
• HP DevInspect 4.0, 5.0, 5.1
• IBM Tivoli Netcool/OMNIbus 7.0

Products Reaching End Of Life In December 09

• Borland StarTeam v9.0
• CA Directory v8.1
• Ingres Ingres Database v2.6
• McAfee VirusScan Enterprise v8.0
• MySQL AB MySQL Database Server v5
• Oracle Berkeley DB v4.3
• Oracle Database Lite v10.2
• Sybase Adaptive Server Enterprise v12.5
• Sybase jConnect for JDBC v5.5
• Sybase Replication Server v12.6
• VMware Consolidated Backup v1.1
• VMware VirtualCenter v2.5
• VMware High Availability v2.5
• VMware VMotion v2.5

Go to Oracle October CPU web page

Oracle recommend that organizations “have a complete inventory of Oracle products across the IT enterprise, with full version numbers” however do not appear to have a solution for this. As a result we hear that many organizations are unable to apply the CPUs and as a result are exposing themselves to critical security vulnerabilities.

Follow these links to the find out more about how Tideway identifies Oracle Application Server , Oracle Database , E-Business Suite , Weblogic and JRockit inventories.

Or why not try out Tideway Foundation by downloading the Community Edition

Steve