Suffering CPU/DB/memory problems in CF? Spiders, monitor pings, and more may be to blame

Note: This blog post is from 2006. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

What if the problem isn't really a "leak" at all, nor "poorly performing code", nor "CF being unable to scale", but instead what if it's really all due thousands or millions of page requests, which (worse) are likely creating (unexpectedly ) thousands or millions of sessions and client variables, each day? They may even be something you're causing (but more likely not). It's a pernicious problem that many never even fathom, or may dismiss too readily.

Curious to hear more? Read on.

So what kind of automated traffic may be hitting your site?

There are many more than you may think, and on the surface they may all seem innocent enough.

First are search bots , which may be visiting your site many times a day to check on your content, to index it so that people can use those search engines and find your content. These may be outside search bots (Google, Bing, Yahoo, and more) or perhaps internal ones (like a Google search appliance). Search engines are good, right? They help people find your content. And the search engine spiders need to visit your site to see what you have, and know when it's updated. But the impact (and volume) may be far greater than you realize.

Did you know that they (Google, Bing, Yahoo, and perhaps many others) come to your site (if it's open to the public) likely visiting every page they can find on your site. EVERY page. And they come EVERY day (asking for all of them)? And if you have multiple domains, they do it for EACH domain. Multiply this by more and more such spiders (many you may have never even heard of), and it may account for tens of thousands of page visits per day, or more. (And the problem is not just THAT they come, but what happens that you may not expect, WHEN they come. More in a moment.)

Or such excessive automated traffic may be caused by simple site monitoring tools (that you or your clients may have setup), checking some page(s) on your site perhaps many times per minute (they want to know if your site is "up", after all).

Or maybe you're in a load balanced environment. The load balancer (whether outside the CF box, or inside it) typically sends requests to some page (that someone configures), to "check if CF or the app is up and running". Well, that too happens, perhaps several times per minute.

And each of the above two may be going against more than one domain on your server, right? And maybe checking both http and https and so perhaps causing many requests WITHIN SECONDS (and this is on top of the search engine spiders above). But that's not all.

Maybe you have site security scanning or PCI compliance scanning which run frequently (perhaps even every day), or load testing tools which run occasionally. You may not even be aware that such things are running, as others may kick them off. Those, too, create high rates of frequent requests to pages.

Or you likely have CF scheduled tasks which run daily. I've seen some people with several of them, running very frequently (perhaps monitoring something internally, to act upon it, like files put in a certain directory--better monitored with a a file watcher gateway, but that's another topic, or watching tasks put in a database perhaps.)

Those scheduled tasks are similar to the automated requests I'm getting at above (and the reason they are a concern is not really about their volume, which may pale in comparison to the rest of the things above, but about something else I'll get to in a moment.) But let's keep thinking about what other kinds of automated requests may be happening on your server (this is just step one in my discussion here).

Perhaps you may have code yourself (running in CFML) which uses CFHTTP to request templates--which may be on your own site. No, that's usually not logical or optimal, but I see it all the time.

Or certainly you could have requests coming from other servers in an automated fashion running pages on your CF server, whether using their equivalent to scheduled tasks, or CFHTTP, or web services, or API calls into your server. My point is that they may be happening repeatedly and frequently.

And some of those may be from automated tools used by folks to scrape your site to get your content, to use for whatever purpose (steal it, resell it, do espionage against you, whatever). I'm not trying to be alarmist here. I'm just saying I see it all the time, when I help people troubleshoot "performance problems" in CF.

Or let's pick something more innocent: you may have hundreds or thousands of folks who have signed up with RSS readers to watch your RSS feeds . Those may come to check your site for changed content, each day, or even more often.

OK, so we've established THAT such automated request tools can come in at potentially high rates and perhaps far higher volume that we realized. And ok, perhaps we now may realize/find that they account for a large percent of CF page visits (I've seen it as high as 80%).

That's bad enough (and you may be tempted to block some, or may want to address some, and we'll talk about that.)

But there's really a more insidious problem, with all these automated requests.

The Problem(s)

The problem is that if the pages being visited by these tools are CFML-based, then the bots and RSS readers will not likely track the session/client cookies CF sends, which starts a real waterfall of problems. There are two potential impacts, sessions and clients.

Update: Rather watch than read? Since writing this blog entry in 2006, I have since given a talk on the subject (in 2009), which was recorded. More here.

Creation of many new sessions

Assuming you have SessionManagement="yes" enabled, each such new request without a cookie will cause the CFML server (CF, BD, Lucee, Railo, or other) to create a new session. Normally, that's not a problem. The browser for a typical user will usually store these cookies for reuse on a future page request.

But since these bots do not typically track cookies, then this causes the CFML server to create a new session for each page request.

And that's not one new session per bot, but one new session per bot per page requested.

And these new sessions will live as long as your sessiontimeout is set--which could be minutes, hours, or days. That could become a substantial resource for CFML server to manage, even if there's "nothing" in the session.

(Props to Mark Kruger whose blog entry, Sessions and Cookies and Bots (oh my), was the first I saw to point this out. As I pointed out in his comments, the problem could be still worse with respect to client variables also, as I'll explain below. And since then, I've realized that RSS Readers could be another, and different problem, since the number of individuals running them may be far greater than the number of search engines.)

So the challenge is to find out how many sessions you have currently. More in a moment.

Creation of many new sessions with large amounts of memory per session

Further, beyond the number of sessions is the question of how large the sessions are. If you DO for some reason put a lot of data into "new sessions" when they're created, then this could become a huge memory burden. And as memory use increases substantially, so could the cost of garbage collection. Eventually the CPU to manage that GC could become problematic, and your system could become unresponsive or even unstable.

Creation of many client records

Finally, there's even a more pernicious (and more persistent) problem due to client variables. Now, even if you'd say "we don't use them", consider this: if you've got ClientManagement="yes" set in your CFAPPLICATION, and you've not disabled in the CF admin the "Disable Global Client Variable Updates" option for your client repository (and they are enabled by default), then the CFML server will create/update fixed client variables (hitcount, lastvisit, etc.) for every page visit. Besides the possible I/O burden (whether in a database or the Registry), given the problem of bots discussed above, this would also mean a new set of these client tracking records would be created for each visiting bot request!

If you're storing these in the Registry, any great increase in the number of entries is clearly bad enough.

But whether you're storing these auto-created variables there or in a database, the problem is quite different from the wildcat creation of sessions. At least those are removed when their sessions timeout or the server is restarted. With client variables, though, these are typically set to expire in days, weeks, or even months! That means they last that long on your server.

So either your registry or client database tables (or both, since each app can choose its own) could become very large and burdensome to be managed. Note as well that CF wakes up every 67 minutes (by default) to go through the client repositories to find any clients that can be expired.

Between that and the initial creation of client records, the volume of updates (or inserts) into the DB can cause significant contention, preventing legitimate CF queries from processing. Do you see how very pernicious this problem becomes?

(What's worse, if you're on Linux/OS X/Unix, note that there is no real registry on your server, but CF stores client variables--if told in the Admin or code to use the "registry"--in a simple file file! It's called cf.registry, typically in /opt/coldfusion/registry/cf.registry. All this discussed here applies all the more for *nix users who find their code using "the registry" for client variables.)

Diagnosis

If you think this may be happening to you (and even if you don't), you should set up monitoring to see how many unique new sessions or clients are being created. You have a few ways to do this.

Sadly, there's no documented mechanism for CF to tell you how many new sessions or clients have been created in a day.

Update: since writing this entry in 2006, there are now documented and undocumented solutions for both CF 8 and earlier.

Some ways to track the count of sessions

First, if you use J2EE sessions, you can get a report of them in the JRun metrics. Just search for any of many articles on using JRun metrics.

But what if you use regular CF sessions? Well, here's good news. Whether running CF 6, 7, or 8, you can in fact get a count of sessions. See my later blog entry about some simple code by Mark Lynch that uses undocumented functions to report the number of sessions (J2EE or plain CF sessions), in total and per application. Very helpful for this challenge.

And if you use CF 8 Enterprise, its Server Monitor (and Admin API) also report the number of sessions, and you don't need to turn on memory tracking for this. Just see the Statistics tab>Active sessions, and click the chart icon on the right, which will show a screen with the total number of sessions.

That will tell you how many sessions you have, which can be very enlightening. I helped a client recently where we found they had 90,000 active sessions. Yikes!

Determining the source of high session/client counts

Of course, the next question, then, is to find out why and how. Is it search engines bots? External or internal? Monitor pings to CF pages? RSS feeds? CFHTTP requests? Load testing tools? Security scanners?

If you can access them, you could analyze your web server logs to find out how many pages are coming in that have no CFID/CFTOKEN values sent in their cookies or query string. (Web server logs can be set to report incoming cookies.) That would be a clue, as it would cause CF to create new ones in response. Of course, it could be legitimate first time users.

Update: You can also view the "user agent" being presented by the browser requests, which can be set to be tracked in web server logs, and is also viewable in CF's CGI scope, as well as in the CF 8 Enterprise Server Monitor and FusionReactor's Request Details page. While bots and other automated request tools can mimic (report) any user agent, some do identify themselves plainly. You may be surprised to find how many visits you get from such requesters. A good log file analysis tool like Log Parser, or any others listed at my CF411 site list of Log Analysis tools.

You could also look at either the registry or client variable database to see how many entries there are (or the cf.registry file if on *nix.) You may be shocked by the number.

Just be careful not to open the Windows Registry location for CF client variables using Regedit or similar tools. If there are a huge number defined there, just trying to view it could bring down your server. A tool like Log Parser, mentioned above, can help, though, as it is in fact able to read and count data in the Registry. You could also export the key from the registry, to see how large it is, but again it could take a long time if it's large, so it may be best to do this during a scheduled outage. More on some of these things below.

Remediation

The simplest thing is to ensure that any code that may be hit by bots, search engines, RSS readers, etc. do not use code that has CFAPPLICATION SessionManagement="yes" or ClientManagement="yes" (or the equivalent "this." properties in application.cfc). That may be trivial, or it may be a hassle, depending on the complexity of your application.

Update: here's a thought that's come up from discussions on a private mailing list. One solution to consider would be to detect if the request has the CFID cookie--which it wouldn't for bots, and if not, set the session timeout to a short value. Be careful, though, because legitmate first-time visitors will also have no CFID token, so if you set any session vars on the page they request or in the application.cfm/.cfc file, don't set it so short that those will be gone when they go to the next page they visit. Perhaps set it to a minute, but realize the implications. I hope to create a blog entry showing the simple code to do this, but until then I hope the idea is simple enough for you to run with it.

Update: As far as the global client variables updates are concerned, you should first check if in fact your client variable repositories (in the CF Admin Client Variables page) have been modified to disable global client variable updates. If not, it's not an obvious thing to disable that. You need to make sure there's no code trying to use client.hitcount or client.lastvisit, which are the variables created by this feature. Assuming there is none, then you could disable this. That doesn't remove the entries created to this point, but it does stop the inserting and updating of entries there, except for pages that really DO use client variables. Just beware that you still need to get rid of those old unneeded entries, or CF will continue to purge them every hour.

Update: As far as purging client variables if perhaps a large number have been created, there are two different solutions, for the Registry and for any datasource. Before doing it, though, note that you would technically want to purge only that data related to requests that were not "really" using client variables. More on that in a moment.

Remediating client variables in a database

First, as for client variables stored in a database, see an explanation and code offered in Adobe technote 18514 (again, no longer available at http://www.adobe.com/go/tn_18514, but available via archive.org.) Bottom line: there are two tables used for client management (in any datasource used to hold client variables). CDATA holds any "real client variables" created on a CFML page. CGLOBAL holds the "global client variables" discussed above, hitcount and lastvisit, as well as the CFID, CFTOKEN, and application name. That's the data used to manage the retention of client variables. While you may be tempted to "blow away" both tables, note that if there are real client variables in the CDATA table, you would not want to delete that if you are really still using client variables. And you'd only want to delete records in CGLOBAL that are NOT related to those CDATA records.

Remediating client variables in the registry

Second, as for the Registry, it may be tempting to go in and delete the key that holds the client variables, but beware of that. For one thing, you still have the dilemma, like above, that you would be deleting all the client variable entries, including any that you really meant to write. But unlike with the database, there's no easy way to look up which entries are NOT associated with "real" client variables, so I can understand the temptation to just delete them all. That's a call you have to make.
(Before you set off to delete them, consider also that you could help your cause by changing CF to not even regard the registry as a client variable option. See below for more on that.)
As for deleting them, note that it could take a very long time, if there are hundreds of thousands or millions of keys. You may want to do it overnight or during system downtime.

Now, while you could use RegEdit to try to view how many there are (and delete them), beware: if you open the location where they live (HKEY_LOCAL_MACHINE\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Clients) and you expand the tree for this key, it may bog down or crash your server. You could choose to export the keys, to see how many there are. That too may take a long time.

You can use the command-line Reg tool (built into Windows), as shared elsewhere by the inimitable Dave Watt's. This would export the specific registry keys for the client variables, though this too may take a long time. At least you'll know, in the end, just how many client vars were in the registry. To do that, do this:

If it says that there are no keys there, then you are fortunately not storing client vars in the registry. BTW, that really should be Macromedia, even for releases of CF from Adobe (8 and above). They never changed the registry key that they wrote to.

If you're prepared to just delete them, you have choices. You could right-click and delete it, but that could lock up regedit for a long time. It would be better to delete them from the command line. It may still take a long time, but you at least remove the risk of tying up resources in the RegEdit GUI.

Here's the command-line command to do this:

(While there is also code available from Adobe to help view/purge client variables in the registry, it's CFML that uses CFREGISTRY, which may not be wise to tie up a CF request thread for a long time. But if you're interested, see the zip file in Adobe technote 17881. The old link, http://www.adobe.com/go/tn_17881, fails, but here it is via archive.org.)

Stopping CF even trying to use client vars in the registry

Finally, if you may just want to stop CF from being able to even TRY to write to this directory, consider either of the ideas offered by Mark Kruger (to remove CF's even knowing of the registry as a client vars option), or that offered by Russ Michaels (to disable CF's ability to write to this part of the registry) and prevent therefore any attempt to use it.)

Other thoughts

If you use load testing tools or security scanning tools, be sure to enable any option to have them honor cookies. Otherwise your testing results may not be accurate, as you're imposing this burden on the server of it creating new sessions and clients for each user request you're simulating, which would not happen in production (except for the bots and RSS readers, etc.).

Hope this info may help others.

For more content like this from Charlie Arehart:

• Signup to get his blog posts by email:
• Follow his blog RSS feed
• View the rest of his blog posts
• View his blog posts on the Adobe CF portal

Need more help with problems?

• If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
• See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed