Suffering CPU, DB, or memory problems in CF? Spiders could be killing you in ways you'd never dream

It may seem innocent enough: search bots or other automated tools may be visiting your site many times a day, due to many different search engines, and perhaps even many times per day per bot. These may be outside search bots or internal ones (like a Google appliance). Or it may be caused by load testing tools, or simple monitoring ping tools. Or you may have hundreds or thousands of folks signed up with RSS readers to watch your RSS feeds. You may have code that uses CFHTTP to request templates on your own site (or others may be scraping your site).

The problem is that if the pages being visited by these tools are CFML-based, then the bots and RSS readers will not likely track the session/client cookies CF sends, which starts a real waterfall of problems. There are two potential impacts, sessions and clients.

Update: Rather watch than read? Since writing this blog entry in 2006, I have since given a talk on the subject (in 2009), which was recorded. More here.

Creation of many new sessions

Assuming you have SessionManagement="yes" enabled, each such new request without a cookie will cause the CFML server (CF, BD, Railo, or other) to create a new session. Normally, that's not a problem. The browser for a typical user will usually store these cookies for reuse on a future page request.

But since these bots do not typically track cookies, then this causes the CFML server to create a new session for each page request.

And that's not one new session per bot, but one new session per bot per page requested.

And these new sessions will live as long as your sessiontimeout is set--which could be minutes, hours, or days. That could become a substantial resource for CFML server to manage, even if there's "nothing" in the session.

(Props to Mark Kruger whose blog entry, Sessions and Cookies and Bots (oh my), was the first I saw to point this out. As I pointed out in his comments, the problem could be still worse with respect to client variables also, as I'll explain below. And since then, I've realized that RSS Readers could be another, and different problem, since the number of individuals running them may be far greater than the number of search engines.)

So the challenge is to find out how many sessions you have currently. More in a moment.

Creation of many new sessions with large amounts of memory per session

Further, beyond the number of sessions is the question of how large the sessions are. If you DO for some reason put a lot of data into "new sessions" when they're created, then this could become a huge memory burden. And as memory use increases substantially, so could the cost of garbage collection. Eventually the CPU to manage that GC could become problematic, and your system could become unresponsive or even unstable.

Creation of many client records

Finally, there's even a more pernicious (and more persistent) problem due to client variables. Now, even if you'd say "we don't use them", consider this: if you've got ClientManagement="yes" set in your CFAPPLICATION, and you've not disabled in the CF admin the "Disable Global Client Variable Updates" option for your client repository (and they are enabled by default), then the CFML server will create/update fixed client variables (hitcount, lastvisit, etc.) for every page visit. Besides the possible I/O burden (whether in a database or the Registry), given the problem of bots discussed above, this would also mean a new set of these client tracking records would be created for each visiting bot request!

If you're storing these in the Registry, any great increase in the number of entries is clearly bad enough.

But whether you're storing these auto-created variables there or in a database, the problem is quite different from the wildcat creation of sessions. At least those are removed when their sessions timeout or the server is restarted. With client variables, though, these are typically set to expire in days, weeks, or even months! That means they last that long on your server.

So either your registry or client database tables (or both, since each app can choose its own) could become very large and burdensome to be managed. Note as well that CF wakes up every 67 minutes (by default) to go through the client repositories to find any clients that can be expired.

Between that and the initial creation of client records, the volume of updates (or inserts) into the DB can cause significant contention, preventing legitimate CF queries from processing. Do you see how very pernicious this problem becomes?

Diagnosis

If you think this may be happening to you (and even if you don't), you should set up monitoring to see how many unique new sessions or clients are being created. You have a few ways to do this.

Sadly, there's no documented mechanism for CF to tell you how many new sessions or clients have been created in a day.

Update: since writing this entry in 2006, there are now documented and undocumented solutions for both CF 8 and earlier.

First, if you use J2EE sessions, you can get a report of them in the JRun metrics. Just search for any of many articles on using JRun metrics.

But what if you use regular CF sessions? Well, here's good news. Whether running CF 6, 7, or 8, you can in fact get a count of sessions. See my later blog entry about some simple code by Mark Lynch that uses undocumented functions to report the number of sessions (J2EE or plain CF sessions), in total and per application. Very helpful for this challenge.

And if you use CF 8 Enterprise, its Server Monitor (and Admin API) also report the number of sessions, and you don't need to turn on memory tracking for this. Just see the Statistics tab>Active sessions, and click the chart icon on the right, which will show a screen with the total number of sessions.

That will tell you how many sessions you have, which can be very enlightening. I helped a client recently where we found they had 90,000 active sessions. Yikes!

Of course, the next question, then, is to find out why and how. Is it search engines bots? External or internal? Monitor pings to CF pages? RSS feeds? CFHTTP requests? Load testing tools?

If you can access them, you could analyze your web server logs to find out how many pages are coming in that have no CFID/CFTOKEN values sent in their cookies or query string. (Web server logs can be set to report incoming cookies.) That would be a clue, as it would cause CF to create new ones in response. Of course, it could be legitimate first time users.

Update: You can also view the "user agent" being presented by the browser requests, which can be set to be tracked in web server logs, and is also viewable in CF's CGI scope, as well as in the CF 8 Enterprise Server Monitor and FusionReactor's Request Details page. While bots and other automated request tools can mimic (report) any user agent, some do identify themselves plainly. You may be surprised to find how many visits you get from such requesters. A good log file analysis tool like Log Parser, or any others listed at my CF411 site list of Log Analysis tools.

You could also look at either the registry or client variable database to see how many entries there are. You may be shocked by the number. (Update: Just be careful not to open the Registry location for CF client variables using Regedit or similar tools. If there are a huge number defined there, just trying to view it could bring down your server. A tool like Log Parser, mentioned above, can help, though, as it is in fact able to read and count data in the Registry.) You could also export the key from the registry, to see how large it is, but again it could take a long time if it's large, so it may be best to do this during a scheduled outage.

Remediation

The simplest thing is to ensure that any code that may be hit by bots, search engines, RSS readers, etc. do not use code that has CFAPPLICATION SessionManagement="yes" or ClientManagemenet="yes" (or the equivalent properties in application.cfc). That may be trivial, or it may be a hassle, depending on the complexity of your application.

Update: here's a thought that's come up from discussions on a private mailing list. One solution to consider would be to detect if the request has the CFID cookie--which it wouldn't for bots, and if not, set the session timeout to a short value. Be careful, though, because legitmate first-time visitors will also have no CFID token, so if you set any session vars on the page they request or in the application.cfm/.cfc file, don't set it so short that those will be gone when they go to the next page they visit. Perhaps set it to a minute, but realize the implications. I hope to create a blog entry showing the simple code to do this, but until then I hope the idea is simple enough for you to run with it.

Update: As far as the global client variables updates are concerned, you should first check if in fact your client variable repositories (in the CF Admin Client Variables page) have been modified to disable global variable updates. If not, it's not an obvious thing to disable that. You need to make sure there's no code trying to use client.hitcount or client.lastvisit, which are the variables created by this feature. Assuming there is none, then you could disable this. That doesn't remove the entries created to this point, but it does stop the inserting and updating of entries there, except for pages that really DO use client variables. Just beware that you still need to get rid of those old unneeded entries, or CF will continue to purge them every hour.

Update: As far as purging client variables if perhaps a large number have been created, there are two different solutions, for the Registry and for any datasource. Before doing it, though, note that you would technically want to purge only that data related to requests that were not "really" using client variables. More on that in a moment.

First, as for client variables stored in a database, see an explanation and code offered in Adobe technote 18514 (again, no longer available at http://www.adobe.com/go/tn_18514, but available via archive.org.) Bottom line: there are two tables used for client management (in any datasource used to hold client variables). CDATA holds any "real client variables" created on a CFML page. CGLOBAL holds the "global client variables" discussed above, hitcount and lastvisit, as well as the CFID, CFTOKEN, and application name. That's the data used to manage the retention of client variables. While you may be tempted to "blow away" both tables, note that if there are real client variables in the CDATA table, you would not want to delete that if you are really still using client variables. And you'd only want to delete records in CGLOBAL that are NOT related to those CDATA records.

Second, as for the Registry, it may be tempting to go in and delete the key that holds the client variables, but beware of that. For one thing, you still have the dilemma, like above, that you would be deleting all the client variable entries, including any that you really meant to write. But unlike with the database, there's no easy way to look up which entries are NOT associated with "real" client variables, so I can understand the temptation to just delete them all. That's a call you have to make.
As for deleting them, note that it could take a very long time, if there are hundreds of thousands or millions of keys. You may want to do it overnight or during system downtime.

Now, while you could use RegEdit to try to view how many there are (and delete them), beware: if you open the location where they live (HKEY_LOCAL_MACHINE\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Clients) and you expand the tree for this key, it may bog down or crash your server. You could choose to export the keys, to see how many there are. That too may take a long time.

If you're prepared to just delete them, you have choices. You could right-click and delete it, but that could lock up regedit for a long time. It would be better to delete them from the command line. It may still take a long time, but you at least remove the risk of tying up resources in the RegEdit GUI.

The inimitable Dave Watt's has noted that the Reg command-line tool (built into Windows) can do this:

REG DELETE HKLM\SOFTWARE\Macromedia\ColdFusion\CurrentVersion\Clients

There is an available /f argument you can add at the end to cause it to delete without prompting for each key. Note as well that this tool also offers query, export/save, and even compare arguments.

(While there is also code available from Adobe to help view/purge client variables in the registry, it's CFML that uses CFREGISTRY, which may not be wise to tie up a CF request thread for a long time. But if you're interested, see the zip file in Adobe technote 17881. The old link, http://www.adobe.com/go/tn_17881, fails, but here it is via archive.org.)

Finally, if you may just want to stop CF from being able to even TRY to write to this directory, consider the idea offered by Russ Michaels, where you can disable CF's ability to write to this part of the registry and prevent therefore any attempt to use it.

Also, if you're on Linux, note that there is no real registry on your server, but CF stores client variables (if told to use the "registry") in a file called cf.registry, typically in /opt/coldfusion/registry/cf.registry. You can edit or delete that file as desired to resolve client variable problems.

Other thoughts

If you use load testing tools, be sure to enable any option to have them honor cookies. Otherwise your testing results may not be accurate, as you're imposing this burden on the server of it creating new sessions and clients for each user request you're simulating, which would not happen in production (except for the bots and RSS readers, etc.).

Hope this info may help others.