CF911: Easier thread dumps and stack traces in CF: how and why

Note: This blog post is from 2009. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

(If you're not familiar with the value of thread dumps or stack traces, read on. The resources I point to get help you to appreciate their usefulness.)

Update 1: I did a several minute youtube video on this topic.
Update 2: I did a presentation on this topic, posted here. After reading the below, you may want to view that with more details on stack tracing any of many ways.

Well, I was tooling around looking for some help on a topic when I happened upon an old entry from the awesome and ever-valuable blog of Steven Erat, formerly of Adobe and now of Webapper. The entry is An easier way to take ColdFusion thread dumps on Windows , and he actually points to entry in another classic blog by Brandon Purcell (formerly of Adobe and now with Universal Mind). Steven's entry also points to a classic Adobe technote on interpreting dumps and stack traces.

But both entries are from 2005, and since then there are still easier solutions for getting either a thread dump or stack trace, and I wanted to point them out (I'll also add a link to this entry in Steven's.)

For those on CF 6, 7, 8

First, for those running CF 6, 7, 8 (and other CFML engines, like Lucee, Railo and Open BlueDragon), you can use either FusionReactor or SeeFusion, commercial monitoring tools, which each offer a simple single button to produce a thread dump (a stack trace for all threads running inside the JVM that underlies CF).

Better still, they also offer an option to get a stack trace for a single running CFML page request. While looking at running/active requests, just click on the button in the respective tool's interface to get a stack trace at taht moment of a given running request. As for how to make use of the result, see the resource above or the presentation I mention below. It's a powerful way to know what's happening at a point in time for a long-running request.

For those on CF 8+ Enterprise

If you're on CF 8+ Enterprise, you can take advantage of its Server Monitor, which also offers the ability to get a thread dump, by way of its "snapshots" feature.

I discuss the Server Monitor in a 4-part series of articles for Adobe, starting here.

Part 3 of the series discusses Snapshots in particular.

(I also have an an article introducing FusionReactor.)

The CF 8 Server Monitor can also provide the stack trace of a given request, but you need both "start monitoring" and "start profiling" to be enabled. If they are, you can see currently running requests in the Active Requests screen, and if you click on one, the stack trace appears in the middle of the request detail page.

But maybe you don't need to do a thread dump at all

All that said, I should point out that one of the values of thread dumps was to see what requests were running, which is incredibly valuable.

But if you do get either FusionReactor or SeeFusion, or can use the CF Enterprise Enterprise Server Monitor, note that they all have a feature to show what requests are running.

That, alone, can be what you really want to know, and their interface is a LOT easier to read than a thread dump. :-) And the stack trace feature let's you see exactly what line of CFML code a request is running at the time you request the stack trace.

Still, thread dumps can be valuable.

Getting thread dumps by email

Of course, you may not always be on (or be able to get on) the server when you want to see what's running. Here's good news:

All three of the monitors (FusionReactor, SeeFusion, and the CF Enterprise Server Monitor) offer an option to trigger sending you a thread dump by email when certain conditions are met (too many requests, requests taking too long, too little memory, etc.)

In FusionReactor, it's in the "crash protection" notification feature. In SeeFusion, it's the "active monitoring rules", and in the CF Server Monitor, it's the Alerts feature. See the docs or other resources for each for more information.

Non-request threads could be an issue

Also, sometimes a problem is not caused by a running CF request. It could be another thread that's having a problem, whether one of the scheduler threads (which are used for background tasks like the client variable purge process, and which have nothing to do with CF "scheduled tasks", which run as normal jrpp* or web* threads), or cf-thread threads (used by CFTHREAD as of CF8), and so on. This is another area where thread dumps (dumps of all the stack traces for all the threads) can be valuable.

Watching thread dumps over time

And whether watching the activity of running requests or other kinds of non-request threads, the key to understanding thread dumps (in addition to what is mentioned in the articles and presentations mentioned elsewhere here) is to view them over time (such as doing a refresh while viewing one interactively, or looking at two in a row if received by email.)

A tool to help analyze thread dumps

Indeed, Steven would also want to point out that Webapper (makers of SeeFusion) also have a free online tool called SeeStack which can help analyze thread dumps to make them a little easier to read.

Still more on thread dumps and stack traces

Further, the Adobe technote that Steven points to, Debugging thread dumps and server problems in ColdFusion MX 6.1 and 7.0, also gives considerable insight into understanding stack traces and thread dumps. It's also quite old and doesn't talk about things with respect to the more modern tools I've mentioned, but the information is valuable whether you use them or not.

In fact, much of it is about understanding stack traces (remember: a thread dump is a list of stack traces for all threads in the JVM), and connecting the dots in what they report and specific possible problems in your code or configuration.

I plan to do some entries in the near future, walking through use of these tools to solve common problems of CF servers being unavailable. Until then, check out also the "related entries" listed at the bottom of Steven's entry.

Of course, there are many other fine classic blog entries (and bloggers) who talk about CF troubleshooting, too. I also plan to offer something to help with finding those.

Update: I did a presentation on this topic, posted here. After reading the above, you may want to view that with more details on stack tracing any of many ways.

For more content like this from Charlie Arehart:

• Signup to get his blog posts by email:
• Follow his blog RSS feed
• View the rest of his blog posts
• View his blog posts on the Adobe CF portal

Need more help with problems?

• If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
• See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed