CF911: Tips for dealing with Windows service timeout, useful when CF's taking too long to stop/start
Note: This blog post is from 2011. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.In my last entry, CF911: Is your ColdFusion service taking too long to shut down? Find out why, I discussed the first of a two part answer to helping people who are finding that sometimes ColdFusion takes too long to shut down (or startup). That one talks about how to find out where CF may be logging info to explain why it's taking so long.
In this second entry, I'll address the separate but related problem, particularly if you're running CF as a Windows service, that you may find you get timeout errors from the Windows Services panel itself. I'll share some tips to help with that, which I share often with clients of my independent consulting as a CF troubleshooter.
The first thing to understand is that there is indeed a timeout (configurable) for how long the Windows Services control panel is willing to wait. Beyond that, though, there are some 3 more tricks you can use to avoid that timeout. (And I would say again that sometimes you really want to focus instead on WHY CF is taking so long to startup or shutdown, but until you do, the following info may help give you some breathing room.)
Update: I should add that a commenter suggested that another solution was just to kill the task. Um, yeah. I knew that, of course. :-) But the point of this entry, like the last one, is when you DON'T just want to kill it. In particular, I made the point in the last entry that if you want to find out *why* CF is taking so long to shutdown, then you need to let it run to completion to see if it may log some diagnostics. See the last entry for more on that.But yes, as Russ Michaels noted, sure, you can also just kill the task. I will say I try to avoid doing that whenever possible, to let CF end gracefully, unless I'm forced to kill it for urgency sake.
Note that these tips can be helpful for everyone who runs CF as a Windows service, and indeed for anyone running any application as a Windows service.
Tip 1: There is a timeout set for how long the Windows Services panel will wait
So the first thing to note is that, yes, the Windows Services control panel has a fixed duration for how long it's willing to wait for an action (start, stop, or restart) to take, which is configurable in the registry. If the service control action you request does not finish in time, you'll get an error like "The service did not respond to the start or control request in a timely fashion", with associated event log errors like 1053, 7000, 7009, 7011, and so on.
And technically, you should be able to control that timeout via a registry key, called ServicesPipeTimeout, which is discussed in this Microsoft technote and defined in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control.
Sadly, though I've tried it on multiple machines, I'm not finding it to work.
The technote shows clearly how to either update or add it (if it doesn't exist), including noting that you need to choose the "decimal" option to see the value, which is in milliseconds. The technote does note that you must restart Windows for that change to take effect (and I have done that, and double-checked the value, keyname, and location.)
If anyone has more thoughts on this, why it may not be working for me, feel free to share. Still, you will find many blog entries and discussions proposing this, for CF and other servers, so maybe it will work for you.
The technote doesn't clarify what the default is if the key is not there at all, though I've found some mention that it's 30 seconds. I suppose it could vary by version of Windows (anyone know for sure?).
(Note that there is another key you may see (or hear about) set in the same location: WaitToKillServiceTimeout. That is NOT related to this discussion. That key, instead, controls how long Windows should wait for any/all services to stop when you shutdown Windows itself, which is interesting to consider for itself, but not related to this particular blog entry about Services control panel timeouts.)
Of course, all the usual warnings apply about messing with the registry. But it's the solution, if you need it.
So again, this ServicesPipeTimeout registry key would seem to be the one that controls how long the Services panel is willing to wait for a requested action to finish.
Do beware that it's system-wide setting, controlling all services. What may be good for you (CF) may not be good for all services, so be careful. Second, again, note that the server will require a restart for that to take effect.
Whether it works for you or not, the remaining tips should be helpful.
For instance, if you can't change that registry settings, or don't want to, or you find some day the Services panel is still taking too long to stop a service and you worry it may timeout (the control panel), then there are 3 more tips I often share with my customers (and now you, dear readers).
Tip 2: Use stop/start, instead of restart
Again, if you're being plagued by a service start or stop taking a long time, here's something you may not have considered.
Most often when people want to restart CF (or some other service) they will use the "restart" action in the Services panel. That's logical, of course.
But consider that when you have a problem with the action taking too long (stop and/or start, since a restart does both), both have to finish in that configured timeout time (above).
You can help matters by doing not a restart but a stop and start separately. That way, each action can take whatever the max time is that's configured (per above) to do just the one action.
Sure, it's a little painful. You do have to wait for (and notice) the stop before then doing the start. There's also the risk that even the stop may take too long. I'll offer two more tips related to that.
Tip 3: Close the progress bar before its "times out"
This one may not be as obvious, but it does really help. When you have a start/stop/restart action (for the Windows Services control panel) taking a long time, you risk hitting that timeout above.
But notice that Services panel shows a progress bar indicating the duration of the action you've requested. Here's the tip: if/when it's about to reach the end of its indicated time, note that you can hit the "close" button.
That will prevent it complaining about not being able to stop the service. (And BTW, I've seen situations where hitting that timeout limit not only gives an error but even causes the start/stop/restart buttons to be disabled, requiring a restart of the box!)
Once you close the progress bar, you can use the refresh button in the Services panel (or f5) to watch for when the service goes from "stopping" to blank, meaning it's down. Same for when you "close" it while starting a service: it will go from "starting" to "started".
Just be aware: If you were doing a "restart" (versus a stop or start), and you hit the close button while the service (CF) is still stopping, note that when it does finally "stop", your having "closed" the window will in fact keep the "start" step of the restart from happening. You need to manually click the "start" button to get it to proceed.
Tip 4: Consider not using the Services Panel at all
Still another tip (which I use regularly) is that you don't actually HAVE to use the Windows services panel to stop/start Windows services. You can do it from the command line (or a shortcut or batch file).
To be clear, I'm not talking about starting the program itself from the command line (CF, as I've been discussing in these last 2 entries), but the Windows service that starts the program. It's a subtle difference.
As an alternative to starting the Windows Service (for CF) using the Service control panel (Control Panel>Administrative Tools>Services, to be clear), you can use a command line command like this, which I use to stop CF:
net stop "ColdFusion 9 Application Server"
The service name is whatever is shown in the properties for the service you want to stop. (And of course you would use a "start" command instead to start the service. There is no "restart" command, buy you could create a batch file to do both.)
You can run this from the command line (Start>Run>cmd) or as a batch file, but I prefer instead to just setup a Windows shortcut, which you can do by right-clicking the desktop and choosing new>shortcut.
That will prompt you for the command (it just asks you to "type the location of the item", but just enter that command above). It will then ask for a name for the shortcut. I call mine simply "stop cf9".
And while you can then execute it by going to the desktop and double-clicking it, a niftier tip is that with more modern Windows versions (2008, 7, Vista), when you hit the the Window Start menu, it now offers a search feature, and you should be able to type "stop cf9" (or just type "stop " and leave it to pop any shortcuts you've created this way to stop various services.)
So why do I like this approach, of stop/starting the service from the command line instead? Because what happens is that a window pops opens (the traditional black background for DOS windows, so easy to observe) and it shows the stop (or start) taking place, however long it takes (which may be very fast). The key benefit is that the window will close on its own when the action is done (however long it takes). I just watch for the stop to be done, and then I do the start.
There is a possibility, of course, that there will be an error in the shortcut or batch file (or perhaps as a response to the correct command), in which case the window will close leaving no chance to see the error. In that case, I just go to the command line (Start>Run>cmd) and execute the same net stop or start command to see what the error is.
To be clear, using this command-line doesn't stop you or others from using the Services panel.
Conclusion
All that said, though, again, if the reason you need to worry about all this is that your CF Service is taking a long time to stop, and you want to find out why, then I would point you again to the last entry I did, In my last entry, CF911: Is your ColdFusion service taking too long to shut down? Find out why.
Hope all that was helpful, whether you use CF or not. Please do let me know in the comments below.
For more content like this from Charlie Arehart:Need more help with problems?
- Signup to get his blog posts by email:
- Follow his blog RSS feed
- View the rest of his blog posts
- View his blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
- See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
The solution is actually really simple. If the service seems like it is not going to restart before it times out Just open the task manager and kill jrun, the service will then restart successfully.
Anyway, yeah, I realize one COULD kill the task. I assumed people knew that. This entry was a follow-up to the last, explaining how one could let it shutdown (despite the services timeout issue) if you wanted to.
But just to avoid any more confusion, I've just updated the entry to make that clear and credited you for making the pointing.
In case you don't know the command prompt command it is
net stop "coldfusion 9 application server"
net start "coldfusion 9 application server"
the quotes didn't seem to always be required, but they never hurt to have them there.
I know: perhaps too much info to read it all. :-)
http://www.adobe.com...
If the other info above doesn't help with an unexpected problem of timeouts during startup or shutdown, consider this (whether the impact of the defaults, or the prospect of modifying these values.)
What would account for the huge amount of time required for CF to re-start on one of my servers, but not the other?
ServicePipeTimeout has nothing to do with killing a service. It has to do with the start-up process and tells the service manager to wait the specified amount of time before giving error (7000 etc ...) that something couldn't start or whatever because something else didn't start. If you have something that starts a little slow you increase the variable in SPT to 60 or greater to tell the service manager to chill. This only pertains to the starting side of a start or restart not the shutting down.
And I will admit I may have gotten something wrong (if you are referring to something in the blog entry or a comment from me), but I am not seeing the immediate connection (and at the moment I haven't the time to reread the blog entry, from 2011, and all the comments) to try to connect those dots.
Thanks.
This article is about Service Control Panel timeouts and about killing a service faster. (in this case, CF when run as a service) In the (second) paragraph of 'Tip 1' you state
[QUOTE]
And technically, you should be able to control that timeout via a registry key, called ServicesPipeTimeout, which is discussed in this Microsoft technote and defined in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control.
[END QUOTE]
The Service Control Manager (SCM) controls the starting/stopping of services. The above mentioned SystemPipeTimeout ONLY controls a STARTUP timeout before the SCM will issue an error for a slow or non starting service. That is why, as you stated, you could not get it to work on several different machines because, the reg key solution you were trying has NOTHING to do with shutdown timeout errors or the Service Control Panel's speed at which it kills services. As for the speed that services are killed, that is controlled by the "WaitToKillServicesTimeout" that you said "is NOT related to this discussion".
So, I'm sure you can see my 1st comment WAS actually referring to something in your article, namely Tip 1 and consequently what my point would be that you were asking about.
michael clyde
So thanks for connecting that dot. And thanks again for trying to help clarify where you feel I was mistaken regarding that. I have now reread the blog entry, and your comments. I still think we're not on the same page. Since you want to make your points clear, let me respond to them.
Your first comment says this ServicesPipeTimeout setting "has nothing to do with killing a service". Well, I didn't say it did. Instead I referenced an MS article which said that value "represents the time in milliseconds before a service times out". And I was discussing it in a section entitled, "There is a timeout set for how long the Windows Services panel will wait". So I was not talking about "killing a service".
i then pointed out in the same section an available WaitToKillServiceTimeout registry entry, and that does refer to "killing" a service, but I noted that it was "NOT related to this discussion". And you reference this later, so I'll get back to that in a moment.
So I guess I'm wondering first what led you to comment that I was asserting the ServicesPipeTimeout DID kill a service. It seems I did not.
Second, you conclude your first comment saying that this setting "only pertains to the starting side of a start or restart not the shutting down." OK, we're on the same page there. The blog entry begins "Tips for dealing with Windows service timeout". I was indeed focused on "the starting side of a start" as well as the stopping.
Your next comment now says that you feel this blog entry is "about killing a service faster". Well not quite. Again, it was about understanding why a service may not start/stop when you expect it should, and some possibilities related to that.
Then you clarify in this second comment that the ServicesPipeTimeout setting "has NOTHING to do with shutdown timeout errors". Well, again, I didn't say it did. Even having reread things now, I just don't see me saying what you say I am.
As for my having "tried it on multiple machines" and "not finding it to work", I was referring to its control of the time for how long it takes the service to start, in the control panel. Again, I wasn't saying it had anything to with "killing" the service. It's a subtlety, but I just want to clarify what I was saying.
Now, I gather you have a sincere concern about misinterpretation of this setting, and fair enough. I just don't see anything I need to change, but if you (or anyone else reading this) sees something I did misstate, do let me know and I'd change it.
BTW, you switched to referring to "SystemPipeTimeout" but I assume you meant ServicesPipeTimeout.
That said, you also say that this "has NOTHING to do with shutdown timeout errors" and " ONLY controls a STARTUP timeout before the SCM will issue an error". Are you sure it doesn't cover both starts and stops of a service? Even if it does control only one of them, I don't see that changing what I said, but if it does control both, then it would change what you said.
Again, I'm just trying to be as clear as you, for the sake of those reading along.
And in fact, you conclude saying that "As for the speed that services are killed, that is controlled by the "WaitToKillServicesTimeout". Well, hold on there. First, again to be clear, that's NOT to be pluralized. It's WaitToKillServiceTimeout. (I'm not picking on you, it's just that it really would help readers for us both to be as clear on these things as possible).
Second, though, I said it's "not related to this discussion" because according to the docs I read (and point to when I referenced it), that affects how long Windows is willing to wait when it, Windows, IS SHUTTING DOWN, and is trying to stop services in that process, before it will just kill a service that's taking too long to stop properly. And that does indeed have "nothing to do" with this discussion about "Tips for dealing with Windows service timeout", in the context I have presented this topic.
If you feel I still have said something wrong, after these clarifications, let me know. I'm open to correcting mistakes. If you just read things differently than I now show I wrote them, and you want to let it be, that's cool too. I'm open to dialogue and want to put further accurate information. Thanks for your points of clarification.
https://adminsissues...