>Well a reboot (i.e. a shutdown) is not a 'brute force termination' - it's either initiated by a user who is logged on with sufficient privileges or maybe something like a UPS during a power failure. In either case it should be assumed that there is a valid reason for the action and your service should be designed to shut itself down safely in a short period of time - failure to do so can place the state of other services at risk.
Yes, there are valid reasons for restarting one of the servers in the virtual environment and this is done by a specific person. But, that person is not allowed to and should not take action on our services. Thus, when network operations require such action, it is coordinated with the tech team. But, in either case, even if it would be done us, this requires that the service be terminated in a smooth mode.
Here is more information to help you understand the context.
We have about 40 Windows Services. They are spreaded across multiple application servers. Application servers serve just for that purpose because the load of what has to be processed has to be leveraged into more than one server to allow the proper operations to execute in the timely fashion allowed time and to avoid a back log. So, this is a big thing. Yes, there are other big applications in the world and some which are considered much more up to the point that they are doing enormous amount of transactions into a geometric scale, far more than what we have to do. However, no matter the infrastructure in place they use, in our case, for the payload, we are up to several servers right now.
Those Windows Services execute either at every minute, quarterly, hourly, daily or at a precised time, usually overnight. It depends on their needs, the payload and various other factors. You can guess that those who are executing quarterly or more are doing more intensive operations. Those at midnight will execute a round trip which may take from 5 to 7 minutes. So, the 20-second delay you mention does not apply here. If this has just been started and someone has to restart the server, he has and will wait 5 to 7 minutes until the Windows Service in progress finishes what it has to do.
I am not sure where the 20-second mention comes from however.
Yes, we have a controlled environment which will shutdown those services when needed and this is what we do when a restart is necessary. From our interface, we send a signal to each service. Each service, if not running, with shut down smootly. If they are running, they will complete, they will check for the signal and will shutdown by themselves if they see it once they finish their loop.
So, based on the above, this is what I need to automate. Each service, upon receiving a notification of shutdown, if not in a loop, can shutdown. Otherwise, it has to finish its loop and then it will shutdown. But, because Windows will shutdown as it sees fit, we have no guarantee that the service has finished its work in progress. This is why we need to control it.
It is done on a variety of services. Some are OS-based services, such as 3rd parties. But, they interact, by some kind of mechanism with the user, and advise that a service is not ready for shutdown and a wait is suggested. The user can however force an immediate shutdown by ignoring that fact. I usually do not do that. If I see something like that, I allow the service to complete before restarting.
The approach of using a slave is not any better, after I thought more about it. Maybe I missed something, but I do not have any guarantee that the slave will kick in on shutdown before all the services that it will monitor. So, if a service is running, on progress process, it may receive the shutdown notification before the slave will see it first. So, this can create a situation that the service with on progress work might be forced to shutdown, even if it is in a loop, before the slave had time to receive that notification. One way, maybe, would be to implement the actual part of code we have worked on since yesterday in each service, detect the shutdown event, avoid acknowledging the proceed of the shutdown if in progress, recognize it however by setting up a global property, and upon completion, it will know that it has to shutdown. The problem here is because there are no interaction with the user, the shutdown might occur before that specific service had time to complete. Thus, where the 20-second delay comes in. Maybe you mentioned 20 seconds because this could be a default of Windows to wait that time if it sees that a service is in progress, not within its timer event. Even if I could increase the delay, I would never have any guarantee that all services have successfully ended before the shutdown occurs. Another way I thought, and maybe this is possible, is that such service would interact with an already existing Windows service, where that one may have authorization to interact with the desktop, and upon receiving a notification will then submit it to the user. This would be basically similar to what I originally proposed but by the use of an already Windows OS-based service which would offer more protection.
>Look at it from another perspective : what you want would equally allow a rogue service to prevent shutdown indefinitely until someone pulled the plug :-}
Yes, this is what we need. We have numerous reasons for that. And, within the actual environment, this is a requirement that that part becomes automated and allows the completion of ongoing operations with success. Those shutdowns are controlled. They happened for a very precise reason. When they do, deployment procedures apply, regulations and sign off. So, yes, if the one doing the shutdown has to wait 4 minutes, he will.
Some would say "Why did you just not do a desktop application". Well, we had. But, new regulations had us to switch all those deskop applications into services. So, this is what we are dealing with now. I believe it is good. I have seen that this is much simpler to negotiate with, remotely manageable, start/stop/create/delete. This is faster to execute those operations and I have seen also additional benefits. We just need to complete this last part about this shutdown issue.
>I'd look at tweaking the service kill time registry setting and maybe experiment with the RequestAdditionalTime method. But it should surely be possible to design a service capable of shutting down gracefully within 20 seconds ?
As per what I mentioned, you now have additional information which explains that in our case, this 20 seconds is very limited and this is a boundary that we cannot negotiate with. Some would say that a service should remain relatively small. Some could be right saying this, some would argue. In our case, most of them are really small. But, we have others that are doing some complex work. And, when they do, they go way over that 20-second mark. We are talking about millions of database transactions a day and an enormous amount of data that has to be processed with automation, in the background in regards to their payload.
>FWIW, there's a low level flowchart here of the communication between the SCM and a service:
http://msdn.microsoft.com/en-us/library/windows/desktop/ee126211(v=vs.85).aspxI will take a look at that. Thanks
>Isn't shutting down the service 'manually' the same (from the services POV) as just shutting down the system ?
No, we never shut down the service manually. We remotely create, delete, start and stop them from the Web interface and my design is secure as it is based on Robot table related fields which will only shut one down if it is not in progress, thus not within the timer event, and will shut down at the end of the event, assuming at the time I sent the signal it was just not ready to be shutted down.
