Avoiding Queue Starvation in CruiseControl.NET
Recently I've been solving some issues with queue starvation in CruiseControl.NET. Since I haven't found much information about this problem I've decided to round up my findings in this blog post.
So, what is queue starvation? It's a problem that can occur when scheduling tasks in priority queues if there are too many high priority tasks so that lower priority tasks never get scheduled and are just waiting indefinitely. For example, let's say we have three queues: high, medium and low priority. Tasks in the high priority always get processed first. Once this queue is empty the tasks from the medium priority queue get processed. Of course the tasks in the low priority queue get processed only if the other two queues are empty. If there are too many high and medium priority tasks incoming all the time so that the queues never get empty, then the low priority tasks are not processed at all.
And how is this related to CruiseControl.NET? As you might be aware, projects can be assigned to different queues. Only projects in the same queue (as the name implies) are built sequentially while projects in different queues are built in parallel to one another. Queues themselves therefore cannot cause starvation because different queues don't depend on one another and projects are always put at the end of its queue. This means that sooner or later each project will be built.
Now, queue priorities enter the stage. Each project can be given a priority defining where the project will be put into the queue: before all of the lower priority projects already in the queue instead of at the end. Effectively this partitions a single sequential queue into multiple queues with different priorities as described at the beginning of the post. We have all the necessary prerequisites for queue starvation. If too many projects with higher priorities have to be built (typically caused by checking in code or triggering a forced build), the projects with lower priorities will never get processed.
What I have observed is that this can happen even if higher priority projects don't have to be built at all. The reason for that is in the way CruiseControl.NET is processing the projects to determine whether there were any source code changes. This is taken care of with an interval trigger which puts a project in the queue, but it only gets built if a modification exists. Nevertheless, even checking if such a modification does exists, takes some time. In extreme cases this can be enough to cause queue starvation and this is also what happened in my case. Lower priority projects never got built, not even during the night and no matter how often a build was forced for them.
Fortunately there is a solution for this problem: the seconds attribute which specifies how often the project is put into the queue for checking if a modification exists. By default it is set to 60 seconds. Increasing the value can solve the queue starvation issue. Instead of simply defining the trigger:
<intervalTrigger />
A longer period can be specified:
<intervalTrigger seconds="300" />
Of course this has its downside: the latency before the project gets built after a modification increases. Therefore you should use this solution with caution. Only increase the value as much as you have to. And don't do it at all if you can apply any of the other solutions:
- Increase the performance of the build server to speed up the processing and building of the projects.
- Parallelize the build process by using more queues. Put the projects in the same queue only if you have to.
- Don't use queue priorities if it's not necessary. In most cases you can get by without them.
