20160703 CISystem
This week I start writing about my development of CISystem, a continuous integration execution system based on a peer-to-peer network of nodes providing resources required for executing jobs.
I am now working for some years with Jenkins and I really love. It is rock solid, easy to understand, very flexible and there are a lot of plug-ins which usually fix all your problems when Jenkins cannot do this alone.
But ... there is one point I simply miss in Jenkins and that is the support of job-lists. Yes, you can configure dependencies between jobs and I guess, you can configure your jobs using the gradle plug-in and yes Jenkins 2.0 comes with a lot of new features where maybe even the gradle plug-in is not needed anymore.
But ... for me it feels like I have to make the decision that Jenkins is my all-in-one-swiss-army-knife-for-everything and simply hope that Jenkins really can do how I want to work with what I call job-lists. I would need to make this decision, because it takes time to learn how it works and if it does not work I still don't have it.
Now, working with Jenkins 1.x, when I need to use resources, which are only available on specific nodes, I often come to the point where it looks like you need to have exactly one job instance for every resource used to be able to release a resource ASAP. The problem is, when you issue 300 job instances and then decide to stop the execution, how do you stop all the running jobs and all the jobs waiting in the queue? Maybe there is a plug-in for stopping jobs or removing them from the queue or you can solve it with a Groovy script, but I still have the feeling, that there is something missing in Jenkins,something related to job-lists.
Jenkins provides a plug-in API, so you can write your own plug-ins to control parts of Jenkins but what do you do when the concept how Jenkins works is not what you want? You would end up in workarounds, because it is not a missing feature you are working on, it is a concept you are working against.
I want to have an execution system which not only works for a list of jobs, I want to have a system which handles lists of jobs but where a list is not only like a super-job, which needs to be finished before the next list can be started.
I always wanted to implement multi-threaded software, I always wanted to write software communicating via the network.
So I have decided to implement my own execution network.
I call it CISystem, which stands for something like Continuously Integrating System.
Yes, I know, CI is old school, now it is more hip to do continuous delivery, but CISystem does not mean Continuous Integration as a way to test and deploy your software. It means the nodes are continuously integrating themselves into the network and when they are taken away from the network, they can reinsert themselves later back into the network and execute jobs of an already running job-list. I say they reinsert themselves, because I want the nodes in the network to inform each other about how the network looks like, for example which nodes are online and which resources are available at each node. This information must be continuously updated, some of it changes with every new job being executed, e.g. a job uses a resource, so it is not available for other jobs.
Regarding the make-or-buy decision, does it make sense to reinvent the wheel once again? Yes, for me, because I can gather experience in implementing a multi-threaded distributed software communicating over the network.
Does this mean I don't like Jenkins anymore or will not continue blogging about using it? No. As mentioned, I really like Jenkins, it is really good at what it does. I have already planned to connect my CISystem with Jenkins, for example I don't plan to implement something like a cron-job feature, I will use Jenkins for that.
Select where to go ...
|
The Blog
|
|
|
|
|
|
My Technical Blogs
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Projects
|
|
|
|
|
|
Blogs Of Friends
|
|
|
|
|
|
|
|
CV/About
|
|
|
|