Use unsupported Jenkins plugins with Jenkins DSL

In a previous post I wrote about how to Automate Jenkins with the use of the plugin Job DSL Plugin. If you didn’t read it, I highly suggest you do that as it will help you understand better what I’ll be explaining here.

When you start using the Job DSL Plugin you’ll probably sooner or later need to configure your job with a plugin that is not yet supported. And by “not yet supported” I mean that there aren’t (yet) DSL commands that will generate a job for that specific plugin. Fortunately they provide you with a way to add them ‘manually’ through the Configure Block.

This part is a bit more complex than using simply the DSL commands, because you’ll have to understand how it works. Now you did notice I wrote “a bit” … that’s because it seems complex, but in fact it isn’t. The only thing you need to know is that the plugin will, with the DSL commands, generate the config.xml of your job containing the full configuration of the job.

To have an idea, this is the config.xml of an empty job

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties />
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders/>
  <publishers/>
  <buildWrappers/>
</project>

Let’s see an example of a basic DSL command and the corresponding config.xml.

job {
    name 'Test Job'
    description 'A Test Job'
}
<?xml version='1.0' encoding='UTF-8'?>
<project>
  ...
  <description>A Test Job</description>
  ...
</project>

So you see that every DSL command will generate some part in the config.xml.

Knowing this you’ll understand that we will have to study the config.xml of an existing job to see how the “unsupported” plugin is configured in the config.xml.

Let’s make it a bit more fun by integrating the HipChat Plugin. I created a simple job in jenkins and opened the config.xml file. (I assume you know how to install and configure the plugin in Jenkins)

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties>
    <jenkins.plugins.hipchat.HipChatNotifier_-HipChatJobProperty plugin="hipchat@0.1.4">
      <room></room>
      <startNotification>false</startNotification>
    </jenkins.plugins.hipchat.HipChatNotifier_-HipChatJobProperty>
  </properties>
  <scm class="hudson.scm.NullSCM"/>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <builders/>
  <publishers>
    <jenkins.plugins.hipchat.HipChatNotifier plugin="hipchat@0.1.4">
      <jenkinsUrl>http://jenkins/</jenkinsUrl>
      <authToken>ABCDEFGHIJKLMNOPQRSTUVWXYZ</authToken>
      <room>76124</room>
    </jenkins.plugins.hipchat.HipChatNotifier>
  </publishers>
  <buildWrappers/>
</project>

The values in the publisher section are being copied from the jenkins administration. That’s a bit annoying because it means you’ll have to expose that in the DSL scripting. At the moment of this writing, I didn’t find a way to configure that as variables.

Looking at the config.xml, we see that 2 nodes were modified, the properties and the publishers node. Both are children from the root project node. With the Configure Block we can obtain the XML Node to manipulate the DOM.

Get hold on the project node:

job {
  configure { project ->
    // project represents the node <project>
  }
}

Now that we can manipulate the project node, let’s add the properties node:

job {
  configure { project ->
      
    project / 'properties' << 'jenkins.plugins.hipchat.HipChatNotifier_-HipChatJobProperty' {
      room ''
      startNotification false
    }

  }
}

What we did here is tell the parser to append (<<) the block ‘jenkins.plugins.hipchat.HipChatNotifier_-HipChatJobProperty’ to the node project/properties. And finally in the block we simply enumerate the parameters as key[space]value as you can see them in the config.xml.

Hint 1: Do not specify the plugin version plugin=”hipchat@0.1.4″ otherwise it doesn’t work.
Hint 2: I append the properties (and below the publishers), because there will/can be others configured through other DSL blocks.

Let’s do the same now for the publishers part:

job {
  configure { project ->
      
    project / 'publishers' << 'jenkins.plugins.hipchat.HipChatNotifier' {
      jenkinsUrl 'http://jenkins/'
      authToken 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
      room '76124'
    }
  }
}

As with the properties, we tell the parser to append (<<) ‘jenkins.plugins.hipchat.HipChatNotifier’ (without the plugin version) and enumerate the parameters.

Following is the full DSL for adding HipChat Plugin support:

job {
  name "Job with HipChat"
  
  configure { project ->
      
    project / 'properties' << 'jenkins.plugins.hipchat.HipChatNotifier_-HipChatJobProperty' {
      room ''
      startNotification false
    }

    project / 'publishers' << 'jenkins.plugins.hipchat.HipChatNotifier' {
      jenkinsUrl 'http://jenkins/'
      authToken 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
      room '76124'
    }
  }
}

Once you grasp the Configure block, you’ll be able to generate any job you want. The example below uses the configure block to add a missing functionality in an existing predefined GIT DSL:

job {
  scm {
    git {
      remote { 
        url("")
      }
      branch("refs/heads/${branch}")
      configure { node -> //the GitSCM node is passed in
        
        // Add the CleanBeforeCheckout functionality
        node / 'extensions' << 'hudson.plugins.git.extensions.impl.CleanBeforeCheckout'  {
        }
        
        // Add the BitbucketWeb
        node / browser (class: 'hudson.plugins.git.browser.BitbucketWeb') {
          url 'https://bitbucket.org/my-account/my-project/'
        }
      }
    }
  }
}

A handy tool to play with (or test) the generation of your DSL is http://job-dsl.herokuapp.com/. It will prevent your from constantly running your DSL and open the config.xml from your jenkins to see if the xml is generated correctly!

Although the Configure block is really awesome, it doesn’t beat the predefined DSL commands, so if you have the time I suggest to contribute to the project by making it a predefined DSL :) https://github.com/jenkinsci/job-dsl-plugin/blob/master/CONTRIBUTING.md

If you have some other great Configure Block example, share them in the comments :)

Automate Jenkins

jenkins is cool

Jenkins is a powerful continuous integration server which has been around for some time now. I’ve been personally using it for years and it never let me down.

However, there will come a time where adding/updating/removing jobs will have an impact on your internal processes. Take for example feature branches. Logically you will (and should) test them, so you will start making new jobs for each feature, and once they are done, you will remove them. Sure using the duplicate helps a lot, but it’s yet another (manual) thing on the todo list of a developer. Or it could even be that you don’t allow (junior) developers to administrate the jenkins, and thus making it the job of the “jenkins manager”.

If you are facing such situation then you will embrace the Job DSL Plugin.

The Job DSL Plugin allows you to generate jobs through some simple Groovy DSL scripting. I’m not going to explain here how everything works, because the wiki of the plugin does a very good job at that, but instead you’ll find some DSL scripts which I’m currently using on a project. I do however suggest reading the wiki first in order to fully grasp the meaning of the following examples.

Generate jobs based on subversion branches

This following DSL will create a job based on the branches it finds on a subversion repository.

svnCommand = "svn list --xml svn://url_path/branches"
def proc = svnCommand.execute()
proc.waitFor()
def xmlOutput = proc.in.text

def lists = new XmlSlurper().parseText(xmlOutput)

def listOfBranches = lists.list.entry.name

println "Start making jobs ..."

// iterate through branches
listOfBranches.each(){
  def branchName = it.text()

  println "- found branch '${branchName}'"

  job {
    name("branch-${branchName}")
    scm {
        svn("svn://url_path/branches/${branchName}")
    }
    triggers {
      scm('H/5 * * * *')
    }
    steps {
      maven("-U clean verify","pom.xml")
    }
  }
}

Generate jobs based on a static list

If you have several libraries which need to be configured exactly in the same way, you could also make use of a static list.

def systems = ["linux","windows","osx"];

// Configure the jobs for each system
systems.each() {

def system = it

job(type: Maven) {
  name("mylibrary-${system}")
    scm {
      git {
        remote {
          url("git@bitbucket.org:some_account/mylibrary-${system}.git")
        }
      }
    }
    goals("-U clean deploy")
  }
};

As you can see you can achieve some very powerful automations with the Job DSL Plugin.

They already support a lot of plugins but in case the one you use is not (yet) supported it is always possible to configure it through Configure Blocks. I had to do that for the HipChat Plugin which I will explain in detail in a following blog post.

Hope this convinced you to stop creating, editing and removing jobs manually and start doing all that automatically ;)

RabbitMQ Messaging Application Development How-to [Instant]

I had the opportunity to get a copy of this book for review by Packt Publishing.

As I already have some experience with RabbitMQ, I was more interested to see if this book could be used as a learning material within the company I work for. Not every new developer knows about RabbitMQ and it’s not easy to start with, especially if you have no background with other queuing systems.

And I was pleasantly surprised while reading the book.

The author first explains what RabbitMQ is all about and what it can do for you. Then quickly goes over the installation process and continues with many different use cases by using an e-commerce shop as example.

The different examples used in the book are carefully put in order of complexity. You will first learn, as in all other langues, how to do a simple “hello world” and continue with more complexer use cases. What I liked a lot is that during each example the author carefully explains each concept used. Exchanges, bindings, queues, brokers will not sound like Chinese any more.

To summarize, the book is a very good candidate to learn RabbitMQ quickly without cutting corners! I highly recommend this to beginners. That being said, more experienced developers can still learn one or two things from the advanced examples.

The only negative point I have is that all examples are written in Node.js. Maybe adding a second or third language can be helpful for some readers.

By the book: http://www.packtpub.com/rabbitmq-messaging-application-development/book

Easy SSH tunneling with Putty

Need to access a MySQL server, a RabbitMQ management server or any other (web) server configured on a dedicated/virtual server which is not (or you don’t want to) open to the outside world?

A simple SSH connection and Putty makes this possible:

1. Requirements

  • Putty SSH client (download)
  • SSH access to a server

2. Configure Putty

Open the Putty Configuration client, and fill in the field ‘Host Name (or IP address)‘ with the IP or host name of your remote server. In the Connection type section you should select SSH which will set the port automatically to 22.

Once this is done, open the Tunnels section in the menu Connection -> SSH. In this section we are going to configure the port forwarding. Assume we want to access the RabbitMQ management web server which runs on the port 55672. We need to tell Putty to listen on a Source port (=local), and to forward it to the Destination (=remote).

For this example we will configure putty to forward port 1234 to the remote port 55672.

Fill in the ports as displayed in following screen and then click on Add:

If everything is correctly configured click Open. This will open a SSH connection with your remote server and at the same time the configured port forwarding. The major drawback is that you cannot see the port forwarding from that screen. If you also use putty to manage your remote server, you will not see the difference. Of course you can always configure the putty screen to be a different color (if you know alternative tricks, please share ;) )

While the putty connection screens remains open you should be able to access the RabbitMQ management web server by calling http://localhost:1234 in your favorite browser.

As you can see the procedure is really simple and opens a lot of possibilities without having to open different ports on your remote server.

Enjoy

How to use a symfony 1 layout from a different directory

In symfony 1 it is possible to have different layouts for an application. But they all have to be put into the directory ‘myproject/apps/frontend/templates/’. But what if you want to use a layout from another location?

Assume you make a plugin with a specific layout, it would be nice to load the layout from the plugin directory, and not to have to copy the file to the global directory.

Here is how you can achieve this:

$template = $this->getContext()->getConfiguration()->getTemplateDir('MODULE', 'LAYOUT_FILE.php');
$this->setLayout($template . '/LAYOUT_FILE');

Let’s say you have the following:

my_project/
  plugins/
    my_plugin/
      modules/
        MyUser/
          actions/
            actions.php
          templates/
            indexSuccess.php
            MyUserLayout.php

The actions.php class could be something like this:

class MyUserAction extends sfActions {

  public function preExecute() {
    $template = $this->getContext()->getConfiguration()->getTemplateDir('MyUser', 'MyUserLayout.php');

    $this->setLayout($template . '/MyUserLayout');
  }

  public function executeIndex() {

  }
}

Have fun!

Symfony Live 2010 – day 1 #sflive2010

So far everything went great, well not great but ok, the periferique in paris was hell this morning, and we had a 5 hour drive from brussels to paris, thx to a train accident. But… most important… we got here.

We mist the first session on i18n but the second one on admin generator gave me som gr8 ideas on layout. After that, the Microsoft crew came on to stage, and pffff.. shitty presentation… just a promotion and selling of their Azure product. And I wasn’t impressed at all!!!

now lunch, simple but ok. and we’ll see what the next hours bring…

sessions are about to start…

well well well, that was an interesting session by Mr Bachelet. I regret it that he didn’t do the session in french, it would have been better!!! a bit of an overview of all handlers and helper, just isn’t enough I guess…

Questions are answered quite good…. He knows his stuff real good! take your time to present next time.

doctrine migrations was nice. interesting and in particular the question session with a very kind host, answering all questions asked. And believe me, there were a lot of questions :-)

Symfony Live 2010 #sflive2010

On the 16-17th of February the Symfony Live 2010 conference will be held in Paris. A lot promising developers and evangelists will be speakers on this conference like:

  • Fabien Potencier
  • Jonathan Wage
  • Dustin Whittle
  • Stefan Koopmanschap

My colleague and I will attend this conference and, if everything goes well, we will try to update our DevExp blog with live feedback or summarized posts on some of the most interesting sessions.

Read from Excel in ASP

I always had a hard time finding correct documentation on the old ASP (active server pages) language and I needed it one more time in the last few days to create a script that read from an excel file.

As I believe that it might be handy for a handful of people around the world (is there actually anybody else still developing in ASP these days?), I hereby share my little piece of code with you guys.

Any suggestions or comments are always welcome.

In my example we read from an excel file that has 1 small table that has 1000 lines and columns from A until G.
The first row contains all the column names.

'initialize variables
Dim objConn, strSQL
Dim x

Set objConn = Server.CreateObject("ADODB.Connection")
objConn.Open "DRIVER={Microsoft Excel Driver (*.xls)}; IMEX=1; HDR=NO; "&_
 "Excel 8.0; DBQ=" & Server.MapPath("filename.xls") & "; "
strSQL = "SELECT * FROM A1:G1000"

Response.Write("<table border=""1"">")
Response.Write("<tr>")
'write all columnNames
For x=0 To objRS.Fields.Count-1
 Response.Write("<th>" & objRS.Fields(x).Name & "</th>")
Next
Response.Write("</tr>")

Do Until objRS.EOF

' write as much columns as there are in your excel file
Response.Write("<td>" & objRS.Fields(0).Value & "</td>")
Response.Write("<td>" & objRS.Fields(1).Value & "</td>")
Response.Write("<td>" & objRS.Fields(2).Value & "</td>")
Response.Write("<td>" & objRS.Fields(3).Value & "</td>")
Response.Write("<td>" & objRS.Fields(4).Value & "</td>")
Response.Write("<td>" & objRS.Fields(5).Value & "</td>")
Response.Write("<td>" & objRS.Fields(6).Value & "</td>")

objRS.Close

Response.Write("</table>")

Set objRS=Nothing

Interview: Jean-Baptiste Jung

You can never get enough interviews, at least if they are a bit interesting…
This time we got an interview with Mr Jean-Baptiste Jung, a very friendly and ambitious guy with little spare time as he mentioned in his interview.

My name is Jean-Baptiste Jung (Pronounce: Young) and I’m a 27 years old blogger, web developer and web designer from the French-speaking part of Belgium.
I started blogging in 2006 by writing a blog in French about the Python programming language. I quickly dropped this blog and created a new one, Lyxia.org to discuss a lot of nice things about the Internet: Web development, Web design, blogging, WordPress and much more. After this successful experience, I decided to create a similar blog, but in English. This is how, in June 2008, I created Cats Who Code.
I’m also an author on several popular blogs as such as WpHacks, ProBlogDesign and Smashing Magazine where I provide various tutorials, how-tos and lists, mostly about WordPress. I have released some WordPress themes as well, the most popular is called OpenBook and the most recent NYC Theme. Apart from blogging, I work as a web developer in a French company and have a lovely wife and a very cute cat.

What projects are you currently working on?
Lots! I’m making big changes to wprecipes.com with a new design and lots of new featur

es. I’m working on CatsWHoCode.com to improve the blog even more. And at last, I’m on the point to publish my first blog which is not about web dev things!

Continue reading

Interview: Jacob Gube

blogthumb4Well, well, well after our first interview I guess our ball got rollin’ and we are proud to present our second interview.

This time we got in contact with Jacob Gube who is the founder and maintainer of the Six Revisions weblog having over 35K RSS readers. Let’s hope we reach the same amount one day!!!

Jacob Gube, a web developer and designer who works with PHP, .NET, Flash/ActionScript, XHTML, JavaScript/jQuery/MooTools and MySQL, is the creator and primary author of Six Revisions – a weblog the provides practical, useful information for the modern, standards-compliant web designer and web developer.

What projects are you currently working on?

I’m working on a PHP/MySQL social news site start-up that’s self-funded. No release dates yet and there’s still a lot of work to be done! Also, I’m trying to finish up a JavaScript book.

Continue reading