JavaOne – Improve your Groovy Kung-Fu

“Improve your Groovy Kung-Fu”

Speaker: Dierk Koenig

For more blog posts from JavaOne, see the table of contents


Use cases

  • Spock for testing
  • System automation – Docker
  • Jenkins scripting console and pipelines

Recommends Groovy ref card

Groovy idioms

“a” == “b” – check if two strings are equal

a in col – check if element in a collection

${‘0’*10} – print 10 times

use Collections, {
time.shufle()
}

type .xyz vs .getXyz()

use “it” (default) instead of creating a variable for grep

catch(all) – vs catch(Exception e)

gpars dataflows – specify what want to do in tasks, but not the order [I had trouble following this part]

@Grab(“groupid:artifactid:version”) – get dependency from Maven Central. Can specify a resolver as first parameter if want to use an alternate one. Not worth it though because then as much work as setting up a project.

Command line

  • groovy -e “print 1” – command line eval
  • groovyconsole – interactive UI

My take: I like that he showed code that he wrote and pointed out idioms. Seeing well written code is helpful for seeing good practices. I also realized that I know more than I thought. Many of the items in this post I knew existed but aspire to use more. Or things I do and would like more of the team to do.

upgrading from jenkins 1 to jenkins 2

The first Jenkins LTS (long term support) release fro Jenkins 2.0 came out this week.  It is Jenkins 2.7.1 and is available for download here. CodeRanch is running 1.640 and we decided to upgrade shortly after the LTS release came out. There have been numerous weekly releases for 2.0. We wanted the LTS release so it is a little more stable.

Backup

Another moderator at CodeRanch installed Jenkins and Sonar originally. Unfortunately, they are mixed together in the data directory. Anyway, I started by taking a backup in case I want to go back to 1.X at any point or mess it up/ I’m not terribly worried because our Jenkins install has 5 jobs on it. Worse case, I can do a fresh install and re-create them. I want to try upgrading first though since it is easier (at least in theory.)

tar -cvf jenkins1-20060709-bkp.tar *

The tar file was 1.2 GB. This is because it includes the job workspaces and Sonar.

Install

Conveniently our Jenkins install is just the war. So I backed up the old war and uploaded the new one:

mv jenkins.war jenkins1-bkp

Starting up with Jenkins 2

Loading Jenkins showed a nice banner with a big button to click

jenkins2-splash

So I clicked the “upgrade now” button. That led to a prompt to ask if I wanted to install some pipeline related plugins.

I said yes so they installed.

Then I got another button that Jenkins was ready. Still looks the same and my jobs are still there.

jenkins-ready

When I went to look at the plugins that were installed, I saw “Warning: This Jenkins instance requires a restart. Changing the state of plugins at this time is strongly discouraged. Restart Jenkins before proceeding.”. Which makes sense. Odd the upgrade process didn’t prompt me to restart. I did it myself at this point.

Existing Jobs

The five existing jobs ran the same way as they did before upgrade.  That odd phrasing was because what really happened was:

  • Three jobs were successful
  • One job hasn’t been run in ages and was clearly an old experiment. I deleted it.
  • One job has been failing for a few days and nobody noticed. It doesn’t have email notification and is triggered by the main job. (It was failing because it was relying on an Ant 1.9 feature and Jenkins was using Ant 1.8.) I upgraded Ant and then it was good. I also added email notification so that doesn’t happen again.

Jenkins 2 claims to be fully backwards compatible. As near as I can tell, this is true. I didn’t run into any issues related to migration. I even felt comfortable deleting the giant backup file at this point.

What did change

What I noticed very quickly

  1. The job configuration screen now tab navigation which takes you right to a section on the page.
  2. The configuration of external tools is now under Maven Jenkins > Global Tool Configuration instead of Manage System.

Actually using pipelines

Granted Pipelines were available in Jenkins 1. Looking at the jobs we had, we have three distinct steps (the main build runs CI and Sonar and then a JavaDoc build runs.) It would be good to have the Sonar one as a top level citizen and equal to the JavaDoc one. I decided to try to re-write the whole thing in Groovy (rather than calling the existing jobs) to see how hard it would be. I also split the Sonar step out into its own stage. This would have been more difficult to do as a separate job because I’d have had to pass the Sonar revision number and hack at it to use the same workspace. (I know how to do both those things – after all the original JavaDoc job uses an absolute path to the Ant script to run in the original workspace – but still.)

As an aside, I had trouble finding the workspace.

Figuring out email notification was a pain. In particular, getting it to email on every failure and first success just like the default used to be. But the rest worked pretty easily and end to end it only took me a couple hours (and 37 test builds) – it’s a pretty simple set of jobs. And now the email list is one place for all the JForum stages so we won’t have the JavaDoc build failing and nobody knowing again. The script wound up being:

def emailRecipients = 'xxx@gmail.com'

node {
 // hack to get workspace variable - https://issues.jenkins-ci.org/browse/JENKINS-33511
 env.WORKSPACE = pwd()
 
 try {
 stage 'Build'
 // poll SVN and check out updates if changes
 checkout poll: true, scm: [$class: 'SubversionSCM', additionalCredentials: [], excludedCommitMessages: '', excludedRegions: '', excludedRevprop: '', excludedUsers: '', filterChangelog: false, ignoreDirPropChanges: false, includedRegions: '', locations: [[credentialsId: 'e10831de-302f-4c1d-9518-fa7fb81bab9e', depthOption: 'infinity', ignoreExternalsOption: true, local: '.', remote: 'https://xxxx/svn/xxx']], workspaceUpdater: [$class: 'UpdateUpdater']]
 // call Ant
 def antHome = tool 'apache-ant-1.9.7'
 sh "${antHome}/bin/ant clean compile jacoco dist-work"
 step([$class: 'JUnitResultArchiver', testResults: 'qa/reports/*.xml'])
 
 stage 'Sonar'
 sh "${antHome}/bin/ant sonar -DprojectVersion=${env.SVN_REVISION}"
 
 stage 'JavaDoc'
 sh "${antHome}/bin/ant javadocs"
 // archive only if run successfully
 step([$class: 'JavadocArchiver', javadocDir: "${env.WORKSPACE}/docs/api", keepAll: false])
 
 // without this, you don't get emails
 currentBuild.result = 'SUCCESS' 
 
 } catch (err) { 
 // without this, you don't get emails
 currentBuild.result = 'FAILURE' 
 throw err
 } finally {
 echo "currentBuild.result: ${currentBuild.result}"
 // send email whether job failed or not
 step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailRecipients, sendToIndividuals: false])
 }
}

 

from java to groovy – thinking in functional programming (via jenkins)

I found my Java knowledge to be a crutch when writing Groovy scripts. Luckily, I was able to use this crutch less and less as I gained more exposure. I was asked about this today so I decided to write up a blog on my journey. If you know how to run a Maven project and want to follow along, I’ve included instructions at the end.

Why I started using Groovy

Jenkins and Hudson are both continuous integration servers that provide a Groovy console for writing queries and updates against the object model. This is really cool as you can do something with/to hundreds of jobs with marginal effort. I had read “Making Java Groovy“, but  needed this real scenario to jump into it.

Phase 1: The “problem” with Groovy (and Java 8)

A valid Java program *is* a valid Groovy program. This is nice and convenient because you can call Java methods. It’s also a crutch because you can write Java methods rather than actually using Groovy. My first Groovy program in Jenkins was something like this:

def jobs = Jenkins.instance.items;
for (job in jobs) {
  if (job.isDisabled() ) {
    println(job.getName())
  }
}

At least it uses two pieces of Groovy. “def” to define a variable of unknown type. And “in” within the for loop. But this is an imperative program. I loop through the jobs checking a condition and printing as I go.

Incidentally, I didn’t have this problem in Clojure or Lisp when taking classes. There was nothing to fall back TO.

Phase 2: Translating after the fact

My next phase along the way to functional programming was translating my “imperative” program into functional programming. Now this step may seem like a waste of time given that I had a working program. I did it anyway so I would expose myself to the idioms of how I *wanted* to be coding this.  At the time, I was hoping it would take. And it did after a while.

But for phase 2, I was doing it very incrementally. First I made my imperative code one line. Still just as imperative. It still has a loop and if statement.

Jenkins.instance.items.each({ j -> if (j.isDisabled()) println(j.getName()) })

Then I started thinking about operations. It’s still not great because it still has “each”, but at least I’ve gotten the “if” statement out of there. I’m using “grep” and thinking about filtering first rather than looping.

Jenkins.instance.items.grep({ j -> j.isDisabled() }).each( { j-> println(j.getName()) })

My next step felt like a nice mental jump. I was able to think about the entire task in functional programming. I filter the list and then I translate to the field I want. And I’ve finally succeeded in getting the print statement out of my logic and just happening outside of it!

print(Jenkins.instance.items.grep({ j -> j.isDisabled() }).collect( { j-> j.getName() }))

Phase 3: Getting rid of some more Java syntax

Next I focused on getting rid of stray Java syntax. A lot of things required in Java are redundant in Groovy. I had omitted semi-colons from day 1. Getting rid of the rest took more time.

First I trained myself to get rid of the unneeded parens. I already have {}. No need to have parens there too. The parens make it a method call. But in Groovy, they are implied.

print(Jenkins.instance.items.grep{ j -> j.isDisabled() }.collect { j-> j.getName() })

Then I trained myself to use the shorter form of method names. Groovy automatically adds is/get to your method names so you can write them as if they are properties.

print(Jenkins.instance.items.grep{ j -> j.disabled }.collect { j-> j.name })

Finally I trained myself to indent consistently. I don’t know if this is the best way, but it helps me remember my chain of actions:

print(Jenkins.instance.items
.grep{ j -> j.disabled }
.collect { j-> j.name })

Phase 4: Thinking functionally

For a while, I’d be able to write easy code like this functionally, but have to fall back to imperative programming for debugging. Then I got used to adding print statements within steps to be able to continue on from wherever I was.  This let me think of the whole exercise as a series of steps and start coding them functionally. When I got stuck, I’d just proceed within an each statement to figure out what to do. Then I’d translate the each statement into the correct method.

They key is that I didn’t start thinking in loops anymore. i started with what I needed to do. “I want to find all the disabled jobs and get their names.” It’s a long way from “I want to loop through all the jobs and print the names of the disabled ones.”

For those wanting to run this with Maven

Jenkins is open source and has a robust plugin model. There’s lots of documentation for writing a plugin, but you don’t need to know any of it to run the console.

1) Create a Maven project/file with this pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>jb</groupId>
	<artifactId>play-jenkins-plugin</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>hpi</packaging>

	<parent>
		<groupId>org.jenkins-ci.plugins</groupId>
		<artifactId>plugin</artifactId>
		<version>1.573</version>
	</parent>

	<repositories>
		<repository>
			<id>jenkins-releases</id>
			<url>http://repo.jenkins-ci.org/releases/</url>
		</repository>
	</repositories>
</project>

2) mvn install (wait a long time for Maven to “download the internet”)

3) mvn hpi:run

4) http://localhost:8080/jenkins

5) Click Manage Jenkins

6) Click Script Console

7) Play!