jenkins convert to pipeline plugin

I needed to convert a freestyle job to a pipeline job for “Automating your Ci/CD Stack with Java and Groovy” Hands On Lab at Oracle Code One. It’s a really simple project, but it is still a pain to convert by hand.

I had read about the Convert to Pipeline plugin so decide to give it a shot. It’s easy enough to install a plugin.

Generated Pipeline

Below is what got generated. While I’d like to assume that credentials binding isn’t in wide use, that one didn’t surprise me. In hindsight, it makes sense because the credentials binding would need to wrap the Gradle block. I was surprised that Gradle wasn’t supported.

I am glad that the tool leaves comments for what you need to do by hand.

// Powered by Infostretch 

timestamps {

node () {

	stage ('osprey - Checkout') {
 	 checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/boyarsky/OracleCodeOne2018-HOL-Automating-Stack-Groovy.git']]]) 
	}
	stage ('osprey - Build') {
 	
// Unable to convert a build step referring to "org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper". Please verify and convert manually if required.
// Unable to convert a build step referring to "hudson.plugins.gradle.Gradle". Please verify and convert manually if required.
		// JUnit Results
		junit 'osprey-project/build/test-results/test/*.xml' 
	}
}
}

Using the pipeline generator

On the pipeline job, you can click the “pipeline syntax” link. It opens a new browser tab with a pull down for common pipeline operations.

Credentials

I choose “withCredentials”, entered the variable names and selected the credential from the pull down. Then I clicked “Generate Pipeline Script” and got:


withCredentials([usernamePassword(credentialsId: 'nexus', passwordVariable: 'nexusPassword', usernameVariable: 'nexusUserName')]) {
// some block
}


Gradle

That brings us to Gradle. As far as I can tell, there are only objects for Gradle if you are using Artifactory. I am using the gradle wrapper though, so I followed the advice of this Stack Overflow post and just called gradlew.


withCredentials([usernamePassword(credentialsId: 'nexus', passwordVariable: 'nexusPassword', usernameVariable: 'nexusUserName')]) {
  sh "./osprey-project/gradlew clean build -b osprey-project/build.gradle -PnexusBaseUrl=http://nexus:8081 -PnexusUserName=${nexusUserName} -PnexusPassword=${nexusPassword}"
}

I decided to separate the publish step into a different stage and rename the stages as well.  Resulting in:


// Powered by Infostretch 

timestamps {

node () {

	stage ('Checkout') {
 	 checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/boyarsky/OracleCodeOne2018-HOL-Automating-Stack-Groovy.git']]]) 
	}
	stage ('Build') {
	    
	   withCredentials([usernamePassword(credentialsId: 'nexus', passwordVariable: 'nexusPassword', usernameVariable: 'nexusUserName')]) {
          sh "./osprey-project/gradlew clean build -b osprey-project/build.gradle -PnexusBaseUrl=http://nexus:8081 -PnexusUserName=${nexusUserName} -PnexusPassword=${nexusPassword}"
       }

		// JUnit Results
		junit 'osprey-project/build/test-results/test/*.xml' 
	}
	
	stage ('Publish') {
	    
	   withCredentials([usernamePassword(credentialsId: 'nexus', passwordVariable: 'nexusPassword', usernameVariable: 'nexusUserName')]) {
          sh "./osprey-project/gradlew publish -b osprey-project/build.gradle -PnexusBaseUrl=http://nexus:8081 -PnexusUserName=${nexusUserName} -PnexusPassword=${nexusPassword}"
       }
	}
}
}

Note: My actual build file was bigger. The parts outside the pipeline (ex: project security settings) got preserved.

building motion profile generator with gradle and/or eclipse

Last year, I blogged about how to build Smart Dashboard with Gradle for FRC. Today, a student asked me how to build Motion Profile Generator using gradle. It’s similar, but seemed like a good time to update the post for this specific project.

You can use pure command line when reading/following this. Alternatively, you can use Eclipse as little or as much as you like. I include the Eclipse “gotchas.” (Spoiler: It’s faster to just use the command line)

Pulling from git

You can pull the code from git in any way you’d like. Common ways are:

  • From the command line (or git bash): git clone https://github.com/vannaka/Motion_Profile_Generator.git
  • From Eclipse, go to the git perspective, and paste in the URL.
  • Download the project from github as a zip file. (Please don’t choose this option. It is better to use git properly. The only reason I’ve ever done the zip option is if I need to look at a code on a computer where I can’t install anything.)

Running Gradle at the command line

The discussion was centered around building Motion Profile Generator. If you aren’t involved with a FIRST Robotics Competition team, this is used to help with autonomous programming. As with many Gradle projects, the project comes with Gradle right in the project. So all you have to is:

  1. cd to directory you cloned Motion Profile Generator in
  2. chmod 700 gradlew
  3. ./gradlew build

Make sure you have internet before running this the first time. If you want to build directly from Eclipse, see my Eclipse Gradle post and skip to the BuildShip section.

Where is the output when building SmartDashboard?

Running gradle creates a build directory. You can look at it on the file system. Or if you are using Eclipse,

  1. Configure the filters so show the Gradle build folder
  2. Add/remove the JRE library as described here to get rid of the Eclipse errors about access restrictions.
  3. Hit refresh before looking for it.

I don’t care how gradle works, I just want to run the program!

No  worries. Just:

  1. cd build/libs
  2. java -jar Motion_Profile_Generator-v3.0.0.jar

Ok now I’m curious, what gets generated?

This build directory  contains a number of things:

  • distributions – a zipped up and tarred up version of Motion Profile Generator which includes the jar file, all need dependencies and the script to kick it off.
  • classes – the compiled .class files
  • libs – a jar file with the compiled code in this project
  • resources – the JavaFX file (user interface files)
  • scripts – to launch Motion Profile Generator
  • tmp – would you really expect to find something important in a directory named tmp?

What does the Gradle build file do?
Gradle is a build tool. The build.gradle file in Motion Profile Generator contains Groovy code that declares what the build should do. Both Gradle and Maven follow “convention over configuration” which means that a lot is implied! I went over most (but not all) of this out loud in the classroom.

A plugin is like a helper tool that Gradle is going to use in order to build. For example, ‘java’ is going to compile among other things.  Most of these plugins are in common use. Gradle needs to download files in order to run. This includes some default plugins. These files are hosted in a repository for binary files. A really common one is Maven Central. This is used by both Maven and Gradle builds to obtain binaries. This build file is really simple. It says that Gradle needs to know how to build a standalone java application.

plugins {
    id 'application'
    id 'java'
}

This tells Java to put some information in the manifest. In particular, which class should run automatically when you try to run the jar. It also includes a version number for reference.

mainClassName = "com.mammen.main.Main"
version = "v3.0.0"

Remember the convention over configuration thing I mentioned earlier? Convention says the source code is in the nested directories src/main/java. This project decided to put the source code directly in the src directory. Therefore the project needs to inform Gradle of this non-standard location.

sourceSets {
    main.java {
        srcDirs = ['src']; include '**/*.java'
    }

    main.resources {
        srcDirs = ['src']; exclude '**/*.java'
    }
}

Normally, Gradle projects include references to dependencies on the internet. This project decided to supply them directly in a lib directory. Again, a non-standard thing to inform Gradle about. (I think this decision is because Pathfinder has extra files)

dependencies {
    compile fileTree(dir: 'lib', include: '*.jar')
}

Next the script specifies that we should create a jar file and an uber jar file including the special files..

jar {
    from configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
    // Set up manifest to point the main class to the right class.
    manifest {
        attributes 'Main-Class': mainClassName
        attributes 'Version': version
    }
	from ("lib/native_libs") {
		into ("")
	}
}

The end! The very last thing is to declare which version of Gradle this entire script should be run with.

task wrapper(type: Wrapper) {
    gradleVersion = '3.3'
}

Output from building in gradle

I’m including the output from a “good” run in case you have problems and want to compare. This is for the first time you run it. After that the download steps won’t be in the output and it will run much faster.

JeanneBrskysMBP:Motion_Profile_Generator nyjeanne$ ./gradlew build
Downloading https://services.gradle.org/distributions/gradle-4.7-bin.zip
.......................................................................

Welcome to Gradle 4.7!

Here are the highlights of this release:
 - Incremental annotation processing
 - JDK 10 support
 - Grouped non-interactive console logs
 - Failed tests are re-run first for quicker feedback

For more details see https://docs.gradle.org/4.7/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :compileJava
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:4: warning: OutputFormat is internal proprietary API and may be removed in a future release
import com.sun.org.apache.xml.internal.serialize.OutputFormat;
                                                ^
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:5: warning: XMLSerializer is internal proprietary API and may be removed in a future release
import com.sun.org.apache.xml.internal.serialize.XMLSerializer;
                                                ^
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:435: warning: OutputFormat is internal proprietary API and may be removed in a future release
            OutputFormat format = new OutputFormat(dom);
            ^
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:435: warning: OutputFormat is internal proprietary API and may be removed in a future release
            OutputFormat format = new OutputFormat(dom);
                                      ^
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:439: warning: XMLSerializer is internal proprietary API and may be removed in a future release
            XMLSerializer xmlSerializer = new XMLSerializer(
            ^
/Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/main/ProfileGenerator.java:439: warning: XMLSerializer is internal proprietary API and may be removed in a future release
            XMLSerializer xmlSerializer = new XMLSerializer(
                                              ^
Note: /Users/nyjeanne/git/Motion_Profile_Generator/src/com/mammen/ui/javafx/MainUIController.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
6 warnings

BUILD SUCCESSFUL in 24s
6 actionable tasks: 6 executed

DevNexus 2018 – Gradle – the basics and beyond

Title: Gradle – the Basics and Beyond
Speakers: Ken Kousen

For more blog posts, see the DevNexus 2018 live blogging table of contents


Install

  • requires Java 7+
  • Gradle 4.2.1+ supports Java 9
  • Gradle releases frequently but is very backward compatible
  • Just unzip and add to path
  • The complete distribute includes the samples. The sample sources are also available online.

General

  • Gradle is now much faster than Maven
  • gradle.com is the enterprise version which is built on top of open source gradle.org.
  • Gradle is mostly implemented in Java. Groovy is for the DSL.
  • Uses Ant behind the scenes as well
  • Groovy comes with Gradle. Don’t need to install Groovy separately
  • gretty is jetty for gradle; also contains tomcat

Docs

Future

  • Kotlin
    • Creating a Kotlin DSL for simple build files.
    • Kotlin is not replacing Groovy
    • Not yet at 1.0.0
    • Goal is better IDE integration
  • Generator
    • Looking at being able to generate projects on web project like Spring Boot has

gradlew

  • Recommend using gradlew. Checking in the file guarantees users are on version of gradle that you intended
  • Has effect of creating many gradle installs on your machine. Not that big.

Creating a project

  • Can generate new project or port basic pom files.
  • Use porting as a guide and then hand edit
  • Generate new project in current directory (make empty directory by hand first)
    • mkdir proj
    • cd proj
    • gradle init –type java-application (or java-library)
  • Files/directories in project
    • gradlew/gradlew.bat – wrapper script
    • gradle/wrapper – jar and properties
    • settings.gradle – lists multi project builds
    • gradle.properties – key/value pairs used by build
  • In home directory’x .gradle
  • wrapper/distx – the binarry/installs.

Daemon

  • Improves performance
  • Cache a lot of the JVM startup
  • For a few hours, same daemon used if using same version of gradle on every project.
  • Tip: turn off daemon on CI servers so every build is a clean build

Gradle file

  • repositories
    • can use jcenter or maven central.
    • or can specify a different one for any Maven or Ivy repo; like internal corporate one
    • can list multiple repos and searches in order
  • dependencies
    • ‘group:name:version’ – same concept as Maven GAV. Or can split into three sections. Ken likes the former better; either fine.
    • can write compile (‘g:n:v’) to specify fully. Groovy parens are optional except when they are not. The DSL can interpret without parens and easier to read.
    • New Java plugin has api/implementation to use as scope instead of compile.
    • compile files – lets you add files not in the classpath
    • compile fileTree – lets you add a directory not in the classpath
  • plugins
    • if registered on plugins.gradle.org, can list id and version numbers.
    • built in plugins (ex: java) don’t get a version number

Commands

  • gradle build -t – rebuilds and then sits idle until you change code. Like a rebuild loop
  • gradle build –dry-run – shows what would run (can use -m instead of –dry-run)
  • gradle -b… – use different build file name
  • gradle –stsatus – shows processes

Build scan

  • Starting Gradle 4.3, will prompt you if you haven’t accepted the license agreement and try to use.
  • It’s hosted at Gradle so sends some info out.
  • Gradle Enterprise does build scans in house

My take

Good structured intro/review. Happy that Ken changed the background to white when we were reading code. He also made the code sufficiently large. I didn’t even need my glasses. Go Ken! The only time I needed my classes was when he showed the manual which uses dark gray on a light gray background for the comments. I knew more of the basics than I realized. But good to have it gel better in my head! And I definitely learned things. Rushed at the end though and that was where most of the new stuff is. Glad I’m a New Yorker and can handle talking fast. Can’t type that fast though so stopped taking notes.