ant and junit 5 – outputting test duration and failure to the log

In JUnit 5, you use the junitlauncher Ant task rather than the junit Ant task.¬†Unfortunately, this isn’t a drop in replacement. In addition to needing a workaround to set system properties, you also need a workaround to write the test results to the console log.

JUnit 4

With JUnit 4 and Ant, you got output that looked list this for each test run. In the console. Real time while the build was running. This was really useful.

[junit] Running com.javaranch.jforum.MissingHeadTagTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.685 sec

The problem

JUnit 5 itself provides a number of built in options for this. However, Ant integration renders this “not very useful” for a few reasons.

    1. I want to use the legacy listeners. For example:
      <listener type="legacy-xml" sendSysOut="true" sendSysErr="true" />
      <listener type="legacy-plain"sendSysOut="true"sendSysErr="true"/>
      
      

      However, if I enabled sendSysOut and sendSysErr to them, all the listeners present redirect away from System.out. This means if I try to use the built it LoggingListener, it also redirects to the file and I don’t see it in the console. This means, I have to chose between the legacy listeners and seeing real time data.

    2. Ant seems to pass a fileset as individual tests. This means that running this code, prints a summary for each test rather than the whole thing at the end.
      <junitlauncher haltOnFailure="true" printsummary="true">
           <classpath refid="test.classpath" />
           <testclasses outputdir="build/test-report">
      	<fileset dir="build/test">
      	  <include name="**/*Tests.class" />
              </fileset>
      </junitlauncher>
      

      It looks like this. The summary tells me about each run, but not with the test names so not useful anyway.

      [junitlauncher] Test run finished after 117 ms
      [junitlauncher] [         3 containers found      ]
      [junitlauncher] [         0 containers skipped    ]
      [junitlauncher] [         3 containers started    ]
      [junitlauncher] [         0 containers aborted    ]
      [junitlauncher] [         3 containers successful ]
      [junitlauncher] [         0 containers failed     ]
      [junitlauncher] [        15 tests found           ]
      [junitlauncher] [         0 tests skipped         ]
      [junitlauncher] [        15 tests started         ]
      [junitlauncher] [         0 tests aborted         ]
      [junitlauncher] [        15 tests successful      ]
      [junitlauncher] [         0 tests failed          ]
      [junitlauncher] Test run finished after 14 ms
      [junitlauncher] [         2 containers found      ]
      [junitlauncher] [         0 containers skipped    ]
      [junitlauncher] [         2 containers started    ]
      [junitlauncher] [         0 containers aborted    ]
      [junitlauncher] [         2 containers successful ]
      [junitlauncher] [         0 containers failed     ]
      [junitlauncher] [        10 tests found           ]
      [junitlauncher] [         0 tests skipped         ]
      [junitlauncher] [        10 tests started         ]
      [junitlauncher] [         0 tests aborted         ]
      [junitlauncher] [        10 tests successful      ]
      [junitlauncher] [         0 tests failed          ]
    3. If I use JUnit Launcher’s fail on error property, it fails on the first test rather than telling me about all of them. (I don’t use this feature anyway.)
    4. Custom extensions also have their system out/err redirected to the legacy listeners.

The solution

This feels like a hacky workaround. But I only have one project that uses Ant so I don’t have to worry about duplicate code. The legacy listeners are useful so I don’t want to get rid of them.

I wrote a custom listener that stores the results in memory for each test and writes it to disk after each test class runs. That way it gets all the data, not just the last one. And then Ant writes it out.

<target name="test.junit.launcher" depends="compile">
		<junitlauncher haltOnFailure="false" printsummary="false">
			<classpath refid="test.classpath" />
			<testclasses outputdir="build/test-report">
				<fileset dir="build/test">
					<include name="**/*Tests.class" />
				</fileset>
				<listener type="legacy-xml" sendSysOut="true" sendSysErr="true" />
				<listener type="legacy-plain" sendSysOut="true" sendSysErr="true" />
				<listener classname="com.example.project.CodeRanchListener" />
			</testclasses>
		</junitlauncher>
		<loadfile property="summary" srcFile="build/status-as-tests-run.txt" />
		        <echo>${summary}</echo>
	</target>

And the listener

package com.example.project;

import java.io.*;
import java.time.*;

import org.junit.platform.engine.*;
import org.junit.platform.engine.TestDescriptor.*;
import org.junit.platform.engine.TestExecutionResult.*;
import org.junit.platform.launcher.*;

public class MyListener implements TestExecutionListener {
	
	private StringWriter inMemoryWriter = new StringWriter();

	private int numSkippedInCurrentClass;
	private int numAbortedInCurrentClass;
	private int numSucceededInCurrentClass;
	private int numFailedInCurrentClass;
	private Instant startCurrentClass;

	private void resetCountsForNewClass() {
		numSkippedInCurrentClass = 0;
		numAbortedInCurrentClass = 0;
		numSucceededInCurrentClass = 0;
		numFailedInCurrentClass = 0;
		startCurrentClass = Instant.now();
	}

	@Override
	public void executionStarted(TestIdentifier testIdentifier) {
		if ("[engine:junit-jupiter]".equals(testIdentifier.getParentId().orElse(""))) {
			println("Ran " + testIdentifier.getLegacyReportingName());
			resetCountsForNewClass();
		}
	}

	@Override
	public void executionSkipped(TestIdentifier testIdentifier, String reason) {
		numSkippedInCurrentClass++;
	}

	@Override
	public void executionFinished(TestIdentifier testIdentifier, TestExecutionResult testExecutionResult) {
		if ("[engine:junit-jupiter]".equals(testIdentifier.getParentId().orElse(""))) {
			int totalTestsInClass = numSucceededInCurrentClass + numAbortedInCurrentClass
					+ numFailedInCurrentClass + numSkippedInCurrentClass;
			Duration duration = Duration.between(startCurrentClass, Instant.now());
			double numSeconds = duration.getNano() / (double) 1_000_000_000;
			String output = String.format("Tests run: %d, Failures: %d, Aborted: %d, Skipped: %d, Time elapsed: %f sec",
					totalTestsInClass, numFailedInCurrentClass, numAbortedInCurrentClass,
					numSkippedInCurrentClass, numSeconds);
			println(output);

		}
		// don't count containers since looking for legacy JUnit 4 counting style
		if (testIdentifier.getType() == Type.TEST) {
			if (testExecutionResult.getStatus() == Status.SUCCESSFUL) {
				numSucceededInCurrentClass++;
			} else if (testExecutionResult.getStatus() == Status.ABORTED) {
				println("  ABORTED: " + testIdentifier.getDisplayName());
				numAbortedInCurrentClass++;
			} else if (testExecutionResult.getStatus() == Status.FAILED) {
				println("  FAILED: " + testIdentifier.getDisplayName());
				numFailedInCurrentClass++;
			}
		}
	}
	
	private void println(String str) {
		inMemoryWriter.write(str + "\n");
	}
	
	/*
	 * Append to file on disk since listener can't write to System.out (becuase legacy listeners enabled)
	 */
	private void flushToDisk() {
		try (FileWriter writer = new FileWriter("build/status-as-tests-run.txt", true)) {
			writer.write(inMemoryWriter.toString());
		} catch (IOException e) {
			throw new UncheckedIOException(e);
		}
	}

	@Override
	public void testPlanExecutionFinished(TestPlan testPlan) {
		flushToDisk();
	}
}

 

ant and junit5 – simulating sysproperty

In JUnit 5, you use the junitlauncher Ant task rather than the junit Ant task.¬†Unfortunately, this isn’t a drop in replacement. For example, junitlauncher doesn’t offer the option to fork and run the JUnit tests. As a result, it also doesn’t have the nested sysproperty tag so you can pass system properties. This is a problem.

For the CodeRanch, we set a system property for the default file encoding. Since developers are around the world, we can’t assume everyone “just has” the encoding set.

Disclaimer

Since JUnit 5 functionality for Ant was introduced this year, I’m hoping what I did in this post is a short term workaround.

Option 1 – pass to Ant

You can pass the the properties to Ant itself as described on Stack Overflow. For example:

__JAVA_OPTIONS=-Dfile.encoding=ISO8859_1

pros:

  • simple

cons:

  • subpar – all the developers need to remember to do this. The reason we had it in Ant in the first place is so folks wouldn’t need to remember
  • for some use cases, the desired system properties could be derived and not know when calling Ant

Option 2- Nashorn Code

Since JUnit 5 is being run in the same process as Ant itself, you need to set the system property in memory. Luckily, Ant allows you to run scripting in various languages. I chose Nashorn because it is built into Java. (There are other variants of this; see below)

<script language="javascript">
  <![CDATA[
    var imports = new JavaImporter(java.lang.System);
    imports.System.setProperty('file.encoding', 'ISO8859_1')
  ]]>
</script>

Pros

  • Short and simple
  • Nashorn is deprecated for removal starting Java 11. This means at some point, it can be removed. (I’m hopeful that the Ant task itself will support system properties by then.

Cons

  • Requires Java 8 or higher
  • The System property is set for the remainder of the build (you could write another code block to null it out after the test if this is a problem)

Variants of option 2

If you are running Ant with a version of Java below Java 8, you could use¬†this technique, but use Rhino instead. I didn’t test this, but I think the code is


importClass(java.lang.System);
System.setProperty('file.encoding', 'ISO8859_1')


JUnit 5 itself uses Java 8 so nobody would be in the situation of pre-Java 8 and trying to use this blog post.

If you are running Ant with a version of Java where Nashorn has been removed, you could use Groovy or Jython as the embedded language. The code is simpler. I didn’t chose this because it requires adding another jar to the Ant directory. I prefer to minimize these set up extensions. Especially for a feature like this which is likely to be temporary.

System.setProperty('file.encoding', 'ISO8859_1')

JavaOne – Project Jigsaw – Integration with Tools

“Project Jigsaw – Integration with Tools”

Speaker: Alexandru Jecan (author of Java 9 Modularity Revealed)

For more blog posts from JavaOne, see the table of contents


What expect build tools and IDEs to do

  • Code completion (IDEs only)
  • Make things easier
  • Work with class path and module path

Maven

  • Only Maven 3+ supports Java 9.
  • No changes to Maven core to run on Java 9
  • presence of module-info.java decides if Maven uses module path
  • Need Maven compiler plugin 3.6.1 to compile Java 9 code
  • Discourages using automatic module name
  • GAV not related to module name. Continue to use GAV.
  • Use module name in module-info.java – double work
  • Maven does not generate “requires” clauses for module-info.java
  • Can add <compilerArgs> options for -add-modules or -add-exports
  • New tag <release> which takes precedence over source/target tags. Recomended compiling twice or backward compatbility. once for java 9 modules and once for java 8. [why? isn’t that what multi release ars are for?]
  • maven-exec-plugin – can configure module path
  • taked abut updating maven toolchain file [not clear how this differs than prior versions of java]
  • maven-jdeps-plugin – can automatically generate module descriptor [but said earlier couldn’t]
  • maven-jmod-plugin – new plugin; not yet released. Creates and lists content of jmod files. Merge native code into jar
  • maven-jlink-plugin – new plugin; not yet released. Creates runtime images
  • maven-depdendency-plugin – said lists GAV [not clear how changed

Gradle
No first class support for Java 9 yet. Coming soon. [but said using; puzled?]

Ant
Ant 1.9.8+ support JDK 9 modules for java/javac/junit. [yet Ant doesn’t support JUnit 5]

Loom

  • New build tool for Java 9
  • Uses YAML config. Uses Maaven repo.
  • Supports JUnit 5,

loom.builders

Moditect
Maven plugin that generates module-info.java based on dependencies. Says helps a little but doesn’t do all the work. [how?]

SonarJava
SonarJava 4.11+ recognizes module keywords

Graphviz
Open source visualization software
Reads jdeps input to generate graphical representation of Jigsaw. As can IDES

IDEs

  • NetBeans, Eclipse and IntelliJ all understand modules clauses [if you count pre-release versions]
  • Also offer autocompletion, syntax higlighting, etc
  • NetBeans 9 – not yet released; must build yourself. Showed editing module path, visual of module dependencies, jshell, jlink
  • Eclipse Oxygen with Java 9 support – came out 9/27/17. Ten second demo
  • IntelliJ 2017.1+ – Mark directory as source root so directory name matches module name. Errors on common module errors, Showed module editing, module dependencies and other features

Open source readiness

How to modularize your app

  • Introduce modue-info.java by creating manually or running jdeps to generate
  • Problems: using JDK internal APIs (ignore warning or fix), using odule not availalbe such as xml bindings (add-modules at compile and runtime), cyclic dependencies (can have at runtime but not compile time), split packages (Oracle plans to make change in later version of Java to deal with this)
  • [Note: He recommended renaming packags as a solution to spit packages. That sounds like a horrible idea unless you can guarantee only you call that code]

    My take before session: You know how they say that first impressions matter? The speaker is wearing a suit. 90% of the people in the room are wearing jeans. Two people in the room are wearing a suit The speaker and someone he knows. Then he showed the table of contents. There are 29 points in 45 minutes in the outline. Preparing to have my head spin!

    My take after session: He is in fact technical. The suit was misleading! The pace was way too fast though; drinking from a fire hose. Not enough time to understand/process many of the points. He talks fast (as do I), but key is to *pause* if you talk fast so people can catch up. Also the side transitions were distracting. A cube transition is cute. But if you are reading when it moves, it is disorienting. And due to the speed, there was a good chance of being reading when transitions started. This was good information, but should have been two sessions so split up and a decent pace. And omiting how to migrate your libraries; that’s a talk on its own There is a Maven BOF tonight; maybe folks can discuss more then!

    One minute after the official end time, he asked if there were questions. My head was spinning with questions. [I didn’t even have time to process which was most important]. Another attendee asked if having JUnit tests in a parallel directory with the same package name is a split package. The speaker said yes and went on to say to wait for Java 10 or rename. I interjected at that point. Unless you are distributing a test jar, I don’t think this is a problem. In fact, most IDEs compile both the /src/main/java and /test/main/java directories to the same folder. I stated this and asked the speaker if he agreed. He said yes.