reactive programming for java developers – rossen stoyanchev – qcon

For more QCon posts, see my live blog table of contents.

General notes

  • Spring 5 has reactive features.
  • Maybe a third of the audience is doing something with reactive, a little more have read about it (that’s me) and a lot haven’t heard of it. Which is good given the title of this talk!
  • Moore’s law slowing down. No more free lunch from the hardware
  • Now apps more distributed/independent/cloudbased compared to 10 years ago
  • Now we expect latency
  • Thread now has to do more async things. Hit limits of scale. imperative logic more complicated.

Other approach

  • Async/non-blocking
  • Use very few threads. node.js has a single thread. Must be careful because blocking those few threads blocks the whole app
  • Reactive programming – think async. Not everything is a callback

Comparing styles

  • Imperative
    • call a method, get a result, do something with that result
  • Async
    • return a Future instead of a value. Don’t throw a checked exception because my happen in a different thread. They are asynchronous in that you can get a result later. But when you eventually call feature.get(), it throws different checked exceptions.
    • In Java 8, can return a CompletableFuture and call future.whenComplete((user, throwable) -> {})
    • If doing a findAll(), the future/completable future approach doesn’t give you a callback/return anything until all of them are ready. You can’t stream or could run out of memory.
    • Async results as stream – get one notification per data item and another for completion or error
  • Declarative
    • Focus on what, not how
    • Declare what should happen rather than focusing on callbacks
    • Java 8 stream API uses this style. It is meant for collections and pull based/usable once
    • Hot streams – latency sensitive streams, data need to be pushed for you
    • Cold streams – pull based

Reactive libraries

  • Stream like API
  • Can be used for hot/cold streams
  • Project Reactor – Similar to ReactiveX, easy to bridge to Java 8 streams. (ReactiveX is like XUnit – commonality for different languages)

Abstractions

  • Flux – sequence of 0 to n – equivalent of Java 8 stream – can convert to/from Java 8
  • Mono – sequence of 0 or 1 – can convert to/from CompletableFuture

Reactive streams spec

  • Back pressure – producers must not overwhelm consumers. Communicates if downstream isn’t ready for more items. Ex: a slow client like a mobile device isnt able to handle more data and server isn’t blocking
  • Small API – only 4 interfaces
  • Rules
  • Interoperability across reactive components so can compose chain

Java 9

  • Reactive streams included (the four interfaces)
    • Publisher
    • Subscriber
    • Subscription
    • Processor
  • Use a reactive library. Don’t implement the four interfaces yourself
  • subscribe() triggers the flow of data
  • “push” was missing until now. Want in JDK to have foundation
  • Classes
    • java.util.concurrent.Flow – the four interfaces
    • SubmissionPublisher – bridge to reactie streams
    • Tie-ins to CompletableFuture and Stream

Reactor

  • GA release scheduled for July
  • Currently called 2.5. Might changed to 3.0

Reactive Spring MVC

  • Apps annotate controllers even now.
  • Return a Flux type.
  • Spring MVC itself needs to change a lot – called Spring Web Reactive
  • The servlet API assumes blocking. There are async workarounds. Servlet 4.0 might support reactive spring integration, but probably just the basics. Using own bridge to Servlet 3.1 in meantime.
  • Can still use Jetty/Tomcat. They are non-blocking behind the scenes.
  • Don’t need servlet container. Can use Netty.
  • HandlerMapping – change to return a Mono so can be non-blocking
  • Focusing on REST/microservices scenarios

Other reactive efforts

  • MongoDB – reactive driver
  • Couchbase – reactive driver
  • Thymeleaf – split templates into chunks and throttle to respect backpressure

using spring RestTemplate and 2 factor authenication to add issues to github

Last month, I figured out how to use Spring’s RestTemplate to add issues to github programmatically figuring it would save me time in the future.  Well, the future is here.  I needed to add 16 issues (one per milestone.)  I run my program and get a 401.  In particular, I got

Apr 20, 2014 6:42:18 PM org.springframework.web.client.RestTemplate handleResponseError

WARNING: GET request for "https://api.github.com/repos/boyarsky/repoName/issues" resulted in 401 (Unauthorized); invoking error handler

Exception in thread "main" org.springframework.web.client.HttpClientErrorException: 401 Unauthorized

at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)

at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:588)

at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:546)

at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:502)

at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:444)

at jb.Trial2.main(Trial2.java:29)

Right.  I’ve changed my password AND enabled two factor authentication on github in the past month.  Luckily, it was easy to switch my program to two factor.  Here’s what I did.

Create personal use token

I created a second personal use token just like I did for for my git commmand line use by going to the applications page and clicking “generate new token”.  I chose to create a separate token so I could revoke access as soon as I’m done running the web service.

Testing the token

Per the getting started with oauth doc, I ran a command line to test the token:

curl -i -H 'Authorization: token notMy40CharToken' \

https://api.github.com/user

 Wrote another trial program

I adapted my second trial program from the initial iterations to test out two factor.

package jb;

import org.springframework.http.*;
import org.springframework.web.client.*;

/**
 * Call a web service that uses authenticated user to test passing credentials
 *
 * @author jeanne
 *
 */
public class Trial3 {

	public static void main(String[] args) {
		String url = "https://api.github.com/repos/boyarsky/oca-ocp-book/issues";

		HttpHeaders headers = new HttpHeaders();

		String personalToken = "notMy40CharToken";
		headers.add("Authorization", "token " + personalToken);

		// ignore result because just testing connectivity
		HttpEntity<String> request = new HttpEntity<String>(headers);
		RestTemplate template = new RestTemplate();
		template.exchange(url, HttpMethod.GET, request, Object[].class);
		System.out.println("success");
	}

}

I then replaced the authentication part of my real program and it worked like a charm.  Even with the changes to the program, it was faster than creating 16 issues by hand with the proper text/assignee/milestone.

Delete the personal token

I deleted the token to ensure it never gets used again.  I don’t want to run the program by accident with my credentials.  Or accidentally post the token here.

using spring RestTemplate to add issues to github

I’m using github as a project management tool for a (non-work) project.  That’s been working well for being able to track things and have it close to the project artifacts.  Milestones have dates (to remember what was committed to) and issues correspond to tasks or follow up items within the milestone.  I reached a point where I needed to add the same three tasks to over 15 milestones.  (And possibly more in the future.)

It probably would have taken an hour to do this manually.  It’s a boring task so I imagine I’d have gotten distracted.  The boredom of even contemplating the task plus the possibility that I might have to do again make me realize I should spend a little longer and write code.  Which meant the prep time was a little over an hour but the time to run the program was less than a minute.

I worked in iterations because I wanted to learn early on if this was feasible (or a time sink.)

Iteration 1 – setup a project and call a github web service

I used Maven because I knew I’d need a couple libraries and dependencies.  In Eclipse, I created a new Maven project with

  • group id = jb
  • artifact = github-issue-creator

I then added one dependency.  I knew I’d need Spring Web because I wanted to use RestTemplate.  At this point, my pom.xml looks like

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>jb</groupId>
  <artifactId>github-issue-creator</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <dependencies>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-web</artifactId>
      <version>4.0.2.RELEASE</version>
    </dependency>
  </dependencies>
</project>

It was then time to actually call RestTemplate.  I quickly reviewed the docs to confirm my memory of the flow.  Then I wrote the simplest program I could that would call a Github web service.  I had learned from the GitHub API docs of the Octocat Hello World test repository.   I confirmed the issues query web service command in the GitHub docs and went to https://api.github.com/repos/octocat/Hello-World/issues in a browser.  Perfect – I get JSON back.

I then wrote the RestTemplate code using this URL.

package jb;

import org.springframework.web.client.*;

/**
 * View a public repo's issue via a web service on github to make sure can
 * connect. This particular repo is referenced int he documentation.
 *
 * @author jeanne
 *
 */
public class Trial1 {

	public static void main(String[] args) {
		String url = "https://api.github.com/repos/octocat/Hello-World/issues";

		RestTemplate template = new RestTemplate();

		// ignore result because just testing connectivity
		template.getForObject(url, Object[].class);
		System.out.println("success");
	}

}

You’ll notice I’m telling Spring to return an array of Objects.  I don’t need a more specific class since I’m ignoring the result.  It’s just a proof of concept – my real code will do a POST and own’t care about getting the data back.  I knew an array was needed because the JSON in the browser returned a list.  If you do need to deal with the returned values, look at this Jackson example with Spring and RestTemplate.

Iteration 2 – use my credentials to access a github web service

So far so good.  I have two more steps left to accomplish my goal.  I can either test passing credentials or try a POST.  I decided to test passing credentials first so I would be doing harmless GET operations during the testing.

I’ve only used basic authentication with RestTemplate once so I had to look up what to do.  This blog post was helpful showing how to use the Apache Commons Codec library to deal with credentials.

First, I had to add this dependency to my POM.

<dependency>
  <groupId>commons-codec</groupId>
  <artifactId>commons-codec</artifactId>
  <version>1.9</version>
</dependency>

This code obtains all the issues for myRepoName.  And no, that isn’t my GitHub password.  It isn’t my repository name for that matter.

package jb;

import org.apache.commons.codec.binary.*;
import org.springframework.http.*;
import org.springframework.web.client.*;

/**
 * Call a web service that uses authenticated user to test passing credentials
 *
 * @author jeanne
 *
 */
public class Trial2 {

	public static void main(String[] args) {
		String url = "https://api.github.com/repos/boyarsky/myRepoName/issues";

		String plainCreds = "boyarsky:notMyGitHubPassword";
		byte[] plainCredsBytes = plainCreds.getBytes();
		byte[] base64CredsBytes = Base64.encodeBase64(plainCredsBytes);
		String base64Creds = new String(base64CredsBytes);

		HttpHeaders headers = new HttpHeaders();
		headers.add("Authorization", "Basic " + base64Creds);

		// ignore result because just testing connectivity
		HttpEntity<String> request = new HttpEntity<String>(headers);
		RestTemplate template = new RestTemplate();
		template.exchange(url, HttpMethod.GET, request, Object[].class);
		System.out.println("success");
	}

}

As you can see, I encode the bytes of my username/password combination. I then set that as the basic authentication header. I had to switch to “exchange” instead of “getForObject” as noted in that blog post for this to work.

Iteration 3 – actually creating an issue

There are only four differences between this and iteration 2 from a web services perspective.

  1. From the github create issue docs, I learned the POST URL is/repos/:owner/:repo/issues.
    private static final String URL = “https://api.github.com/repos/boyarsky/myRepoName/issues”;
  2. Use HttpMethod.POST instead of HttpMethod.GET.  I actually wasted some time with this step.  I forgot to change it so the program was running without error and returning  issues rather than setting them.
  3. Pass in the JSON for the values you want to set in the issue.
  4. Now I tell Spring to excpect an Object instead of an array because the JSON returned is no longer an array.
String json = "{\"title\": \"" + title + "\", \"assignee\": \""
				+ assignee + "\", \"milestone\": " + milestone + "}";

		// ignore result because just testing connectivity
		HttpEntity<String> request = new HttpEntity<String>(json, headers);
		RestTemplate template = new RestTemplate();
		template.exchange(URL, HttpMethod.POST, request, Object.class);

I tested this once and it worked. I then added the logic to get the proper values (it is formulaic) and let it fly.

Getting the milestone id was easy – it is in the URL: https://github.com/boyarsky/myRepoName/issues?milestone=4&state=open

Conclusion
While it would have been faster to add all the issues by hand, this was certainly more interesting and more fun. And the next time I need to add an issue to each milestone, it will be trivial. Thanks github for the wonderful documentation!

 

edit: if you need to do this for two factor authentication, see my updated post