product review: infinitest

Want to run your JUnit tests as soon as you change the code?  That’s what infinitest strives to do.  The starting point is which directs you to the corporate site.

What does it do?

Infinitest installs as a plugin to your IDE.  In Eclipse, it adds a bar to the bottom of your screen.  I’d say I lost about an inch of real estate.  Not much, but if you have a lot of other plugins, this could be a problem.

Every time you change Java code, it sees which unit tests should be re-run and changes the bar to the appropriate color:

  • black – waiting for changes – when open workspace and haven’t done anything yet
  • green – all is well
  • yellow – no tests were affected by this change
  • red – one or more tests are failing (could be from this change or a previous change)

The bar, also tells you how many “test cases” were run.  They mean the number of test classes rather than the actual number of tests so this number isn’t as useful as it sounds.

Does it require a lot of configuration?

No.  The entire user guide is two screens long.  Configuring it took me less than two minutes – which were spent looking up the name of my integration test runner and package so infinitest didn’t run them.   I also looked at the preferences and saw it had a default of half a second before giving up on a test.  That’s a good length.  No unit test should take that long.

What do I need to run/install it?

Infinitest runs as an Eclipse or IntelliJ plugin.  I tried it in Eclipse.  You install from an update site.  My only concern is they don’t provide a manual zip file style local update.  Now granted, neither do many other products, but they take the time to write:

If you are behind a firewall, you may have limited access to the Internet. This can cause problems accessing the update site. If you use a web proxy to connect to the Internet, make sure that you have correctly entered your proxy settings into Eclipse under Preferences->General->Network Connections.

If it’s enough of a problem to document, I think it’s enough of a problem to realize you have customers that aren’t allowed do download any plugins without having an approved body do it.

How much does it cost?

The initial install is for a 30 day trial.  After that, you have to decide if you are an “individual” or “corporate” user.  Once you decide you either fill out a form saying you are an individual or pay $150 per seat (perpetual license) and a license is e-mailed to you.  Just copy/paste the license into your IDE and you are done.

individual – you own the copyright.  In other words you aren’t selling what you write.  From talking to the vendor, it sounds like open source and code I write for JavaRanch fall into this category.

corporate – code you write for someone else to make money – also known as your job.

How well did it work?

I have three Java projects in the workspace I used to test:

  1. JForum – The JavaRanch fork of JForum.  About 200 classes and a bunch of unit tests.
  2. PickWinners – The program JavaRanch uses to pick winners in the book promotion.  About a dozen classes and about the same number of tests.  Both unit and integration tests.
  3. Play – this has one class and no unit tests.  I use it to test out “what does this code do”

My attempts:

  1. Edited a file in my Play project – it ran all the test cases in JForum and PickWinners.  At first, it looked like it wasn’t using dependencies, but I think this was because it was the first run and it though nothing had run yet.
  2. Edited the file in Play project again. This time it correctly realized no tests were affected.
  3. PickWinners now has 5 errors of type “Infinitest Test Failure” – All are errors accessing the database from my integration tests.
  4. After adding infinitest.filters, I had to clean the project to remove the failures.  (This was documented in the manual page.)  It does re-run tests that are still unfiltered.  It took me to passes to get the right filters.
  5. Made another change to PickWinners – it’s still running JForum tests
  6. Cooked and ate dinner (in other words, a bunch of time passed)
  7. Tried again – all of a sudden it  started picking up only the truely dependent tests.  Maybe there was a learning curve?  Cool.
  8. Almost everything I tried after dinner did pick up the correct tests based on dependencies.  It (usually) found tests that use the class indirectly.  It got faster and finding dependencies when I repeatedly edited the same class.  I assumed it cached the result.  Which is good because when doing real development, this is what happens.

What problems did I encounter?

  1. The JForum project doesn’t show the test failures as compiler errors.  The PickWinners project does.  The JForum project also complains sporadically about a class not being found that is in the same project.  Note: this problem temporarily went away after I restarted my workspace.  It came back after I introduced a failing test and corrected it.
  2. As I wrote above it usually finds the relevant tests.  For example, I changed the implementation of post.setText() to use the value “hi” instead of the passed in value.  I expected a whole gaggle of tests like the following to run and fail.  None of them did.

    public void noCodeTags() {
    String actual = post.getText();
    assertEquals(“transformed post”, “test”, actual);

  3. I have to assume this won’t work for reflection based code.  I don’t have any on hand.  While I could write some, it doesn’t sound like it work from the vendor’s description.  And if the vendor admits it won’t work, I’m inclined to believe them.

Final thoughts

I like the emphasis on “keeping the bar green.”  If you always see it on the screen, it reminds you it exists.  I haven’t had it installed for long, but I’m already used to it being there and relying on it.  Even if it isn’t perfect, it’s great to have present.

It worked well enough to be useful on my home projects.  I’m not sure how well it would function in my work environment where compiles involve a lot more code.  The “find the proper tests phase” was noticeable at home.  It could be better at work (faster machine) or worse (much bigger workspace.)

not sure how well it would work in my work environment where compiles involve a lot more code

Great Developers Are Not Afraid to Experiment

As a moderator on the JavaRanch, I often come across people asking “What would happen if I executed the following code”. Many times the author of such posts can answer the question by copying and pasting his/her code into a Java main() method and running it. Some might chalk these posts up to being lazy, but, clearly, taking the time to write a post on a message board – often signing up as a member for the first time – takes some amount of effort as well. With that, I’m going to be go with the assumption developers avoid experimenting with code because they are scared or unsure of their own knowledge. Besides which, if it is a matter of laziness, there’s not much advice I can give except to say “Don’t be lazy”.


Why is experimenting with code can be scary

In my experience as a teacher, development lead, and moderator I often come across developers who are unsure of their own knowledge. Often times they don’t fully understand what it is the code is doing and are afraid to experiment for fear it will either demonstrate their own personal weakness or harm the existing code. To them, I believe that experimenting is most crucial, if only because some of the doubts and questions that linger in their head can often be answered in an surprisingly short amount of time. If you are staring at a piece of code, puzzled by what you do not understand, take my advice: Step back and play 20 questions, ask yourself “What are questions I have about this code that can be answered with a simple yes/no?” then set up test code to answer each question. Once you have your answer, your doubt about the application should vanish, replaced by first-hand knowledge of what is going on. Keep in mind that sometimes these experiments lead to even more questions, but that’s good; it’s part of the learning process. In those cases, perform even more experiments on the code.

What is an experiment?

What constitutes an experiment? Oftentimes, it involves just writing a short line or two of code, then writing a logging statement, or, more commonly, if you don’t have a logger, a System.out.println() statement that displays the value of some variable or object. For example, if you don’t know why a method is behaving a certain way, add a dozen output statements throughout the code so you can follow two things: 1) the path the code is taking and 2) the value of the data throughout the code. Many times, you may be staring at a section of code, wondering why it’s not working, only to find out that section of code is never reached at run-time. Experimenting can be about changing values and recording the inputs, but sometimes its just about outputting where/what you think a process is doing.

Some people will recommend a debugger for experimentation but I’m not one of them. Aside from often being unwieldy and confusing to use, especially for beginners, sometimes running code through a debugger can affect what it outputs. For example, in a server environment, remote debuggers can be especially difficult to use. Services may have transaction timeouts of 30 seconds, and pausing the code in the debugger can cause an exception to be thrown before the method is complete. If you like using debuggers, more power to you, but as someone who has used both output statements via logging tools and debuggers, I greatly prefer the logging tools. Primarily, this is because the output statements give you more of a trial and error structure to work with: either a test succeeded, or it failed, and the output is all there in front of you.

Stand-alone Safe Experiments

The easiest and safest experiments are those you create that are completely separate from any other code. In terms of Java and Eclipse, this is akin to creating a new Workspace and a new Java Project to run the test code in. Every developer should have a temporary, throw-away workspace like this to perform low-level Java tests with. Simply create a file which in some way asks the question you want answered, and execute the program to evaluate the results.

Safe Experiments within Existing Code

Let’s say the code is highly framework dependent, such as often is the cases with J2EE and database-driven applications. For example, creating a temporary workspace to house your test case may be too cumbersome to implement. In such a case, you can run the experiment inside your existing code provided you are careful and follow some general guidelines:

  • If you are working with code that is backed by a repository, such as Subversion, CVS, ClearCase, etc, make sure your experimentation code does not get checked in or you may end up with applications such as this, this, or even the impossible (the last one). The DailyWTF has literally thousands of such cases. It is perfectly fine to experiment, just be careful to cleanup when you are done!
  • If you are working with code that is *not* backed by a repository, then install a developer repository! I cannot tell you enough the power and value of using a coding repository for all development work. In lieu of that, though, you should just make a copy of the project and/or workspace and experiment with the copy, keeping the original intact.
  • If you are working with code that connects to a database, make sure it’s not one other developers use. Making changes to a shared database as part of a test could affect other developers, so, if possible, you should have your own local copy of the database. This does not mean you should have a production database, but merely a copy of a QA or Development database

Final Thoughts
In my experience, one of the things that separates a great developer from the rest of the pack is that the great developer fully understands the code they are writing. When a bug appears (even great developers can cause bugs), this person often has a good idea where to look for the problem right away, and save valuable support time. Ultimately, if you ever find yourself mystified by your own code base, run some experiments and learn why things work the way they work. You’ll be a better developer for it!