ken kousen on groovy

I saw Ken Kousen deliver his intro to Groovy talk today at the NY Java SIG.  While I’ve done some small things with Groovy, I’ve forgotten a lot of what I once knew.   Some of it, i remembered/used but am including here so this isn’t random facts but a summary of what was presented. Blogging about it this time in hopes I can retain more.

First of all, Ken is a great speaker.  If you have the chance to see him in person, go.  He:

  • is entertaining – programmer humor
  • presents info clearly
  • has well thought out examples
  • switches well between the groovy console, command line, browser and slides.  While there were frequent switches, it was easy to follow
  • live examples. Surprises and mistakes and all    Icndb.com for humor in json examples provided energy towards the end

Useful links

Now on to Groovy

What is groovy

  • Groovy is like Java 3. Other jvm languages are simulating other languages like jythong or special purpose like functional languages. Groovy is a general purpose like a next generation OO language. Plus you can fall back to Java and make it “groovier” later [which is great for learning]
  • Compiles to bytecode.  Also a C compiler is available.
  • If the name of your groovy file matches the name of the class, you can only have a class in the file. Otherwise you can put multiple classes or even just scripts in the file. Ken’s convention is to use underscores in filenames for scripts to identify them.
  • Hello world is println “hi” [yes, that’s it]
  • Parens are optional “until they are not”. They have a low precedence.  Can leave out if obvious where they go
  • Semicolons are always optional
  • Perls goal is to write the shortest code possible. Groovy’s goal is to write the simplest code possible.
  • Testing and build are the place to start with groovy when introducing into a reluctant organization

Options to run

  • groovy foo.groovvy – don’t have to compile first. Should compile first when use it for real though. If want to integrate with java, must run groovyc to compile since java requires class files.
  • The class files require groovy in the claspath to run with “java” rather than “groovy”. In groovy home, embedable directory contains groovy-all and groovy-allindy. The later is for invoke dynamic in Java

Tools

  • Groovy Grails Tool Suite – for Eclipse
  • Groovy menu has convert file extensions to groovy where right click java class. This lets you refactor slowly. [Or if create java class by accident like often do]

Operator overloading

  • Every operator corresponds to a method.  This means if your class implements that method, you can use that operator.  This lets you write new Date() + 7 to get a date a week in the future.
  • ** is power operator
  • [] goes with getAt or putAt methods.  This is what lets you say s[-1] or s[0..2] instead of pure array indexes.  The getAt/putAt methods understand how to navigate.  The methods also know to call next for 0..2 and prev for -3..-1.  I also learned you can specify multiple ranges such as as s[0..2,4..5]
  • In groovy == calls the equals method. Which means just converting a java class can result in an infinite recursion. Run your tests when refactoring
  • To append: list << value

Groovy Types

  • 123.getClass().getName() is an Integer. If you make it bigger becomes long or BigInteger dynamically.
  • 2 – 1.1 uses BigDecimal so accurate unlike 2d – 1.1d which has Java roundoff error from using doubles.
  • GString “test ${name}” uses interpolation to substitute the variable name. You don’t need braces if just one variable
  • Single quotes are Java strings
  • Double quotes are groovy strings
  • Triple quotes are for multi line strings.  Useful for SQL.
  • Def means don’t know or care about type
  • Duck typing. Can call any methods on def oject. If method exists at runtime ,call it. Else throw MissingMethodException at runtime.
  • Map keys assumed to be Strings

Groovy translation

  • S.class.name works. When it looks like you are referencing a private field, groovy adds get or set to call appropriate methods. The runtime environment provides getter and setter methods so  you don’t need to write them
  • Don’t need to type return belcause last expression is automatically returned

Plain Old Groovy Object (POGO)
Generates at runtime:

  • getter/setter – If don’t want setter, add final to instance variable.  If don’t want getter or setter on instance variable, type private.
  • map based constructor
  • class and methods public by default
  • instance variables private by default

All you have to write is a class and the types.  If you implement a method, Groovy will use yours and not generate it.  If using the map based constructor, Groovy calls the default constructor and any appropriate setters.  This means you don’t need overloaded constructors.

class Person{
String first
String last
}

Person p = new Person()
p.setFirst("f")
println "$p.first"

new Person(first: 'a', last: 'b')

Walkthru of converting a POJO to a POGO

  1. Remove getters and setters
  2. Remove private from  instance variables
  3. Remove semicolons
  4. Remove constructor – need to use map constructor – affects callers [not backward compatible to simplify all the way]
  5. Remove public on class
  6. Remove import for java.util.Date
  7. Add @Canonical which includes @ToString, @TupleConstructor and @EqualsAndHashCode.  All are AST transformations
Changing types
  • [1,2] creates list
  • LinkedList a = [1] makes the desired type
  • [1] as Set also creates the desired type

Closure

listOfStrings.each { /* code using it as default loop variable */ }

map.each{ k,v -> /* code */ }

list.collect{ /* transform to new list where results make up the new list */ }.join(',')  // map reduce

Closure coercion can be used for an anonymouse inner class
{ /* code */} as FilenameFilter
Useful when an interface has only one method because uses implemetation for all methods in interfaces.  Could pass a map of closures but might as well implement interface then.

Helpful classes/methods

  • Groovy adds toURL to String class and getText to Url class masking query.toUrl().text a one liner
  • XMLSlurper lets you get a dom tree from a URL which you can then navigate as tree.root.result[o].geometry.location.lat 
  • db.rows(“selet * from table”)
  • db.eachrow(” query “){ /*code */}
  • groovy.json.JsonSlurper().parseText(json)
  • groovy.xml.MarkupBuilder to create. Similarly for xml, json and ant
Testing
  • Must test code since constraints we are used to are gone
  • Spock over GroovyTestCase – very nice framework.  See example in github/book

Random interesting tips

  • Can use map based constructor on a pojo too because groovy will call default constructor and set methods
  • foo.with{} calls foo.xxx for any methods not found in scope
  • Meta programming: add methods to class that need. Even to a java class. Complex.metaClass.plus= {}.  Every class has a metaclass.

getting computer vision to work on a raspberry pi

This year, I helped a local high school FIRST robotics team get computer vision working on a Raspberry Pi.   Two students worked on the problem.  At the beginning, they both worked on computer vision.  Later, one specialized on the Pi and the other on CV.  I learned a lot from a book and from playing with it.  We encountered some interesting and memorable problems along the way.

Recognizing a target

We started with sample code from another team from last year.  This helped learn how to write the code and understand the fundamentals.  It also helped with the “plumbing” code.  As hard as it was to recognize a target, this didn’t prove to be the most frustrating part.  The Pi itself mounted a number of challenges.

Parts for the Pi

We bought/scavenged parts for the Pi.  A USB keyboard, USB mouse and cell phone micro charger where donated.  A HDMI/DVI cable we needed to buy.  We borrowed a computer monitor and ethernet cable.

Finding the right jars

The Pi is built on ARM.  We needed javacv-0.2-linux-arm.jar.  It turned out there is no linux arm version in the latest javacv (0.3).  There is one in 0.2 which we used.  Which was incompatible with the versions of other tools.  (see next problem.)

Setting up the Pi

Compiling opencv on the pi takes 4 hours.  Since that’s how long a meeting is, this meant running the compile overnight.  Having to wait overnight to find out if something worked was like a taste of what punchcard programmers had to go through!

Then it turned out we couldn’t even use our compile.  We were missing the libjniopencv_core.so file. We spent a few days trying to solve this.  We wound up  using a pre-compiled for Pi version.  This is how we got version compatibility.

Updating a NetBeans Ant script

Since speed matters in a competition,we wanted to change the build run target to not compile first.  Netbeans comes with an empty looking build.xml and a useful build impl xml file.  (This is actually my favorite feature of NetBeans – that the build can easily be run outside of NetBeans.)  We easily found the run target in the build impl file.  We copied it to build.xml, renamed it and removed the compile dependency.  This wasn’t actually a problem, but it was interesting how NetBeans sets up the build file.

Starting a script on startup

We wanted the Pi to start the computer vision script automatically on boot up.  We created a file in /etc/init.d since this is a Linux (Debian) install.  Then we made a fatal error.  We forgot to add the & to run the script in the background.  So when we tested rebooting, the Pi hung.  And we couldn’t SSH to it because it hadn’t booted.  The solution was to take the pi’s sd card to another computer and edit the bootup script to use single user mode.  We could then login and edit the startup script to add the missing &.

Networking

We used Java sockets to transfer the “answer” from the Pi to the robot.  The answer being a single number representing the number of degrees off from the center of the target.  We made the mistake of testing this with both ends on a regular computer.  When moving to the robot it didn’t compile because the robot uses J2ME.  We then refactored to use the mobile version (code here).

Performance – CPU

Success.  Computer vision works.  The problem is it took 3 seconds per image.  We reduced it to 1.3 seconds per image by reducing the resolution to the smallest one the camera supports.  We shaved off another 0.1-0.2 seconds by turning off file caching in ImageIO.  We learned the problem was full CPU usage when calling ImageIO.read.

I found an interesting thread showing the “old way” of creating a JPEG using ImageIcon was much faster.  We tried it and the thread was right.  It even created an image we could open in a photo editor.  The problem is that it didn’t work with our code/libraries for image processing.  We don’t know why.  Obviously ImageIO has some default we are unaware of.  A 1 second time is acceptable so this isn’t  a problem.  But it’s an interesting conundrum.

Another interesting CPU note.  We tried compiling image magik.  It took over 3 hours on the Pi.  By contrast, it took 2.5 minutes on a Mac.

fixing csrf for jforum and csrf filter analysis (part 1)

This post goes through how we fixed CSRF (cross site request forgery) in JForum, issues encountered and approach.  It is useful reading for anyone who needs to protect against CSRF on their website.

Background

Stock JForum has a number of security vulnerabilities.  We’ve fixed a lot of the XSS ones.  We hadn’t fixed CSRF as of early January 2013.  (It is fixed now – don’t bother trying.)  We had captchas enabled for creating ids.  And I think our working theory was that other CSRF problems would be annoying, but easy to undo.

On January 14th, we learned there was going to be an announcement of XSS and CSRF vulnerabilities  on January 16th (later changed to February – High-Tech Bridge asked me to post this before their announcement.)  I did a code review and learned that someone could exploit a CSRF attack to cause all sorts of damage including deleting the forums – which would require a database restore and likely lost posts from the duration.  Eek.  Nothing like an urgent need to have a release.

What is CSRF

Cross Site Request Forgery is a security attack where the attacker tries to do something in your name.  For example, if you open an email (with images enabled) or go to another website in a tab of your browser while being logged into your bank in another tab, the bad guy could try to move money between accounts for you.  Not something you want.  In a forum, the bad guy could post spam or ads in your name.  Or even delete the forum.  OWASP has an excellent overview of CSRF along with suggestions of “fixes” that do and don’t work.

Analysis

The first thing I did was write code (AllJForumActions.java) to output the names of all actions in JForum that can be accessed from a browser.  To see how much analysis was required.   In stock JForum, the 112 such method names are listed here..  In JavaRanch/CodeRanch forked JForum, there were 191.  (we’ve added a lot of functionality)

I then classified them in a property file by whether they were safe (read only) or needed protection.  I wrote a unit test to check this property file:

  • has all action names represented
  • doesn’t have any action names more than once
  • only lists valid case.
A sample property file shows the expected format.  The unit test helps future proof the CSRF functionality in addition to checking for typos.  (With almost 200 actions, it is easy to miss something by hand.)  Unit test failures direct developers to a few paragraph guide I wrote up on whether the new action needs CSRF protection.

Choosing a CSRF filter

Ok.  Analysis done.  Now on to coding. CSRF is a common problem.  I’d like to use an existing tested filter and just adapt it to JForum.

1) Google code CSRF Filter (StatelessCookieFilter)

From website: this one says it is better than the OWASP one by being simpler and using cookies rather than session (memory) usage.

Analysis

  • It is simpler, I’ll give them that.
  • I’m not terribly concerned with the memory implication.  It’s just one token per user.  We store other things in the session in memory.
  • Disadvantage: the code requires you to manual add the CSRF token to all POST forms.  That’s time consuming and error prone.
  • HUGE disadvantage: the filter assumes that only POST methods need to be protected.  This is only true if you strictly enforce not accepting GET requests for any requests that change state.  While I agree all code should be that way, it isn’t in the real world.  Many frameworks (including JForum) treat get and post requests the same by directing them to the same code.  If I was using this code, I’d need to change it avoid the check for POST.  (albeit not a big deal since it is less than 100 lines long).  We’d need to protect both get and post actions and exclude the expected GET URLs from the filter.  This checks the expected POST actions regardless of their actual submit type.
Summary: the code is smaller and simpler because it does a lot less.
Appears to only support “token per request.”  I want token per session.  It’s less secure than per request, but users on coderanch do a lot of “let me open 20 tabs at once.”  Besides, we aren’t running a bank here.

3) OWASP filter

Well documented.   Supports many types of checking.  Sets up Javascript DOM injection of token so don’t have to hand edit each page.  It supports a choice of only injecting the token into forms or whether to include src/href links as well.  This is great as I know that each and every update to the database isn’t in a form.  I’m also confident the developers know what they are doing because the documentation says:

 However, most JavaEE web applications respond in the exact same manner to HTTP requests and their associated parameters regardless of the HTTP method. The risk associated with not protecting GET requests in this situation is perceived greater than the risk of exposing the token in protected GET requests. As a result, the default value of this attribute is set to true. Developers that are confident their server-side state changing controllers will only respond to POST requests (i.e. discarding GET requests) are strongly encouraged to disable this property.

Comparing this to the google code which only checks POST, it is a very easy decision which one to go with.  The one that actually secures you against CSRF in the face of code that isn’t perfectly written.  OWASP CSRF Guard!

Starting out with OWASP CSRF Guard Version 3

The documentation was clear and self explanatory so let’s get into the problems I encountered:

  1. Minor problem: the OWASP CSRF Guard source code from github isn’t the same with the download offered on owasp.  It was an easy Ant build from the github version.  I rebuilt because I needed to call a method that existed in the github version but not the download/packaged one
  2. Major problem: JForum doesn’t consistently use URL names for actions.  There are a lot of Ajax calls that pass the “url name” as a parameter so I can’t filter these out through the OWASP API.    And the “unprotectedPages” logic in the OWASP filter is deep within the component in a final class.  Very secure, but I can’t extend it so must find another approach.
Clearly customization will be needed.  Which brings us to part 2 – extending the OWASP filter or skip right to part 3 – changing JForum.