Bozho's tech blog

A while ago I asked the following question on stackoverflow: “Why would I use a template engine?”. I compared template engines to simple JSP includes.

Although I didn’t particularly fancy any of the answer, it was obvious that templating is the preferred way of defining the structure of websites, especially if they are bigger ones. Since then I have used velocty, freemarker, tiles, sitemesh and grails. I hate freemarker, because of its cryptic syntax, its illogical behaviour at times (e.g. only strings as map keys) and because of throwing exceptions when something is null (even if nothing is invoked on the null). I don’t particularly like velocity (apart from using it for email templates) – there’s a learning curve and you don’t get more than a JSP. I didn’t like tiles and sitemesh because they require some external xml configuration that makes navigating through the code harder. And while all of these worked and I wouldn’t say “don’t use them” (apart from freemarker), they didn’t feel good.

When working with a grails application, I very much liked GSP (groovy server pages) – it is simple but powerful and has decent IDE support. It has all the good parts of JSP combined with simple if/else constructs and with built-in templating and some extra features. Using it in a non-grails application, I thought, would be an issue. It seems that it is possible, and I’d recommend that, although I haven’t used it outside of grails – here’s how to do it.

But there seems to be a way to have templates with plain JSPs as well (since v2). Everything you need to know is explained in this answer. In short – you use custom .tag files to define de-facto templates. Just make a WEB-INF/tags/templates folder and put your templateName.tag there. Then use in a regular JSP. It is really simple and you don’t need any 3rd party library.

These are the two options I recommend – GSP and JSP. They are simple, usable by front-end developers that do not have experience with Java and the code is rather readable.

Your system is probably sending some emails. Sometimes these emails contain links to the public part of the site, sometimes they have links to the authentication-protected part. Either way, if the email is sent to registered users (as opposed to just subscribed emails) you should not make the user type in username and password. Even if it’s the public part of the site, the user may then want to do something that requires authentication with the content you display – e.g. add it to favourites. It’s a bit tricky with public content though, and you might not want to have it there, as users tend to forward “digest” or “best this week” emails, and then the recipients will be able to impersonate them, without even knowing.

Anyway, when users click on a link in an email sent by your system, they could be logged in automatically. That’s the product requirement I think many systems should have, and it sounds pretty reasonable. But I’ve rarely seen in. Let’s assume you like it and you want to implement it in your system. How to do that, as there are certainly some security implications?

1. For each email sent, generate a token. The token should be the HMAC of (the user’s email (or username, or id) concatenated with the token generation timestamp)
2. Use an application-wide, configurable per-environment key for the HMAC. Also use a good hash algorithm, like SHA-256 (and not MD5)
3. Append the token to each link in the email. You are probably using an email template, so just have ?emailLoginToken=${token} for each link. Also pass the email and the timestamp as parameters.
4. Have a filter/interceptor that looks for that specific parameter name, and if it is found, invoke the authentication-by-token logic (described below)
5. Redo the HMAC with the parameters passed (email and timestamp) and compare it to the token parameter
6. Check if the token is still valid – you can have an expiration period and don’t allow tokens to be used after more than 2 weeks after generating them.
7. If everything is OK (the token passed and the result of redoing the HMAC are the same), login the user (send a session cookie). But add a flag (in the session) that the login was via a token. Then do not allow changing password, email or any “sensitive” action without password confirmation. Use the same conditions as in your “remember me” login, if you have implemented that.
8. Have all such links go through https

There’s another, slightly different scenario. Instead of using hmac and sending all the parameters, just generate the token as the hash of the email and timestamp and store it (and the timestamp) in the database, with foreign key to the users table. In this case, the authenticity confirmation comes from the fact, that it’s found in your database, rather than through verifying an HMAC. Then, when the token is presented as a parameter, simply look it up in the database. You should still pass the user’s email as parameter and compare the passed email parameter to the email of the user corresponding to the persisted token (to avoid guesses). On successful login you can delete the token. You may decide not to do this, and have a scheduled job that cleans up tokens older than X weeks. If you delete it immediately, it will be more secure. If you leave it, it will make it possible for the user to open the same mail again, and the login will still work. This approach lets you invalidate tokens on demand. For example if users decides to change their password or email, you can invalidate all their tokens.

It seems like a tough process, but it’s fairly easy to implement. The tough part is taking all the points into account and not compromising security.

Note that security is an important aspect here. As pointed out on reddit, email may be insecure – when the user downloads emails via unsecured POP3 (with Outlook, Thunderbird, etc), an attacker may obtain the links and impersonate the user. That’s why you should restrict the actions the user can perform when logged in via a token. This may not matter that much, since most users either use webmail or secured POP3, or internal email server. But you should not do that for banking software, for example. Having tokens in links is probably less problematic than password reset links, though, which can also be intercepted the same way.

One extra step you can take when addressing security is restrict the token authentication to the most often used IP addresses by the user.

Overall, the above approach has more pros than security risks, so I believe any mainstream site should implement it (having in mind all the security implications).

For every project I’ve participated in, there has been a need for augmenting some third-party library. A couple of reasons:

• there’s a bug
• there’s missing functionality
• there’s functionality that doesn’t work as you need it to

What do you do in these cases? Obviously, you report the problem or even suggest a fix. But until that goes into the next version, you need to get your work done. To do that, you have many options, depending on the severity of the situation and the quality of the library:

• extend the problematic class (or inject a custom implementation of some piece into it) – that’s the best way, as it is a purely object-oriented approach and doesn’t require any hacks. But it’s possible only if the problematic library is of good quality and is designed for extension
• get the sources, fix the problem, repackage the jar. That sounds like a nice approach, but it has some issues. If you use a dependency manager (and you should use one), it would need either installing the jar in a local repo, or using system scope + copy-dependencies plugin, or having a nexus installation and adding the jar there. The thing I dislike the most with this approach is that you don’t actually have the fix in your project – and you can’t quickly iterate when fixing it
• use reflection to access some hidden parts of the library
• extend a class, but copy-paste some methods that cannot be modified otherwise (because they are monolithic). I’ve done that many times, and although it’s a hacky approach, it is not that bad. Of course, try to minimize the copy-pasted code from the library.
• if you can’t use subclasses of the problematic class, then make your own versions with exactly the same name and package, and pray to the classloader to pick yours instead of the packaged ones (it usually does, but you should really test that on multiple environments)
• use AspectJ or other bytecode “magic” to modify the behaviour of the problematic classes

Problems may occur when you upgrade the library, so watch out for that. The approach picked up should be the least hacky one, even if it requires writing a few Adapters and subclassing a couple of classes – always prefer writing an Adapter or plugging in a custom implementation to reflection or bytecode manipulation.

I’ve done most of the above things, depending on the urgency of the fix and the quality of the library. And while writing this post, I played a bit with java instrumentation, and came up with a very simple (and limited) tool, called quickfix – it allows you to replace a target method with a custom method of yours – just annotate it with @ReplaceMethod and specify the target to be replaced. This would work in rather simple scenarios (and you can do the same thing with AspectJ), but it may be handy sometimes, so feel free to try it and report problems.

Bitcoin is gaining popularity, and last week I decided to implement it as a payment option for my computer-generated music service. Why I decided to do that is detailed here (payment is only needed for commercial use of the music), but here I’ll share the technical details for implementing the bitcoin integration.

First, you need to pick a payment gateway/provider. You can pick one from this list. They differ in terms of their API and specific flow, but the general flow should be almost the same. Some of the implementation details below are relevant only to what I picked up – Coinbase – but they should be similar for other providers. I picked Coinbase, because it seemed the most developer-friendly and straightforward. It has one bug that I had to workaround, but apart from that it’s ok.

The general flow (probably containing some Coinbase specifics) is below. Note that I am describing the most complex scenario, where you have multiple items with variable prices and quantities. You can use the html/js code I used in my cart page. On the other hand, it’s assuming a best-case payment provider that does not redirect the user to external pages and uses simply a javascript dialog.

1. Place a custom “Pay with Bitcoin” button on the checkout page
2. Clicking it submits a form and thus triggers a back-end checkout process
3. The back-end code saves a new purchase/order in the database and returns its id
4. Then the back-end code invokes the payment gateway API, providing the order id (or any custom application data) and the price, and gets a code to identify the transaction
5. The price can be provided in BTC or any currency. Depending on the API, you may want/need to convert a fixed price in a popular currency to BTC before passing it via the API. If you do so, be sure to cache the conversion rates for a while – no need to fetch them every time. Providers may do this conversion automatically, but if you want to display it to the user first, you may want to do the conversion yourself.
6. On returning the code to the front-end, the javascript gets it, and opens a dialog (using a js function/event provided by the provider).
7. The dialog may contain multiple payment options, but it usually shows a bitcoin payment address to which the user should send the money. He can do that with his mobile or desktop wallet
8. After the transaction is completed, a callback URL is invoked on your application, which contains the order id that you passed when generating the code. Then you can confirm the purchase and send the items purchased.
9. Then the dialog is automatically closed (or the user presses “transaction complete”). At that point you need to send an (ajax) request that clears the cart contents, and redirect to a “Thank you” page.

Note that the documentation of the payment provider should provide all the implementation details. Below I’ll share some specifics of my case: Coinbase and Java:

The default scenario described in the docs is to put a button with a predefined code on the page. That doesn’t work if you need to increase quantities or have multiple items. That’s why you should dynamically generate the button code. But the javascript library only handles this on page load. So I had to copy some minified javascript code and invoke it when the code is returned from the back-end. Here is the whole javascript code (invoked when the user presses “Pay with bitcoin”):

$(document).ready(function() {
  $('#payWithBitcoin').click(function() {
      var email = $('#email').val();
      if (${!userLoggedIn} && (!email || email.length == 0 || email.indexOf("@") == -1)) { // simple validation here; actual - on the server
          alert("Please enter a valid email");
      } else {
          $.post("${root}/cart/bitcoinCheckout", {email: email}, function(data) {
              $("#bitcoinPurchase").attr("data-code", data);
              $(".coinbase-button").each(function (b, d) {
                  var f, g, h, i, j, k;
                  return f = $(d), h = f.data(), h.referrer = document.URL, j = $.param(h), g = f.data("code"), k = f.data("width") || 195, i = f.data("height") || 46, f.data("button-style") !== "none" && f.replaceWith("<iframe src='" + c + "/buttons/" + g + "?" + j + "' id='coinbase_button_iframe_" + g + "' name='coinbase_button_iframe_" + g + "' style='width: " + k + "px; height: " + i + "px; border: none; overflow: hidden;' scrolling='no' allowtransparency='true' frameborder='0'></iframe>"), $("body").append("<iframe src='https://coinbase.com/buttons/" + g + "/widget?" + j + "' id='coinbase_modal_iframe_" + g + "' name='coinbase_modal_iframe_" + g + "' style='background-color: transparent; border: 0px none transparent; overflow: hidden; display: none; position: fixed; visibility: visible; margin: 0px; padding: 0px; left: 0px; top: 0px; width: 100%; height: 100%; z-index: 9999;' scrolling='no' allowtransparency='true' frameborder='0'></iframe>");
              });
              $("#coinbase_modal_iframe_" + data).load(function() {
                  $(document).trigger('coinbase_show_modal', data);
              });
          });
      }
  });

  $(document).on('coinbase_payment_complete', function(event, code){
      $.post("${root}/cart/clear", function() { //clear the contents of the cart
          window.location = "/?message=Checkout successful. Check your email.";
      });
  });
});

    <a id="payWithBitcoin" href="javascript:void(0);"><img src="${staticRoot}/img/bitcoin.png"/></a>
    <div class="coinbase-button" id="bitcoinPurchase" data-button-style="none"></div>
    <script src="https://coinbase.com/assets/button.js" type="text/javascript"></script>

Another thing to have in mind is that there are no official clients – you need to invoke the RESTful API manually. This is simple, of course. You can use a RestTemplate and Jackson, for example.

public String getButtonCode(BigDecimal price, long purchaseId) {
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);

        ButtonRequest buttonRequest = new ButtonRequest(); //this is a value-object specifically defined for this request
        buttonRequest.setCurrency("btc");
        buttonRequest.setCustom(String.valueOf(purchaseId));
        buttonRequest.setPrice(price.toPlainString());
        buttonRequest.setType("buy_now");
        buttonRequest.setName("Computer-generated tracks");
        ResponseEntity<String> entity = template.postForEntity("https://coinbase.com/api/v1/buttons?api_key=" + coinbaseKey, buttonRequest, String.class);
        String json = entity.getBody();

        try {
            JsonNode node = jsonMapper.readTree(json);
            return node.get("button").get("code").asText();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
}

And finally, when the callback URL is invoked, you need to get the order id and finalize the payment:

    @RequestMapping("/confirmBitcoinPurchase")
    @ResponseBody
    public void confirmBitcoin(@RequestBody String json) throws Exception {
        JsonNode root = mapper.readTree(json);
        JsonNode order = root.get("order");
        if (order.get("status").asText().equals("completed")) {
            String paymentId = order.get("id").asText();
            .....
        }
     }

Overall, it takes some time to figure out. Previous experience with payment provider integration would be a plus, but there’s one important difference – you do not submit anything user-specific to the payment provider (like credit-card details). Instead, the user makes the payment to a target bitcoin address, which is handled by the payment provider. Whenever the transaction is complete, the provider invokes your URL.

Then you can get your money from the payment gateway either via a bank transfer, or by transferring them to your own bitcoin wallet.

There’s a trivial problem that every project has – keeping the database structure in sync with the application. That is, when your code relies on a new table or column, that table/column must appear in the database. This is especially a problem when there are multiple environments, on which the code gets deployed (production, stage, integration, local), as usually additional steps need to be performed in order to update the database schema.

The quickest way to achieve the desired effect is to provide SQL upgrade scripts every time the code needs a new database element (or no longer needs one). But that’s tedious, as these sql scripts have to be executed manually on each environment the code gets deployed to. Automating this process has been achieved by a couple of frameworks, so that you don’t need to do it. Here is a list of options for the Java world.

Why do we actually need such a solution? We aim for frequent, one-click deployments to staging and production environments. We can’t do that without an automated way of syncing the database (otherwise there will be missing columns, or obsolete non-null columns will prevent data insertion). And once in a while we may need to revert a release, and we’d have to rollback the database changes accordingly. While you can just drop and recreate the schema while developing locally, you can’t do that in production.

Rails developers will rightly say “We’ve had that for a long time, out of the box”. Indeed, rails migrations seem like a perfect implementation of the idea of keeping the application code in sync with the database schema. But I had to pick a technology for a Java project. Actually, it is a grails project, and even though there appears to be a grails plugin for that, I picked another one – Liquibase (note: the grails plugin is built ontop of liquibase). I’ll explain how to get it working with JPA in a separate post, but it seems to have everything rails offers: human-readable (XML) format for the upgrade, automatic rollback script execution, and also tools that can generate the upgrade scripts based on a database diff.

In fact, I’ve never used a liquibase-like solution for my personal projects. I use hibernate hbm2ddl.auto even in production, and sometimes manually fix stuff. Yes, that’s not a workable solution for a big project and it has risks. Virtually nobody recommends using `hbm2ddl.auto` in production, because it’s not perfect and it may mean losing data or downtime. For those unfamiliar with hibernate, hbm2ddl.auto is a configuration switch that makes hibernate (and ORM framework) create tables and columns that correspond to the entity mappings. In a sense, it is a way to automatically sync the application code with the database structure. And it can do that, because we have the expected database structure in our annotated POJOs (entities), in a structured way that can define exactly what tables, columns and keys we need. But, as I said, it’s not recommended for production (it is recommended to set it to “validate”, so that it can verify whether the database and the code are in sync).

But I’m an idealist. I want something even better than a good migrations tool like liquibase. We have the prerequisites – a structured way for our code to define what’s the schema it requires. That’s the JPA annotations (with some vendor extensions). We just need a tool that works as if it had a manually created sql upgrade script. It needs to:

• create tables, columns, indexes, foreign keys
• drop tables, columns, indexes, foreign keys
• alter tables, columns, indexes, foreign keys
• fix the data before/after the structural change
• make sure no data is lost

hbm2ddl.auto does only the first, and can partly do the fourth. It will be relatively easy to implement the “drop” and “alter” functionality. How do we make sure no data is lost? Whenever a column or table is dropped, a new temporary table (e.g. prefixed with _hbm_temp) is created that stores the data from that column/table. If it happens that the data should not have been deleted, you can get it back. Otherwise, you can cleanup all the temp tables once in a while. I can’t see any obvious problem with that approach, and it will save a lot of time, effort and mistakes.

While on the topic, I can’t skip the main reason for this complexity – the fact that relational databases have a rigid schema. NoSQL database, on the other hand, are mostly schema-less. It eliminates the exceptions arising from missing columns or tables – they automatically “appear” in the database. However, unused fields stay there and pollute the database, which you should clean up manually. And lacking any database-imposed constraints means that they should be handled in the application code. So, there’s a tradeoff (as usual).

Overall, avoid having to execute sql scripts manually in order to fix the database. Always aim for the application to be the main point, and make everything else (the schema) follow. This can’t happen always (e.g. if you need database triggers and stored procedures, or you have multiple applications using the same database, that can’t share their codebase), but at least put these scripts in the same repository as the code and make it possible to have quick deployments to any environment, with as little manual tasks involved as possible.

We’ve all done pagination – it’s sensible for both lists in the UI and for processing data in batches. “SELECT (columns) FROM table LIMIT X, Y”. Possibly “SELECT (columns) FROM table ORDER BY some_column LIMIT X, Y”.

What’s the caveat? Especially when working with an ORM and when processing batches, where order doesn’t matter, you are likely to omit the ORDER BY clause. You would expect the results to be ordered by the primary key. And they are in most cases (though it is not guaranteed), but when you EXPLAIN the query, you can see that the index is not used for the query – a full table scan is performed to fetch the results, which is slow. Note that if we omit the LIMIT clause, again a full table scan is used, but this is something you’d rarely do – i.e. a query with no WHERE clause and no pagination.

So, rule #1: always include an ORDER BY clause in order to use the index.

For lists in the UI you’d usually order by some column – date modified, name, etc. That also performs a full table scan, unless you have an index on that column.

So, rule #2: always have an index on the column you ORDER BY

Kind of obvious things, but they may be silent performance issues lurking in your project. And now onto something not so obvious, that surprised me. When using LIMIT X, Y, MySQL scans the whole table (up until X+Y), regardless of the index. Note that even if the “type” (in the result of EXPLAIN) is not ALL (full table scan), but “index”, MySQL may still scan most of the rows – check the “rows” column. But it can also be misleading (in case of LIMIT it shows that rows = everything, but stops when it fills the result with the desired number of rows). So if you have LIMIT 100000, 50 MySQL will scan 100050 rows. This makes the query slow (the query time reflects that – the larger the offset, the longer the query). This is due to MySQL not maintaining an index on the row number. It cannot use the primary key, because even if it is auto_increment, it has gaps. Luckily, in huge tables you normally have a WHERE clause which forces the use of an index and reduces the number of rows to scan. Also, in a UI you will rarely find yourself querying for the 100000th record. But anyway it is something to consider, especially in the case of batch processing jobs which need to go through the whole table. You do that in batches, because 1. ORMs may store everything in memory and just kill your application 2. the transaction will become really long.

So, rule #3: when performing batch operations on tables with a lot of rows, do not use LIMIT. Use WHERE id > X and id < Y (or BETWEEN)

This does not guarantee that each batch will be the same size, but that doesn’t matter.

The rules above are “rule of thumbs”, rather than “must dos”, but it’s good to have them in mind. The bottom-line here is that you should analyze your performance, find specifically slow queries, then EXPLAIN them and see how they can be optimized and what would be the impact of the optimization.

P.S. Here’s a presentation on the low level details of MySQL engines regarding indexes.

If you are using spring and JPA, it is very likely that you utilize ehcache (or another cache provider). And you do that in two separate scenarios: JPA 2nd level cache and spring method caching.

When you configure your application, you normally set the 2nd level cache provider of your JPA provider (hibernate, in my case) and you also configure spring with the “cache” namespace. Everything looks OK and you continue with the project. But there’s a caveat. If you follow the most straightforward way, you get two separate cache managers which load the same cache configuration file. This is not bad per-se, but it is something to think about – do you really need two cache manager and the problems that may arise from this?

Probably you don’t. So you have to get rid of the redundant manager. To do that, you need to set your spring cache manager as shared:

<bean id="ehCacheManager"
    class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
    <property name="shared" value="true" />
</bean>

This means that spring won’t create a new instance of cache manager, but will reuse the one already created by hibernate. Now, there’s something to think about here – it would depend on the order of bean creation – whether the JPA factory bean or the cache manager factory bean will be first. Luckily, this doesn’t matter for the end result, because SingletonEhCacheRegionFactory reuses an existing cache manager instance if it finds one.

So, now you have made your cache manager jvm-singleton. But then there’s another problem that you may encounter if you have multiple applications deployed and you are using JMX. The Cache manager registers itself as a JMX bean. But when you have singletons, multiple applications will try to register the same cache manager multiple times, and that will fail. The result will be a couple of exceptions in the log and the inability to control the cache manager of multiple modules. A side effect of the same problem gets in the way if you use something like Terracotta (there cache manager identity matters). Luckily, you have an easy fix for that. Just add one property to the bean definition shown above:

    <property name="cacheManagerName" value="${module.name}" />

${module.name} is a property resolved with a PropertyPlaceholderConfigurer and is configurable per webapp, so each webapp can have a different module name. That way the cache manager will be accessible under the specified name via JMX.

Overall, be careful with your cache managers. Even in case you are using different cache, jpa and DI provider, you should verify the scenarios described above.

Every now and then a “hardcore” programmer comes and mocks you for using a “sissy” tool like Eclipse or IntelliJ. Real programmers use emacs and vim. If you are lucky and the other developer has some arguments and social skills, you are not mocked, but rather explained how you are wasting a lot of precious time with your IDE and once you get used to emacs or vim, you’ll realize how irreplaceable they are – it will be like going into the 3rd dimension and wondering how you’ve ever lived in 2D.

Zoom out. Eclipse and IntelliJ are pretty much Java-oriented (though there are plugins for other languages). So if you are using ruby, python, php, or whatever, then probably emacs is the best thing you can get. But let’s see what you absolutely need your editor/IDE to do for Java:

• debug – number one priority. If you can’t put a breakpoint and run your whole application in order to simulate a specific case, then you are left with poor man’s System.out.println(..)
• quick navigation through the code – if you can’t go to a class with 5 keystrokes (no mouse), if you can’t go back to where you previously edited, if you can’t see which classes subclass a given class or, most importantly – see which methods call a given method, then your code analyzing efficiency drops remarkably. And in a typical software project you read and analyze existing code way more often than you write new code
• refactoring – you want to change a name of a method. Add or remove arguments. Move a class to another package. The IDE makes the appropriate changes everywhere in the entire project and you don’t have to go and rename stuff manually (missing some of the occurrences)
• auto-completion – that’s important, but not because it saves you 4 keystrokes. It shows you the possibilities you have. When you work with an unfamiliar (or forgotten) API, you are not sure which method does exactly what you want. So you experiment by typing a name you suppose you need and read the attached documentation to that method
• coloring and highlighting- syntax coloring is a basic thing, and is not that important. But highlighting – when you select a variable you must be able to see where it is used.
• tools support – your project is build and managed with maven. You need your editor to be able to pull dependencies from the repository and attach them to the classpath. You need to be able to save parameterized build configurations as well. You need Tomcat/web server hot-swap/hot-deploy – changed code to be directly replaced at runtime. Checkstyle. Unit-test integration.
• advanced search and replace – regex search, method search, wildcard search, etc. Priceless in many cases, in which you should otherwise create a separate program that handles to complex find/replacement logic

All of the above are of huge importance to productivity. I wouldn’t drop any of them (especially the “call hierarchy” feature, which shows which methods call a given method). How many of them are supported by Emacs? In its basic form – zero, but that’s not a problem – there’s JDEE, which provides some Java support. Does it provide all of the above? No.

At this point I can safely wrap-up the publication, because I’ve given the answer – I’m not going to use emacs because it doesn’t give me what an IDE does. But I know the arguments that people will bring up, so I’ll add some more paragraphs.

In emacs you can do everything. You can write a Lisp script that sprays pixie dust on the code and makes all bugs disappear. And you can implement anything that’s missing in JDEE. You can. You can also probably write your own servlet container and web framework. But that’s what you do at home, when bored and have nothing else to do. If you are working on a project, you don’t do these things. They take time, and this is time nobody is going to grant you in order to play with your favourite editor. And even if you do, they will be buggy and feature-poor at first.

In emacs you can type code really fast. I don’t know, probably faster than in Eclipse. You can define all sorts of shortcuts, macros and templates and write code blazing fast. And that’s great, if you are a typist. Programming, on the other hand, requires way more thinking and analysis than simply writing code. I don’t know the percentage of time of a developer is needed for the actual typing, but it’s less than 20% for sure. And by the way, you can define templates and shortcuts in eclipse, too.

So, you want me to use an editor that makes me use more time and effort to perform day-to-day development tasks, and to do that I have to spend a lot of time learning and customizing my editor (and then probably store the configuration somewhere safe, so that I can reuse it across computers/jobs)? And I know why you like that. Because it’s tough, challenging, and makes you stand out of the “masses” that use these lame IDEs. It’s not something most people would admit, it’s a psychological effect. And I have no problem with anyone using emacs or vim. Just don’t feel elite for that.

So, why not emacs for me? Because Eclipse is better for Java.

In a previous post of mine I asked the question: do programmers get bored? And yes, sometimes they do, especially if there are no challenges. And the usual software project out there is trivial – implementing business case after business case. But programmers still find challenges even in the trivial applications and websites they are writing:

• Complex user interface – implementing a complex UI on the web is a good challenge – often good components come out of that, that are later open-sourced and have the potential of getting popular. And if you are really good, you may solve a problem that many people have and make your component a standard solution.
• Architecture – if a system is big, the architecture matters. Separating components and deciding on the way they communicate is a challenge that requires both advanced skills in concrete technologies and the ability to get a good overview on a complex system.
• Big data – if you are lucky enough to work at a company that handles billions of records, you would be required to solve problems related to that previously unusual amounts of data. That requires deep understanding in databases and strategies for distributing and synchronizing data. As a result tools or even new databases can emerge. Look at Cassandra, for example, which was initially developed inside Facebook.
• Algorithms – it is rare that a business application requires complex algorithms, but in case there is, this is a great point for putting your efforts into and get satisfaction from your work. For example, you can optimize the delivery routes of a transport company, which in turn saves thousands of dollars for fuel. If you can do that, not only the clients will be happy (which is rarely an incentive for developers), but you will be happy as well for having cracked down a complex problem.
• Low-level implementations. Normally you work with tools and frameworks that are already developed and stable. However, sometimes you need to implement some low-level detail yourself. My most recent experience is the connection pooling and high-availability for a RabbitMQ client. It is a gratifying experience when you finally see that thing working properly in production, under heavy load, and allowing for any node to go down without affecting the system health. Other examples may include distributing scheduled jobs, augmenting a distributed cache solution with additional eviction policies, etc.
• Open-ended problems. There are problems that don’t have a solution yet. There are problems that are not known to be problems yet. Web search was such an unsolved (or poorly-solved) problem back in the nineties, when Google solved it. You rarely find these problems in a regular company, so you usually have to take them up yourself at home. My most recent example is my algorithmic music generation service

The list is not exhaustive, of course (feel free to add to the list in the comments), but you get the idea – even in regular companies there can be tasks that go out of the standard business-requirement box which makes you bored. If you are good at what you do, you have a chance of picking these tasks yourself. Note a common pitfall here – if something sounds cool and challenging it isn’t automatically something to go for. Writing an ORM is challenging, but you should instead reuse an existing one. Writing a web framework is also cool, but it’s most likely to take a lot of time without providing the benefits and stability of existing frameworks. Evaluating whether an existing solution works is also a challenging task, by the way.

Certainly, there are things you won’t be able to work on in your company. You have two options. Either change your job or do interesting things at home (and then, possibly, reuse them at work). The computer-generated music is nothing related to my current job, but I did it. The RabbitMQ pooling on the other hand was a task in my current job.

Overall, if you are really interested in doing interesting stuff, rather than just “completing business goals and getting the paycheck”, then you should put some thought into what is interesting for you and how can you get to working on it.

Having only twitter and facebook authentication so far, I decided to add Mozilla Persona to the list for my latest project (computoser, computer-generated music). Why?

• I like trying new things
• Storing passwords is a tough process, and even though I know how to do it, and even have most of the code written in another project, I don’t think that I should contribute to the landscape of every site requiring password authentication
• Mozilla is an open foundation that has so far generated a lot of great products. Persona implements a BrowserID protocol that may be supported natively in browsers other than Firefox in the future (for now, you need to include a .js file)
• 3rd party authentication has been attempted many times, and is a great thing, but isn’t mainstream for a couple of reasons. Being a bit different, Persona might succeed in becoming more popular.
• This explanation by Mozilla makes sense

So, I started with the “Quick setup” guide. It is looks really easy. Way easier than OpenID or OAuth authentication – you don’t have to register anything anywhere, you don’t need 3rd party libraries for handling the verification on the server, and you don’t need to learn a complex authentication flow, because the flow is simple:

1. user clicks on the signin button
2. a pop-up appears
3. if not authenticated with Persona, the user is prompted for registration
4. if authenticated with Persona, the user is prompted for approval of the authentication to your site
5. the popup closes and the page redirects/refreshes – the user is now signed in

Of course, it is not that simple, but there are just a few things to mind that are not mentioned in the tutorial. So let’s follow the official tutorial step by step, and I’ll extend on each point (the server-side language used is Java, but it’s simple and you can do it in any language)

1. Including the .js file – simple. It is advisable to get the js file from the Mozilla server, rather than store it locally, because it is likely to change (in order to fix bugs, for example). It might be trickier if you merge your js files into one (for the sake of faster page loading), but probably your mechanism allows for loading remote js files.

2. The signin and signout buttons. This looks easy as well. Probably it’s a good idea to add the logout handler conditionally – only if the user has logged in with Persona (as opposed to other authentication methods that your site supports)

3. Listening to authentication events. Listening to events is suggested to be put on all pages (e.g. included in a header template). But there’s a problem here. If your user is already authenticated in Persona, but his session has expired on your site, the script will automatically login the user. And this would require a page reload in a couple of seconds after the page loads. And that’s not necessarily what you or the users want – in my case, for example, this may mean that the track they have just played is interrupted because of page refresh. It can be done with AJAX of course, but it is certainly confusing when something in the UI changes for no apparent reason. Below I’ll show a fix for that. Also, the logout listener might not be needed everywhere – as far as I understand it will automatically logout the user in case you have logged out of Persona. This might not be what you want – for example users might want to keep open tabs with some documents that are not accessible when logged out.

4. Verifying the assertion on the server. Here you might need a 3rd party library in order to invoke the verification endpoint and the parse the json result, but these are pretty standard libraries that you probably already have included.

Now, how to solve the problem with automatic authentication? Declare a new variable – userRequestedAuthentication – that holds whether the authentication has been initiated explicitly by the user, or it has been automatic. In the signin button click handler set that variable to true. Here’s how the js code looks like (btw, I think it’s ok to put the code in document.ready(), rather than directly within the script tag. Assuming you later need some DOM resources in the handler methods, it would be good to have the page fully loaded. On the other hand, this may slow down the process a bit). Note that you can include an empty onlogin handler on all pages, and have the complete one only on the authentication page. But given that the login buttons are either on the homepage, or shown with a javascript modal window, it’s probably ok having it everywhere/on multiple pages.

<script type="text/javascript">
    var loggedInUser = ${context.user != null ? '"' + context.user.email + '"' : 'null'};
    var userRequestedAuthentication = false;
    navigator.id.watch({
        loggedInUser : loggedInUser,
        onlogin : function(assertion) {
            $.ajax({
                type : 'POST',
                url : '${root}/persona/auth',
                data : {assertion : assertion, userRequestedAuthentication : userRequestedAuthentication},
                success : function(data) {
                    if (data != '') {
                        window.location.href = '${root}' + data;
                    }
                },
                error : function(xhr, status, err) {
                    alert("Authentication failure: " + err);
                }
            });
        },
        onlogout : function() {
            window.locaiton.open("${root}/logout");
        }
    });
</script>

As you can see, the parameter is passed to the server-side code. What happens there?

@RequestMapping("/persona/auth")
@ResponseBody
public String authenticateWithPersona(@RequestParam String assertion,
        @RequestParam boolean userRequestedAuthentication, HttpServletRequest request, Model model)
        throws IOException {
    if (context.getUser() != null) {
        return "";
    }
    MultiValueMap<String, String> params = new LinkedMultiValueMap<>();
    params.add("assertion", assertion);
    params.add("audience", request.getScheme() + "://" + request.getServerName() + ":" + (request.getServerPort() == 80 ? "" : request.getServerPort()));
    PersonaVerificationResponse response = restTemplate.postForObject("https://verifier.login.persona.org/verify", params, PersonaVerificationResponse.class);
    if (response.getStatus().equals("okay")) {
        User user = userService.getUserByEmail(response.getEmail());
        if (user == null && userRequestedAuthentication) {
            return "/signup?email=" + response.getEmail();
        } else if (user != null){
            if (userRequestedAuthentication || user.isLoginAutomatically()) {
                context.setUser(user);
                return "/";
            } else {
                return "";
            }
        } else {
            return ""; //in case this is not a user-requested operation, do nothing
        }
    } else {
        logger.warn("Persona authentication failed due to reason: " + response.getReason());
        throw new IllegalStateException("Authentication failed");
    }
}

The logic looks more convoluted than you’d like it to be, but let me explain:

• As you can see in the javascript code, an empty string means “do nothing”. If anything else is returned, the javascript opens that page. If not using spring-mvc, instead of returning a @ResponseBody String from the method, you would write that to the response output stream (or in php terms – echo it)
• First you check if there’s an already authenticated user in the system. If there is, do nothing. I’m not sure if there’s a scenario when Persona invokes “onlogin” on an already authenticated user, but if you are using other authentication options, Persona won’t know that your user has logged in with, say, twitter.
• Then you invoke the verification url and parse the result to JSON. I’ve used RestTemplate, but anything can be used – HttpClient, URLConnection. For the JSON parsing spring uses Jackson behind the scene. You just need to write a value-object that holds all the properties that Persona might return. So far I’ve only included: status, email and reason (jackson detail: ignoreUnknown=true, spring-mvc detail: you need FormHttpMessageConverter set to the RestTemplate). It is important that the “audience” parameter is exactly the domain the user is currently on. It makes a difference if it’s with www or not, so reconstruct that rather than hardcoding it or loading it from properties. Even if you redirect from www to no-www (or vice-versa), you should still dynamically obtain the url for the sake of testing – your test environments don’t have the same url as the production one.
• If Persona authentication is “okay”, then you try to locate a user with that email in your database.
• If there is no such user, and the authentication action has been triggered manually, then send the user to a signup page and supply the email as parameter (you can also set it in the http session, so that the user can’t modify it). The registration page then asks for other details – name, username, date of birth, or whatever you see fit (but keep that to minimum – ideally just the full name). If you only need the email address and nothing else, you can skip the registration page and force-register the user. After the registration is done, you login the user. Note that in case you have stored the email in session (i.e. the user cannot modify it from the registration page), you can skip the confirmation email – the email is already confirmed by Persona
• If there is a user with that email in your database, check if the action has been requested by the user or whether he has indicated (via a checkbox in the registration page) that he wants to be automatically logged in. This is a thing to consider – should the user be asked about that, or it should always be set to either true or false? I’ve added the checkbox. If login should occur, then set the user in the session and redirect to home (or the previous page, or whatever page is your ‘user home’)(“context” is a session-scoped bean. You can replace it with session.setAttribute("user", user)). If the authentication attempt was automatic, but the user doesn’t want that, do nothing. And the final “else” is for the case when the user doesn’t have an account on your site and an automatic authentication has been triggered – do nothing in that case, otherwise you’ll end up with endless redirects to the registration page
• in case of failed authentication be sure to log the reason – then you can check if everything works properly by looking at the logs

A cool side-effect of using the email as unique identifier (make the database column unique) is that if you add Persona later to your site, users can login with it even though they have registered in a different way – e.g. facebook or regular registration. So they can set their password to something long and impossible to remember and continue logging in only with Persona.

The details I omitted from the implementation are trivial: the signup page simply gathers fields and submits them to a /completeRegistration handler that stores the new user in the database. The /logout url simply clears the session (and clears cookies if you have stored any). By the way, if automatic signin is enabled, and Persona is your only authentication method, you may not need to store cookies for the sake of keeping the user logged in after the session expires.

Overall, the implementation is still simple, even with the points I made. Persona looks great and I’d like to see it on more sites soon.