A Problem With Convention-Over-Configuration

November 15, 2015

Convention-over-configuration is a convenient thing. Instead of writing tons of configuration in xml/yaml/json/whatever, you simply know that something will have a given default value. For example, the restful endpoint URL may have a value of /class-name/method-name, or a join table will be named mainEntity_joinField. A “view” can be populated by default with the input parameters of the controller method or other values.

And this is all very nice – we don’t want to annotated each field of a java bean, and we don’t want to configure explicitly our maven directory structure. But there is a “darker” side of convention-over-configuration, and it comes whenever the knowledge of the default behaviour is needed in order to understand the code.

Imagine you open a project for the first time. I have had the need to do that recently – as a government adviser I sometimes have to do a quick-fix or a quick investigation of some software that would otherwise require a tender, a contract and at least 4 months to handle. I agree it’s a rare usecase, but bear with me.

If, for example, a web framework automatically includes /views/header.ext into your views, and you try to find “where the hell is this menu item coming from”, you may have a hard time. If you try to figure out what controller handles the /foo/bar/baz URL, and you don’t find any configuration or mapping file, nor you find any part of the URL using the search functionality, you’re lost.

Convention-over-configuration can roughly be split in three groups: straightforward removal of boilerplate code, specific configuration logic and stuff that doesn’t matter for investigating the project. But there is no obvious line between them. The fields of the java-bean can obviously be referred to in a JSP by name, or spring beans are automatically named using the uncapitalized . It doesn’t matter whether Maven has the java classes in src/main/java, or in src/java – you’ll click through the folders anyway. But if there is specific logic of mapping URLs to controller methods, then you’d have to read about the framework being used. In rare cases like mine you may even not know the framework being used, so you have to find that out first.

That’s a problem, in a sense – in order to understand the program flow, you need to know the framework details.

I know this is rarely a big problem – normally you join a team which already has knowledge of the framework and if you find yourself wondering “how is that configured”, you can always ask someone. But as a general advice – try not to use convention-over-configuration with complicated, specific logic. It may save a few keystrokes, but typing is not what takes time in software development. And any complicated convention-over-configuration logic makes the project harder to read and navigate.


E-voting [presentation]

November 13, 2015

Last week I gave a talk on the OpenFest conference about (remote) e-voting (or internet voting/i-voting). The talk was not in English, but here are my translated slides:

I’ve addressed some of the topics from the presentation in a previous post.

Overall, it’s a hard problem, with a lot of issues to be addressed and it must be treated seriously. However, we must do it, sooner or later, as it will allow for a more dynamic and direct democracy.


Setting Up CloudFront With S3

November 1, 2015

Yesterday I decided to setup CloudFront for Computoser. I store all the generated tracks in AWS S3, and every time a track is played or downloaded, I was making a request to S3. Not that the traffic is that much, but still it sounded like a good idea to use a CDN (CloudFront) – it would save a little bit of money (not that the current bill is any big) and it would make download faster accross the globe. To be honest, I didn’t do it for any of these – I was just curious how would a CloutFront setup work.

There is enough documentation on “How to setup CloudFront with S3″, and besides, the UI in the AWS console is pretty straightforward – you create a “distribution”, in “Origin” you specify your S3 bucket, and that’s it. Of course, you can use your own server as origin server, if you don’t store the content in S3.

Then you wait for around 10-15 minutes, and things should work – i.e. when you access http://randomkey.cloudfront.net/foo.png, it should be opened. But for me it wasn’t – the response was “Access denied”. Which meant the bucket policy had to be changed (as described here):

  "Version": "2008-10-17",
  "Statement": [{
    "Sid": "AllowPublicRead",
    "Effect": "Allow",
    "Principal": { "AWS": "*" },
    "Action": ["s3:GetObject"],
    "Resource": ["arn:aws:s3:::bucket/*" ]

Then the application had to be configured to use CloudFront. This can be done in two ways:

  • In the view – in your pages you can set the root of the CloudFront, and make all references to CDN-available resources absolute
  • In a controller (or anything back-end) – if it is not about static resources, but (as in my case), files that are generated and stored by the software, then configure the path, and instead of fetching from S3, redirect to the CloudFrount URL.

Both approaches can be useful and easy to roll-out. For the former to work out-of-the-box, you’d need some pre-existing handling mechanism for static resources (e.g. a ${staticRoot} prefix, or a custom tag. It is generally a good idea to have a good static resources setup, regardless of whether you use a CDN or not.

But for bigger systems, a CDN is useful and apparently – easy to setup.


In Defence of Monoliths

October 22, 2015

The first Microservices talk I attended was a year and a half ago. My first reaction was “why is that something new?”. Then I realized it is already getting overhyped, so I listened to some more talks, read a bit more articles, so that I can have a good reason not to like the hype.

What are microservices is probably defined here or by Martin Fowler, or at any of the first google results for “microservices”. It is basically splitting up your functionality into separately deployable modules that communicate with each other in order to complete a business goal, and each microservice is limited to just a small, well-defined scope. Product purchasing is one microservice, user registration is another microservice, “current promotions” is another microservice, and so on. Or they can be even more fine-grained – that appears to be debatable.

And whenever I encounter a new “thing”, I try to answer the questions “what’s the point” and “does this apply to me”. And I’d like to point out that I’m not the person that rejects anything new, because things can be done “the old way”, but there’s a fine line between a good new technology/architecture, and hype. And besides, microservices is nothing new. I remember several years back when we had an application split into several parts that communicated with web services. When I joined, I refactored that into a single “monolith”, which improved response times by around 20%. It turned out we never needed the split.

And that’s why I’m going to write about – that you probably don’t need microservices. And Martin Fowler has phrased this very well:

…don’t even consider microservices unless you have a system that’s too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application. Do pay attention to good modularity within that monolith, but don’t try to separate it into separate services

There, done. Now go build a monolith. But microservices advocates wouldn’t agree and will point out all sort of benefits of a microservices architecture (or will point out that your system is too complex, so you have to use microservices. And pay them for consultancy). So let’s examine a few alleged advantages that microservices have (for example around 30:57 of this video). The question I’m going to ask is – can this easily be done in a monolith? (I have to clarify, that a “monolith” is what is generally perceived as the opposite of microservices – e.g. one codebase, one deployment.)

  • modeled around the business domain – absolutely. You can structure your packages and runtime dependencies around the business domain.
  • culture of automation – that has nothing to do with the architecture – you can automate the deployment of any application. (We are doing an automated blue-green deployment for a monolith, for example).
  • hide implementation details – that’s what object-oriented programming is about. Your classes, and your packages, hide their implementation details and expose interfaces. Microservices bring nothing to that (except the network overhead). You can even still have multiple projects, built as dependencies for the main project.
  • decentralize all things – well, the services are still logically coupled, no matter how you split them. One depends on the other. In that sense, “dcentralized” is just a thing that sounds good, but in practice means nothing in this particular context. And is maybe synonymous with the next point.
  • deployed independently, and monitored independently. That alone doesn’t give you anything over a monolith, where you can gather metrics (e.g. with statsd) or get profiling and performance information about each class or package.
  • isolated failures – now that’s potentially a good thing. If one module “fails”, you can just display “sorry, this functionality doesn’t work right now” and handle the failure. A monolith, however, doesn’t have to fail completely either. It is the details of the failure that matter, and I’ve never seen any detailed explanation. A server goes down? Well, you have a highly-available cluster for that, regardless of how your code is structured.

Some more, loosely defined benefits like “easy to modify” and “easy to understand” are claimed. But again, a well written, structured and tested monolith can be as easy to understand and modify.

Basically, a lot of commons sense, software engineering, continuous integration/delivery and infrastructure management best practices are claimed to be a bonus of microservices, while in fact they work perfectly fine with a monolith.

The ability for a graceful degradation is possibly an important aspect of microservices, but again, you can handle it with a monolith as well – it would require a little bit of extra code – e.g. feature if’s that are toggled in case of failures. But it’s nothing compared to the extra effort you have to put in place in order to get a working microservices application.

And that’s a lot – you have to coordinate your services. You have to decide what to do with common data. And the usual suggestion is “duplicate it”. If two microservices need some common data, they can’t just use the same shared database – each microservices has its own database, so it has to duplicate the data. Which sounds easy, unless you have to keep that data in sync. Which you always have to do. And when you have to keep duplicated data in sync accross 5 microservices, the overhead possibly exceeds any possible advantages.

Another big problem are transactions. You either don’t need transactions (in which case – lucky you), or you end up with (if I may quote Martin Kleppmann) “ad-hoc, informally-specified, bug-ridden slow implementation of half of transactions” (here’s the whole talk about transactions).

The microservices communication overhead is also not to be underestimated, including the network overhead and all the serialization and deserialization. And my controversial advice to use a fast, binary format for internal communication, rather than JSON/XML, is rarely seen in practice.

So, I would again recommend to follow Martin Fowler’s advice and just stay with a monolith. Do follow best practices, of course, including:

  • Modularity – separate your logic into well defined modules (will be easier with project jigsaw), define your class public interfaces carefully, and use loose coupling within your codebase
  • Continous delivery and automation – automate everything, deploy often
  • Be highly available – make your application horizontally scalable, be as stateless as possible, and make failures undetectable by end users

But don’t believe when someone tells you these best practices are features of the microservices architecture. They aren’t, and can be done pretty easily in a monolith, without the side effects – you don’t have to think about keeping duplicated data in sync, about network overhead, about writing a half-assed two-phase commit mechanism, about coordinating the services, etc. And if you really think you have to do “microservices”, be sure to have pretty good reasons for it.


Retryable operations

October 12, 2015

In every project that I’ve worked on, there’s always a need of a certain piece of functionality: retrying an operation. Normally it’s about calls over the network that can fail once, but then succeed. It can be about a lot of other stuff, mostly including communication with another system (be it over the network or not). It is functionality that you absolutely need in most applications, especially if you want them to be highly available (as pointed out here, for example).

And every time I have to introduce this functionality in a project, I check the standard libraries we have imported, and there’s no such thing. So I always end up copy-pasting the same piece of code from previous projects of mine. I don’t even remember when was the first time I introduced it, but it’s “traveling” with me ever since. So here it is:

 * Class that provides retrying functionality. Example:
 * <p></p>
 * <code>
 * Callable&lt;String&gt; callable = new Callable&lt;String&gt;() {..};
 * String result = RetryableOperation.create(callable).retry(5, IOException.class);
 * </code>
 * @param <T> the return type of the operation
public class RetryableOperation<T> {
    private Callable<T> callable;
    private Runnable runnable;
    private boolean exponentialBackoff;
    private int backoffInterval = 500;

     * Create a retryable operation based on a Callable instance. The return
     * type of retry(..) is the type parameter of the Callable instance.
     * @param callable
     * @return
     *      a new instance of RetryableOperation
    public static <T> RetryableOperation<T> create(Callable<T> callable) {
        return new RetryableOperation<T>().withCallable(callable);

     * Creates a retryable operation based on a Runnable instance. In this case
     * the retry(..) method always returns null.
     * @param runnable
     * @return
     *      a new instance of RetryableOperation
    public static RetryableOperation<?> create(Runnable runnable) {
        return new RetryableOperation<Object>().withRunnable(runnable);

     * Retries the operation. Retrying happens regardless of the exception thrown.
     * @param retries
     *      number of retries before the exception is thrown to the caller
     * @param exceptions
     *      the operation will be retried only if the exception that occurs is one of the
     *      exceptions passed in this array
     * @return
     *      the result of the operation (null if Runnable is used instead of Callable)
     * @throws Exception
     *      the exception that occurred on the last attempt
    public T retry(int retries, Class<? extends Exception>... exceptions) throws Exception {
        if (callable == null && runnable == null) {
            throw new IllegalStateException("Either runnable or callable must be set");
        Set<Class<? extends Exception>> retryFor = new HashSet<Class<? extends Exception>>();
        for (int i = 0; i < retries; i++) {
            try {
                if (exponentialBackoff && i > 0) {
                    int sleepTime = (int) ((Math.pow(2, i) - 1) / 2) * backoffInterval;
                if (callable != null) {
                    return callable.call();
                } else if (runnable != null) {
                    return null;
            } catch (Exception e) {
                if (retryFor.isEmpty() || retryFor.contains(e.getClass())) {
                    if (i == retries - 1) {
                        throw e;
                } else {
                    // if the exception is not the expected one, throw it immediately
                    throw e;
        // can't be reached - in case of failure on the last iteration the exception is rethrown
        return null;

    private RetryableOperation<T> withCallable(Callable<T> callable) {
        this.callable = callable;
        return this;

    private RetryableOperation<T> withRunnable(Runnable runnable) {
        this.runnable = runnable;
        return this;

    public RetryableOperation<T> withExponentialBackoff() {
        this.exponentialBackoff = true;
        return this;

It is very simple, and yet works pretty well. You can either retry every failure, or you can retry a specific exception (you don't want to retry NullPointerException, but you have to retry network failures, having configured the proper timeouts, of course):

   Result result = op.retry(3);
   Result result = op.retry(3, IOException.class);

I had even proposed this piece to guava for inclusion, and then saw some other similar proposals, but to my knowledge there is no such functionality - neither in guava, nor in apache commons yet. And I'm not creating a new github project, because that would require managing an entry in maven central, and it's too much of an effort for just a single utility class.

Of course, there are other attempts to solve this, which have a bit bigger APIs and footprints - the retry guava extensions and the recently extracted as a separate project spring-retry. They are worth checking, and they have maven dependencies ready to import.

Whatever option you choose, check if it supports anonymous functions (since Java 8). It probably does automatically, but check anyway.

The point is to have this functionality available and with a very easy API, so that you can spare the users avoidable failures - retrying calls to external systems a few times is a must.

P.S. Depending on the complexity and the requirements, this can be an underlying component, or used in conjunction with a "Circuit breaker" (an implementation here)

P.P.S The reddit discussion brought some other nice utilities


Why All The Fear of Electronic Voting?

October 2, 2015

Due to an upcoming referendum in Bulgaria about whether we want “remote electronic voting”, I, as a technical person, and at the same time, as government adviser, argue a lot about electronic voting. A year and a half ago I’ve given a very brief overview of what I think has to be done and is doable.

But now I’d like to ask a more general question – why all this fear of electronic voting? I have heard literally hundreds of versions of the same bunch of arguments: anonymity is not guaranteed, someone can change everything with one command, I can’t be sure what happens to my vote, it’s a black box, someone may easily compromise everything with a virus, you’ll be DDoSed, etc.

This for example is one giant strawman video. Every single bit it in, spoken very quickly and very assertively, is wrong. Votes can be anonymous, you can verify your vote, there are ways to prevent massive result changes, there are ways to protect even the clients (hardware keypads, virtualized voting environments), and good operational security gives you, ontop of the essential security, a way to detect every attempt to attack the system, and there are ways to prevent DDoS.

And horribly few people have actually read anything on the topic. I recommend, for example, reading this 136 pages report. And these papers: paper, paper, paper, paper, paper, paper.

But you won’t, will you? Because you are sure that it cannot ever be secure, the ballot secrecy cannot be guaranteed, and you can’t be sure whether your vote is counted. Even though the literature hints otherwise. Let me just outline a few key things:

  • One man – one vote. This relies on a national e-id infrastructure, that some countries (like Estonia, Belgium, and soon Bulgaria) have. The id card has a smartcard built-in. That guarantees the one man – one vote principle. A person can only have a single id card with a single keypair to use for voting.
  • Ballot secrecy. David Chaum has proposed the so-called “blind signature” that, using cryptography, allows the voting officials “stamp” your vote without seeing it and count you as “already voted”, and then on a second step you send your stamped vote, without the identifying information. There’s also the double-envelope approach used in postal voting, that is applied to electronic voting (but it relies on good operational security). And then there are the anonymous credentials schemes which I haven’t looked into in details.
  • Mass replacement of votes. Sending an SQL query to the vote count server is probably what you imagine can happen. OpSec, of course, is tasked to prevent it, but that is not enough. A very determined attacker can break into almost any system. But recent research is being done on using the bitcoin blockchain data structure. A de-facto distributed, unchangeable database. The votes on the central server can then be compared to the public ledger, in order to verify that there are no discrepancies. Which is, by the way, what happens with paper voting, as you’ll see below.
  • Black box. Of course, proprietary, closed-source solutions are a no-go. But a fully open-source, peer-reviewed, pilot-tested, in-person tested (as recommended in the report) system is not a black box. It is a fully and constantly auditable system.
  • I don’t know whether my vote is counted at all. That’s a major concern, addressed by a lot of E2E research (end-to-end verifiable voting). All sorts of approaches exist. For example a receipt, that you can later verify against a central system. The receipt doesn’t have to contain the actual vote (because someone may have paid to and then wants to check), but a number that you get while voting should match with a number that you see later on a website. The receipt can be issued via a smartphone app, sms, the screen, or any combination of those for a higher level of assurance.
  • Client side malware. Now that’s hard. But there are ways to address it. Hardware keypads for entering the PIN for the smartcard, with a small screen to show the actual information that is required to be signed/encrypted. Then come the multiple-factor authentication and validation. You can use a mobile phone, where receipts (as mentioned above) are sent. If a malware replaces your vote (if you are allowed to cast replacement votes, methods for which are also described in papers), you’ll get notified. You may even have to cast your vote from two devices – one computer and one smartphone (identification with a smartphone is a separate topic). That way a large-scale malware attack becomes unlikely. If you add that the client-side software used for voting can be digitally signed, or can be changing itself constantly, then a generalized malware has to target millions of combinations of versions of desktop and mobile OSs, the voting software, etc. And if you instead vote from a remote virtualized environment, to which you login via a sort of a VPN client (with a reasonable assurance on the other end that it is not a fake virtualized environment), then yes – individuals can be targeted, but large-scale attacks may hit a brick wall.
  • How will we avoid coercion and vote-buying in remote, uncontrolled environments. That’s a good question, and although it doesn’t sound technical, it is. First, biometric factor as part of the identification may defend against mass collecting of smartcards for voting. Then there’s the concept of a “panic PIN”, which allows a coerced voter to appear to have voted, but to instead send an alarm to the authorities that he is being coerced, which has been discussed in papers as well.

You probably won’t notice the last recommendation of the report, which says that at the present moment there is no voting system that is secure enough to be deployed for national elections. And that is true (as well as the other recommendation). Yes, it is very hard to build a proper e-voting system. You have to take into account at least all the thing listed in the 136 page report. And even more. You have to be paranoid and expect a state-level attack, insider attack, botnets, etc. But that makes it very hard, not “a bad idea” or “impossible”. I’ll quote the comment by Matthew Proorok on the above youtube video:

The thing is, none of this makes electronic voting a bad idea. It makes electronic voting a problem with a lot of hurdles to overcome. After all, you start out the video pointing out that physical voting, too, has its weaknesses. And that attempt after attempt has been made to defraud the system. And that, over time, we’ve found ways to defend against those attempts. Effectively, you’re saying that electronic voting hasn’t had that kind of proving period yet, and thus it’s a bad idea, and thus we shouldn’t use it. That sounds like a great mindset for NEVER DOING ANYTHING NEW.

And at the same time nobody realizes how flawed the paper based system is, and how the same type of loosely defined arguments can be used against the paper based system as well. Saying the “we’ve found ways to counter all types of fraud in a paper based system” is entirely wrong, as I can prove to you if you come and visit just a single Bulgarian election. By the way, do you know that at the moment, paper voting results are finally combined on a computer? Possibly using excel somewhere. How are we sure these computer system are not attacked? How are we sure that the computers that send the the protocols from the local centers to the central committees are not compromised with a malware? There is a paper trail, I hear. Recounting rarely happens, and discrepancies, even if discovered, are often buried, because otherwise the whole election may have to be rerun. My point is, these are problems not inherent only to remote, electronic voting. They exist even now.

So, ultimately, I don’t understand all the fear in e-voting, even from people that are moderately tech-savvy. The mantra “if it’s a computer, it can break” is in fact “if it is anything in the real world, it can break”. But when has that stopped us from progressing and fixing broken systems (and paper voting is broken; the fact it doesn’t appear so in western democracies is because society doesn’t exploit it, and not because it’s unexploitable).

But I do understand the psychology that leads to accepting all pseudo-arguments thrown in the air, as a massive FUD campaign (sometimes even coordinated, by the way) – it is way easier to throw these fears, than to debunk them, one by one, especially when debunking them requires linking scientific papers. It’s easy to tell people “this can’t be done”, because sometimes it sounds counterintuitive that it can, and then it’s hard to explain why it can.

I’m not saying we should be all voting online by now, I’m saying we should push in that direction, and we should agree that this is the direction to push, because it feels like it’s right behind the corner and it’s a way to increase participation, especially for future generations, and therefore enhance not only the legitimacy of the democracy, but the opportunities for more direct democracy.

And it will come down to trust in the system. For which, the whole FUD-technical explanation cycle will be repeated many times. But I believe that in due time we will have trust in such systems (as we do in many other electronic systems) and that will enable us to do more with our democratic rights.


Common Sense Driven Development [talk]

September 17, 2015

A few months ago I spoke on a conference (jPrime) about common sense driven development (and what it isn’t).

Here’s the video:

And here are the slides:

As you’ll see, I am not a very good and engaging speaker. I hope at least the content is meaningful, although this time it felt a bit chaotic. Nevertheless, the points I’m trying to make are still worth noting, and I hope they help identifying the lack of common sense we are often surrounded with.


“Forget me” and Tests

September 10, 2015

Your users have profiles on your web application. And normally you should give them a way to delete their profiles (at least that’s what the European Court has decided).

That “simply” means you need to have a /forget-me endpoint which deletes every piece of data for the current user. From the database, from the file storage, from the search engine, etc. Apart from giving your users at least partial control over their own data (whether you can have it or not is their decision), it is also a benefit for developers.

Apart from your isolated unit tests, containing a lot of mocks, you have other sorts of tests – integration test, acceptance tests, Selenim tests. All of these need a way to leave the database in the same state that it was before they were executed. In some cases you can use read-only transactions (e.g. with spring-test you get that automatically), or you can use an in-memory database and hope it will work the same way as your production one, or you can drop the database and recreate it on each run. But these are partial solutions with some additional complexity.

The best way, I think, is to just reuse the “forget me” functionality. From your acceptance/selenium tests you can call the /forget-me endpoint at the end of the test (tearDown), and for your integration tests y. If you distribute client-side APIs (or a third-party is building them against a test deployments of your system), you can again call the forget-me endpoint.

That, of course, doesn’t cover non-user-related data that you need in the database. If you have such data (apart from enumarations and data that should be always there), you have to take care of it separately.

Doesn’t that bring some additional complexity as well, and the constant need to update your forget-me functionality? Isn’t having read-only transactions, or a shell script that recreates the database after each run, simpler to support? Assuming that you need to have a properly working forget-me functionality anyway – no. It’s better to reuse it. That would also make sure the endpoint is indeed working properly, and your users can be fully forgotten.


The “Software Engineer” Mindset

September 7, 2015

What is a software engineer? And what is a senior software engineer? Many companies define a “senior software engineer” as a person who has spent more then 6 years as a programmer. And that’s not always correct.

The other day I was asked whether I recommend becoming a “generalist” or a “specialist”. Whether one should stay focused on one particular technology and become really proficient with it, or do a little bit of everything. I once wrote that if you do a little bit of everything, you become no expert at all. And while I still partially hold that view, it needs to be elaborated.

The software engineer mindset never results in narrow specialists. But it doesn’t mean you don’t “drill” into a particular technology. In fact, you drill into many particular technologies/frameworks/levels of abstractions. You become proficient with them, then you move on to the next one. Probably with side projects, probably as transitioning from one job to another, where something unfamiliar is used alongside the known bits. Over time, you gather enough experience that each new technology is familiar and you get into it pretty quickly. On the other hand, staying focused on one particular technology for a long time doesn’t let you see the full spectrum of possible solutions to a problem. So no, doing mostly jQuery/Rails/Spring/Android/… for 15 years doesn’t make you a “senior software engineer”.

The software engineer mindset is about solving the problem. The more senior you are, the faster you are in finding simpler solutions. The more technologies you are familiar with, the more non-localized solutions you are able to produce – in a multi-technology project (web, android and iOS frontends, with a java backend, a public API, for example) a solution that looks okay in one particular technology, may be a hack in the rest.

The software engineer mindset is not saying “I don’t know about that, another colleague was doing it”. I’ve been getting answers like this on interviews – people have even been implementing JSR specs, and only knew the part they were working on for the past 2 years. How it fits with the rest is what the software engineer should be concerned with.

Isn’t that the role of the architect, some people may ask. But the architect is a role, not a job. Each software engineer with the right mindset and knowledge is an architect, and should be. Maybe one will represent the team in front of committees (if such are needed at all), but the top-down architect approach is broken. Mostly because an architect-only position doesn’t get to write code, and loses grip with reality soon enough.

Maybe I’m trying to label what I like doing (going into all parts of the application, from the high-level architecture to the low-level details) as a “software engineering mindset”. And maybe I’m just adding yet another synonym for the “full-stack developer” cliche. Anyway, I think it’s good to encourage people to see the broader technology landscape, and it is equally important to encourage them to spend time focusing on particular problems and technologies. Otherwise they may become one of those architects and seniors, that pretend to know a lot, but haven’t actually seen the intricate details. And the devil is in the detail. The software engineer has both.


Comments on The Twelve-Factor App

August 22, 2015

The Twelve-Factor App is a recent methodology (and/or a manifesto) for writing web applications that hopefully is getting quite popular. Although I don’t agree 100% with the recommendations, I ‘ll quickly go through all 12 factors and discuss them in the light of the Java ecosystem, mentioning the absolute “musts” and the points where I disagree. For more, visit the 12factor.net site.

  1. Codebase – one codebase, multiple deploys. This means you must not have various codebase for various versions. Branches is okay, different repos are not. I’d even go further and not recommend Subversion. Not because it’s not fine, but because git and mercurial do the same and much more. You can use git/mercurial the way you use SVN, but not the other way around. And tools for DVCS (e.g. SourceTree) are already quite good
  2. Dependencies – obviously, you must put as many dependencies in your manifests (e.g. pom.xml) as possible. The manifesto advices against relying on pre-installed software, for example ImageMagick or Pandoc, but I wouldn’t be that strict. If your deployments are automated and you guarantee the presence of a given tool, you shouldn’t spend days trying to wrap it in a library of your working language. If it’s as easy as putting an exetuable script in a jar file and then extracting it, that’s fine. But if it requires installation, and you really need it (ImageMagick is a good example indeed), I don’t think it’s wrong to expect it to be installed. Just check on startup if it’s present and fail fast if it’s not.
  3. Config – the most important rule here is – never commit your environment-specific configuration (most importantly: password) in the source code repo. Otherwise your production system may be vulnerable, as are probably at least a third of these wordpress deployments (and yes, mysql probably won’t allow external connections, but I bet nobody has verified that).

    But from there on my opinion is different than the one of the 12-factor app. No, you shouldn’t use environment variables for your configuration. Because when you have 15 variables, managing them becomes way easier if they are in a single file. You can have some shell script that sets them all, but that goes against the OS independence. Having a key-value .properties file (for which Java has native support), and only passing the absolute path to that file as an environment variable (or JVM param) is a better approach, I think. I’ve discussed it previously. E.g. CONFIG_PATH=/var/conf/app.properties, which you load on startup.

    And in your application you can keep a blank app.example.properties which contains a list of all properties to be configured – database credentials, keys and secrets for external systems, etc. (without any values). That way you have all the properties in one place and it’s very easy to discover what you may need to add/reconfigure in a given scenario. If you use environment variables, you’d have to have a list of them in a txt file in order to make them “discoverable”, or alternatively, let the developers dig into the code to find out which properties are available.

    And last, but not least – when I said that you shouldn’t commit properties files to source control, there is one very specific exception. You can choose to version your environment configurations. It must be a private repo, with limited access and all that, but the (Dev)Ops can have a place where they keep the properties and other specifics for each environment, versioned. It’s easier to have that with a properties file (not impossible with env variables, but then again you need a shell script).

    The 12-factor app authors warn about explosion of environments. If you have a properties file for each environment, these may grow. But they don’t have to. You can change the values in a properties file exactly the way you would manage the environment variables.

  4. Backing Services – it’s about treating that external services that your application depends on equally, regardless of whether you manage them, or whether another party manages them. From the application’s perspective that should not matter. What I can add here is that you should try to minimize this. If an in-memory queue would do, don’t deploy a separate MQ. If an in-memory cache would do, don’t deploy a redis instance. If an embedded database would do, don’t manage a DB installation (e.g. neo4j offers an embedded variant). And so on. But if you do need the full-featured external service, make the path/credentials to it configurable as if it’s external (rather than, for example, pointing to localhost by default).
  5. Build, release, run – it is well described on the page. It is great to have such a lifecycle. But it takes time and resources to set it up. Depending on your constraints, you may not have the full pipeline, and some stages may be more manual and fluid than ideal. Sometimes, for example in the early stages of a startup, it may be beneficial to be able to swap class files or web pages on a running production server, rather than going through a full release process (which you haven’t had the time to fully automate). I know this sounds like heresy, and one should strive to a fully automated and separated process, but before getting there, don’t entirely throw away the option for manually dropping a fixed file in production. As longs as you don’t do it all the time and you don’t end up with a production environment for which you have no idea what version of the codebase is run.
  6. Processes – this is about being stateless, and also about not relying on any state being present in memory or on the file system. And indeed, state does not belong in the code.

    However, there’s something I don’t agree with. The 12-factor preferred way of packaging your assets is during build time (merging all css files into one, for example). That has several drawbacks – you can’t combine assets dynamically, e.g. if you have 6 scripts, and on one page you need 4, on another page you need 2 of the ones used on the first page, and another 2, then you have to build all this permutations beforehand. Which is fine and works, but why is it needed? There is no apparent benefit. And depending on the tools you use, it may be easier to work with CDN if you are dynamically generating the bundles.

    Another thing where further Java-related details can be given is “sticky sessions”. It’s not a good idea to have them, but note that you can use your session to store data about the user in memory. You just have to configure your servlet container (or application server) to share that state. Basically, under the hood it still uses a distributed cache like memcached or ehcache (I guess you could also use a redis implementation of the session clustering). It’s just transparent from the developer and he can still use the session store.

  7. Port Binding – this is about having your application as standlone, instead of relying on a running instance of an application server, where you deploy. While that seems easier to manage, it isn’t. Starting an servlet container and pushing a deployment is just as easy. But in order to have your application bind to a port, you need to have the tooling for that. They mention jetty, and there is also an embedded version of tomcat, and spring-boot (which wraps both). And while I’m not against the port binding, I’d say it’s equally good to have it the other way around. Container configuration is done equally easy, regardless of whether you drop an environment-specific xml file, or do it programmatically and load the properties from the file mentioned in point 3. The point is – it doesn’t matter – do whichever is easier for you. Not to mention that you may need some apache/nginx functionality.
  8. Concurrency – it’s about using native processes. This, I think, isn’t so relevant to a Java runtime, which uses threads under the hood and hides away the unix process. By the way, another explicit reference to unix (rather than staying OS-independent).
  9. Disposability – that’s about embracing failure. Your system must work fine even though one or more of application instances die. And that’s bound to happen, especially “in the cloud”. They mention SIGTERM, which is a *nix-specific signal, whereas the general idea of the 12-factor app is to be OS-independent. There is an apparent leaning towards Linux, which is fine though.
  10. Dev/prod parity – your development environment should almost identical to a production one (for example, to avoid some “works on my machine” issues). That doesn’t mean your OS has to be the OS running in production, though. You can run Windows, for example, and have your database, MQ, etc. running on a local virtual machine (like my setup). This also underlines the OS-independence of your application. Just have in mind to keep the versions the same.
  11. Logs – the 12-factor app recommends writing all logging information to the system out. A Java developer will rightly disagree. With tools like loggack/slf4j you can manage the logging aspects within the application, rather than relying on 3rd party tools to do that. E.g. log rotation and cleanup, or sending to a centralized logging facility. It’s much easier to configure a graylog or splunk adapter, than having another process gather that from system out and push it. There can be environment-specific log configurations, which is again just one file bundled together with the app.properties). If that seems complicated, consider the complications of setting up whatever is going to capture the output.
  12. Admin processes – generally agreed, but in addition I’d say it’s preferable to execute migrations on deployment or startup, rather than manually, and that manually changing “stuff” on production should preferably be done through something like capistrano in order to make sure it’s identical on all instances.

Overall, it’s a good set of advice and an approach to building apps that I’d recommend, with the above comments in mind.