July 26, 2015

CompletableFuture: Clean callbacks with Java 8’s Promises (part 4 of 4)


Pages: 1 2 3 4

CompletableFuture callbacks, advanced (continued)

Collecting and combining results

A typical use case for asynchronous computation is to have multiple functions executed in parallel and then collecting their results once they’re all finished. For instance, you might run several remote method invocations in parallel to receive different entities, and then combine the result into one entity collection.

Fortunately, CompletableFuture comes with an API which specifically covers these cases.

Wait for all promises to finish, and collect result

This is the implementation of an “AND”-combination of several promises. It can be achieved by combining several callbacks with the #allOf(…) method:
// 1a - build the task
final CompletableFuture<String> promise1 = new CompletableFuture<>();
final CompletableFuture<String> promise2 = new CompletableFuture<>();
// 1b - define task result processing
final CompletableFuture<Void> promiseCombined = 
        // Note that "it" is of type Void.
        CompletableFuture.allOf(promise1, promise2).thenAccept(it -> inform(it));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    promise1.complete("Future 1 explicitly fulfilled");
    promise2.complete("Future 2 explicitly fulfilled");
});
Well, kind of. Unfortunately, the resulting combined promise is of type <Void>, leading to the ridiculous need to create a listener which takes a single parameter of type Void. Typically, we would much rather like to have a resulting promise working with the collected results of the combined promises, i.e. of Type <List<…>>.

This blog post (go take a look; they cover CompletableFuture in much more detail) presents a solution for that which I have adapted here.

Now, we can use the new #allOf(…) to collect the results of multiple promises:
final CompletableFuture<Void> promiseCombined = 
    CompletableFutureUtil.allOf(promise1, promise2).
        thenAccept(all -> all.stream().forEach(it -> print(it)));

Wait for both promises to finish, and collect result

If you just want to wait for exactly two promises, there’s an alternate syntax to combine the second with the first more explicitly through #thenCombine (…):
final CompletableFuture<String> promise1 = new CompletableFuture<>();
final CompletableFuture<String> promise2 = new CompletableFuture<>();

promise1.thenCombine(promise2, (v1, v2) -> combine(v1, v2)).thenAccept(it -> print(it));
There are actually “combining” variations of all three main callback methods:
  • To AND-combine two #thenApply(…) (intermediary) into one operation, use #thenCombine(…)
  • To AND-combine two #thenAccept(…) (terminal) into one operation, use #thenAcceptBoth(…)
  • To AND-combine two #thenRun (…) (terminal) into one operation, use #runAfterBoth(…)
Again, you are apparently not supposed to find any inherent logic in that naming scheme…

Wait for the first promise to finish

The opposite use case of waiting for all promises to finish of course is to wait just for the first one – this would then be the “OR”-combination. As one might expect, there’s an API for that as well: it’s the #anyOf(…) method.
// 1a - build the task
final CompletableFuture<String> promise1 = new CompletableFuture<>();
final CompletableFuture<String> promise2 = new CompletableFuture<>();
// 1b - define task result processing
final CompletableFuture<Void> promiseCombined = 
        // Note that "it" is of type Object.
        CompletableFuture.anyOf(promise1, promise2).thenAccept((it) -> print(it));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    promise1.complete("Future 1 explicitly fulfilled");
    promise2.complete("Future 2 explicitly fulfilled");
});
Unfortunately, this method has a drawback as well (sigh): Unfortunately, the resulting combined promise is of type <?>, thus of Type Object. However, I think that typically, the result type of the combined promises would match and then it would be sensible for the combined promise to be of that very type. Hence, I created another helper method which just stupidly casts the combined promise.

This method can then be used without explicit casting:
final CompletableFuture<Void> promiseCombined = 
    CompletableFutureUtil.anyOf(promise1, promise2).thenAccept((it) -> print(it));

Wait for the first out of two promises to finish

Again, there is a special syntax for the case where there are exactly two promises you want to combine. Consider this example of #applyToEither(…):
final CompletableFuture<String> promise1 = new CompletableFuture<>();
final CompletableFuture<String> promise2 = new CompletableFuture<>();

promise1.applyToEither(promise2, it -> transform(it)).thenAccept(it -> print(it));
Yes, this is actually the counterpart of #thenCombine (…), with the callback not applied to the return value of both promises, but only to the one which returned first.

Of course, there are “combining” variations of all three main callback methods again:
  • To OR-combine two #thenApply(…) (intermediary) into one operation, use #applyToEither(…)
  • To OR-combine two #thenAccept(…) (terminal) into one operation, use #acceptEither(…)
  • To OR-combine two #thenRun (…) (terminal) into one operation, use #runAfterEither(…)

CompletableFuture API overview

For your reference, here’s a brief overview of the CompletableFuture methods covered in this article. I’ve arranged them into the three groups of intermediary, terminal and terminal (with Runnable parameter) methods. Note that only the method names are used here; parameters are omitted.
  • A (asynchronous) means that there’s an additional method starting with the same name, but ending with …Async for asynchronous callback invocation.
  • E (Executor) means that there’s an additional asynchronous (A) method with an additional Executor parameter for explicit Executor assignment.
  intermediary terminal terminal (Runnable) A E
Construct from task static supplyAsync   static runAsync   X
complete listener thenApply thenAccept thenRun X X
completeExceptionally listener exceptionally        
Combined complete and completeExceptionally listener handle whenComplete   X X
Dynamic callback chaining thenCompose     X X
AND all combining static allOf        
AND both combining thenCombine thenAcceptBoth runAfterBoth X X
OR any combining static anyOf        
OR either combining applyToEither acceptEither runAfterEither X X

Conclusion

As I wrote in the introduction, I am sure promises are a great addition to the Java language and I’d love to involve them in a more complex application when given the opportunity. They really help writing clean, functional, reactive code which is especially useful in a multi-threaded, asynchronously run environment. They are just a great concept!

On the other hand, their current implementation in Java 8 leaves quite much to be desired, at least in my opinion. If you thought that the Lambda API is a mess (and you have every right to do so), CompletableFuture takes this to a whole new level. The API is bloated, unintuitive and some features are just very poorly implemented.

This really is a shame because instead of helping us to properly apply the somewhat complex concept of promises, it further obscures it, depriving it of its potential as a straightforward threading tool. I cannot understand why this highly complex API is not at least accompanied by an official tutorial!

Nevertheless, I hope that this article helped you to better understand the concept of promises and their application in Java 8 through the CompletableFuture API. By all means, if anything is unclear or if you miss any important information, please let me know in the comments below.

You may also want to take a look at the GitHub repository which contains the full source code of these code samples as JUnit tests.

Update September 6, 2015: Meanwhile, I’ve created a small wrapper API for CompletableFuture which is much more concise, consistent and contains fixes for the flaws discussed in this article. It's part of the Collection API wrapper library LambdaOmega. Check out the accompanying blog post here or visit its GitHub page. Version 0.1 is now RELEASED!



Pages: 1 2 3 4

CompletableFuture: Clean callbacks with Java 8’s Promises (part 3 of 4)


Pages: 1 2 3 4

CompletableFuture callbacks, advanced (continued)

Chaining callbacks

Now comes the interesting part which is to chain callbacks. This is what in a classic, pure callback-based implementation would quickly devolve to deeply nested callback hell.

The most basic implementation is to chain two fulfill callbacks, “pipeing” the result of the first one to the input of the second one:
promise.thenApply(it -> transform(it)).thenAccept(it -> print(it));
Now we see the difference of intermediary and terminal callbacks discussed earlier in action. Because the return value of the transform(…) method must be “piped” into the input value of the print(…) method, this method must return any value (not void). The CompletableFuture API in this case forces us to register the intermediary callback with #thenApply(…) and the terminal callback with #thenAccept(…).

But wait – it this any different from naïvely chain callbacks like this…?
promise.thenAccept(it -> { print(transform(it));} );
Yes, it is.
  • First of all, you would have to manually do exception handling, i.e. differ whether the inner or the outer function threw an exception, and react accordingly. In the example, there is no exception handling (through callback rejecting) at all, but this applies especially when registering a combined callback handler with handle(…) (intermediary) / whenComplete(…) (terminal).
  • Through the use of asynchronous callbacks which we covered very briefly (the methods ending on …Async(…)), we can make any part of the callback chain run asynchronously with complete control over Thread management.
Let’s not fall back into the old “callback” way of thinking.

Remember that inside the actual callback function, the control flow is no longer controlled by invoking #complete(…) / #completeExceptionally(…) on a promise object, but implicitly by the callback’s return value / by the exception it throws. More on that in the next section:

Callback chaining fall-through with eventual exception handling

It’s important to realize that the callback chain is traversed in sequence to find the next fulfill / reject handler. Thus, in order to make sure no exception thrown within the callback chain gets missed, it’s good practice to finalize a callback chain with either a rejection handler or a combined handler:
promise.<String> thenApply(it -> {throw new RuntimeException("Promise rejected");})
        .thenApply(it -> transform(it))
        .thenApply(it -> transform(it))
        .whenComplete((it, err) -> {
            if (it != null) {
                print(it);
            } else {
                logAccidentalException((CompletionException) err);
            }
        });

startThread(() -> {
    sleep(WAIT_MILIS);
    promise.complete("Future explicitly fulfilled");
});
In this example, none of the it -> transform(it) will get invoked. After the first callback threw the RuntimeException, it goes straight through to the “error” case of #whenComplete(…).

Note that in this example, promise rejection really happens implicitly in a callback handler, i.e. by throwing an exception, as mentioned in the previous section. However, in this case, the exception “piped” into the next rejection handler would not be the original business exception, but that business exception wrapped in a CompletionException:
private String logAccidentalException(CompletionException ex) {
    // do something with input, e.g. print it
    throw new RuntimeException(ex);
}
(Because of CompletableFuture API shortcomings, we still have to cast it when registering the callback with #whenComplete(…).)

Using callback chaining for exception handling

If you think about it, you could really use promises as a mini-DSL (domain specific language) for exception handling:
String output = CompletableFuture.supplyAsync(() -> {
    // do stuff
    return "Promise";
}).whenComplete((it, err) -> {
    if (it != null) {
        print(it);
    }
    else {
        log(err);
    }
}).get();
That’s pretty straightforward. If the method succeeded, you proceed the output in the if branch of #whenComplete(…); otherwise, you handle the exception in the else branch.

(I wouldn’t recommend using that in practice though as you should stay with Java’s globally accepted exception handling standards).

Using callback chaining to recover from exceptions

A useful application of callback chaining is the possibility to recover from an exception in a previous function call. Remember that when introducing the reject callback, we emphasized that the respective callback handler must throw an exception itself?
private String log(Throwable ex) {
    // do something with input, e.g. print it
    throw new RuntimeException(ex);
}
If an exception is thrown by a previous function call, the reject callback gets invoked, but any subsequent fulfill callback would be ignored.

In this example code, fireing an exceptional callback, log(it) gets called, but not the subsequent second print(it):
// 1a - build the task
final CompletableFuture<String> promise = new CompletableFuture<>();
// 1b - define task result processing
promise.exceptionally(it -> log(it)).thenAccept(it -> print(it));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    promise.completeExceptionally(new MyPromiseRejectedException("Promise rejected"));
});
However, by returning a proper value from a reject callback, we tell the chain mechanism that we have “recovered” from the exception, thus the subsequent fulfill callback would then be invoked.

Consider this implementation of fix(…):
private String fix(Throwable ex) {
    // do something with input, e.g. print it
    return "Recovered";
}
In this example code, fireing an exceptional callback, fix(it) gets called which returns a proper value again, triggering the subsequent call on the second print(it):
promise.exceptionally(it -> fix(it)).thenAccept(it -> print(it));

Dynamic callback chaining

If you need the power to dynamically define which promise to “pipe” into, e.g. based on the outcome of the previous promise, use #thenCompose(…) rather than #thenAccept(…) / #thenRun(…) / #thenApply(…):
// 1a - build the task
final CompletableFuture<String> promise1 = new CompletableFuture<>();
final CompletableFuture<String> promise2 = new CompletableFuture<>();
// 1b - define task result processing
promise1.thenAccept(it -> print(it)).thenCompose(it1 -> promise2);
promise2.thenAccept(it2 -> print(it2));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    promise1.complete("Future 1 explicitly fulfilled");
    promise2.complete("Future 2 explicitly fulfilled");
});
For #thenCompose(…), the function provided must thus return the next promise in the chain.

Pages: 1 2 3 4

CompletableFuture: Clean callbacks with Java 8’s Promises (part 2 of 4)


Pages: 1 2 3 4

Promises basics with CompletableFuture

Fulfill a promise

This is covered in the first introductory example in the section above.
  • We explicitly fulfill a promise by calling #complete(…).
  • The respective listener is registered with #thenAccept(…) (takes the promise outcome as an argument) / #thenRun(…) (takes a Runnable as the argument, no access to the promise outcome).

Reject a promise

Rejecting a promise means signaling an error occurrence, i.e. the future did not complete normally but throws an exception. As with return values, you wouldn’t let your actual task throw the exception but instead, you inform the CompletableFuture event listener that an exception occurred:
// 1a - build the task
final CompletableFuture<String> promise = new CompletableFuture<>();
// 1b - define task result processing
promise.exceptionally(it -> log(it));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    promise.completeExceptionally(new PromiseRejectedException("Promise rejected"));
    
    try {
        promise.get();
    } catch (InterruptedException ex) {
        throw new RuntimeException(ex);
    } catch (ExecutionException ex) {
        Assert.assertEquals(PromiseTestUtil.PromiseRejectedException.class, ex.getCause().getClass());
    }
});
  • We explicitly reject a promise by calling #completeExceptionally(…).
  • The respective listener is registered with #exceptionally(…).
Note that the exception listener handler throws an exception (here, it re-throws the original exception) once its job is done.
private String log(Throwable ex) {
    // do something with input, e.g. print it
    throw new RuntimeException(ex);
}
This is important. We will come back to this presently.

Fulfill and reject a promise?

As promises replace callbacks which signal successful or exceptional completion of a function, they must be fulfilled at most once. As I mentioned earlier, this is implicitly built-in into CompletableFuture. You can fulfill or reject a promise only once; a fulfilled promise cannot be rejected and vice versa; any additional call is ignored and the callback is not invoked. The API provides access to the current status of a promise:
  • #isDone() returns true if the promise has been either completed normally or exceptionally already.
  • #isCompletedExceptionally() returns true if the promise has been completed exceptionally already.
Moreover, the return value of the event fire methods indicate the original status of the promise as well:
  • When fulfilling a promise by calling #complete(…), the return value is true if the promise has already been “done”, thus no listener will be invoked.
  • When fulfilling a promise by calling #completeExceptionally(…), the return value is true if the promise has already been “done”, thus no listener will be invoked.

However, note that if you specify a promise to have multiple callbacks like this:
promise.thenAccept(it -> print(it));
promise.thenAccept(it -> print(it));
you essentiall say: If this promise gets fulfilled, fire both these callbacks. This is perfectly valid. Fulfilling a promise will then invoke both callbacks, but any subsequent fulfilling will not invoke any callbacks.

Having said that, Java wouldn’t be Java if there wasn’t any hackish way to undermine this mechanism. There actually is:
  • #obtrudeValue(…) sets or re-sets the value retrived by #get().
  • #obtrudeException(…) sets or re-sets #get() to throw the exception provided.
Note that neither of those methods does fire any callbacks. They really just change the outcome of calling #get(). Because they mess with about everything promises are built for, I strongly advice you to never use these methods at all.

CompletableFuture callbacks flavors

If you ventured into the CompletableFuture API already, you may have noticed that some of the myriad of methods defined for the class really are different flavors of the basic methods we already covered here. I’d like to call them flavors rather than types, because you can intermix them, resulting in a vast amount of different variations.

Intermediary and terminal callbacks

We will later explore how to chain or “pipe” callbacks i.e. use the return value of one callback as the input value of another one. However, so far we only used terminal callbacks, i.e. methods which do not return a value (void) (these implement the Consumer functional interface). These are not eligible for chaining.

Of course you must define a callback function which returns a value of any type in order to chain / pipe it to another callback. That’s what intermediary callbacks are for (these implement the Function functional interface).

There’s an intermediary equivalent for these CompletableFuture methods we covered already:

  intermediary terminal terminal (Runnable)
Construct from task static supplyAsync   static runAsync
complete listener thenApply thenAccept thenRun
completeExceptionally listener exceptionally    

(Excerpt from the complete CompletableFuture callback API overview on the last page.)

Yes, unfortunately, the method names of terminal / intermediary callback registrators differ, and their resemblance is not obvious nor intuitive.

Thread-independent callbacks

Most callbacks offer an “async” flavor as well; these are the methods ending with …Async(…). They will run in a new thread rather than re-using the thread from a previous callback. The somewhat intricate details are discussed in this blog post. We’ll come back to this when we cover callback chaining.

Callbacks with explicit Executor

Asynchronous-flavored callbacks optionally take an additional Executor argument which will spawn the callback thread if provided; otherwise, according to the API, ForkJoinPool.commonPool() is used.

CompletableFuture auxiliary API

As we’re about to cover the entire CompletableFuture callback API, I’d like to briefly present some additional methods of CompletableFuture as well:
  • #get(…) blocks the thread and eventually retrieves the return value of the promise. Throws InterruptedException (threading-related) and ExecutionException (when the promise was rejected) as checked exceptions. You should always prefer this method over #join() if you’re interested in the return value as it forces you to check for rejection as well.
  • #getNow(…) tries to retrieve the return value of the promise immediately, or returns the provided default value; never blocks.
  • #join(): same as #get(), but throws unchecked exceptions only.

CompletableFuture callbacks, advanced

Combining callbacks

Of course, we can register both a fulfill callback as well as a reject callback. One way to do so is by registering them individually like this:
promise.thenAccept(it -> print(it));
promise.exceptionally(it -> log(it));

Note that both methods return a modified CompletableFuture object; yet, we need to register both on the same original object, thus do not use builder-style method chaining here.

An equivalent way is to provide a combined callback with #whenComplete(…):
promise.whenComplete((it, err) -> {
    if (it != null) {
        print(it);
    }
    else {
        log(err);
    }
})
Note that it’s important to register a rejected callback whenever you expect an exception to happen; because the “normal” fulfill callback would not be triggered when a promise is rejected and thus you might “miss” completion information.

We can add to our list of CompletableFuture methods:

  intermediary terminal terminal (Runnable)
Combined complete and completeExceptionally listener handle whenComplete  

Multiple callbacks

Just as you can register both a fulfill and a reject callback, you can actually register an arbitrary number of both fulfill and reject callbacks!

For instance, defining these two callbacks:
promise.thenAccept(it -> print(it));
promise.thenAccept(it -> print(it));
would fire it -> print(it) twice when the promise is fulfilled. Note that this is not the same as chaining callbacks as covered on the next page.

Pages: 1 2 3 4

CompletableFuture: Clean callbacks with Java 8’s Promises (part 1 of 4)



I’ve recently stumbled across the concept of promises and the fact that they were somewhat secretly introduced in Java 8. The CompletableFuture API in my opinion has the potential to change functional, asynchronous programming in Java as much as Lambdas do. I’ll cover the basics of this API in this article.

TL;DR: There’s a human-readable overview of the CompletableFuture callback API on the last page. Maybe that’s all you’re looking for.

Java 8’s CompletableFuture finally brings the concept of promises (as e.g. included in the Scala language and in the most recent ECMAScript 2015 standard) and as such monadic functions to the Java language. They are a brilliant choice to deal with asynchronous computations as they let us trade “callback hell” for a declarative, reactive API. Even if that sounds very academic, they offer real benefits for event-driven asynchronous programming you don’t want to miss.

With this article, I hope to cover the CompletableFuture API exhaustively; there’s also an accompanying GitHub repository which contains JUnit tests with example applications of the API.

Why should I make a Promise?

A promise is a future

According to the CompletableFuture’s Javadoc, a promise is a Future that may be explicitly completed. Well… Okay, let’s start the other way around. A Future represents the result of a computation which may or may not be completed at some point in the future. In case you are not already familiar with futures, here’s an example:
// 1 - build the task
FutureTask<String> retrieveName = new FutureTask<>(() -> {
    sleep(WAIT_MILIS);
    return "Future";
});

// 2 - start the task
startThread(() -> {
    retrieveName.run();

    // 3 - collect the result
    try {
        print(retrieveName.get());
    } catch (InterruptedException | ExecutionException ex) {
        throw new RuntimeException(ex);
    }
});
Here, a future task is created which does some calculations and eventually returns a String. FutureTask is an implementation of the Future interface. It wraps around a Callable (a Callable is a Runnable with doesn’t return void but a return value; here, the Callable is implemented as a lambda expression). It provides methods to start and cancel a computation, query to see if the computation is complete, and retrieve the result of the computation (quote from the API).

In the example, the future task is scheduled for execution asynchronously, thus on a new thread (FutureTask#run() starts the task’s computation, let’s assume that #startThread(…) does Thread#start() to start the Thread). You would typically use an ExecutorService for threaded execution, but I neglected this facility to keep the code as simple as possible.

After it has been scheduled, the task may or may not be completed at some point in the future. That’s where one of those methods mentioned earlier come in handy.

Here, we use FutureTask#get() to query for the computed result. This method waits if necessary for the computation to complete, blocking the caller thread, and then retrieves its result. We then use the result in a subsequent function (here, it’s a #print(…) function). After all, the whole point of building up this Future value setup is that we want to do anything with the return value once we receive it. We always want to do anything with a Future’s return value; otherwise, we would rather start a simple Runnable for a fire & forget asynchronous task.

Typically, we already know what we want to do with the return value before we even start the task. In the example, it’s invoking the #print(…) method with the return value. It seems silly to wait for the task to complete in order to then instruct it what to do next (and repeat this for all subsequent steps). The conceptual problem with a future here is that you have to actively query for its completion using #get(). That’s polling. Wouldn’t it be much more elegant and reactive to have a way to declaratively react on the completion event? Through an… event listener, a callback? Now, we’re slowly getting there.

The problem with callbacks

So we want to define an event listener – a callback – which is invoked as soon as the asynchronous runnable has completed. It will collect the result and process it.

In order to define a task which executes a callback function rather than returning any value, it’s enough to derive from Runnable (for technical reasons, we have to derive, we cannot create a derived Runnable implicitly with lambdas). And we have to derive because we want to be able to provide the callback function via the constructor. The specialized runnable could look like this:
private static class RunnableCallback implements Runnable {
    private final Consumer<String> callback;
    
    public RunnableCallback(Consumer<String> callback) {
        this.callback = callback;
    }

    @Override
    public void run() {
        sleep(WAIT_MILIS);
        callback.accept("FutureCallback");
    }
}
It decouples task execution from the callback: Once the computation is finished, the callback is invoked with the computation outcome (a.k.a. the static String for the sake of this example).

We can then instantiate the task by providing the callback function, and eventually start it:
startThread(() -> {
    new RunnableCallback(it -> print(it)).run();
});
Here, the callback is again defined using a lambda expression. (Of course, the callback function would typically execute much more sophisticated operations which require the context of the original caller of the asynchronous function. Otherwise, you wouldn’t use a callback function, but implement its operations just inside the asynchronous function itself.)

This looks pretty straightforward and it works. The task is executed asynchronously and the calling thread never blocks with polling. The callback gets invoked as specified.

This solution seems fine for this very simple case. But what about more complex cases? For instance,
  • We should probably have a dedicated callback function in case an Exception occurs during processing (an “error callback”).
  • We may even have special requirements such as starting a range of asynchronous tasks and fire a callback only when the last task has completed.
  • Most importantly though, the situation will quickly become messy once we begin to chain callback functions, e.g. if one callback triggers another asynchronous function which in turn takes its own callback, and so on. This is what especially in the JavaScript / Node.js world (where asynchronous operations are quite frequent) is known as callback hell.
  • Also, a callback must not be called multiple times as it is really thought to signal that the function has completed which happens only once; we’d have to implement this mechanism ourselves.
Promises provide us with a sensible mechanism to mitigate callback hell and write clean asynchronous code. With CompletableFuture, the respective API is now part of Java.

Make a promise

The default use case for a promise which is equivalent to above implementations with a future or a callback is to specify a function to be executed with the result returned by the original asynchronous function after its successful completion. Let’s see how to do this.
// 1a - build the task;
final CompletableFuture<String> retrieveName = CompletableFuture.supplyAsync(() -> {
    sleep(WAIT_MILIS);
    return "Promise";
});
// 1b - define task result processing
retrieveName.thenAccept(it -> print(it));

// 2 - start the task
startThread(() -> {
    try {
        retrieveName.get();
    } catch (InterruptedException | ExecutionException ex) {
        throw new RuntimeException(ex);
    }
});
First of all, we have to wrap our asynchronous function into a CompletableFuture which is the actual promise object. Here, the asynchronous task is implemented as a lambda expression.

The CompletableFuture / promise object now offers a multitude of methods which help us specify the promises’ “monad” behavior.

This basic use case is covered by the #thenAccept(…) method which executes the Consumer function with the result of the wrapped asynchronous function as soon as the latter has completed successfully. Here, the Consumer is implemented as a lambda expression, once again.

Finally, we have to execute the task. Therefore we use #get() to explicitly query for the return value of the CompletableFuture; this is actually the equivalent of a Future#get() call. Note that just as Future#get(), the method blocks the caller thread, so we put it in a separate thread.

This solution works fine. We were able to specify any subsequent action for the return value of the asynchronous task declaratively and independently of its actual invocation.

Nevertheless, we missed an important point here. Going back to the CompletableFuture’s API, it is defined as “a Future that may be explicitly completed”. But actually, we didn’t explicitly mark the promise as completed. Instead we invoked the underlying future which, upon completion, called the CompletableFuture’s callback function(s). This is essentially still the same as a vanilla Future. Let’s see how much more elegant asynchronous tasks get with explicit promise completion.

Fulfill a promise

This is the code necessary to define a CompletableFuture and complete it explicitly:
// 1a - build the task
final CompletableFuture<String> future = new CompletableFuture<>();
// 1b - define task result processing
future.thenAccept(it -> print(it));

// 2 - start the task
startThread(() -> {
    sleep(WAIT_MILIS);
    future.complete("Future explicitly fulfilled");
});
First of all, we instantiate a new CompletableFuture. Note that this really is an “empty pod”. The actual asynchronous task we want to manage with this promise object can and will be instantiated independently of the CompletableFuture object.

As in the previous example, we have the whole CompletableFuture API available on our promise object which we use to declaratively specify its callbacks. Once again, we use a lambda expression in order to specify to invoke print on the return value.

Then we build and start the thread we will observe with the CompletableFuture object. We will now explicitly call the respective callback function of the CompletableFuture object to signal that the task has completed. We do this by using #complete(…) with the desired return value. Of course, the asynchronous task must have a reference of the CompletableFuture to invoke this method. Because now, the asynchronous task does not need to have a return value because we’ll never query for it explicitly, we can just implement it as a fire & forget Runnable (which is here implemented as a lambda expression).

And that’s it. Explicitly completing the promise fires the respective event handler. Thus, CompletableFuture really is a useful API to save you from building asynchronous function callbacks and it covers a lot of use cases as we shall see in the next section.

OK, but what is it actually good for?

As we just saw, CompletableFuture really is just a callback handling facility for a Future. A Future is a means to collect output data from asynchronous tasks. Hence, a CompletableFuture is useful whenever you want to collect the result of an asynchronous task or multiple asynchronous tasks. The two most typical use cases are:
  • Parallel computing on multiple CPU cores
  • Remote method invocation through e.g. SOA or REST calls through a remote interface
In both of these cases, you want to eventually collect the computation output, and CompletableFuture gives you a means to specify what should be done with the output once it is received in a functional, declarative and reactive programming style which is well-suited for this situation.

All you have to do to enable promise-style programming for any asynchronous method is to provide the method with a promise object and fire its completion callbacks accordingly.

For the rest of this article, I will stick to the “execute a Thread with a Thread#sleep(…) invocation” just because it’s the most simple thing to demonstrate any asynchronous use case.

Pages: 1 2 3 4

July 19, 2015

Scrum: 8 signs that you’re doing it wrong (part 2 of 2)


Pages: 1 2

The Sprint is not fixed

This is a very obvious violation of one of Scrum’s most basic rules and the most easy one to detect: The Sprint length, considered to be fixed by the Scrum Guide, is shortened or, more often, prolonged at the discretion of a dedicated team member, a “boss”, management or any other person with the authority to do so. This typically happens in a situation where Sprint planning drastically underestimated development efforts and time is running out to complete the Sprint’s objectives. By lack of understanding of how Scrum works, it is then assumed that extending the Sprint will fix the problem. Of course, it will not, but instead, it will make things much worse.

Defining a single development cycle (the Sprint) as a time-boxed unit of fixed length cleverly enables both transparency and control of the development cycle: With the time length fixed, development effort can be adjusted by changing the scope or the quality (with the scope clearly being preferred) to achieve the Sprint goal. Naturally, we can easily manage changes to the scope whereas adjustments to the time frame are really only possible in one way. Most importantly, a constant, fixed time-box for a Sprint gives us a means of comparing one Sprint’s performance with another’s. This enables us to judge or make predictions on the current Sprint’s performance or the overall project progress, and with each additional Sprint, collecting more empirical performance data, our predictions will become even more accurate over time.

As soon as we begin to change the length of a Sprint, we lose this tape measure. Also, prolonging a Sprint breaks the Product Owner’s expectations and disturbs the processes associated with Product Increment delivery which would then be postponed.

Prolonging the Sprint is just no option! Instead, what you should do in a situation where you realize that you can’t implement all the predicted functionality until the end of the Sprint is to finish the Sprint in time and then use the Sprint Review and Sprint Retrospective to inspect why you weren’t able to meet your Sprint planning estimates. Yes, in this situation, Scrum will force you to re-assess your planning, your development workflow and your overall performance rather than just hiding what went wrong by tweaking Sprint planning. This is the only way good teams will get even better over time!

The most extreme violation to the fixed Spring length rule of course is to not have any well-defined Sprints at all which basically makes the project become a waterfall or a death march. This clearly is not Scrum nor agile at all and because this process lacks any feedback loop, it is clearly unmanageable and very ill-suited for custom software development. Because there is no way to inspect or adapt team performance, project management really becomes superfluous is such a scenario. I assert that if in this situation, project management is laid off, eventually a self-organizing team with a (customer) feedback-controlled development process will emerge.

There’s no well-defined Product Increment

This one is closely related to not having fixed-length Sprints, as explained in the previous section. In Scrum, every Sprint has a goal, and the goal is create a fully-functional, shippable product of actual business value, the so-called Increment, built by further evolving the Increment created in the previous Sprint. If your Sprint Goal is not defined like this, you’re not doing Scrum.

Typical deviations of a Sprint Goal’s definition are:
  • there is no well-defined goal or outcome at all;
  • there is an initial goal, but it is only loosely defined or it gets changed arbitrarily throughout the Sprint;
  • there is a well-defined outcome, but it is not built upon the outcome the previous Sprint.
Not having a fixed Product Increment introduces the same lack of manageability as not having a fixed Sprint length, as explained in the previous section. Note that Scrum does allow to re-negotiate a Sprint’s scope during the Sprint; however, massively altering the outcome would certainly invalidate Sprint planning and thus diminish Sprint manageability.

It is absolutely essential to have the Sprint Goal and every single task (a Sprint Backlog item) well defined and to make sure this definition is well understood by every team member. This really is the precondition to Scrum’s transparency. More precisely, every single Sprint Backlog item needs a “definition of done” which allows you to “tick” finished tasks off as the Sprint progresses. Ambiguity about task acceptance criteria leads to lengthy discussions and conflicts.

Lastly, having the outcome not built upon the outcome out the previous Sprint actually violates the idea of building an “increment” as this doesn’t match its definition. By building something “completely different” than in the previous Sprint, you deprive yourself of the possibility to learn from past experience which is the big advantage agile processes introduce.

To sum this section up, it is vitally important to make sure that the Sprint Goal and the tasks required to reach it are well defined and understood. Tasks with an incomplete or ambiguous “definition of done” must not be deemed completed. During Sprint Review / Retrospective, it must be discussed why the “definition of done” was inaccurate in order to introduce changes to the next Sprint Planning.

Planning and reporting is either not transparent or not updated daily

This is arguably the most vague anti-pattern presented here as it comes in many different shapes. Typical signs of its application include:
  • there is no Sprint Backlog or it is either not visible for the whole team at all time, it is unclear or it is not updated daily with the input of the entire team;
  • there is no Product Backlog or it is either not visible for the whole team at all time, it is unclear or it is not updated after a Product Backlog Refinement Meeting;
  • information about the current Sprint progress is not presented on the Sprint Backlog, but distributed or hidden in external resouces such as bug tracking systems;
  • the Daily Scrum is not held daily (Ha!).
Obscuring information about team performance and project progress diminishes the Team’s ability or work in a self-organizing manner and for each individual to work efficiently. With information locked away, knowledge gets centralized, introducing dependency on single individuals which keep the knowledge and thus become information bottlenecks. There’s the enemy of a self-organizing team again.

In a Scrum team, it is for any Team member vitally important to have this information updated on a daily basis:
  • what tasks any other Team member is working on and how long these tasks will keep him occupied;
  • what the current status of every single Sprint Backlog item is, and how long it will take for each one to get finished;
  • what the current status of every single Product Backlog item is, and how long it will take for each one to get finished.
He does need to have this information in order to make sensible decisions about how the tasks he is currently working on do interfere with tasks others are currently working on; what are the dependencies of the tasks he is currently working on; what problems others are facing which may concern him as well or which he may help solve; what task he should start next; and other decisions he must make as a member of a self-organizing team.

This is yet another issue which typically concerns organizations that are not used to agile processes, and it may either hint that agile thinking is not clearly understood or that it’s actually undermined. Scientia potentia est – knowledge means power, and one way to stay in power is keeping knowledge locked. Needless to say that this violates any agile thinking. Scrum cleverly forces us to break this kind of thinking by introducing these transparent artifacts. Your Scrum team must use them, or you’re not doing Scrum.

It’s usually most efficient to keep them as simple as possible, e.g. use a whiteboard and sticky notes as the Sprint Backlog. If your team doesn’t use them but you really want to have them, volunteer to keep them updated. Your team will quickly have to acknowledge that the time effort is negligible and the profit is remarkable. Impress them by drawing a Burndown Chart. If you lack any information to properly keep the backlog updated, ask for it. That way, you will make more and more information transparent over time.

There’s no Sprint Retrospective

Of all of Scrum’s events and artifacts, the Sprint Retrospective meeting gets most typically neglected in my experience; this is especially true for organizations with scarcely any prior agile experience.

However, the Sprint Retrospective is an essential part of Scrum, and without it, you’re not doing Scrum. The Sprint Retrospective is an opportunity to inspect the development processes and to identify impediments and improvements for the next Sprint. These ideas are kept track off, and at the end of the next Sprint, the Team investigates how well they have been realized. Thus, the Sprint Retrospective really is the key opportunity for Scrum’s central process of inspection & adaptation.

Hence, the Sprint Retrospective is arguably the most inconvenient event introduced by Scrum as it forces the Team to identify and discuss development process impediments, thus admitting that the processes are not working perfectly yet and to challenge itself to address and resolve problems.

The Sprint Retrospective’s effectiveness is diminished if one of the following applies:
  • the Sprint Retrospective is not held at all or not held at the end of each Sprint;
  • not all Development Team members participate in the Sprint Retrospective;
  • people are not free to talk about any aspect of the development process or input of certain people gets judged higher than the input of others;
  • the Team fails to decide concrete, measurable improvements for identified problems or their realization is not tracked and inspected in the next Sprint.
I think it’s also important to realize that as with all human interactions within a Scrum process, Scrum is organized democratically with respect to the individual, not to the majority. Every single individual’s opinion must be respected and may lead to adaptations. This is important for as soon as someone’s opinion gets ignored, he will not be able to take responsibility for what others decide which impairs the all-important self-organization of the Team. It’s of course difficult to find solutions which please everyone, but a Team which works as a unit will eventually find compromises on which every single Team member can agree. If your Team fails to do so, well, you’re actually in trouble, and Scrum reveals this.

Again, these aspects are typically neglected in non-agile teams, leading to poor collaboration within the team and diminished motivation of individuals.

What if your team fails to find common agreements for aspects of the development process? In a Scrum-driven process, this must not be ignored. Eventually, you will have to come to a decision. The team must work as a unit, otherwise, Scrum is no more applicable. You must take actions accordingly.

Conclusion

Being an intrusive framework, Scrum’s aim is not to provide concrete solutions to problems in your processes, but to show what is going wrong. Scrum does that by promoting and facilitating communication (both team-internally and with the customer) and transparency. But in order for Scrum to succeed, your team and organization must be ready to open up for Scrum’s processes and to embrace evolution and adaptation.

Scrum is an inconvenient framework in that it just doesn’t allow you to hide away any problems in your organization; if you try to do so, you will break Scrum’s processes, rendering it useless or even harmful.

In my opinion, organizations who fail to or are straight unwilling to properly apply Scrum where its application would be appropriate are typically suspicious. It oftentimes turns out that they try to modify the framework in order to hide what they think should stay unrevealed.

However, this of course is the worst thing you can do. Scrum is an industry-proven framework which works as a whole. Changing parts of it without knowing its details is really like introducing your own process framework without actually knowing what it does. You must either introduce Scrum as a whole or otherwise recognize that it doesn’t fit your organization and head for a better-suited process framework. Unfortunately, if Scrum’s inapplicability is caused by underlying problems concerning team hierarchy or communication you will most likely not find any “magic” framework which fixes these problems for you; at most, you will find one which hides them away in the short run.

Applying Scrum just because it’s fancy to do so or because it does seemingly involve less overhead than traditional process management methodologies without understanding its purpose or ignoring the changes it is introducing will leave you with an ill-suited, dysfunctional and poorly defined process with will harm the team’s efficiency and the quality of the product outcome. When you thus make things worse rather than improving them, agility becomes “ugility”.

Do you agree with these Scrum anti-patterns? Have you experienced others? Let me know in the comments section below – I’m highly interested in your experience.


Pages: 1 2

Scrum: 8 signs that you’re doing it wrong (part 1 of 2)



Holding a 15-minute meeting every day is not enough to make your team agile! Throughout my career, I’ve seen many Scrum anti-patterns applied, ultimately leading to rendering Scrum less effective or even counter-productive. In this blog post, I’d like to share some of the most dramatic anti-Scrum “smells” I’ve encountered so far. If any of these resemble your project situation, you should quickly take counter-action as these situations will not lead you to agility, but to what I like to call “ugility”.


After all, the Scrum Guide consists of just 16 pages. That is Scrum, and anything else is not. Is it actually that difficult to follow this recipe? Yes, because “Scrum is easy to learn, but difficult to master.”

By promoting transparency, collaboration and fast feedback loops, Scrum is a perfect match for development processes with high risks and high uncertainties. This typically applies to any custom software development process. Still, Scrum doesn’t fit every situation. For e.g. a pure “bug fixing project”, other, well-defined process models may actually be more appropriate. Here’s the catch: Failing to apply Scrum or any agile process on a project which is unmanageable by traditional process frameworks may break the project, but improperly applying Scrum by lack of understanding its principles may make the situation even worse.

Here I’ve congregated eight easily identifiable Scrum anti-patterns as I have observed them in real-world projects. I’ll try to explain what is wrong about them, why it is wrong and what you can do about it.

The Daily Scrum is conducted by the boss

Intriguingly, I have found that whenever a team introduces Scrum, the Daily Scrum is typically always held, however incomplete Scrum’s overall application is. Nevertheless, the way the Daily Scrum is held also reveals much about how well Scrum and its underlying principles are understood.

The most wide-spread misapplication of the Daily Scrum is to have it conducted by a “boss”. A boss may here refer to one or multiple persons, which may be the “Scrum Master”, a management team member or have any other role. Typical symptoms of the Daily Scrum being conducted by a boss are:
  • the Daily Scrum is held by the boss’s discretion or at his explicit invitation of the other team members;
  • the Daily Scrum is not held or significantly shorter if the boss is not present;
  • the Team Members do not communicate with each other, but report to the boss.
This is not what the Daily Scrum is all about. The Daily Scrum provides an opportunity to transparently discuss the current status of each Team Member’s work with the rest of the team and to synchronize the Team’s knowledge. Abusing the Daily Scrum as a pure reporting platform to the boss defeats its purpose and is a certain time-waster. This kind of one-way reporting can easily be replaced by a daily status e-mail to the boss.

Teams that misinterpret the purpose of the Daily Scrum typically lack understanding of two of the most fundamental objectives of Scrum:
  • Reporting to a boss implies that there is a single instance in control and in charge which violates Scrum’s philosophy: Scrum’s strength really comes from the entire Team being in charge and working together as a self-organizing unit. Having a dedicated person in charge for anything impairs the team’s self-organization and thus its ability to perform work.
  • Reporting to a boss rather than communicating with the entire team also implies that the team’s internal communication is impaired and information is held by a single instance rather than by the team in its entirety. Having the team cut off from some kind of information prevents it from taking responsibility and performing its work and introduces a huge risk to the project.
In the worst case, the boss will even use the Daily Scrum to delegate tasks to team members rather than having them collaborate in a self-organizing manner. This clearly totally defeats the purpose of Scrum.

What I’m trying to show here is that in any of these situations, the fact of rendering the Daily Scrum less effective is far less problematic than the potential underlying misconceptions of the Scrum process. This situation needs immediate action in terms of a root cause analysis: Find out why your team needs a single person in charge, and why information doesn’t flow freely within the team. If you’re bold enough, suggest that the boss doesn’t participate in the Daily Scrum anymore, or that it is conducted by another person to mitigate the boss’s influence. Observe how these changes affect execution of the Daily Scrum and overall team collaboration.

If either of the two underlying misconceptions of Scrum mentioned above apply to your project, this is indeed a very dangerous situation, and trying to resolve it, or rather to fight it, may break your team if it turns out that the “boss” or management is actually unwilling or unable to accept a transparent, team-centered project workflow.

There’s no dedicated Product Owner

The Product Owner is one of three Team member roles established by Scrum; yet in my experience many teams seem to have trouble filling this position adequately or they really just view the role as somewhat “optional”. Well, it is not.

A similar situation is that a Product Owner is actually set but the person this role is assigned to is unwilling or unable to fulfill it or just doesn’t have enough time. Another common anti-pattern, and probably the most dangerous one, is that the Product Owner is the Team’s “boss” or the Scrum Master.

Indications that the Product Owner’s position is fills inadequately include:
  • the Product Owner does not take part of every event where its participation is required (e.g. the Sprint Planning);
  • the Product Owner is unable to make a final commitment when giving input, e.g. it’s common that his decisions get revised at a later point.
As his name suggests, the Product Owner is in charge of specifying the product which is the outcome of the Scrum project, by making decisions on the Product Backlog. It is vitally important that he has both the ability and authority to do so and to not be conflicted with other charges.

Scrum is all about collaboration, and the most important form of collaboration in a software project happens between the developers and the client which in Scrum is established by the collaboration between Development Team and Product Owner. The Product Owner cannot make technical decisions nor can he make sensible business decisions without the Development Team’s input on the technical aspects. On the other hand, the Development Team cannot make business decisions nor can it make sensible technical decisions without the Product Owner’s input on business aspects.

Finding a Product Owner which fulfills above criteria is not simple and it’s one of the most important aspects of initially setting up a Scrum project. Not having an able Product Owner dooms the project. Typically, improper Product Owners include:
  • People who do not have the required business knowledge of the product’s application domain. This typically applies to any person which is not part of the department or even the company which will use or business-support the product.
  • People who do not have absolute authority over the product.
  • People who have conflicting interests within the development process. This typically includes any person which is part of the Development Team or even the company which develops the product.
If the Product Owner is recruited from within the development company (e.g. a business analyst), because the actual client company reputedly cannot provide a Product Owner, all these points typically apply.

In the extreme case of the Product Owner being the Development’s team’s boss (he may claim himself Scrum Master although in this position, he clearly isn’t) the conflict of interest is obvious. Taking both these roles, he has absolute control over what is done, and how it is done, leaving developers no choice but to obey his will. This of course is the complete opposite of a self-organizing team.

The quest to find an appropriate Product Owner must have succeeded before the project even begins. The most important criteria typically is the Product Owner’s authority. If he has to ask his boss for whatever business decision, this is a strong indication that actually, his boss should be the Product Owner. In fact, multiple persons can internally make business decisions, but for the Scrum process it’s important that there is only one single definite “voice” giving input to the Development Team. The customer may literally send anyone as the “Mouth of Sauron”.

If the quest to find an appropriate Product Owner does not succeed, the project is doomed, at least as far as Scrum’s applicability is concerned. Yes, Scrum is not applicable for any conceivable situation. Actually, it does define some fundamental constraints to its applicability, and having the central role of the Product Owner filled is one of them. You can then only resort to going for a non-agile approach or to teach the customer that without a Product Owner, you’re not able to do your job. That’s professionalism in action.

Developers aren’t in direct contact with the Product Owner

This is a special case of the Product Owner not fulfilling its role or being impaired (e.g. by management) to do so. In this case, the Product Owner is not or scarcely in direct contact with the Development Team. Most typically, in this situation, there’s a “middle man”, e.g. the Development Team’s “boss”, the Scrum Master, a business analyst or a key account manager which acts as a proxy between the Development Team and the Product Owner.

This is a very clear Scrum anti-pattern. Having a “proxy” between Development Team and Product Owner doesn’t add any value, but instead, weakens communication, mutual understanding and collaboration efficiency, and is overall harmful to the Scrum process. Management may introduce this anti-pattern by lack of understanding of Scrum’s principles or by lack of confidence in having the developers communicate directly with the customer, or because it would force adaptations in the team structure, e.g. re-evaluation of the role of a business analyst.

In Scrum, having direct contact between the Development Team and Product Owner is recognized as being essential to build mutual trust and understanding. If a developer doesn’t know the customer, how is he supposed to understand his requirements? Scrum promotes and facilitates Dev-PO-communication by providing recurrent formal events for meetups.

If your team has any of above situations in place, i.e. a “proxy” of any kind between developers and customer, the team must take immediate action to change that. If the situation is caused by management’s restraint to have direct developer-customer communication, this will be tricky to solve. However, if Scrum events are held, you can use these as an opportunity to introduce the Product Owner more informally in the development process. Having these events not conducted by a “boss”, but executed in a self-organizational manner, may help the Product Owner understand that he would profit most if he was able to freely communicate directly with the people who actually build his stuff.

Development Team members have any role other than Developer

A Developer is a Developer is a Developer. No other roles or titles are recognized for any member of the Development Team. There is no “boss”, no “architect”, no “business analyst”, and no “senior programmer”. This is very different to how “traditional” development teams are organized, that is hierarchical and very ill-suited for software development.

Having team members with distinct functions centralizes knowledge, authority and responsibility which seriously diminishes the team’s ability to form a self-organizing, interdisciplinary unit which is absolutely essential to Scrum. If for instance, there was a single “software architect” in the team which is in charge of making “fundamental technology decisions”, e.g. which database to use, why should any other team member feel responsible for that decision? In this very moment, the team as a unit died.

The Scrum Guide explicitly prohibits any Developer role except Developer. This forces a Scrum Team to share knowledge and responsibility. There’s no single point of failure, and there are no silos. Everyone is kept accountable for the work of the entire team. It’s important to realize that Scrum still recognizes people’s individual strengths. Through the self-organizing nature of a Scrum Team, each individual will bring in its own strength, resulting in diversified task assignments. Or, as it is put in this stackoverflow answer, “a "lead" developer will continue "leading," but as a natural consequence of their (sic!) experience and not as a mandated function of their title“.

Breaking a traditional hierarchy is one of the most difficult things when introducing Scrum. I’ve seen it succeed in some teams which worked together disregarding any existing hierarchies, and fail in others who were unable to adapt to the new thinking. Also from above stackoverflow answer: “That's in an ideal world where people can put their egos aside. Good luck!” The most sensible advise I could give here is to educate people on how true collaboration will increase productivity and efficiency. Having people in the team which are unable or unwilling to collaborate will harm the team’s overall performance and may introduce technical debt which is well worse than not having their manpower in the team at all.

Pages: 1 2

July 12, 2015

Using HTML5 right now: Web components by example (part 2 of 2)

Native HTML5 JavaScript

When targeting browsers with native HTML5 custom components support only, there is no need to import 3rd party JavaScript polyfill shims. You can just rely on the browser’s native JavaScript API to implement custom elements.

Check availability

First of all, you want to check whether the user’s browser supports native HTML5 custom components. This JavaScript code does that:
if (!document.registerElement) {
    alert('Custom elements not supported!');
}
else {
    // build custom elements here
}
When running in a vanilla Firefox 39 installation, the alert will pop up. By enabling experimental HTML5 custom elements support with about:config setting
dom.webcomponents.enabled = true
or when using Chrome 36+, the alert won’t show up indicating native support is provided.

Implementation

Now we are ready to implement our custom element! This is all the JavaScript code we need:
var MyInfoboxProto = Object.create(HTMLElement.prototype);

MyInfoboxProto.createdCallback = function() {
    var stars = this.getAttribute('stars') != null ? this.getAttribute('stars') : 0;
    var name = this.getAttribute('name') != null ? this.getAttribute('name') : 'Rating';
    this.innerHTML = "<span class='my-infobox my-rating" + stars + "'>" + 
        name + "<i class='fa fa-lg'></i></span>";
};

document.registerElement('my-infobox', {
    prototype: MyInfoboxProto
});
We can then use the custom element in HTML code, as in
<my-infobox stars="2"></my-infobox>
which renders
Rating
There are a few important things happening in this JavaScript code which enable the magic:
  • var MyElementProto = Object.create(HTMLElement.prototype); lets you derive your custom element from the base HTML element’s prototype. You can actually use any existing HTML element (tag) as a base for your custom element.
  • Building the component usually happens in the createdCallback function. There are three more callback functions available: attachedCallback, detachedCallback, attributeChangedCallback(attrName, oldVal, newVal).
  • You can manipulate any property of your element’s prototype in the callback. The most concise way to update the rendered HTML code certainly is to set this.innerHTML.
  • With this.getAttribute('name'), you can retrieve a tag instance’s attribute value. In the example, we check whether they are set and use a sensible default value otherwise.
  • The actual registration happens in document.registerElement(…). Note that a custom tag name must include a dash character (-), e.g. my-element.
That’s all we need for this example! You may want to take a look at the complete example code on GitHub.

For more information about how to write custom elements, check out these useful tutorials as well:

webcomponents.js

Installation

If you happen to work in a Node.js infrastructure, you can use the bower dependency manager to install the required JS file. Otherwise, the simplest way to download the file I found was to download the ZIPed distribution from the Polymer project which contains a minified webcomponents-lite.min.js file. This is the only file we need for pure webcomponents.js.

Implementation

As I briefly explained earlier, webcomponents.js is a fully standard-compliant shim for the HTML5 web component JavaScript API. This means that you can write the same JavaScript code and it will work either with native browser support present, or with the JavaScript shim fallback.

Try it out! Link the webcomponents.js script file in the “native” implementation:
<script src="webcomponents-lite.min.js"></script> 
In Firefox, you can then disable experimental support for HTML5 custom components with
dom.webcomponents.enabled = true
the “not supported” alert will never pop up, and the custom element will perfectly work.

Of course, you will then remove the support test as it is performed by the shim already.

Click here to see the complete source code on GitHub.

Polymer

Installation

Again, use the bower dependency manager to install the required JS files. Or download the ZIP from the Polymer homepage. For custom element, we just need to link to the polymer.min.js and webcomponents.min.js files in our HTML page.

Implementation

With Polymer, you can define custom elements either in JavaScript, similar to how it works in plain HTML5 code, or you can do it the declarative way using specialized HTML tags. For illustration purposes, I will here use this second approach.

The complete solution reads as follows. This Polymer element goes in the HTML page’s <header>:
<polymer-element name="my-infobox" attributes="name stars">
    <template>
        <link rel="stylesheet" href="mystyle.css">
        <link rel="stylesheet" 
            href="https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
        <span class="my-infobox my-rating{{stars}}">{{name}}<i class="fa fa-lg"></i></span>
    </template>
    <script>
        Polymer({
            name: "Rating",
            stars: 0
        });
    </script>
</polymer-element>
We can then use the custom element in HTML code, as in
<my-infobox stars="2"></my-infobox>
which renders
Rating
What is happening in the Polymer code?
  • The <polymer-element> umbrella tag specifies the element with the name provided and “publishes” an arbitrary number of attributes for use with the tag.
  • The <template> sub-tag defines the rendered content. Note that we have to link to all the required external files, e.g. stylesheets. Having them in the containing HTML file itself will not suffice! What’s strange is that apparently, the link to mystyle.css could then be completely removed from the containing HTML file whereas the link to font-awesome.min.css must be present in both the containing HTML file and the polymer-element’s template section in order to render correctly. I don’t understand that yet.
  • Most importantly, in the <template> sub-tag, we can refer to the element’s attributes by putting them in double angle brackets (handlesbars style).
  • The <script> sub-tag allows us to define default values for the element’s attributes.
If you go for this declarative approach, you’ll have to include your custom elements in other HTML files by using HTML imports, rather than <script> links:
<link rel="import" href="elements/my-element.html">
Note that this does not work when loading from file://, thus it’s not an option for static HTML resources.

Of course, Polymer works either with native browser support or by falling back on the webcomponents.min.js shim. In the latter case, however, performance impact is clearly noticeable. Because the declarative <polymer-element> solution works only with the fourth part of the web components specification, Shadom DOM, enabled, we must use the full-blown webcomponents.min.js rather than the “lite” version. Also note that you must link to that JS file first, before linking to the polymer.min.js.

Click here to see the complete source code on GitHub. 

X-Tag

Installation

The required x-tag-core.min.js file is available in the X-Tag core GitHub repository.

Implementation

X-Tag supports imperative JavaScript-based custom element creation only. The source code will thus typically look like a more simplified version of vanilla HTML5 custom element code.

This is the complete solution:
xtag.register('my-infobox', {
    lifecycle: {
        created: function() {
            var name = this.getAttribute('name') != null ? this.getAttribute('name') : 'Rating';
            var stars = this.getAttribute('stars') != null ? this.getAttribute('stars') : 0;
            this.innerHTML = "<span class='my-infobox my-rating" + stars + "'>" + 
                name + "<i class='fa fa-lg'></i></span>";
        }
    }
});
The interesting part here is that
  • xtag.register registers the element in question.
  • Although the second argument to the register function (here: {lifecyle: …}) accepts a content element to define the rendered markup as a plain String, I don’t think this works with integrating the element’s attribute values. There is just no templating mechanism as provided by Polymer. Thus I included the attribute values the imperative way.
Again, X-Tag either uses built-in native web components support or uses its internal JavaScript fallbacks. Performance impact seems hardly noticeable.

Click here to see the complete source code on GitHub.

Bosonic

The Bosonic Polyfill offers declarative API similar to the one offered by Polymer.

Because it doesn’t seem to offer more features than the other polyfill libraries, and because it doesn’t stand out particularly when looking at its file size or its project health, I decided to not give it a closer look here.

document-register-element

Installation

The required document-register-element.js file (minified) is available in the GitHub repository.

Implementation

document-register-element.js uses an imperative syntax very similar to the one used by X-Tag to register custom elements:
document.registerElement(
    'my-infobox', {
    prototype: Object.create(
        HTMLElement.prototype, {
            createdCallback: {value: function() {
                var name = this.getAttribute('name') != null ? this.getAttribute('name') : 'Rating';
                var stars = this.getAttribute('stars') != null ? this.getAttribute('stars') : 0;
                this.innerHTML = "<span class='my-infobox my-rating" + stars + "'>" + 
                    name + "<i class='fa fa-lg'></i></span>";
            }
        }
    })
});
At this point, I don’t think I have to explain this code any further.

The true miracle about this polyfill is that with its tiny 7 KB footprint it supports even more browsers than the original webcomponents.js. Note however that this polyfill really only contains the custom elements part of the web components specification!

For pure custom elements however, it works great with both native support and its own fallback without noticeable performance impact.

Click here to see the complete source code on GitHub. 

There’s more

You may also want to take a look at the other three parts of the web components specification. Some custom elements polyfills, most prominently Polymer, have additional support to facilitate usage of that functionality as well. Especially the Shadow DOM specification nicely interplays with custom element as it allows you to encapsulate or opt-out DOM elements on the fly. I will potentially publish another post about these topics at some later point.

Also, the pure existence of these custom element generators makes it possible to actually create business-case agnostic reusable components. Thus alongside these polyfill libraries, custom element repositories have emerged, from which you can take any custom element of your choice and include it in your HTML pages. This really enables a whole new level of native HTML reusability. Some of these catalogues are featured on the webcomponents.org homepage in the “Discover” section – go have a look!

Conclusion

I was surprised to find a very diversified exosystem all up and running, implementing what is commonly still considered the “web technology of the future”. Unfortunately, this diversified ecosystem looks somewhat confusing at first sight. But after taking a closer look, I am now convinced that this not only has great potential for the near future, but is also very useful right now! Apart from simple, static HTML pages (such as this blog), I think one should seriously consider including a web component abstraction layer in a HTML project, and most definitely when targeting modern browsers exclusively. This is especially true for HTML-heavy client side web application which do not already include component-based 3rd party frameworks (as e.g. AngularJS).

The true question, in my opinion, is thus not whether to include a web component solution, but which library to use for it. From what I have tried out so far, I came to this conclusion:
  • If for whatever reason you are lucky enough to target Chrome only, I would consider native web component development with webcomponents.js as a fallback just in case.
  • If you’re planning to use web components excessively, you should consider going with the Polymer library. Its declarative approach and the handlesbars-like inline expressions will certainly increase maintainability even with a large code base. Due to noticeable performance impacts, this should target browsers with native web components support only. This solution is clearly targeted for heavy client-side solutions.
  • If, on the other hand, you just want to use web components sparingly or if you have strong responsiveness constraints, you should really consider the document-register-element library. The only disadvantage of this solution is that you lose standard API compatibility.
Please let me know in the comments section if this article was helpful for you or whether it lacks some important information. Given that I don’t receive too many discouraging comments, I will consider turning this into a whole HTML5-themed article series…


Pages: 1 2

Using HTML5 right now: Web components by example (part 1 of 2)



In this article I’ll cover one of the most exciting and most useful parts of upcoming HTML5 - web components, and show you how to use them today already. This article features a complete example implementation using native HTML5 as well as some of the most prominent polyfill extensions.

DRY – don’t repeat yourself – is one of the most important clean code principles. Even the most simple, procedural programming languages typically support DRY compliant design, e.g. by encapsulating reusable code in functions. Technologies which abstract HTML views typically support the definition of reusable components as well; e.g. JSF features the so-called composite components, and AngularJS features so-called directives. With HTML5, creating custom reusable components will finally be natively supported by every implementing browser. Enter web components.

Its place in the HTML5 ecosystem

Web components are just one part of a multitude of features which are introduced in the HTML5 specification, the successor of HTML4 which is currently implemented by all major web browser engines. What is commonly referred to as “web components” are actually four different parts of the HTML5 specification:
For a basic overview about custom components, see webcomponents.org. Its resources section contains many useful artifacts such as hello world repositories and tooling support.

For an overview of current browser support, see Are We Componentized Yet?

In this article, I will concentrate on the central custom elements part of the specification.

An example problem with HTML4

For illustration purposes, I’m going to present a simple demo application of a web component. Even though the resulting HTML page can be perfectly created using pre-HTML5 technology only, it’s reusability would be clearly diminished, leading to impaired maintainability.

I didn’t have to look too far to find an example application of web components – there’s actually a good little use case on my blog. Consider the small “rating” widget I created for a blog post’s header section:
Rating
If we consider this widget as a component, it has exactly two variables:
  • The name
  • The number of stars
Everything else is the same and thus reusable. Repeating anything but these two variables in the source code would violate DRY. Let’s see how be can increase reusability with HTML prior to version 5.

Naïve, pure HTML & CSS

This is the first attempt:
<span class="my-infobox">Rating
    <i class="fa fa-lg fa-star"></i>
    <i class="fa fa-lg fa-star"></i>
    <i class="fa fa-lg fa-star-o"></i>
</span>
This clearly leaves much to be desired.
  • Good: The overall styling can be done in a central CSS file which will refer to the "my-infobox" class.
  • Bad: The overall syntax to include a rating is very bloated and requires repetition when used multiple times.
  • Bad: The “stars” must be placed imperatively (here, the stars are generated using the font-awesome vector icons library). This is error-prone and obfuscates the code.

More CSS

We could slightly improve readability like this:
<span class="my-infobox my-rating2">Rating<i class="fa fa-lg"></i></span>
Here, we define additional CSS rules: my-rating0my-rating3 which dynamically render the respective number of “stars”. Note that this is only possible with the somewhat hackish ::before and ::after selectors and by disrespecting font-awesome’s best practices to refer to an icon by its class name rather than by its character number. For instance, this generates the rating-2-stars:
.my-rating2>i::before {
    content: "\f005\f005"
}
.my-rating2>i::after {
    content: "\f006"
}
Unfortuantely, the basic problems remain:
  • The overall syntax to include a rating is still very bloated.
  • We cannot get rid of CSS classes declarations because our only way of abstraction is using CSS, and CSS declarations cannot refer to CSS classes; e.g. to get rid of "fa fa-lg", you’d have to extract the "fa fa-lg" class’s declarations and include it in a .my-infobox>i rule which of course would violate DRY as well.

The desired solution

What I’d like to have is a declarative, business case-driven custom HTML tag with a minimum set of busines-driven attributes, and with default values for attributes, like this:
<my-infobox stars="2"></my-infobox>
Well, this is exactly what we will end with using web components.

Library jungle or What the hell is a shiv?

Vanilla JavaScript for HTML5-compatible Browsers

If you target only browsers which natively support HTML5 web components (these are Chrome 36+ and Opera 30+ at the moment) you don’t need any external library at all. The JavaScript API functions required to implement web components are shipped with the browser.

Of course, this is the ideal situation associated with the least development setup effort and, more importantly, with the least deployment / bandwidth / runtime overhead. However, this will not apply to most projects which let the user choose any browser to his liking, however crappy it might have been programmed. In this situation, you have no choice but to fake browser support for web components by introducing a more or less thin JavaScript abstraction layer which provides the features missing in the browser core:

Fallback for older browsers: Polyfills shims / shivs

There are actually a couple of JavaScript libraries which emulate native support for HTML5 web components. They are all able to detect native browser support, and provide a fallback in case there is no native support.

These libraries are referred to as polyfills, and are an HTML5-specific implementation of what is known as a shim (literally a “washer”), which is sometimes also called a shiv. A shim or a shiv is just the general technical term for an API which retrofits an updated implementation into a legacy system; in the case of polyfills, it’s HTML custom elements retrofitted into legacy browsers.

The most basic polyfill available is webcomponents.js which is implemented strictly according to the W3C’s web components specification. The JavaScript API provided with this polyfill thus matches exactly the browser’s native JavaScript API, but it also comes with the most amount of boilerplate code.

All the other available polyfill libraries either build on top of webcomponents.js or recreated it and added additional functionality, mostly with the aim to reduce boilerplate code and increase overall ease of development.

Overview

Here is an overview of some of the most widely-used polyfill libraries.

Library C I T S IE FF Safari Andr iOS size Project health Download
(native) x x x x - - - 4.4.4 - - Implementation on hold in IE, FF, Safari -
webcomponents.js x x x x 10 x 7 x x 113 KB active,
23 releases,
35 contributors
bower / .js file
webcomponents-lite.js x x x 10 x 7 x x 36 KB active,
23 releases,
35 contributors
.js file
Polymer (based on webcomponents.js) x x x x x 7 x x 234 KB active,
63 releases,
55 contributors
.js file
X-Tag x x - - 9 5 4 2.1 ? 15 KB active,
70 releases,
10 contributors
.js file
Bosonic x x x x 9 25 6 ? ? 27 KB active,
0 releases,
4 contributors
bower / .js file (unminified)
document-register-element x - - - 8 x x 2.2 5.1 6 KB active,
33 releases,
3 contributors
.js file

All information is up to date as of the writing of this article (July 2015). C: Custom elements, I: HTML imports, T: Template, S: Shadow DOM. For polyfill libraries, compatibility data is taken from the respective project’s homepage. If there was no explicit version information, the column is simply marked with an “x”. The file size has been ascertained empirically.

On the next page, we will implement the example using the native JavaScript API as well as some of the most widely-used polyfill libraries.

Pages: 1 2