Inverno Framework performance benchmark
In my introduction to the Inverno Framework, I discussed how performance has become more and more important following the Cloud transformation. Indeed, in order to reduce operational cost, modern applications MUST be able to support very high throughput while optimizing the usage of hardware resources.
Following the end of the first development phase, which included core functionalities such as IoC/DI, configuration module, HTTP server module, Web development modules and a RDBMS client module, I have submitted Inverno Framework to TechEmpower Framework Benchmarks in order to see where Inverno stands in comparison to other Web applications frameworks. Results are not yet official and should appear in the next round, but we can already analyze unofficial results.
The objective of the Inverno Framework has always been to provide the best possible platform for modern Java application development, as a result, it is important to assess performance with that in mind. I usually say that performance is a multi-faceted topic that cannot be reduced to the single raw performance. If speed were the only criteria, we would all drive rocket cars, which would be to ignore practicality, safety, environment and the fact that not everybody could or should drive such car. It is therefore important to put things into perspective while analyzing raw performance. Especially since benchmark rules allow for many optimizations that are sometimes inappropriate when developing actual applications.
So let’s have a look at the benchmarks as of 2022–02–21 and see how the Inverno Framework compares to the competition.
I will first consider direct main competitors which are all Java based and well-established actors of the industry, then I will widen the analysis to all JVM based frameworks and finally to all frameworks regardless of the underlying technology.
I have identified the following frameworks as Inverno Framework main competitors:
- Spring Framework which has become the defacto standard in enterprise application development over the past 15 years.
- Quarkus which is the RedHat response to Spring Boot following the end of JEE.
- Micronaut which is a new growing application framework for modern application development.
- Vert.x which has become a reference in the field of Java performance.
- Netty which is not actually an application framework but upon which many frameworks, including Inverno and Vertx, are based for low level I/O and threading model.
Here are the Composite results:
Note that Netty doesn’t appear in that list as it does not implement DB tests. Its scores for JSON and Plaintext tests are 1,094,575tps and 4,028,882tps respectively.
Based on these results, we can identify two groups: a first group composed of Vert.x, Inverno and Quarkus (and Netty), and a second group with Spring and Micronaut. Difference between these groups is huge and both Spring and Micronaut do not resist comparison with the others. For instance, considering raw HTTP performance, Inverno does 2274% better than Spring.
We should note that the Spring boot application used in these tests is not using Spring WebFlux which would probably increase performance but probably insufficient to catch up on both raw HTTP performance and DB access performance since Spring still relies on JDBC which is not using non-blocking I/O and does not take advantage of pipelining.
In the first group, we can see that the difference between Inverno and Vert.x is tight, Vert.x has better DB access performance while Inverno has better raw HTTP performance. Quarkus completes this trio with slightly lower performance. These results are not very surprising considering that both Vert.x and Inverno are based on Netty for low level network I/O, Inverno relies on Vert.x SQL client for DB access and Quarkus is using Vert.x for both HTTP server and DB access. The difference in DB access performance between Inverno and Vert.x can be explained by the overhead brought by the integration of Vert.x SQL client in order to make it reactive and consistent with the rest of the framework.
It is interesting though to note that Inverno’s HTTP server performs extremely well, 11% better than Vert.x and even better than Netty.
Now let’s broaden the scope and consider all JVM based frameworks, including Java, Kotlin and Scala.
Inverno appears in third position behind Vert.x and Officefloor, Jooby appears fourth, and we find Quarkus in fifth position. Further down the list, gaps widen a bit, but we still have quite performant frameworks such as Greenlightning which, like Officefloor, implements its own network stack.
Officefloor appears to be the most performant, with impressive raw HTTP performance and DB access performance comparable to Vert.x which is reasonable as it uses Vert.x PgClient for DB access. However, if we look at the benchmarked application itself, the amount of optimizations and specific configurations used to obtain those results on each test is much higher than for the others. It also doesn’t use a template engine which allows better optimization and can explain the good score on the fortunes test. As a result I’m not sure that we would get the same results with a regular application even if they should remain quite high. Finally, in order to get such raw HTTP performance, it uses its own HTTP server which does not seem to support HTTP/2 protocol and that is not ideal when developing an actual Web application.
As noticed earlier, The gap between Vert.x, Inverno and Jooby is very tight, which is quite understandable given that these frameworks are all based on a reactive threading model (using either Netty or Undertow) and use Vert.x PgClient for DB access. Quarkus lies behind with lower raw HTTP performance, probably due to Vert.x integration overhead. If we now look at the code, we can see that all benchmarked projects use reasonable optimization and are close to what we would see in a regular application. In order to be completely fair, I must recognize that unlike Jooby and Quarkus, both Vert.x and Inverno are routing requests using a simple switch. This is an honest optimization that is made possible by the architecture of these frameworks which provide plugable Web routers. All these frameworks support HTTP compression, HTTP/2 protocol and provide specific features to create REST APIs which qualifies them for Web application development.
Some frameworks do not appear in the Composite Frameworks Scores because they don’t implement all tests, however they perform extremely well and should also be considered. Let’s see how Inverno stands when considering each test individually:
We can see that Inverno is in the top 5 in all DB access tests. It is also interesting to notice that all frameworks in this top five use Vert.x PgClient. Considering raw HTTP performance, Inverno appears in the top 15 (after removing duplicates). None of the frameworks ahead are based on Netty which makes Inverno the fastest Netty-based HTTP server. We can also note that, among these, only Proteus, which is based on Undertow, seems to support HTTP/2. That being said, HTTP performance of frameworks like Redkale, Activej or Rapidoid remains impressive.
Finally, let’s have a look at the Composite Framework Scores regardless of the technology.
Inverno is in twelfth position overall (out of 120) right behind Vert.x. Unsurprisingly, we find mostly C/C++ and Rust based frameworks at the top.
Take away
With a global performance comparable to Vert.x, the Inverno Framework is exactly where I wanted it to be, it is one of the most performant Java Web application framework in the world, outperforming industry standards like Spring or Quarkus.
The choice of Netty as network I/O framework made it possible to create a performant and stable HTTP server supporting HTTP compression, HTTP/1.1 and HTTP/2 protocols, and facilitating the integration of other Netty-based libraries such as Vert.x SQL client or Lettuce. The Inverno Framework appears to be the most performant Netty-based HTTP server, however, when it comes to raw HTTP performance, frameworks based on specific network I/O libraries can be much more performant which indicates that there might be room for improvement for Netty.
It is also important to point out that performance, while being crucial, is not the main objective of the Inverno Framework which has been specifically designed for modern application development. It embraces reactive and modular programming and provides features like compile-time IoC/DI, advanced configuration, programmatic and declarative Web routes definition, RDBMS and Redis clients, a reactive template engine and build tools for building and packaging applications for the Cloud. All this together produces a smooth and efficient development experience to create fast modular maintainable cloud-native applications.
