Developer’s Corner: Achieving Extreme GeoServer Scalability with the new Marlin vector rasterizer

Questa è la didascalia
As you probably know, GeoServer is a Java based software, so its performance is heavily dependent on what the Java Virtual Machine and the Java standard library can offer.
Over the years the GeoServer developers have been pushing the boundaries of what can be done by replacing portions of the Java standard libraries by adopting faster replacements, for example, the turbo jpeg encoder, the native PNG encoder available in ImageIO native extensions, or the recent adoption of the pure java, high performance PNGJ library.
One of the Achilles’ heel that so far we failed to replace is the anti-aliased rasterizer contained in the JDK, that is, the portion of software that turns all vector data into its raster representation while painting maps. But… things are about to change: sit back and get some popcorn!

The choice in Java rasterizers 

So far we were left with two choices, depending on the JDK adopted:

  • The Oracle JDK comes with the Ductus rasterizer, a closed source, native antialiased rasterizer that has serious scalability issues, since it allows only a single shape at a time to be rasterized. Quoting a Oracle engineer: “One of (Ductus) drawbacks is that it had been designed to take advantage of some AA-accelerating hardware that never came to be.  With the accelerator it would have been insanely fast, but hardware went in a different direction.  The problem was that this early design goal caused the entire library to be built around an abstraction layer that allowed for a single “tile producer” internally (because there would be only one – insanely fast – hardware chip available) and the software version of the abstraction layer thus had a lot of native static data structures (there’s only one of me, right?) that prevented MT access. “
    Long story short, while it provides good performance for the single threaded cases, it’s not really thought out for server side usage
  • The OpenJDK comes with the Pisces rasterizer, a open source, pure java rasterizer that has no scalability issues, but that unfortunately is quite a bit slower than Ductus, it takes at least 4 concurrent requests for it to become faster than Ductus. Another issue is that OpenJDK is only available for Linux, so if you’re running a Windows server, you’re out of luck.

A new kid on the block
One not well known detail about the rasterizers is that Oracle, in order to be able to have different implementations depending on the JDK binary build, made the rasterizer pluggable, so it is possible to build your own and use it, in both OpenJDK and Oracle JDK, by setting some startup variables.
But, of course, given that OpenJDK is open source, and open to contributions, it is also possible to provide improvements to it. Back in March 2013 Laurent Bourges contacted the Java “2d-dev” list to inquire about Pisces performance, and shortly after, he followed up with a set of patches to improve Pisces performance.  Andrea Aime followed with interest, and provided a compact way to replicate GeoServer performance issues by serializing the set of java2d commands used to paint the maps in our benchmarks, creating the “MapBench” tool, a system to save the list of commands on disk from a GeoServer, and then play them back as a synthetic benchmark that only deals with map rendering (without protocol handling, data reading, final png/jpeg encoding).
Laurent took the benchmark and started working against it to further improve Pisces (and MapBench itself, in the process).
While the various patches spurred some interested in Oracle employees, it became soon clear that getting them accepted in OpenJDK would have proven difficult, partly due to the OpenJDK 8 getting close to the feature freeze for its final release in 2014.
After a few months of hiatus we decided to move on with the development outside of the OpenJDK community, leveraging the ability to plug other implementation of sun.java2d.pipe.RenderingEngine, and setting up a true open source project to host the evolution of these Pisces patches: the Marlin-renderer was thus born, and it’s now being developed on GitHub for the benefit of all Java users, along with MapBench.
Laurent provided set of single and multi-threaded benchmarks based on MapBench and some map sets, a few easy on the renderer, others quite a bit more demanding. While the detailed results are available, let me just echo Laurent summary:
Marlin 0.3 is the winner:
– 60% faster than pisces for small workload; 300-400% faster for high workload
– 1 thread: 1% slower than ductus for small workload; 90% faster for high workload
– 4 thread: 90% faster than ductus for small workload; 400% faster for high workload
Some impressive speedups for sure, but how do they translate into speedup for a whole WMS GetMap request? Well, it depends a lot on how much time we actually spend rendering, instead of reading data, and encoding images out in some binary format (e.g. PNG, JPEG).
Here are a couple of examples.
The WMS 2010 performance shootout
The last FOSS4G WMS shootout took place in 2010, in Barcelona. Back at the time we were thinking GeoServer performance was badly affected by Ductus scalability issues, and back then, OpenJDK 6 graphic subsystem was definitely slower than what we have today in OpenJDK 7.
Now, we recently managed to get a rather significant speedup by using OpenJDK7 (so, Pisces) plus the new PNGJ based encoder. How much is Marlin adding to it?
While we were here, we prepared a fuller set of comparisons, comparing what we had back at the time, and what we have now:
  • We run again the benchmark with GeoServer  2.2-beta2 (what we used back in Barcelona) with the native ImageIO PNG encoder and JDK 6 (ok, a more recent version of JDK 6 than what we had back at the time, but we don’t expect it to be significantly faster than what was used back then)
  • Then we used GeoServer 2.5-beta (whose PNG encoder defaults to PNGJ now), with an Oracle JDK 7, with a OpenJDK7, and finally, OpenJDK 7 plus the Marlin 0.3 binary release.
Here are the detailed result, in requests per second, at the various load levels (from 1 to 64 concurrent clients):
Threads GS 2.2-beta2 + Oracle JDK 6 GS 2.5-beta + Oracle JDK 7 GS 2.5-beta + OpenJDK 7 GS 2.5-beta + OpenJDK 7 + Marlin 0.3 Speedup over 2.2-beta2 Speedup over 2.5-beta + OpenJDK 7
1 15,27 21,51 21,16 21,10 38% 0%
2 23,47 31,98 30,64 32,37 38% 6%
4 36,08 49,74 52,58 58,17 61% 11%
8 39,62 55,19 59,99 65,55 65% 9%
16 41,61 57,22 69,57 76,18 83% 10%
32 41,89 54,86 63,77 73,00 74% 14%
64 37,61 50,41 61,34 67,86 80% 11%
In summary:
  • For this particular benchmark, not very dependent on the raw rendering speed, the PNG encoder change occurred in GeoServer 2.5-beta provided a great benefit, as well as switching to OpenJDK to leverage the much improved Pisces scalability
  • Marlin still manages to speed up the results up to 14% compared to Pisces
  • Between the various changes in GeoServer and Marlin we improved a lot our performance compared to the 2010 results
A rendering heavy benchmark
While the above results are interesting, they don’t really do justice to Laurent excellent work. Luckily, during the Christmas break Jonathan Moules hit the GeoServer devel mailing list with a more interesting case.
The benchmark in question uses a real world map based on Ordinance Survey Strategi data and a set of SLDs that make for maps with lots of details… and when I say lots, I mean that some maps contain, even if they are hard to see, a few tens of thousands of little polygons (one might say, why don’t we use scale dependencies and avoid that? Well… because people often setup this kinds of maps).
As a further aggravation, the output images are quite a bit bigger than what was used in the WMS shootout, in particular, we are talking about 1200×1200 pixels, compare to the average size of the WMS shootout ones were on average 800×600, that is, 3 times smaller.
Jonathan setup is also different, in that it’s not a scalability benchmark, all runs use 10 concurrent clients, but show how the performance changes as we change the zoom level of the map instead, thus hitting different combinations of layers, information density and symbology used.
We have run that same benchmark in different combinations, using in all cases a single GeoServer 2.5 beta instance, but with different rasterizers:
  • Oracle 7 JDK with Ductus
  • OpenJDK with Pisces
  • OpenJDK with Marlin 0.3
Scale denominator Oracle JDK (Ductus) OpenJDK (PIsces) OpenJDK + Marlin 0.3 Speedup over Ductus Speedup over Pisces
10.000.000 11,92 10,83 13,14 10% 21%
5.000.000 9,66 8,16 11,47 19% 41%
2.500.000 8,21 6,71 11,53 40% 72%
1.250.000 4,59 4,06 8,99 96% 121%
625.000 2,96 3,06 7,21 144% 136%
300.000 4,09 4,77 9,99 144% 110%
150.000 6,63 8,25 13,05 97% 58%
In summary:
  • Marlin based tests are always the fastest ones
  • The harder to draw the map, the more information we have to render, the greater the benefit in using Marlin
Now, the loyal reader of this blog might ask, ok, but what if I setup multiple copies of GeoServer with Oracle JDK 7, each running in its own JVM, and load balance them, to mitigate the Ductus scalability issues? Well, the answer is that you’ll get a more complex setup, but one that is still 30% slower than what Marlin provides.
Moving forward
So, are you still sitting there? Please go to the Marlin GitHub pages, download its release, learn how to plug it into the JVM, and give it a kick with your own maps, and let us know how that works via the dedicated mailing list.
Remember Marlin is pretty new, so there might be issues using it, and of course we hope to improve its performance further. But even in its current early state, we’re pretty excited about it.
If you are interested in learning about how we can help you achieving your goals with our Open Source products and professional services, do not hesitate to contact us!

The GeoSolutions team,