Why have hardware-accelerated vector graphics not taken off?

Question

I'm working on an app that involves real-time manipulation of vector paths at 60fps, and I'm very surprised by how little information there is on the subject. At first, I tried to implement my idea using CoreGraphics, but it didn't perform adequately for my purposes. I then discovered that there was a Khronos standard for hardware-accelerated vector graphics called OpenVG, and thankfully a kind soul had written an OpenGL ES semi-implementation called MonkVG.

But despite the fact that OpenVG is a very practically useful API, it seems more or less abandoned by Khronos. According to Wikipedia, since 2011, the working group "decided to... not make any regular meeting [sic] for further standardization". The documentation, best I can find, consists of just a single reference card. And what's more, there are barely any examples of OpenVG anywhere on the internet. I can find hundreds of OpenGL tutorials in the blink of an eye, but OpenVG seems conspicuously missing.

You'd think that hardware-accelerated vectors would be more important in today's world of rapidly-increasing resolutions, and it does seem that many companies are implementing their own ways of doing this. For example, Qt and Flash have schemes for hardware-accelerated vectors, and many of Adobe's tools have hardware acceleration as an option. But it seems like the wheel is getting reinvented when a standard already exists!

Is there something I'm missing about OpenVG that makes it unsuitable for real-world use? Or is it just that the standard didn't catch on in time and now it's destined for obscurity? Do you think there's room for a standardized API for hardware-accelerated vector graphics in the future, or will it just be easier to use traditional raster-based techniques? Or are vectors simply on their way out, before they were ever in?

Answer 1

update: See bottom of reply

This answer comes a bit too late, but I hope to shine light to others (particularly now that C++ standard committee wants to incorporate Cairo into std):

The reason nobody really cares about "accelerated vector graphics" is because of how GPUs work. GPUs work using massive parallelization and SIMD capabilities to colour each pixel. AMD typically works in blocks of 64x648x8 pixels while NVIDIA cards typically work in 32x32 4x4 pixels [See update at the bottom]

Even if they're rendering a 3D triangle, the GPU works on whole quads that this triangle covers. So if a triangle doesn't cover all 8x8 pixels in the block (or 4x4 in the case of nvidia) the GPU will compute the colour of uncovered pixels and then discard the result. In other words, the processing power for uncovered pixels is wasted. While this seems wasteful, it works incredibly good for rendering large 3D triangles when paired with a massive number of GPU cores (more detailed info here: Optimizing the basic rasterizer).

So, when we look back at vector based rasterization, you'll notice that when drawing lines, even if they're thick, there is a massive blank space. A lot of processing power wasted, and more importantly bandwidth (which is the major cause of power consumption, and often a bottleneck) So, unless you're drawing an horizontal or vertical line with a thickness multiple of 8, and it perfectly aligns to 8 pixel boundaries, a lot of processing power and bandwidth will be wasted.

The amount of "waste" can be reduced by calculating the hull to render (like NV_path_rendering does), but the GPU is still constrained to 8x8/4x4 blocks (also probably the NVIDIA's GPU benchmarks scale better with higher resolutions, the pixels_covered / pixels_wasted ratio is much lower).

This is why many people don't even care about "vector hw acceleration". GPUs simply aren't well suited for the task.

NV_path_rendering is more the exception than the norm, and they've introduced the novel trick of using the stencil buffer; which supports compression and can significantly reduce bandwidth usage.

Nonetheless, I remain skeptic of NV_path_rendering, and with a bit of googling shows that Qt when using OpenGL (aka the recomended way) is significantly faster than NVIDIA's NV_path_rendering: NV Path rendering In other words, NVIDIA's slides were "accidentally" comparing XRender's version of Qt. Ooops.

Instead of arguing that "everything vector drawing with hw acceleration is faster", Qt developers are more honest admitting HW accelerated vector drawing is not always better (see how their rendering works explained: Qt Graphics and Performance – OpenGL)

And we've not touched the part of "live editing" vector graphics, which requires triangle strip generation on the fly. When editing complex svgs, this could actually add serious overhead.

Whether it is better or not, it highly depends on the applications; as to your original question "why it hasn't taken off", I hope it is now answered: there are many disadvantages and constraints to take into account, often making a lot of people skeptical and may be even biasing them into not implementing one.

update: I've been pointed out the numbers are completely off base, as the mentioned GPUs don't rasterize in 64x64 & 32x32 blocks but rather 8x8 = 64 and 4x4 = 16. This pretty much nullifies the conclusions of the post. I will soon update this post later with more up to date information.

Answer 2

I don't think it is really true that nobody really cares about "accelerated vector graphics" as written in this answer.

Nvidia seems to care a fair bit. Besides Kilgard who is the lead technical guy on NV_path_rendering (henceforth NVpr to save my fingers), the Khronos president, Neil Trevett, who is also a VP at Nvidia, has promoted NVpr as much as he could in past couple of years; see his talk1, talk2 or talk3. And that seems to have paid off a bit. As the time of this writing, NVpr is now used in Google's Skia (which in turn is used in Google Chrome) and independently [of Skia] in a beta version of Adobe Illustrator CC (beta), according to Kilgard's slides at GTC14; there are also some videos of the talks given there: Kilgard's and Adobe's. A Cairo dev (who works for Intel) also seems interested in NVpr. Mozilla/Firefox devs also experimented with NVpr and they do in fact care about GPU accelerated vector graphics in general as this FOSDEM14 talk shows.

Microsoft also cares a fair bit because they created Direct2D, which is used fairly widely [if you believe the Mozilla dev from the aforementioned talk].

Now to get to the point of the original question: there are indeed some technical reasons why using GPUs for path rendering is not straightforward. If you want to read about how path rendering differs from bog-standard 3D vertex geometry and what makes GPU acceleration of path rendering non-trivial, then Kilgard has a very good FAQ-like post, which is unfortunately buried somewhere in the OpenGL forum.

For more details on how Direct2D, NVpr and such work, you could read Kilgard's Siggraph 2012 paper, which of course is focused on NVpr, but also does a good job surveying prior approaches. Suffice to say that quick hacks don't work too well... (as the text of the PSE question noted.) There are significant performance differences between these approaches as discussed in that paper and shown in some of Kilgard's early demos, e.g. in this video. I should also note that official NVpr extension document details several performance tunings over the years.

Just because NVpr wasn't so great on Linux in 2011 (in its first released implementation), as that 2011 blog post of Qt's Zack Rusin said, it doesn't mean that GPU acceleration of vectors/paths is hopeless as Mr. Goldberg's answer appears to have inferred from that. Kilgard has in fact replied to the end of that blog post with updated drivers showing 2x-4x improvement over Qt's faster code and Rusin hasn't said anything after that.

Valve Corp. also cares about GPU-accelerated vector rendering, but in a more limited way, relating to font/glyph rendering. They've had a nice, fast implementation of large font smoothing using GPU-accelerated signed distance fields (SDF) presented at Siggraph 2007, which is used in their games like TF; there's a video demonstration of the technique posted on YouTube (but I'm not sure who made that).

The SDF approach has seen some refinements by one of the Cairo & pango devs in the form of GLyphy; its author gave a talk at linux.conf.au 2014. The too-long-didn't-watch version is that he does an arc-spline approximation of the Bezier curves in order to make the SDF computation more tractable in vector (rather than in raster) space (Valve did the latter). But even with the arc-spline approximation, the computation was still slow; he said his first version ran at 3 fps. So he now does some grid-based culling for stuff that's "too far away", which looks like form of LOD (level of detail) but in the SDF space. With this optimization his demos ran at 60 fps (and it was probably Vsync limited). However his shaders are incredibly complex and push the limits of hardware and drivers. He said something along the lines of: "for every driver/OS combination we had to change things". He also found significant bugs in shader compilers, some of which were then fixed by their respective devs. So it sounds a lot like AAA gaming titles development...

On another tack, it appears that Microsoft has commissioned/specified a little bit of new GPU hardware to improve their Direct2D implementation with, hardware which is used by Windows 8, if available. This is called target-independent rasterization (TIR), a name which is a bit misleading as to what the stuff actually seems to do, which is spelled out in Microsoft's patent application. AMD claimed that TIR improved performance in 2D vector graphics by some 500%. And there was a bit of "war of words" between them and Nvidia because Kepler GPU's don't have it, whereas AMD's GCN-based GPUs do. Nvidia has confirmed that this is indeed a little bit of new hardware, not simply something a driver update can provide. Sinofsky's blog post has a few more details, including some actual benchmarks of TIR. I'm quoting only the general idea bits:

to improve performance when rendering irregular geometry (e.g. geographical borders on a map), we use a new graphics hardware feature called Target Independent Rasterization, or TIR.

TIR enables Direct2D to spend fewer CPU cycles on tessellation, so it can give drawing instructions to the GPU more quickly and efficiently, without sacrificing visual quality. TIR is available in new GPU hardware designed for Windows 8 that supports DirectX 11.1.

Below is a chart showing the performance improvement for rendering anti-aliased geometry from a variety of SVG files on a DirectX 11.1 GPU supporting TIR: [chart snipped]

We worked closely with our graphics hardware partners [read AMD] to design TIR. Dramatic improvements were made possible because of that partnership. DirectX 11.1 hardware is already on the market today and we’re working with our partners to make sure more TIR-capable products will be broadly available.

I guess this was one of the nice things that Win 8 added that was mostly lost to the world in the Metro UI fiasco...