I'm curious about developing a simple system for GUI application sharing between users on a single computer, which uses Xorg (or perhaps any modern form of X11 display). I'd like user Alice to be able to authorize user Bob to view and control a particular selected application she's running, which will then appear for Bob exactly as it appears for Alice, permitting them to both control the application.
TL;DR: I'd like to know how to gather/scrape the graphical data.
Assumptions and requirements include:
- Assuming Alice and Bob are logged into the same local machine.
- Alice and Bob may be using any desired desktop (Gnome, KDE, XFCE, etc) and any graphics toolkit (GTK, Qt, etc). They may have different configurations.
- Alice and/or Bob may be logged in via separate VNC server (eg. tigervnc), X11VNC framebuffer-grabbing server, X2Go, physical monitor and keyboard, VirtualBox window on a host screen, or some other means. The user should not need to care about how they are viewing their desktop.
- The toplevel window and all descendent windows of Alice's selected application would appear for Bob at the same size/scale, but allow Bob to move them separately from what Alice sees. Resizes would be mirrored to both users.
- For the moment I'm only interested in supporting basic 2D graphical operations -- 3D rendering and video are nonessential.
What I'm looking for is a way to hook or be informed of UI updates to the application, so I can stream those updates (the protocol doesn't matter for this question) and redraw them for Bob. How is this normally done?
I've noticed that Google Hangouts for example, is able to share a user's full screen. It can also share individual apps, which is what I'd like to do -- but only sometimes. It seems to fail on desktops not using a compositor of some kind, and it may fail for other reasons as well. I would not like to require a hardware framebuffer to snoop on, as one of my most important use cases is where one or both users are remote.
Is there a way I can "intercept" drawing/painting API calls at some level, to get a fairly hardware- and desktop-agnostic means of capturing an application's GUI?
EDIT: As an add-on thought, how might the situation have changed or how might it in the future, with regard to compositors and display servers like Wayland? The relationship between X servers and compositors seems a bit confusing at the moment, but might Wayland (or Mir?) offer a more straightforward means of doing this for both X and direct-API applications?