2

I'm curious about developing a simple system for GUI application sharing between users on a single computer, which uses Xorg (or perhaps any modern form of X11 display). I'd like user Alice to be able to authorize user Bob to view and control a particular selected application she's running, which will then appear for Bob exactly as it appears for Alice, permitting them to both control the application.

TL;DR: I'd like to know how to gather/scrape the graphical data.

Assumptions and requirements include:

  • Assuming Alice and Bob are logged into the same local machine.
  • Alice and Bob may be using any desired desktop (Gnome, KDE, XFCE, etc) and any graphics toolkit (GTK, Qt, etc). They may have different configurations.
  • Alice and/or Bob may be logged in via separate VNC server (eg. tigervnc), X11VNC framebuffer-grabbing server, X2Go, physical monitor and keyboard, VirtualBox window on a host screen, or some other means. The user should not need to care about how they are viewing their desktop.
  • The toplevel window and all descendent windows of Alice's selected application would appear for Bob at the same size/scale, but allow Bob to move them separately from what Alice sees. Resizes would be mirrored to both users.
  • For the moment I'm only interested in supporting basic 2D graphical operations -- 3D rendering and video are nonessential.

What I'm looking for is a way to hook or be informed of UI updates to the application, so I can stream those updates (the protocol doesn't matter for this question) and redraw them for Bob. How is this normally done?

I've noticed that Google Hangouts for example, is able to share a user's full screen. It can also share individual apps, which is what I'd like to do -- but only sometimes. It seems to fail on desktops not using a compositor of some kind, and it may fail for other reasons as well. I would not like to require a hardware framebuffer to snoop on, as one of my most important use cases is where one or both users are remote.

Is there a way I can "intercept" drawing/painting API calls at some level, to get a fairly hardware- and desktop-agnostic means of capturing an application's GUI?

EDIT: As an add-on thought, how might the situation have changed or how might it in the future, with regard to compositors and display servers like Wayland? The relationship between X servers and compositors seems a bit confusing at the moment, but might Wayland (or Mir?) offer a more straightforward means of doing this for both X and direct-API applications?

3

With X, there are only three ways to do this: require that programs be linked with a customized Xlib that has the hooks for communication with an application to show it elsewhere, use a customized server to capture single window contents (as VNC might do) or intercept and process the X protocol stream.

I designed and built a program called XTV in the late 1980s that replicated single X clients to multiple hosts and did keyboard and pointer sharing. This was accomplished through protocol interception and modification for each destination display. It was also, as far as I know, the first attempt to do it in a way that used unmodified clients and servers. The closest thing to it that would work with more-modern X is Xpra.

There was a paper about it and subsequent research and development, some of which is covered by other papers that cite it.

  • This is fantastic information; I'd give you an upvote but apparently that's not something new users can do. Thank you. – Chris Robison May 29 '15 at 17:29
  • @ChrisRobison: You can still make it the selected answer. Contact info is in my profile if you want to drop me a line to discuss it in more detail than would be on-topic here. – Blrfl May 29 '15 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.