1

For the past few months I've been looking into developing a Kinect based multitouch interface for a variety of software music synthesizers.

The overall strategy I've come up with is to create objects, either programatically or (if possible) algorithmically to represent various controls of the soft synth. These should have;

  • X position
  • Y position
  • Height
  • Width
  • MIDI output channel
  • MIDI data scaler (convert x-y coords to midi values)

2 strategies I've considered for agorithmic creation are XML description and somehow pulling stuff right off the screen (ie given a running program, find xycoords of all controls). I have no idea how to go about that second one, which is why I express it in such specific technical language ;). I could do some intermediate solution, like using mouse clicks on the corners of controls to generate an xml file. Another thing I could do, that I've seen frequently in flash apps, is to put the screen size into a variable and use math to build all interface objects in terms of screen size. Note that it isn't strictly necessary to make the objects the same size as onscreen controls, or to represent all onscreen objects (some are just indicators, not interactive controls)

Other considerations;

Given (for now) two sets of X/Y coords as input (left and right hands), what is my best option for using them? My first instinct is/was to create some kind of focus test, where if the x/y coords fall within the interface object's bounds that object becomes active, and then becomes inactive if they fall outside some other smaller bounds for some period of time. The cheap solution I found was to use the left hand as the pointer/selector and the right as a controller, but it seems like I can do more. I have a few gesture solutions (hidden markov chains) I could screw around with. Not that they'd be easy to get to work, exactly, but it's something I could see myself doing given sufficient incentive.

So, to summarize, the problem is

  • represent the interface (necessary because the default interface always expects mouse input)
  • select a control
  • manipulate it using two sets of x/y coords (rotary/continuous controller) or, in the case of switches, preferrably use a gesture to switch it without giving/taking focus.

Any comments, especially from people who have worked/are working in multitouch io/NUI, are greatly appreciated. Links to existing projects and/or some good reading material (books, sites, etc) would be a big help.

Improve this question
7
  • 1
    I'm one of the people who posted an answer to your meta question about this. You still haven't clearly stated the problem you are trying to solve, or how we can help you with it. In short, make your question specific and answerable; questions soliciting comments or links are not generally a good fit on the SE network, especially if we can't figure out what material we should be searching for. Are you simply having difficulty figuring out where to start? What have you tried so far?
    – Robert Harvey
    Sep 23, 2011 at 15:13
  • Re problem; I'm clearly not stating myself well. I have some application that accepts input in the midi protocol. Midi protocol consists of some channels (64 iirc) and a range of non-floating-point control values varying continuously on each channel(0-127, in whole number increments). In order to buiold an interface for this app, I need to, A) design the interface taking into consideration multitouch capability of at least 2 cursors, B)create some abstract description (xml or otherwise) of the interface to enable a user to build an interface for other midi apps.
    – jamesson
    Sep 23, 2011 at 15:42
  • OK, sounds good so far. What do you need help with, specifically? What are you stuck on?
    – Robert Harvey
    Sep 23, 2011 at 15:49
  • App controls consist of toggle switches, rotary controls/faders, and continuous controllers. Switches are AFAIK best handled by gestures. Rotary controls and faders are a different matter. What kind of motion is best suited to manipulating a rotary control, and how to detect it, is beyond my current skill set. Note that finger tracking is not yet implemented, but even if it were, I have no idea what the math would look like for "2 fingers turning a knob" from 2 cursor points. Further difficulties arise when determining the "best" way to place controls onscreen in order to maximize usability,
    – jamesson
    Sep 23, 2011 at 15:54
  • finally, and most importantly, there has to be some kind of way to enable end user to make interfaces themselves. Whatever ideas I may or may not have about usability, the "target market" will want as much flexibility as possible. Furthermore no single interface is suitable for all apps. So in terms of my most direct problem, the xml descriptor is probably it. I need a way to draw some stuff on screen, setup midi routings, and write the whole mess to an xml file. Since the midi translation is handled in max/msp, anything outside of there will have to redo the whole solution from scratch.
    – jamesson
    Sep 23, 2011 at 15:59

1 Answer 1

Reset to default
0

  1. Every touch controller I have seen that has a knob on it, you turn the knob by moving your finger in a line. So to turn a particular knob, I would touch the knob, and then slide my finger up to turn the knob clockwise, and down to turn it counterclockwise. Lifting your finger releases the knob. Your users will be accustomed to this convention, if they've ever used someone else's control surface.

    Moving a slider with my finger feels like moving a slider. Turning a knob in the usual way does not, because it relies on vertical tactile sensation. This is in large part why I bought an Oxygen 61 for my own personal setup, so I could have some real knobs to turn.

  2. Rendering your control surface and its mappings to an XML file is a matter of creating a schema and writing code to perform the translations between your XML and your control surface. To get some ideas about how this might work, consider studying Microsoft's WPF or Silverlight models; they use XAML (Microsoft's own XML schema) to render user interfaces.

    Rely heavily on third party XML libraries, and consider techniques such as serialization; these tools will greatly speed up your development effort.

  3. In terms of letting people develop their own control surfaces, it seems axiomatic that they're not going to write XML to do it. You will need to provide some kind of drag and drop from a toolbox to a designer surface, and then save that out to the XML file.

    Allow your users to patch the controls into a MIDI channel by touching them after selecting a control element. Many current music packages such as Abelton Live already support this; you click on the item you want to control, and then touch the control on your control surface that you want to map to it, and it will automatically get patched in.

Of course, all of this depends on what sort of gestures for touch (and un-touch) are available for the Kinect. Also, if the Kinect knows how to detect rotary motion of your hand, you could simply translate that into rotary motion for the knob, with say a 2:1 scaling so that you can get a complete turn without turning your hand all the way around.

IMPORTANT: If you haven't yet, as part of your research, watch the movie "Minority Report."

See also http://en.wikipedia.org/wiki/User_interface_markup_language

Improve this answer
5
  • Thank you very much for your interesting and insightful comments
    – jamesson
    Sep 23, 2011 at 16:53
  • Re user expectations, while I definitely agree with you that they must be met, at the same time I feel like the technology offers and hints at a variety of possibilities that are as yet completely unexplored. Your minority report example, for instance, is completely unrelated to current consumer sota for touch/multitouch interaction. Only ms surface has anything approaching this, and a limited subset of that functionality has been passed to win7 mt. This to me is extremely surprising, because multiple academic projects have made great strides in creating a truly "intuitive" user experience.
    – jamesson
    Sep 23, 2011 at 17:01
  • Examples include MIT sixthsense and the original developer of the minority report interface (whose name I forget atm). There is nothing about that system that is technically unfeasible today (ncis: la features a more realistic solution) but nobody seems to have any interest in doing it commercialy. I am assuming the cause is user intransigence, which IMO can be overcome with an interface that sufficiently accurately reproduces natural motions. In terms of my project, I think the best solution would be implementing several manipulation schemes and letting the user choose among them.
    – jamesson
    Sep 23, 2011 at 17:05
  • 1
    Looking at the Kinect in greater detail, it seems as if it is a souped up sensor device only; all you get is the data stream. That being the case, my suggestion for detecting rotary motion is to see if you can find the tips of the fingers (like they do in Minority Report), and look for those points moving in opposite directions. There's nothing depicted in Minority Report that couldn't be achieved on the Kinect, given sufficiently sophisticated software.
    – Robert Harvey
    Sep 23, 2011 at 17:06
  • Your evaluaton of the kinect is probably accurate, but opposite directions I think is a mistake - that would be the scale gesture. Your stuff about xml and serialization are spot-on; I will now have to study ms silverlight more closely.
    – jamesson
    Sep 23, 2011 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.