I am a final year undergraduate student of Information Technology. My team and I have taken up "Sign Language Recognition" as our Final Year Project. We have just started with it and we are in the phase of information gathering (gathering data). We plan to use Instrumented Gloves as the input device. But we do not have much knowledge in the area.

Also, we came across the following methods for training the system for actual recognition of gestures and hence sign language.

  1. Neural Networks
  2. Symbolic Learning Algorithms
  3. Hidden Markov Models
  4. Instance Based Learning
  5. Grammar based techniques

Please tell me which of them I should use for sign recognition.

Also, tell me about Instrumented Gloves and is there any specific variety which we should choose for our project?

  • 7
    Ask for help understanding any of the techniques you cited if you need it. But this is your project; you and your team should decide how you want to attack it. You'll learn more by making a bad decision on your own than by having someone here tell you what to do. Don't assume that anyone here knows more about this problem than you do! You're creating a new thing, after all, so do it in your own way. – Caleb Aug 6 '11 at 17:37
  • okieszzz..thanks @Caleb...I am just new around here....and got a vote down :( – shahensha Aug 6 '11 at 17:58
  • 1
    I know of at least two projects that implement some degree of sign language translation. I guess you could just look at how those projects work and do the same thing, but if you're doing a research project you'll probably want to look at those, see what they can do and what they can't, and try to build a system of your own that works better than those that came before. – Caleb Aug 6 '11 at 18:22
  • 2
    Any plans to use a Kinect (if it can capture enough detail)? I'm assuming they're more widely available than the gloves, so your solution may be more portable. – StuperUser Aug 11 '11 at 16:03
  • 1
    @StuperUser I did end up using Kinect and we built an HMM-based system for 12 signs in the Indian Sign Language. It also worked for simple sentences made from those signs. Had to use basic NLP for that. After 5 years of grad school, I am so glad I did something like this. It opened my mind and I learnt so much and here I am well into my PhD. A big thank you to the community. – shahensha Mar 14 '17 at 0:04

Thad Starner (now at GA Tech) was able to get this to work with HMMs and a cap-mounted video camera back in 1995. I imagine you'll have some good luck along those lines with the massive increase in processing power and sensors since then.

Thad Starner and Alex Pentland. Real-time american sign language recognition from video using hidden markov models. In Proceedings of International Symposium on Computer Vision, Coral Gables, FL, USA, 1995. IEEE Computer Society Press.

You're going to have to do a lot of data cleaning/filtering before it gets to the HMM, however. This is a signal-processing task and it may be suitable to use fuzzy logic to categorize the inputs (look at how mouse-gesture recognition is done) and something like a NN to downsample the data.

| improve this answer | |
  • 4
    Does the Kinect SDK have anything that would be useful for recognition out of the box? – StuperUser Aug 11 '11 at 16:05
  • +1: a friend made a sign language recognition system using a regular, low-cost webcam for his thesis last year. Worked well enough. I also believe he was using hidden Markov models in it. – scrwtp May 12 '12 at 13:10

I know some ASL (?) has been captured and recognised by computers in the past, and that was a while ago. I remember watching videos of this on VHS. 1994 at the latest.

Sign Languages (there's more than one) are typically multi-channel media — it's not just the handshapes, so the gloves may not be quite enough (depending on your aims). My ASL is almost non-existent, but British Sign Language uses something like this (pinch of salt required, I'm very rusty):

  • ‘Phonology’:
    • 26 hand-shapes (configurations of the fingers).
    • Various hand orientations
    • Various hand starting positions
    • Various types of hand movements
    • Shoulder shapes.
    • Head position and tilt. (in BSL, a slight ‘nod’ indicates first person)
    • Eyebrows and eyes.
    • Mouth shapes.
    • Overall expressions which are context-sensitive. Something bad may have happened, but your expression is expected to change briefly to the expression required by the sign.
  • Grammar:
    • Signing space (where, in relation to the body, signs are allowed)
    • Very different from English grammar. E.g. BSL generally follows an Object-Subject-Verb word order, where sentences like CAT DOG CHASED (‘the dog chased the cat’) are the norm.
    • Spatial pronouns: hand-spell a name, point to a location in your signing space, and you can then point back there rather than hand-spelling or signing the name.
    • Verb aspect: I don't know about ASL, but BSL has some seriously complex aspect. You can stress a sign by lengthening the ‘hold time’ (starting state of the sign) and making the movement abrupt. More aspect I can think of right now indicates whether something takes place slowly, quickly (and how slowly or quickly), direction of travel, the signer's attitude towards what's being signed, etc. ‘They stared at me for a long time and I was put off’ is essentially the single verb ‘SEE’ with lots of aspect.
  • Syntax: again, pretty different to what you be used to. BSL is somewhere between Japanese and Spanish, but that's probably an oversimplification.
  • Complications: fingerspelling follows slightly rules in BSL (not so much in ASL). There's a staggering amount of community-local slang (sign languages aren't written, so they're a lot like spoken languages before the invention of writing — in some cases, boys finishing boys' schools and girls finishing girls' schools graduated to find they had very divergent languages), and quite a few loanwords. My favourite one is (probably only Scottish) slang for ‘I love you’, which is an amalgam of the ASL letters I, L, Y (think standard Spiderman gesture) turned into a transitive verb (the sign moves from close to the signer's torso to the intended person): a completely invalid handshape in BSL, but turned into a BSL verb.

I think you'll be seriously hard-pressed to capture and parse successfully all of these channels. Perhaps reading a suitably limited subset may be a good idea: basic handshapes (ASL fingerspelling is very clear, and should be very easy today) and some basic motions. You may have to limit the complexity of the sentences you parse since you probably won't be able to capture all aspect reliably.

The software will feel to native signers a bit like a hearing learner of sign language, missing out on a lot. I suspect signers will adapt very quickly. After all, native English speakers can still understand a sentence even if some of the articles are missing.

| improve this answer | |

Borrowing heavily from the comments: I think you should look into existing image recognition algorithms. Personally, I would simply implement an existing open source one. Once you have some understanding of and comfort with one, train it on a single sign.

Use the output from your gloves to generate an image. To that end I would look for a glove that provides output easily translatable to an image, such as a table of bit representing the location of the digits. Then pass the images into the image recognition software.

Once those two are working together, your gloves and image recognition software, you should move on to whatever User Interface you want, such as speech or text output.

Finally keep adding signs, testing after each addition, till your semester is up, or the system performance degrades too much.

| improve this answer | |
  • 1
    Why add the intermediate step of image recognition, when it would be much easier to work off of raw data from the gloves? – Austin Hyde Aug 11 '11 at 15:46
  • @Austin My personal experience is severally limited, but I know of several open source image recognition systems that I consider mature. My experience with Neural Networks, Symbolic Learning Algorithms, and Hidden Markov Models is that the available frameworks are difficult to use. I simply laid out how I would start off approaching the problem. I have since +1 Alex's answer for including actual reference work, and directly addressing the question of which model may be preferred. – Joshua Drake Aug 11 '11 at 15:54
  • I'm not experienced at all with any of the above, it just seemed like image recognition was not the "right way", so I was just wondering if there was a particular reason you suggested it is all. – Austin Hyde Aug 11 '11 at 17:52

There is another approach available, recently I've worked on a project that used gabor wavelets in face recognition Wikipedia entry for Gabor wavelets. And I stumbled on a interesting project that fits your need.

The project revolves around a paper, rock & scissors game and the recognition of symbols used inside a game. It tracks the player trough the camera and recognized the hand gesture.

The project can be found on the following link (Ropesi project). There are some videos inside the svn trunk that demonstrate the use and application of the project.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.