Inactive Leap Motion Controller

[Cervator]

Name: Leap Motion Controller
Summary: Introduce support for input mappings using the Leap Motion Controller
Scope: Optional / pluggable Engine piece? Can it be a module?
Current Goal: Re-implement the initial Leap branch prototype using the shiny new Jitter lib (which also needs more work)
Phase: Implementation
Curator: Cervator + begla + potentially one more if somebody is interested?
Related: Giant mess of structure overhaul fun - not sure if this would go best in an alternative input directory separate from the main engine (if it can't be a straight module, anyway), Jitter library repo

Updates:

• Feb 8th - Prototype is functional, but pretty awful and rough to control
• Feb 16th - @Begla's first round overhaul got us to two-handed full movement control including camera view. Fly like Superman!
• Feb 18th - We technically have a video now, but it isn't very presentable yet
• Feb 23rd - New Leap SDK out, integrated along with a couple native gestures for attack & god mode
• March 9th - Leap SDK 0.7.5 out, upgraded our setup, stability is a little better. Polished video hopefully just about ready
• March 16th - Video published
• April 11th - Started extracting Marcel's LeapMotionP5 lib to a pure Java version to base Jitter on (with him on board). Structure looking good
• April 21st - First version of Jitter lib published!

Original Post:

So here's my little hinted-at secret - begla was contacted by Leap Motion last week and we met with one of their representatives from San Francisco over Skype on Friday. Very interesting and impressive stuff, both the hardware (trivial motion detection lag, high accuracy, many data points) and their product maturity with 10k dev units prepared and many more pre-orders than that. In a nutshell it is a super Kinect for PC & Mac, and maybe Android in the future (and Linux soonish (tm))

We signed up as interested developers and they mailed us both a dev unit each, free of charge, and I got mine earlier today (Tuesday). begla might get his tomorrow? Unsure. Neat to be able to get a hold of stuff like this, especially now with hardware startups contacting us instead of the other way around via funding kickstarters (OUYA, Oculus Rift)



So again this is another one of those "would kick ass if we could do this, but no promises!" options. Bit cheaper and simpler this time vs OUYA or the Rift, and in a few hours I had Terasology reading the motion of my hand from in-game and triggering movement events - although admittedly said events aren't working and my rush-code is full of dirty hacks just trying to get it working fast. An actual good developer could probably get it working from scratch in an hour

Not to sound like a broken record but this has a lot of potential for nifty controller alternatives. The Leap can pull lots of data from multiple hands down to where individual fingers are pointing, the roll/yaw/pitch of your palm, distance from the controller, etc. I did some thinking and imagine you could do about everything in-game without even touching the mouse or keyboard (though you might need voice chat to go with it in multiplayer). In particular creature commands could be entertaining as hand gestures, and I miss casting spells via mouse gestures in Black & White

If somebody is super curious to play with this it is possible we can get another controller sent out (doesn't hurt to ask, I figure!) since the company is very eager to get interesting games implementing their stuff, especially by launch in about two months (in Best Buy for starters). I'm still not coding enough to get in shape where it'll be worth my time so if push comes to shove I can even send my unit onwards

I'll share some more thoughts on what kind of motions could be behind what movement/action in-game later, but I spent the whole evening trying to get input events working and have given up for now very tired I figured this might be a good consideration to go with the mouse button behavior thread going on (that I also haven't managed to crunch through fully due to this ..)

Spoiler: Mostly resolved/outdated - thanks Immortius!

Immortius - could I ask for a quick bit of review to figure out what I'm doing wrong? I'm thinking it is probably something simple, and me not having messed with input in ages (if ever) is just leaving me missing it. Was also unsure what to initialize when.

• We need to instantiate a Controller class from the Leap SDK - seems to work if I put it in the CoreRegistry but doing so in StateSinglePlayer doesn't seem right ...
• I first created a LeapListener based on their SDK's listener class but when hooked up as a listener (straight in the engine class for testing) it would either exit after 59 frames if I didn't keep the Controller as an instance variable (the max history by default is 59 frames) and if I did keep an instance variable around it would crash Java instead. Huh. It has native hooks just like lwjgl, btw.
• I cloned InputSystem to LeapSystem to see if I could pull frame updates manually in the main loop instead which avoided crashes but doesn't seem to get event communication done right. InputSystem is registered somewhat differently, not sure if that matters or if it is something else
• I'm trying to simply fake a key input event to move the player forward (as if 'w' is pressed) but I'm not quite faking it well enough it seems. The LeapSystem initialize doesn't seem to persist its instance variables and while I can just fetch stuff each iteration from CoreRegistry that doesn't seem right, and maybe it is part of the problem?

                KeyEvent event = KeyDownEvent.create(17, 0f);
                CoreRegistry.get(LocalPlayer.class).getEntity().send(event);

17 is what the "key" variable holds for 'w' - not sure how to just say "forwards button" instead

I can make that trigger but the event just doesn't seem to go anywhere. Or rather, it goes through several registered event handlers none of which seem to really care about it. Any ideas?

Here's the actual logging for me raising my hand (palm flat, fingers together) from a low position above the device to a high position

Frame id: 2096284, timestamp: 18217952018, hands: 1, fingers: 0, tools: 0, framesTotal: 649
Hand sphere radius: 85.49996 mm, palm position: (145.418, 707.702, -183.015)
Previous hand y was 109.12005 while new is 707.70154 so delta is 598.5815
Detecting RAISED hand, triggering move forward
Hand pitch: 6.814376855197349 degrees, roll: 11.944047424249678 degrees, yaw: -20.013106551203098 degrees

 

[metouto]

Well, it looks like fun to me but waaaaaaaay out of my brain range I am thinking

 

[Immortius]

Well, this is very interesting.

Immortius - could I ask for a quick bit of review to figure out what I'm doing wrong? I'm thinking it is probably something simple, and me not having messed with input in ages (if ever) is just leaving me missing it. Was also unsure what to initialize when.

• We need to instantiate a Controller class from the Leap SDK - seems to work if I put it in the CoreRegistry but doing so in StateSinglePlayer doesn't seem right ...

I would suggest either in the RegisterInputSystem load process, or in TerasologyEngine (initControls() or similar). The former is more about configuring input for the current world (taking into account module binds), the latter doing initialisation of the input systems. Presumably you only need to initialise the Leap once during startup, and would stay running forever after that.

• I first created a LeapListener based on their SDK's listener class but when hooked up as a listener (straight in the engine class for testing) it would either exit after 59 frames if I didn't keep the Controller as an instance variable (the max history by default is 59 frames) and if I did keep an instance variable around it would crash Java instead. Huh. It has native hooks just like lwjgl, btw.

Don't know the SDK, so can't say. Obviously could be a buggy SDK, or perhaps you need to drain its data or something.

• I cloned InputSystem to LeapSystem to see if I could pull frame updates manually in the main loop instead which avoided crashes but doesn't seem to get event communication done right. InputSystem is registered somewhat differently, not sure if that matters or if it is something else

• I'm trying to simply fake a key input event to move the player forward (as if 'w' is pressed) but I'm not quite faking it well enough it seems. The LeapSystem initialize doesn't seem to persist its instance variables and while I can just fetch stuff each iteration from CoreRegistry that doesn't seem right, and maybe it is part of the problem?

                KeyEvent event = KeyDownEvent.create(17, 0f);
                CoreRegistry.get(LocalPlayer.class).getEntity().send(event);

17 is what the "key" variable holds for 'w' - not sure how to just say "forwards button" instead

I can make that trigger but the event just doesn't seem to go anywhere. Or rather, it goes through several registered event handlers none of which seem to really care about it. Any ideas?

I can't begin to guess what is going on with your LeapSystem , but I can explain a bit about the InputSystem and its events. The key thing being how binds work.

At the moment the input system starts by detecting mouse button and keyboard key presses. It sends out some events for these, but they're mostly unused - it isn't good to code directly against them as it prevents rebinding. There are also Axis inputs - like mouse left-right and forward-backwards. These are more analog controls, spanning a continuum of values between -1 and 1. These are directly used at the moment.

After this, we have bind buttons. These are like virtual keys not linked to a device directly. They are linked directly to gameplay concepts though - so there is a JumpButton, and a ForwardsButton, and so forth. These are bound to a number of the lower level inputs, as determined by the configuration.

And finally there are BindAxis. These combine two bind buttons to acts as a virtual axis. So for instance, the ForwardsButton and BackwardsButton are combined to form the ForwardsMovementAxis. This provides some convenience for the systems reading the input, as they emulate things like smoothly moving between extremes over time.

All of this happens in the InputSystem, so sending a Key W event won't cause forwards movement. You could send a ForwardsMovementAxis event to the local player though. It probably would conflict with the FowardMovementAxis events coming from the InputSystem proper though.

So I guess "true" integration would involve having the Leap provide low level "button" events similar to the mouse buttons or keyboard keys, which can then be bound like anything else. I'm not sure if the Leap has anything like an axis (though I expect it would), in which case there may be value in refactoring things so that raw axes are mapped to bind axes, and then you could have looking around controlled by the Leap cleanly.

 

[overdhose]

I could totally see my self slapping some miniions with that... very curious how much resources this eats for the pleasure of doing that

 

[Panserbjoern]

Awesome!

 

[Cervator]

So I could swear I had written a post here already with some more details, and being that I just wrote up another one then accidentally changed page and lost that one too ... yeah

There are open source libs popping up already, of course. There's an OpenLeap group on GitHub (that I joined for the fun of it) with a few small options as well as a really nice starting gesture library written in Processing, although I can't find any actual Processing, looks like pure Java. That one is GPL though so we can't use it with our license - but it makes a good example to review.

Immortius - thanks a bunch for the feedback, I had been wondering about some of the other events like ForwardsMovementAxis. Just tried that and it works! I can now make the character start moving forward by raising my hand and instead move backwards by lowering my hand!

The one thing I couldn't figure out - what's "id" ? In

BindAxisEvent.prepare(String id, float value, float delta)

Just tossed some gibberish in there and it worked. Great, now I'm going to have trouble sleeping since I want to code instead

Fully agreed on having to make some "button" type events for the Leap motions. It has the full six 3D axes (palm roll/pitch/yaw and 3d motion) + fingers + measuring the radius of an imaginary sphere held in your hand + ... I don't even know. That one starting gesture library already has a small pile of things you can draw with a finger to then trigger specific gestures. So first step is probably to start writing those, and yeah the axes should be fairly natural to do

More advanced might be making it more intuitive and give feedback to the user since you don't have any resistance with your hands in the air. Suggestions include using colors on the screen for indicating going out of bounds (like the rubberbanding effect on smartphones if you try to move too far), virtual hand indicators on screen, and so on. Gotta get the refresh rate right too and make sure it doesn't eat too many cycles. Then we can start on the fun stuff like casting lightning bolt lightning lighting bolt!

Edit: Right now raising your hand just kicks off a max move forward that never ends while lowering kicks off max move backwards. Interaction with keyboard is graceful, events from there just smoothly adjusts everything like you'd expect. Pressing 'w' and letting go stops the player even if previously forward movement was "stuck" from there being no stop event from the Leap. Tiny fix is on GitHub

 

[overdhose]

virtual hand indicators on screen

will the hand be purple??? because when I looked at some of the leap's vids, seeing an animated hand on screen... I could provide them with some ideas there...

Imagine really picking up oreons in your hand and drop them back in the world... that would be a real neat thing to see

 

[Cervator]

I would absolutely love seeing that!

One of the examples included is an integration piece with Unity where you can grab a block. It is a bit rough but a few more SDK upgrades and the retail controller? It could happen - if somebody does the code

 

[overdhose]

get to work slacker But yeah I'm not sure either how you would calculate the picking up, specially since we can't select the minions yet as is. But I'm sure you'll figure it out, just shoot that wall with your special gun ^^

 

[Immortius]

The one thing I couldn't figure out - what's "id" ? In

BindAxisEvent.prepare(String id, float value, float delta)

Just tossed some gibberish in there and it worked. Great, now I'm going to have trouble sleeping since I want to code instead

It is just an identifier for the event, typically something like "engine:forwardmovementaxis". It... really isn't used (having specific classes negates the need for it), except perhaps if you ever want to log an event. I think it was intended more for the javascript modding.

 

[Cervator]

Here's full movement and jumping working, if awfully written and really hard to control

Pretty sure I'm making the math harder than it should be, but it is past midnight again

 

[overdhose]

so where's the video??? slacker!!!!!

 

[Cervator]

Spent more than half the day at work, getting up around 6:45 am on a Saturday, home around 11 hours later

For bonus points most the time was spent waiting So I did clean up the code a little bit and added running. But something about it isn't working - it doesn't look like I'm triggering it right by merely modifying the character movement component (which is working fine for jumping). Code is over here - probably should work on camera control next.

 

[Cervator]

Quick update!

begla has finally gotten his Leap Motion unit after it got stuck in customs for a little while. He dove in and got way past where I was at in no time, which I totally expected

We'll be playing with it some more over the weekend. In the meantime I've created a Jitter repo under OpenLeap where somebody already wrote a pretty nice almost-Java (Processing) library for the Leap with working gestures and fun stuff like that. By our powers combined we will summon Captain Planet and hopefully get a nice general purpose Java lib for the Leap we can use here in Terasology and others elsewhere for other stuff!

 

[begla]

Got the basic player movement (looking and walking around) implemented as a first draft. Jumping, sprinting and attacking is also possible.

We'll try to get a video up this weekend!

 

[Cervator]

Ended up working on some unrelated topics tonight that I've been putting off for weeks or even months

I like the current leap controls and am seeing the potential for offering a few control schemes. Like:

• Two hands pure Leap (sort of current functionality with a few more gestures)
• One hand Leap + one hand on keyboard (might be more relaxing and flexible, if less cool)
• Two hands Leap and Oculus Rift, which in that setup would handle the camera

But that's fancy future stuff. In the meanwhile we just need to dig up that iconic superman music piece as current movement while flying totally is like that. Need to put that on the video - will try to record something tomorrow, just ended up too late tonight on other stuff

Some other general ideas:

• Might need a slightly bigger "dead zone" in the center
• Discussed (on IRC) the potential to switch behavior when Leap loses track of hands to not auto-cancel each frame but instead either cancel once when the hand is lost or even perpetuate the last action (auto-run, although that could feel funny too)
• Still curious to see what we can do with yaw/pitch/roll (especially if you're trying to control everything with one hand)
• For comfort may need an overall smaller motion zone (more sensitive with outer bounds)
• Maybe a simple gesture to enter/leave god mode? "Jump" with both hands at once?

 

[begla]

Thanks for the input. Will see what I can do with that later this day.

 

[Cervator]

Running a little email thread with a Leap group for work on the library Jitter - some summary notes for what the lib could do:

• Capture and trigger input events (axis events, hand events, maybe basic gestures, maybe something about the "grab sphere"?) that we can map to our events rather than make the actual calculations.
• Provide a LeapListener with a set of methods for said events like a MouseListener
• Provide some sort of settings/expectations for hands, like one or two hands being expected and figure out a natural resting zone said hand(s) might hang out at in a neutral state
• Calibration mode to possibly help with that / train the user a bit in where to hold hands with what expectations
• Frame averaging / smoothing if that might be helpful (Leap native listener runs way above 60 FPS)

A big "hurry up and wait" issue right now is seeing what the Leap company itself releases for the SDK regarding mouse emulation and gesture support / training. Supposedly one round of updates is imminent (based on comments last week from Brandon, our contact with Leap)

 

[Cervator]

We're at a point now where we can make videos of the Leap in action - but there are some challenges left before said videos are good enough to publish

I made a couple first drafts and they aren't pretty but I uploaded one, unlisted and unedited, to YouTube - since we promised a video I suppose I'll share the link here with no real intent to publish it widely just yet. It is hardly a good representation of what the Leap can do so and it hasn't been processed what so ever. Goal is to get to a point where we can do a closeup of just the two hands, maybe with a better background like a light color with a grid, along with a Frapsed recording of the game with one picture being edited inside the other picture.

Good to get some of the notes into this thread as well (including a couple hints from our Leap contact) - so here are a few:

• Consider turning down monitor brightness while recording to get a better lighting balance in the video
• Try to make it obvious that hands are supported? Maybe the Leap could be closer? For me that might mean sitting on the keyboard shelf
• Want to try a smaller motion space to use arms less, hands more. Or just try a better position vs. the Leap
• Palms vs fists? Maybe fist on left hand (or either?) could be another / a different way to stop movement?
• Model how to show off stuff the best in a video by looking at the video(s) already released by the Leap team itself
• Would be good to add a couple more gestures to highlight, like placing blocks
• Stabilize video better
• Next SDK update from Leap is right on the verge of being released, we were told maybe even this afternoon, but no luck It should contain lots of goodness, maybe the Leap developer library community can then start moving forward again rather than wait to see what the base SDK will do

 

[Cervator]

Finally had a little time to play with this again. Combined with a new SDK upgrade that brings basic gesture support. So much for having a clear goal for what Jitter should do, since now we may be in even more of a "wait and see" position for what could be expected in the Leap SDK vs library

I integrated the new gesture stuff from the Java sample (cut out some of the informative stuff so be sure to review the sample for more and include the logging again for fun), changing the current finger tap detection to the native "KEY_TAP" gesture and adding "CIRCLE" to enable/disable god mode depending on whether the player makes a clockwise or counterclockwise circle. Works beautifully.

I also tidied up a little bit and stopped the "lost hand" situation from halting all motion.

The doPlayerLooking method is a bit more boring now and we might want to do something with the ATTACK_THRESHOLD and such again, but it doesn't seem like it is needed.

Either hand can now do either gesture, which may or may not be desirable going forward

Code is a little rough (as always for my stuff) - begla please review and polish as you see fit. I think the added gesture for god mode will be good for the video and having the visualizer up at the same time as the game is nice as there's a new indicator for gestures (with 'o' pressed once). Will aim to find a way to record game + visualizer + actual hands so we can weave the three feeds together in a video with two inserts.

Still need to look at the dead zone / distance sensitivity to make movement plus looking a little more comfortable, but it is past 2.30 am ...

I wonder if another SDK update will bring a "pick up" gesture we can use soon to pick up minions like overdhose
desires