[ALPHA] S9ilents Nostril Head Tracking

[s9ilent]

Note, this is a test version (Test ver 5d)

Download link
http://members.optuszoo.com.au/whyisthat/scripts/SNHT.zip
*Edit*Oh and before I forget, Sources. Made by me, using Aforge.NET DLL's.

What:
A program that uses your webcam to track your nostrils, allowing for mouse movement on the screen

How to use:
1) Make sure you have.. directx... 8.1? or greater (I'm fairly sure the game needs dx9 so you should be fine in this regards)
2) If you don't have the .NET framework of at LEAST v2.0.50727, then get it. link (this is for the 3.5 framework)
3) Position the your webcam somewhere where it can see the black parts inside your nostrils (I'm serious... its how the program works..)


Now, put the program somewhere and run it.
1) Click Start
2) On the webcam should start streaming into screen. On the picture, click on your nostrils. It should light the nostril on the left side of the screen (i.e. your right, its left), red, and green for the other one. There should be a blue + indicating the middle spot.
3) Calibrate the program. Look at the top left of your screen, click the top left button... and similarly with the other 4 corners. You should notice that once all 4 are locked, the mouse will begin to move.

4) (optional) Click on the autobox button. This will restrict movement to inside the box. (As I have black hair, it can loose lock on my nose and lock onto my hair, the auto box restricts it to my nose~ish area)

Stopping the tracking:
Hold the right mouse button. (Or uncheck the "Track" box)

Autobox has boxed wrong:
Check the box-tool, and draw a box on the picture.
(Remember to uncheck the box-tool if you are trying to click on to your nose to relock the tracking. )

What is deviance:
The LOWER deviance is, the more accurate it is and thus, it tracks less pixels, and can be slightly more jumpy (Reduces Type 1 errors~false positives)
The HIGHER deviance is, the less accurate it is, so it tracks more pixels, and can be slightly less jumpy. (Reduces Type 2 errors~false negatives)

General Webcam hints:
DON'T Adjust the camera properties if the camera is running. Stop it first, adjust, wait a second, then start it up again. (This is a hard coded issue w.r.t the webcam~software works, so I will never be able to fix this)
To increase frame rates, turn down exposure, (Turn off auto exposure first). Turn off low light boost)
To decrease cpu usage, just turn off all the auto-features from the camera. (No flicker, light boost, auto white settings, exposure settings)



Considerations:
CPU eatingness.. Unfortunately, Webcams use lots of CPU power. Mine uses 35-55, and with my program that raises to 45-65 (My cpu is rather slow, below X3's minimum sys req's). Some need more, some need less. Lowering the frames will lower MY programs usage, but probably not the webcam's. Lowering/increasing the resolution does affect the webcams usage thou.



Ongoing Ideas:
Multi thread the program (for multi core support)
Attempt to track non-nose movement (to get a 2nd/third point, thus allowing full movement, and no more messing around with calibration)
Auto calibration
Calibration mid run
Re-boxing mid run, or at the very least, un-boxing
Re-centring lock (i.e. if lock has gone crazy, move your head back, press a button, and it starts at the middle position again)
Thinking up a better name... (Originally s9ilent's head tracking, but giving the spelling of that acronym, i decided to try to un-swear word it)

[OOZ662]

LEDs (such as those used by TrackIR) will give you the best accuracy, as not much else emits the same brightness and color that they do. For the UberGeeks(tm), there's actually a method of using UV LEDs and removing a filter from the webcam's lens to make it a UV camera, thereby eliminating almost all interference.

[s9ilent]

I'm not quite as keen on an LED system (as I'm sure if you had track ID, then you could just use that instead). Whilst it is certainly very effective, that fact that it constantly needs power is eh
(And I'm not to sure how to wire up LED's to a computer, although I will mess around with using the sound/usb ports).
LED's do have the advantage however that they work through lenses (e.g. a sunglass lens) as I was using one to block out ambient light from other monitors and it worked well.

Right now I'm planning on using two dots, 1 made with a green highlighter, one with a pink highlighter, both stuck about 90 degrees apart on my headsets microphone (on the side facing the screen/webcam, so it doesn't interfere with the mic's functions)
It seems to contrast fine when it's bright (I have huge amounts of indirect sunlight atm), and I'd imagine it would only get better when it's dark
If you don't have a mic, then I don't know.. stick them on your forehead? (I did it when testing, it looks rather funny)

[OOZ662]

(This isn't suggesting you're wrong or anything, just providing FYI-style info since I play another game in which TrackIR and its homebrew alternatives are popular)

Actually, rigging a USB power device is simple. All USB ports always give off 5V. One pin is 5V, one pin is data, and two are GND (as I remember. A quick search would verify). All that would be needed is a proper resistor inline with each LED.
Since most people abhor futzing with cables, many people use home-made battery packs. Steal a battery connection out of a junky toy, put the right resistor in front of it, and viola.
Also, the common homebrew setup involves attaching all the stuff to a baseball cap. Handy method, I think.

(P.S. Ooooooom... )

[Treelor]

Making a printable image that we could tape on to a hat or something would be a really cost-effective way to do this.

[s9ilent]

You can already download something like that :
http://www.cachya.com/esight/overview.php
(Its someone else stuff)


I'm just experimenting with different ideas and trying to make my own ATM, I didn't like a printout, as its rather large, and fluctuates depending on the room lighting (i.e. light from the monitor, ambient lighting, etc.)

Although, right now I'm going to try what OOoommzz662 suggested. (i.e. LED's). I'm going to try to make a generalised thingy (such that all you will need is 3 LED's, the distance between the tree, and a "centre" calibration)

*shrug* I'm going to have to put this on hold for a few weeks thou

[spur390]

If you're the DIY kind of guy, try the Freetrack program, and googled instructions on the hardware aspect. If you have the cash, TrackIR is a better use of your time.

But are you considering using this for X3? The field of view is too narrow for this to be effective.

[s9ilent]

I've freetrack, but the thing I didn't like about it was that it only locks onto light sources (Cachya is probably better at eliminating false positive locks, but it uses a print out instead of LED's making it questionable under adverse lighting, as well as requiring you to wear a large funny thing). But other wise free track is actually pretty good (but controlling the mouse is harder then I expected)



For my program right now, I switched DLL's from some webcam_capture (I don't know who made this one) to directX's direct show. Frame rate is up, and CPU usage is down... but it's creating... issues... (As I need EACH frame bitmaped, not just one when you press a button).
And because of the bizzare way you actually capture frames (there is no on frame event -afaik) it is causing read and write errors (i.e. the program crashes because its trying to read a bit map whilst it is being written)
I've managed an adhoc to stop "most" of them... but yeh... I'm trying to find a more permanent solution via LOTS of google, but it isn't helping.

On my webcam_capture program, I had made it lock onto colours instead of light sources.

note to self, new idea, colour ratio's (more light/dark sensitive), weight based on SS colour?




*edit* Oh and I wanted this more for shooting at the mouse cross-hair, whilst flying with the joystick, and not so much for looking around.




Progress!!
After mucking about with directshow and getting generally annoyed... (it isn't wpf friendly apparently... and I think I know what that means now) I'm now using Aforge libraries!!! Much better! (there is actually a frame event, as well as lots of other library that are useful, e.g. pixel/colour stuff, edge detection)

[corhen]

yea. i would love to see this, if only for the "in first person Lookaround" like they show on that Demo Vidoe.....

please, get this to work!

[Hexenhammer]

that would be awesome.... awesome to the max!

[s9ilent]

Ok, I've got my program back up and fully working (thanks to aforge libraries, they are awesome cakes) -well back to working at it's old standards.

I've defined the goals of the program as:
1. Offer a head tracking system that other programs do not already offer (for X3 mainly)
2. Must not cause (too) performance loss to X3

And I planned to do 1) by making it track small coloured objects rather then simple light sources (or black and white funny triangle signs). The problem that I'm noticing is that it uses quite a bit more CPU.
Now I can multi thread the app so it doesn't affect X3... in theory.. but I have no idea how it will work in practice (as I have a slow single core CPU). (Also you can do stuff like adjust the frame rate, and I'm trying less accurate but faster routines)


So at this point, is it worth me continuing this project? (As I'm not actually going to use it my self... unless I get a new computer, which might not be for a while)
(Also Remember, that there are other free track programs available, Just wiki trackir, and look at the external links, there is the chachya? program, and free track)


*edit* lunch time~
I am now rather pleased at the programs tracking capabilities (although it is day time at the moment, and the green and pink do work best during the day, although I'm happy to report that the huge amounts of white glare flying in through the window have no affect on the prog).
It current uses about 70% of my CPU (1.8Ghz equiv single core). 35-55% of which is from just running the camera, the rest is from the tracking routines.

Now my todo list is:
1. Figure out a third object to track
It may be possible to do a green-pink-green, and use a blob filter, but that is more CPU intensive, 3 distinct objects would be better. Possibly a light source? (But that sort of goes against the key objective of no external power req.)
2. Convert on screen co-ords to 3d data. Convert 3d data -> mouse/keyboard/what ever
3. Multi thread the program and test it. (This is tricky for me as I don't use a multi core although..... I may be able to borrow some.. but that is a pain in it self...)

And a new problem encountered.... huge changes in lighting conditions (e.g. black screen -> white screen, e.g. looking at a sun, load screens etc.) Cause it to loose its lock (completely, it goes crazy looking for a new one)

[s9ilent]

Out of curiosity, how many people do NOT use a black chair (or have black stuff, around the microphone level of their headset?) (black black, not just dark)


*edit* Never mind...
I should just but a piece of non black apparel over my chair...




After MUCH!!! (And I mean HUGE) amounts of trial and error and testing just about everything I Could think of, the only thing which I can really consistently track is either large white things (but chachyai already does that) or black on non black surfaces.
So, naturally, I need to track something black now, and what is the best thing I could think of (I need 3 points)
1. Microphone from headset
2 & 3. The inside of your nostrils (I'm rather serious ) It is like the only part of you which is devoid of light (except if you open your mouth...) so now i've just got to work out that one, but I think I can just say, get the lowest black part for the mic). However, this does mean that you have to put the camera sort of underneath your face (instead of on top of the monitor) which can present potential problems for laptop cameras (or any inbuilt screen ones)




*edit*
I've managed to make it track a single point (technically two points, your nostrils), and as stated previously, it requires the camera be positioned facing upwards. However, given how the nose doesn't move by a whole lot, the level of precision is rather low (I found that it would split the screen into about 20x15 "boxes, which the mouse moved in between)
(Also, as this is only a single point tracking do hickey, it is sensitive to any movement, e.g. if you move in your chair, it will count that movement incorrectly)

[s9ilent]

Test version 5d released. (It's still in it's alpha stages)


This is sort of just a proof-of-concept/evaluation-of-ongoing-production release sort of thing.
So give feed back!

FEED BACK ME!!! (Mwuhahhahaha)

[corhen]

NOSTRIL TRACKING?????


well, im gonna give it a try!

[s9ilent]

Yup, nostril tracking.

I had to track either something black or white. I tried eyes, but they where not white enough/large enough/too much reflection from my glasses.
So I figured I would track black instead. The first idea was a microphone from a headset, but each time I opened my mouth, I would "swallow" the tracking. Which left just one course of action left, Nostril tracking!