I can't help but feel that most these companies working on these technologies are viewing them in isolation, when none of our peripherals are ever used in isolation. The mouse wasn't developed to replace the keyboard, but to supplement it. You don't generally use a mouse to select characters from a representation of a keyboard on screen (there are reasons when you want this, but it's not the general use case), so why do we always see the equivalent of that in new peripherals?
The two main cases I'm thinking of are eye tracking, and brainwave tracking (like EMOTIV). Individually, neither looks like a compelling way to control a computer when you have a mouse and keyboard, but together, I think that might really yield something interesting (and sooner!).
Instead of using brain waves to move a cursor around the screen, use eye tracking. Instead of using gaze lingering for click, use brainwave tracking. Individually they seem cumbersome and annoying to use. Together they could be a really compelling interface device, IMO.
The point of eye tracking in VR isn't really about being an input control (even if that's what this company was focusing on).
There are many big problems in VR that get solved if you know where the user is looking at. The most important one is, your eyes really only see high resolution in a very small region around what you're staring at, and if the VR headset can only really render that region in high quality, this solves many issues (processing power, framerate, even wireless transmission).
Of course, for any of this to work you need perfect eye tracking, which is very hard.
Either that, or most of the companies know that the technologies they are building are likely to be acquired to build part of the feature set of a larger offering, and are building them for precisely that reason. Too cynical?
I know a person that built a company for acquisition like you mention. Don't think they were ever successful. To be an acquisition target is one thing. But to be acquired by companies like Google most people would have to meet their hiring bar (aka passing their interviews) and have the right composition for the team (mostly engineers they would have a hard time hiring away). Of course there are exceptions, like they're solely purchasing for the technology, but that means the tech they built was pretty good and couldn't easily be cloned (of course this excludes if they're buying for market share of users if the product is super popular). My opinion is incredibly risky building for acquisition. Anyone have examples that it isn't?
This operates on the assumption that eye tracking and brain waves signals are positively or negatively correlated (for which the setup you propose can be thought of as an estimator with control variates). Is this indeed the case?
Eye movements and EEG are actually somewhat tricky to record and/or interpret simultaneously.
Eye movements cause travelling waves to sweep across visual brain areas (which are actually a huge chunk of the brain), potentially swamping whatever was going on before. Some of this is probably supports visual processing (by, e.g., persisting or resetting neural activity across eye movements), and some of it might be epi-phenomenal.
Furthermore, EEG electrodes pick up huge, non-neural signals from the movement of the ocular muscles and the movement of the eyeball itself. This can be reduced with clever pre-processing and more expensive amplifiers (e.g., that don't saturate or recover from saturation more quickly).
I'm not sure what information you're working off that makes you think they need to be positively correlated. I'm referring to using two separate systems in conjunction. Eye tracking to cursor placement (and anything else that it can handle well), and brainwave tracking to handle intent. At the simplest level, eye tracking for mouse x/y coordinates, and brainwave tracking for mouse button presses.
For those wondering about the privacy implications of this technology, eyefluence required a VR headset to function and I'm not aware of them demonstrating their tech using the user-facing cameras found in mobile phones and tablets, so I don't think there is any reason for worry on that front (yet).
Here is a video of a 2016 demo of eyefluence tech:
10 years ago while I was IBM, I submitted a proposed patent that was a sort-of-joke by saying Amazon (or other sites) could get rid of 1-click shopping and go to 0-click shopping by following your eyes and automatically making the purchase if you looked at something for a given amount of time. Sounds like this will be a reality in a few years.
The Samsung phones have a ton of eye and face detection options for brightness and screen wake up based on eye and face detection, which I always turn off to conserve battery. I have never heard of that feature on iPhone though.
Physically removed both front and back integrated cameras from my tablet. If the FBI Director is paranoid enough to cover his webcam then that is good enough reason for drastic measures...
One that I've seen is plugging something into the audio jack, and then breaking it off and taping over it. That way the computer will detect something is plugged in and turn off the mic. Of course this won't be possible once Apple, and then everyone else, gets rid of the audio jack from computers.
Isn't that still software controlled? So, let's say a malware is able to gain privileged access to my Windows machine and can install anything, would he be able to override this default behavior of 'turning off the mic when something is plugged in', especially when the external voices are more audible than something coming from the 'plugged in' dummy audio-in cable?
It would have to be an audio device that has that little "ring" of audio in. Sorry, not knowledgeable in this field, someone smarter will be able to clarify, but I've noticed that my headphones that don't have a mic capability have less little "separator" rings on the 3.5mm jack, whereas the ones that do have mic capability have more.
Better details from a beta tester over on /r/oculus on Eyefluence's tech:
" . . . I'm still not allowed to say how exactly it works but in my mind it's the gold standard for eye based user interfaces . . . Their [main focus] is the UI" (1)
So sounds like a blend between some good eye tracking tech and some GREAT user experience design and implementation.
I'm a machine learning researcher and I also have a lot of experience analyzing eye movement data.
There aren't large labeled datasets that are publicly available in this area. Just to classify eye movements into events, you would need images of the eyes or eye velocity to classify eye events into smooth pursuit, fixation, etc. That means someone needs to label all of those in ground truth. To actually determine what someone is looking at, you have to go beyond this to have information about the scene. When looking at a screen, this is usually done with calibration, but things get more complicated when trying to predict what someone is looking at with a mobile eye tracker and a fixed scene camera. Body and head movements complicate things further (e.g., VOR) and many algorithms ignore them.
My research group has been trying to address this problem by using user interactions with a website to create this "ground truth". One assumption is that you have to look where you click. But there are more subtle user interactions that could be helpful.
Depends on which aspect you mean. E.g. a proper infrared tracker with high framerate, something like >120Hz, can be used to detect microsaccdes. I don't know much about webcam tech, but I doubt it can reach those framerates and at the same time low noise needed to distinguish those microsaccades. Then again, the research of the meaning of those and whether they're predictive or not isn't conclusive afaik so I wouldn't know any use for VR anyway.
If you don't mind me jumping in here, since people with eye tracking experience seem to be quite scarce: is it viable these days to build a working accurate gaze tracker that would, say, work using just an iPad camera? No hardware whatsoever, just using the camera to somewhat accurately position a dot on the screen. Every tracker I've tried is close to absolute garbage when it comes to accuracy.
That may have been one of the reasons why they didn't make it.
Eye tracking is tricky. I worked a bit with two guys developing a commercial eye tracker in the 80's, it was a pretty sophisticated optical set-up that combined the image being looked at and the image of the person looking using a 45 degree angle 50% mirror, that made the job considerably easier.
One of the complicating factors was that eye movements have a lot of noise in them that our brain has become very adept at filtering out so what we think our eyes are doing can be quite different from what they're really doing.
Not easily. Samsung has something like this, which attempts the minimal task of "see if you're looking at the screen" to prevent dimming while you read. It's... not good. Not even passable, in my experience.
That task should be achievable, but anything fancier - even positioning a dot - is probably still a tall order with that single camera.
The hardware based IR solutions (e.g. Tobii) tend to work really well, though. Is it a matter of not having figured out yet how we could achieve similar results using just a webcam image?
You don't need deep learning. Seriously. Not everything in the world needs deep learning. I wrote an eyetracking system for tracking my vision while playing mobile games[0], and really, the eye-tracking is quite simple and you can do it with violajones haar cascade and do it with an accuracy high enough that you can press buttons with your vision.
The hardest part were ellipse curve-fitting algorithms - I had to go pretty cutting edge for that.
At the last Oculus Connect, Michael Abrash covers some of the high level issues they've seen with fast and "virtually perfect" eye tracking that is required for VR [1]. He predicts it should be doable within the next 5 years (including new rendering techniques required for foveated rendering).
His entire keynote is great, particularly if you're interested in the near-future possibilities of VR.
Well, they're not just tracking eyes: they are using eye position, focus, and gestures to control a user interface which is a bit different.
The moment the stuff you want to track (eyes) also makes the enviroment change (ui) stuff can get tricky.
Eye blinking also changes the way we perceive stuff. If we use our eyes to control a UI, blink events are also special because they usually happen when we have processed enough information from a scene (Walter Murch book: In the blink of an eye tells something about this from a filmmaker perspective).
So... not as easy as it looks at first sight (no joke intended ;-).
When I wrote EyeMap, it turned out humans blinked a lot more than I had anticipated. It wasn't much later until EyeMap had been shuttered before I read that same book. What I did was basically ignore the 5 or so frames where the eye was blinking because EyeMap's main purpose wasn't to be a UI - the button pressing bits are only for calibration purposes.
Identifying eyes is fairly easy, of course. Identifying open, approximately centered eyes is the easiest version of this and it's what we want.
Basic eye-tracking efforts say something like "which of these equidistant squares did your line of vision intercept?" That's still enough to produce some cool things like "look at the red box to click it", but it's not a real-world solution.
But the problem is identifying small changes and cross-referencing them with the state of the world to identify what's actually being viewed. The difference between someone looking at a sticker on a window and looking out that window is big, but the evidence available to distinguish them is tiny.
Other 'natural' movements can also spell doom for eye-tracking: turning your head without moving your eyes sweeps your vision, while turning your eyes saccades it and only values the endpoints. And turning your head and body while keeping focused on a single point feels simple, but identifying it means tracking three different axes of rotation as they interact!
Even though this is for hardware and VR, I have to admit the first thought on seeing the headline was that google doubled down on its advertisement business. More so after the recent news that they changed their policy to track users more.
He asks us, "how do you transfer your intent to act without winking and without waiting", but never tells us how its done. Anyone know? This made the video incredibly frustrating for me.
That was super frustrating, I went back a few times to see if he said it and he did not. I was just waiting for him to tell me for the rest of the video and it never happened.
I can see good eye tracking as getting much, much more traction than VR for regular users. While most applications of VR to everyday tasks seem forced and gimmicky, eye tracking can revolutionize reading, which is arguably the task professional and casual users spend at least 80 percent of their online time on.
The two main cases I'm thinking of are eye tracking, and brainwave tracking (like EMOTIV). Individually, neither looks like a compelling way to control a computer when you have a mouse and keyboard, but together, I think that might really yield something interesting (and sooner!).
Instead of using brain waves to move a cursor around the screen, use eye tracking. Instead of using gaze lingering for click, use brainwave tracking. Individually they seem cumbersome and annoying to use. Together they could be a really compelling interface device, IMO.