Text Entry in Virtual Environments using Speech and a Midair Keyboard

Jiban Adhikary, Keith Vertanen

IEEE Transactions on Visualization and Computer Graphics, 2021.

Entering text in virtual environments can be challenging, especially without auxiliary input devices. We investigate text input in virtual reality using hand-tracking and speech. Our system visualizes users' hands in the virtual environment, allowing typing on an auto-correcting midair keyboard. It also supports speaking a sentence and then correcting errors by selecting alternative words proposed by a speech recognizer. We conducted a user study in which participants wrote sentences with and without speech. Using only the keyboard, users wrote at 11 words-per-minute at a 1.2% error rate. Speaking and correcting sentences was faster and more accurate at 28 words-per-minute and a 0.5% error rate. Participants achieved this performance despite half of sentences containing an uncommon out-of-vocabulary word (e.g. proper name). For sentences with only in-vocabulary words, performance using speech and midair keyboard corrections was faster at 36 words-per-minute with a low 0.3% error rate.