Guitar hero

“Some of the kids I work with are always talking about video games … like World of Warcraft … and I can foresee the day that I can participate in that,” says Philip Cooney, 51, a supervisor in the computer sales department of the student bookstore at the University of Nevada, Reno. He’s a computer expert who’s unable to play computer games because he can’t see them.

Legally blind, Cooney suffers from retinitis pigmentosa, a degenerative genetic condition. He started losing his eyesight at age 14. He describes his vision as “like looking through heavy wax paper.”

But he’s relatively good-humored about his vision loss. He laughs when he says, “I’ve had some unfortunate run-ins with fire hydrants.” And he’s optimistic about the future of computing for the visually disabled.

“I know what computers can do, what they’re capable of.” Before losing his sight, Cooney was a computer programmer for the U.S. military and then UNR. And now, as a computer salesman, Cooney stays current on the latest technological developments.

“Guitar Hero is sort of a lark,” he says. “It’s an intermediate step, but important.”

Cooney beta-tested a version of the Guitar Hero video game designed for visually impaired users. It was designed by Eelke Folmer, an assistant professor in the Computer Science and Engineering department at UNR.

“Eelke came down to buy a computer,” says Cooney. “And he noticed my cane and noticed my disability … and, actually, I’d already heard a little bit about the research he was doing … so I agreed to do some guinea-pigging for him.”

Guitar Hero is the game that brilliantly turned playing air guitar, the embarrassing but near universal activity of miming the moves of your favorite musician on an imaginary guitar, into a video game. In the familiar version of the game, users press buttons on a hokey guitar-shaped video controller and respond to color-coordinated visual cues that direct them to what “note” should be pressed next.

Folmer and a graduate student, Bei Yuan, developed a glove that provides visually impaired users with a buzzing sensation on the appropriate finger before each note that should be played. The glove, which resembles Nintendo’s classic Power Glove controller, transposes visual data into tactile vibration, what Folmer calls “haptic feedback.” Since the visual domain is larger (in terms of the quantity of information that can be processed at a single time), some concessions are made. For example, game play has been simplified from five buttons down to three.

Cooney helped with suggestions of where on the fingers to place the buzzers and had another suggestion: “Folk songs.”

“Folk music would be easier to learn because it doesn’t have all that background stuff going on,” he says.

“Activision [the company that publishes Guitar Hero games] thought it was really cool,” says Folmer, but the glove’s current production cost, approximately $1,500, means that it’s not yet commercially viable.

Folmer, 31, is a very tall, very excitable Dutchman. He has that Northern European look that suggests he’ll live peacefully for more than a hundred years. His enthusiasm is unbridled. Like many academics working in highly specialized fields, he’s prone, in casual conversation, to fanciful flights of thought detailing the minutia of his research—even if these lines of thought leave the listener behind. Unlike most academics, he has a Grand Theft Auto poster on the door to his office, innumerable Guitar Hero guitar-shaped game controllers all over the office, and an enormous flatscreen monitor, ideal for magical transportation into virtual environments.

School games
UNR currently offers an undergraduate minor in video games. This should be exciting news to every local thumb-twiddler. This degree, offered by the CSE department, masquerades behind the euphemism “digital interactive design,” but it is a degree in video games nonetheless. The first required course: “Introduction to Video Game Development.” Coursework is focused on the fundamentals of game design and game play. It is only a matter of time before this program is expanded to offer an undergraduate major, as well as graduate degrees.

The “Introduction” class, CS 281, is taught by Folmer and open to all UNR students.

Robby Fuchs, 15, took the class. Fuchs is a student at the Davidson Academy of Nevada, a school held on the UNR campus with an accelerated curriculum for “profoundly gifted” young students.

His emails are written with carefully considered language:

“Dr. Folmer’s class … was the first time that I was able to take my interests in computer science and gaming, and apply them in a constructive manner in an academic setting,” writes Fuchs, who demonstrates on the cover the Guitar Hero for the visually impaired. “Half of it involved group work, which added an interesting dimension to the class.”

“What makes a good game?” asks Folmer. “It has to be fun, and the game play should be intuitive. … Students don’t have the resources to make a Halo game, so they must learn how to create effective game play on a smaller scale.”

He’s proud to show off effective student work. One assignment he gives to classes is to make an original Pac-Man game. Student David Dorn, in what Folmer considers an outstanding example of beginning game design, combined the principles of the classic arcade game (candy-eating, ghost-chasing) with the design of the Japanese slot machine/pinball phenomenon pachinko. The resulting game is called, aptly enough, Pac-Man Pachinko.

“This is thinking outside the box,” says Folmer. “You take an existing concept, change the parameters or change the goal, and you have a new concept.”

The cynical reader might note that Dorn’s innovation is essentially combining a classic and innocuous video game—one of the first games to appeal to women and other users outside of the core video game demographic, teenage boys—with a gambling game. Is the purpose of this academic research to develop a more addictive slot machine?

The UNR program is indeed partially sponsored by an endowment from the locally based “adult” gaming industry giant International Game Technology.

It seems likely that today’s recreational gaming innovations will become tools for tomorrow’s gambling industry. There’s no question that the local presence of the “adult” gaming industry is part of the reason that UNR is one of the first academic institutions to have a program like this. But this doesn’t diminish the fact that the institution is developing a regional talent pool for a burgeoning industry and developing research with applications beyond the confines of the casino industry.

There are schools like Full Sail University in Winter Park, Fla., that offer accelerated, three-year degrees focused exclusively on video game design. The advantage of a degree from a university like UNR is that it pairs the relatively limited applications of a game development degree with a more traditional computer science degree. This offers students better leverage when seeking employment outside of the gaming industry, a notoriously difficult industry that values, above all else, young blood. According to Folmer, 70 percent of employees in the industry have less than six years experience.

Like radio, film and television before it, video games are the advent of a new media technology. Many of today’s young intellectuals, academics and industry leaders grew up with video games a part of the lexicon of their lives. Even the novel—an artform now generally accepted as being capable of the highest intellectual merit—was considered lowbrow when it first appeared. What might have seemed like children’s toys are becoming an artform and an important tool for interacting with the world.

“Games are a hot topic now,” says Folmer. “Sales are increasing and surpassing Hollywood.” And not just financially. Take, for example, the 2003 game Star Wars: Knights of the Old Republic. It arguably has better special effects and a better storyline than the 2005 film Star Wars Episode III: Revenge of the Sith.

A second Second Life
The other difference between the UNR program and programs at more specialized game design schools is that the UNR program emphasizes research. Folmer’s research is on interaction design for physically disabled users: the blind, the deaf and users with severe physical impairments that hinder mobility.

The Section 508 Amendment to the Rehabilitation Act of 1973 requires that all electronic and information technologies maintained by the federal government be accessible to users with physical disabilities (the standards and protocols are similar to those of the more familiar Americans with Disabilities Act of 1990).

At first, it might not seem as though video games, a realm generally associated with flower-petal-shooting plumbers and barrel-tossing apes, might be liable to comply with any government standards of accessibility. But as many institutions begin using video games as teaching and training tools, and schools begin conducting classes in virtual environments, it becomes clear that if video game accessibility is not a legal issue now, it will become one in the very near future.

One model for the sort of virtual environments often used as teaching tools is Linden Research, Inc.'s web-based program Second Life. Second Life is not a game, per se, but a fully interactive, virtual world. (In fact, the UNR Art Department has already conducted a digital media class that used Second Life as a virtual classroom.) Users conduct an avatar through 3D virtual environments that have an appearance similar to those seen in contemporary first-person games and can interact with other users’ avatars. There are no specific quests or goals, but the interactions are similar to those encountered in massively multiplayer online role-playing games (MMORPGs), like Everquest, or even console games like the Grand Theft Auto series.

“Second Life has the potential to replace the traditional internet,” says Folmer. The immersive graphics and greater sophistication could eventually make for a more interactive and informative world wide web. But how can we ensure that these virtual environments remain accessible?

“There are two models for accessibility: replacing stimuli or reducing interaction,” says Folmer. Replacing stimuli is the process of transferring data from one sensory input to another. Closed captioning, for a familiar example, is the transference of audio information into visual information. Reducing interaction is a process of simplification: decreasing the innumerable features of modern game controller—or even a simple computer mouse—into a single toggle switch.

“Instead of ginormous number of inputs, can we reduce the input to one button?” asks Folmer.

He has a single-button controller: one huge button, roughly the size of a dinner plate. A single-switch client is a user who can’t use a mouse. Folmer designed an application for a single-switch Second Life user. The user double-clicks to toggle between motion and interaction. An arrow scans the available directions for motion, allowing the user to click on the direction to move and then click again to stop motion. In the interaction mode, the arrow iterates over nearby objects, allowing the user to scan and select. This process can be slow, but it is effective.

Folmer and Fuchs are collaborating on a single-button strategy game.

“We are attempting to do this,” says Fuchs, “to show that games with a complicated interface, such as real-time strategy games, are able to be played by people who are physically disabled and can only press one button … at a time.”

Folmer has other controllers geared toward disabled users: a one-handed controller and a bulky, unattractive contraption that allows users to play using their mouths—the controller has inputs for the lips, the tongue, the teeth and the breath. This application has severe limitations: It is difficult, for example, to sustain breath for repeated button-pushing. And all this technology is expensive.

The real cyberworld
Folmer’s current and most ambitious project, funded by a grant from the National Science Foundation, is an application to make Second Life accessible for blind users.

“That’s what I am really looking forward to,” says Cooney. “Second Life will certainly allow me to practice and learn new skills that will translate to other parts of computing.”

Visually impaired users interact with computers using a screen reader, a device that translates visual text into audio information using the familiar digital voice associated with Stephen Hawking and the Radiohead song “Fitter, Happier.”

Screen readers are considerably less adept at recognizing non-textual information, which makes it difficult for blind users to navigate virtual environments that employ digital representations. Folmer is developing a digital middleman to translate visual information into textual information that can be read by a screen reader program.

Many virtual objects in Second Life have text names, and Folmer’s program accesses those text names, even though they don’t appear onscreen. Visually impaired users can enter a text command, “describe,” and the software will translate the visual information into text that can be read by the screen reader. The user can then enter more specific instructions—for example, “describe avatars"—for further, more detailed information.

The quantity of initial descriptive information can still be formidable, but many visually impaired users will be accustomed to processing this sort of data.

“What sighted people don’t understand,” says Cooney, “is that they get 60 to 70 percent of their information about their environment visually and without that—it’s not like your other senses get better, I don’t believe that, I think that’s a myth—but the percentage of input that you get from the other senses increases, so your perception gets better.”

Nonetheless, a certain amount of information filtering is required, so the software gives a priority rating to objects in the environment and only objects above a certain priority rating get recognized with a textual description.

The software prioritizes based on userability, proximity and function. Only about 25 percent of objects in Second Life have proper names on the server, so these objects get prioritization, as do larger objects, objects with moving parts or a high quantity of component parts, and other users’ avatars.

Folmer is also developing the software to recognize certain objects based on spatial dimensions, though thus far with somewhat limited results: “It can currently recognize buildings with about 80 percent confidence,” he says.

There are a number of major limitations: The information can either be too much or too general. And the software responds to general, memorized commands, like “move” and “describe,” but not more complex interactive commands like, “drive this car.”

Folmer cites Zork, an RPG-style, text-based computer game released before the advent of graphic monitors, as a model for how the Second Life application for the blind should eventually work. Zork is like a virtual choose-your-own-adventure novel, with more precise feedback from and interaction with the environment, and viable game play.

“The fun aspect is really important,” says Folmer. “It should be enjoyable for the user and others to interact with.”

“Eelke’s research is perfect for me because it has the potential to get me involved with things I don’t normally do … new aspects of the computer,” says Cooney. “It opens up a stratum I would otherwise be restricted from. And it’s a two-way street: He’s making it more accessible on my end, but also helps programmers understand, ‘Hey, this is what this person needs to be able to interface with this software.’ He’s like a mediator … or an ambassador.”

And as Folmer says, virtual environments like Second Life could be the model for the future of the internet. If gaming is the forefront of interactive digital design, can the internet be far behind? Are today’s video games the models for tomorrow’s information highway, our primary tool for communication? Folmer’s research ensures that, if the internet reaches the level of immersive interactivity of contemporary video games, it will be accessible to everyone.