Dan’s Blog (2.0)

Once upon a time, a company called Apple came out with a great concept: a breakthrough consumer device with a new user interface that left the competitors in the dust. It brought UI to a whole new level by introducing a new visual and gestural language which greatly increased ease of use. In doing so, it lowered the barrier to entry for the general public, created new markets for its products and a revolution occured. Sound familiar? It should. I’m talking about the introduction of the Macintosh in 1984. The new visual language of pointing, clicking, dragging and using overlaping windows gradually became the dominant UI paradigm. But here’s the problem: other companies stole Apple’s great ideas (which Apple had actually stolen from Xerox but never mind). What could have rocketed Apple to market dominance instead became a commodity that anyone could implement.

Flash forward to 2007. Apple again comes out with a new UI paradigm, together with a visual and gestural language, and they release it as part of a breakthrough consumer device; the first of a series of devices in different form factors which they think will undo the last 20 years and rocket them to dominance of all things digital. But this time, they’ve got an ace up their sleeve: a string of patents. As Wired reported in February, Apple is trying to patent the gestures that make up the iPhone UI - the iPhone’s equivalent of “point and click.” In fact, if Apple’s efforts succeed, I think they will be shooting themselves in the foot. Why? Because if we are, en masse, to move to a new user interface paradigm, beyond point and click, we are going to have to have some consistency. If “pinching” means “shrink” on one device and “close” on another device, this would be a disaster from a user experience standpoint, and could turn potential users off in a big way.

In fact, we don’t have to imagine for too long because some of new breed of “iPhone killer” devices now hitting the streets exhibit this very problem. I was just looking at a touch UI device manufactured by an un-named Korean company (that also coincidentally manufactured my fridge which now is on the blink after only 3 years of ownership - not that I hold a grudge). The problem with this device was that it was replicating a non-touch UI (a UI controlled by a four-way rocker switch) with a touch-screen overlaid on top. It wasn’t quite as bad as the Prada phone that I wrote up last year, but it was close. For example, instead of scrolling by simply flicking your finger up and down, it required you to (repeatedly) press soft buttons at the bottom of the screen labeled with up and down arrows. I haven’t actually had the chance to test out the Nokia “touch” Series-60 device, but when I read this article in News.com with accompanying spy shot, my blood chilled. A scroll bar? Menu buttons on the bottom? Could it be that Nokia is falling into the same trap - trying to replicate a button-based UI with a touch screen overlaid on top? I sincerely hope not — indeed, I think Nokia has enough UI expertise to understand that touch needs a new visual gestural language.

But this brings us back to Apple and their patents. I am not a lawyer, but I don’t believe patenting gestures is a good idea. It seems like there’s plenty of prior art - take a look at Jeff Hann’s talk at TED on gesture-based UI as an example - but the main thing is: in order for us to move into this brave new world of touch, I would argue that gestures need to be royalty free, and companies need to know that if they implement commonly used gestures they will not be sued. If anything, we need standardization of gestures so that users can have some kind of consistency between touch-based platforms. The people behind Interactivegestures.com are moving in this direction, but it’s unclear to me what the intellectual property around these gestures (if any) is. What is the way forward to ensure that gestural and touch-based UI can flourish and isn’t hobbled by intellectual property disputes before it’s even properly off the ground? We briefly “touched” on this issue during a discussion on the future of mobile user experience at Over the Air led by UIQ’s David Mery and Idean’s Mikko-Pekka Hanski, but this topic alone needs more discussion. At risk may be the very future of human computer interaction.

Filed by Daniel Appelquist at 15:11 under mobile 2.0
Tags: Apple, gestures, iPhone, ipr, patents, touch, userexperience

7 Comments

I’ve been thinking a lot recently about the mobile user experience, particularly the experience of the Web on the (typical) mobile device. I say “typical” because I’m not talking about the iphone here — I’m talking about the kind of mass-market device that billions of real users hold in their hands every day. Increasingly these are devices that are capable of a reasonable data services experience, but they are still not being used to their potential. What is the new user paradigm that will truly kick start the mobile Web?

The essential innovation of the Web itself was putting together two existing technologies: hypertext and the Internet. Hypertext had been around for a while, in library-science and computer science circles and even in such products as Hypercard. Likewise, the Internet was around and widely used (mostly by academics and students) through well understood but essentially plain text paradigms such as FTP, Telnet and Gopher. Both these technologies by themselves were limited in their appeal. But somehow, layering Hypertext on top of the Internet (the Web) created something that was greater than the sum of its parts, and the Web as we know it was born. Yes, there were other factors at work in the birth of the Web but I believe it was the marriage of these two technologies that was the crucial factor.

When we come to the mobile Web, however — that is, usage of the Web on devices which are intended to be used one-handed, often with a four-way rocker switch instead of a universal pointing device — the underlying UI metaphor of hypertext breaks down. If you have to scroll through all the possible links on a “page” one by one in order to get to the one you want, this breaks the hypertext usage model and creates a usability nightmare. This may indeed be the crucial factor holding back the mobile Web. The whole rich Web user experience we’re now used to is based on the idea of hypertext. Take away hypertext as a tenable user experience metaphor and the whole thing crashes down around you.

Yes, there are smart browsers and other technologies coming at us that try to work around this problem, and some of them do a fairly good job. But these technologies are like treating the symptoms instead of attempting to cure the disease. The work of the Mobile Web Best Practices working group has stressed the idea of using existing Web technologies (such as XHTML Basic) to create a mobile-friendly user experience for content. I believe this approach does address some of the root causes as opposed to the symptoms and that it is already influencing a new generation of mobile-friendly Web content. But these guidelines still only go part of the way towards realizing the full potential of the mobile Web because the underlying metaphor is still essentially hypertext.

But what innovations exist that could take hypertext’s place as an underlying user metaphor? Clearly “widgets” (a la MacOS Dashboard widgets, Yahoo! widgets, etc…) are one candidate, but widgets generally exist as islands — an action in one widget does not generally bring you to another widget. Widgets tend to be very “one way” — that is, widgets are not generally rich environments for content creation. I think content creation will be a key part of the future of the Web on the mobile platform. So I think it is a bit facile to think that widgets are the future of the mobile user experience.

In the panel on designing for convergent devices that I attended at South by Southwest, Denise Burton from Frog Design made some very good points about focus (that is, of everything on the screen what is the “active” element — subject to actuation of the “OK” button or other functions such as text input) and the use of animated transitions to clue the user in when that focus changes. She also demonstrated a UI built on top of Qualcomm’s UI1 that brought some of these concepts to life (a kind of continuum of widgets with clear transitions to show which widget is active and the ability to dive deeper into a particular widget to get a more detailed view). The advantage of this approach is that it puts a large amount of information at a user’s fingertips and provides a highly efficient navigational metaphor (as few clicks as possible to get to the information you want). The only problem is that this type of experience exists as a closed universe — another walled garden. How do you take this kind of immersive UI and open it up to a world of developers? How can such a user experience support the kind of user choice that we have come to expect from the Web?

I do believe that WICD will play a role since it was built to support a user experience within a browser context that is also responsive to the kind of user events that are common on the mobile platform and can responded with scalable animations. By embedding multiple interactive SVG objects in a single page, WICD can also enable the kind of experience I’ve described above using (open) Web technologies and standards. But WICD is just an enabling technology. WICD also doesn’t address the content creation challenges on the mobile. The real innovation in mobile Web user experience needs to be built on top of such technologies.

I can’t quite put my finger on what it will be, but I feel I’ll know it when I see it. Any ideas?

Filed by Daniel Appelquist at 20:52 under Mobile Web, mobile 2.0
Tags: FrogDesign, mobile2.0, MobileWeb, sxswi, userexperience, W3C, WICD, widgets

7 Comments

I’ve been working with the folks at Mobile Monday New York and SVG.org to develop an event on mobile SVG for the end of this month. The theme will be SVG on mobile devices and we’ll be featuring presentations from Qualcomm, Opera, Ikivo Sun, Beatware and others. Antoine Quint of has put together an excellent roster of speakers and Lubna Dajani and the folks at MoMo NYC have lined up a great venue (the Samsung Experience at the Time Warner Center).

For more details, or to register for the event, please visit the event web site.

Filed by Daniel Appelquist at 11:50 under Mobile Monday, Mobile Web
Tags: events, mobile2.0, mobilemonday, Mobility, MoMoNYC, SVG, upcoming:135061, userexperience

1 Comment