Beyond Point and Click

Pinch GestureOnce upon a time, a company called Apple came out with a great concept: a breakthrough consumer device with a new user interface that left the competitors in the dust. It brought UI to a whole new level by introducing a new visual and gestural language which greatly increased ease of use. In doing so, it lowered the barrier to entry for the general public, created new markets for its products and a revolution occured. Sound familiar? It should. I’m talking about the introduction of the Macintosh in 1984. The new visual language of pointing, clicking, dragging and using overlaping windows gradually became the dominant UI paradigm. But here’s the problem: other companies stole Apple’s great ideas (which Apple had actually stolen from Xerox but never mind). What could have rocketed Apple to market dominance instead became a commodity that anyone could implement.

Flash forward to 2007. Apple again comes out with a new UI paradigm, together with a visual and gestural language, and they release it as part of a breakthrough consumer device; the first of a series of devices in different form factors which they think will undo the last 20 years and rocket them to dominance of all things digital. But this time, they’ve got an ace up their sleeve: a string of patents. As Wired reported in February, Apple is trying to patent the gestures that make up the iPhone UI – the iPhone’s equivalent of “point and click.” In fact, if Apple’s efforts succeed, I think they will be shooting themselves in the foot. Why? Because if we are, en masse, to move to a new user interface paradigm, beyond point and click, we are going to have to have some consistency. If “pinching” means “shrink” on one device and “close” on another device, this would be a disaster from a user experience standpoint, and could turn potential users off in a big way.

In fact, we don’t have to imagine for too long because some of new breed of “iPhone killer” devices now hitting the streets exhibit this very problem. I was just looking at a touch UI device manufactured by an un-named Korean company (that also coincidentally manufactured my fridge which now is on the blink after only 3 years of ownership – not that I hold a grudge). The problem with this device was that it was replicating a non-touch UI (a UI controlled by a four-way rocker switch) with a touch-screen overlaid on top. It wasn’t quite as bad as the Prada phone that I wrote up last year, but it was close. For example, instead of scrolling by simply flicking your finger up and down, it required you to (repeatedly) press soft buttons at the bottom of the screen labeled with up and down arrows. I haven’t actually had the chance to test out the Nokia “touch” Series-60 device, but when I read this article in with accompanying spy shot, my blood chilled. A scroll bar? Menu buttons on the bottom? Could it be that Nokia is falling into the same trap – trying to replicate a button-based UI with a touch screen overlaid on top? I sincerely hope not — indeed, I think Nokia has enough UI expertise to understand that touch needs a new visual gestural language.

But this brings us back to Apple and their patents. I am not a lawyer, but I don’t believe patenting gestures is a good idea. It seems like there’s plenty of prior art – take a look at Jeff Hann’s talk at TED on gesture-based UI as an example – but the main thing is: in order for us to move into this brave new world of touch, I would argue that gestures need to be royalty free, and companies need to know that if they implement commonly used gestures they will not be sued. If anything, we need standardization of gestures so that users can have some kind of consistency between touch-based platforms. The people behind are moving in this direction, but it’s unclear to me what the intellectual property around these gestures (if any) is. What is the way forward to ensure that gestural and touch-based UI can flourish and isn’t hobbled by intellectual property disputes before it’s even properly off the ground? We briefly “touched” on this issue during a discussion on the future of mobile user experience at Over the Air led by UIQ’s David Mery and Idean’s Mikko-Pekka Hanski, but this topic alone needs more discussion. At risk may be the very future of human computer interaction.

Liked this post? Follow this blog to get more. 

7 Comments on “Beyond Point and Click

  1. Small nit: Apple licensed the tech from Xerox, they didn’t steal it. In return, Xerox got some amount of Apple stock that they later sold for quite a lot of money. Xerox sued Apple during the Apple v. Microsoft days, but that is widely viewed as having been a defensive act that would have allowed them to collect monies from Microsoft if Apple had prevailed.

  2. Hi Dan, one thing I hear from the iPhone developers is that it is very difficult to really test the UI other than loading the app onto the phone for the very reason of multitouch. I am not aware of any Mac/PC (including Tablet PCs) that have multitouch interfaces … so I am not sure how Apple test it themselves? Thoughts?

    Cheers, Alex

  3. Testing is always a key issue with mobile development, and the iPhone is no different. It’s just funny that iPhone developers (read: “used to developing on Macs for Macs”) are now learning the lesson that mobile developers have known for years: you need to test on real devices. It adds a cumbersome step to the development process, but I don’t think it’s avoidable.

  4. I think you have to test on-device for the sake of user experience, but what do you do when something fails, you rely on the device telling you where the problem is? I guess this comes down to how Apple (et al) integrate a development environment into the device so you can automate things a bit better. I just think not being able to debug and fix an application either on device or in a simulator/emulator is a real barrier. As a tech community we should not allow ourselves to go back to the days before good debug/simulation enviornments, worst UX is when an app is buggy and dies.

  5. Manufacturers trade patents all the time. Few are kept to create a temporal monopoly except for a short time. Thus Apple’s patents doesn’t mean the gestures won’t be implemented in other devices if the user’s demand it.

Leave a Reply

Your email address will not be published. Required fields are marked *


This site uses Akismet to reduce spam. Learn how your comment data is processed.