Push Button Phones and the Limits of User Testing

Noel Rappin, Friday, December 5, 2008 @ 10:14 am

Two of the more unlikely layouts,
from mental_floss

I loved this mental_floss blog blog item showing user testing from 1960 of the layout for a push-button phone. (Though I'd be interested to know where mental_floss got the data...) By now we're all used to the layout, and even used to the fact that the phone and calculators have almost directly opposite layouts.

One other thing about the mental_floss article -- the writer suggests that mechanical calculators were around long before this user test. This is true, but as far as I can tell, those calculators did not use the 3x3 layout. I don't think the 3x3 layout became standard on calculators until after 1960.

That detail out of the way, the phone testing data is interesting, both for diagrams of the 18 semi-final designs. Some of those designs are clearly the result of a fevered brainstorm session -- a cross, two diagonal rows, or a circle with the zero in the middle.

Seeing the rejected layouts is interesting from a historical perspective, but what interests me most from a UI standpoint is the data for the five finalists, and what it says about how to interpret user data. The blog post doesn't have the details, but it seems like users were timed on dialing numbers, error rates were checked, and users were asked their favorites and least favorites.

The five finalists are:

• The 3x3 plus one layout that eventually won out
• Two rows of five with 1/2/3/4/5 on top and 6/7/8/9/0 below
• Five rows of two, 1/2 on top, followed by 3/4, and so on
• A layout that mimicked a rotary phone, with buttons in the same positions as the dialing holes.
• A "speedometer", with the numbers laid out in a circle, with 1 at about the 7 o'clock position, moving counter clockwise to 0 at the 5 o'clock position.

All the blog post has is the final aggregate data -- here, let me display the data slightly simplified:

There are a some more trivially interesting pieces here -- for instance, why did everybody hate the five rows of two design -- but I'd like to draw your attention to two facts:

1. There's no way you can come out of this table and make any kind of claim that the 3x3 layout is superior.
2. A strict reading of the data strongly suggests that the rotary layout would have been superior.

Why don't we have rotary style push-button phones? In hindsight, it's easy to make the case that user's objective and subjective preference for the rotary dial was primarily because of their familiarity with the layout. (We're all agreed on that yes? Nobody pining for a rotary pushbutton on their cell phone or anything?) At the time, though, I'd imagine that it was also possible to make the case that the rotary layout was just intrinsically better for dialing -- hey, even the word "dialing" implies a round design, right?

I can even picture the meeting -- the engineers come in with the data and the recommendation to go with the 3 x 3 layout, and one or two fellows in the corner harumphing about following the data where it leads.

The point, though, is that as helpful is it is in general to test design with existing users, you also need to keep in mind that existing users come in with a specific set of expectations and experiences that might not apply to new users. It can be hard in the midst of the user testing to separate the intrinsic greatness of a given design from it's apparent ease due to similarity to what users are familiar with. (Sometimes, of course, similarity with previous experience is a goal in and of itself).

Another point might be that there may not actually be that much difference between the different layouts, given even a small amount of user training. As late as 1980 or so, the Chicago Museum of Science of Industry had an exhibit that compared dialing speed between rotary and (still relatively new) push button phones. Almost any of the push button layouts would be faster than the dial (well, maybe not the weird plus-shaped one...)

Still, though, the data begs the question: why aren't our phones two rows of five? Was compactness or something else considered a design constraint?

agile ajax

Comments: 1 so far

• Leave a comment
• Comment feed for this post
• TrackBack URL

1. 

   [...] you see the article about usability studies for the push-button phone? It’s fascinating. The one we’re used to isn’t the one that tested [...]

   Pingback by Design of everyday things | things that confuse me, Thursday, December 25, 2008 @ 5:12 pm

Leave a comment