4 Problems with Instant Usability Assessments

Last week, ReadWriteWeb published “5 Problems with Gmail’s New Design,” a literal handful of complaints about Google’s new mail interface. The lede:

Gmail’s redesign may come with a bunch of spiffy new themes that look great in screenshots, but the actual usability of Gmail is in steep decline.

I’ll set aside the numerous issues I have with the article to focus on a broader point. It’s rarely a good idea to assess the “usability” of a new interface right after its release, as ReadWriteWeb has done. Here’s why.

4 Problems with “5 Problems With…” Posts

1. Where’s the data?

A very wise man once said about usability assessments, “Yeah, well, that’s just, like, your opinion, man.” Google collects vast amounts of data about the usability of their interface, both prior to launch and post-launch. Unless the author is a credentialed HCI type (and even then, treat skeptically), it’s unlikely that the author can offer much more than an “I don’t like this” opinion about a new interface. Sometimes, designs that seem like poor usability decisions actually turn out to be highly successful, like when Twitter added steps to its new user flow.

2. No context

Any conversation about a product’s usability must refer to the usability of a specific feature for a specific audience. RWW takes issue with the new Google UI styles, likening it to a Mondrian painting and claiming that “it isn’t actually that useful.” Not useful for what? Not useful for whom? In what context?

3. Long-term usage not accounted for

What looks like a usability issue at first might actually be a better way to operate over the long run. Users need time to acclimate to a new interface. My friend David Choi summarized the user research he did in support of GMail’s new design, specifically with regard to longitudinal studies (i.e., studies of usage over long periods of time). If designers have done their jobs, users will discover new and better ways to complete tasks which may not be obvious at first use.

4. Usability is not the same for everyone everywhere

“Usability” is not some single metric that improves or declines for everyone with each new release. Products like GMail serve many audiences, each with different needs and modes of use. Things I don’t like about the new GMail may well be things that help my mom use the interface. Usability rarely increases or decreases uniformly for all users of a system as vast as GMail.

In conclusion

So, while I share a couple of RWW’s concerns about the new GMail interface, I’m not about to claim that the usability of GMail is “in steep decline.” I don’t have anything but my own opinion to stand on to make that claim. What I don’t like about the new GMail may very well make email easier or simpler for someone else.


    Ek |

    “Not useful for what? Not useful for whom? In what context?”

    You sure are self-righteous, aren’t you?  Certainly it must be shocking, but perhaps the writer actually had the unimaginable gall to be making a general statement.  Worse yet, perhaps a statement so general is nevertheless true.

    But hell, what does he know?  He’s just another stupid change-fearing sheeple, right?  After all, being a reactionary luddite is the only reason anyone ever rejects change.  It’s not like complaints can ever be valid.

    Have you warmed up to the Gawker Media redesign yet?

    You see, in contrast to the beliefs of marketing executives, people don’t actually become “comfortable” with new interfaces.  They become tolerant.  Yes, there’s a difference.  People come to grudgingly accept an inferior user experience, and if you ask them years later, they will report that they still find it inferior.  Eventually, the backlash dies down because the users come to terms with their powerlessness, though in many cases consumption drops.  Oh well.

    Keep up the good work.  Corporate America needs more consumers like you.

    Caleb |

    Ek, I hesitate to reply to your comment because it is heated and rude and therefore doesn’t really warrant a reply. I think you’re being a little unfair.

    All that he is trying to say is that it’s easy to not see the forest for the trees. I don’t think the post is about being a consumer for ‘Corporate America’ so much as it is about the ease with which people complain about something new before making an attempt to understand it. He’s not making any commentary about the recent changes with GMail or saying that nobody should complain ever, just that it’s easy to complain and certainly more so when making a snap judgement. What you might not like about something today, you may very well love tomorrow.

    If you feel the need to respond, please at least keep it civil. There’s no need to be a jerk.

    Ek |

    I don’t think I’m being unfair.  Matt is very much making a claim “about the recent changes with GMail” in the sense that, since Google collects traffic data, Google’s ideas about user interface design are more valid than anyone else’s.  In a more general appeal-to-authority sense, Google is right because they represent the website, while the users are just ignorant peasants, which is a position that quite frankly presents little surprise coming from a web designer (or rather, a senior user experience designer).
    When Matt says this:
    > Unless the author is a credentialed HCI type (and even then, treat skeptically), it’s unlikely that the author can offer much more than an “I don’t like this” opinion about a new interface.
    There is hardly any possible reading that doesn’t infer precisely that criticisms are automatically invalid.  It literally says that unless you are credentialed in human-computer interaction (and even then, it hedges parenthetically), your opinion is “unlikely” to have any merit.  Seriously, read it again.  In other words, “Yeah, well, that’s just, like, your opinion, man.”  The truly astonishing thing is that well-researched human interface guidelines do exist, but the unassailable experts just aren’t following them.
    Matt’s statements don’t exist in a vacuum.  Context is important.  Powerful entities are always making changes that upset end user experience, and when the users complain, they are universally scolded for being unreasonable, being reactionary, being “afraid of change”, and told that they will eventually “get used to it” or even that they “may very well love” it “tomorrow”.
    The problem is that these criticisms of criticism, these metacriticisms, almost never take the form of an actual rebuttal.  They are too rarely “you are wrong, because this is actually good, and here’s why”.  Instead, they are principally “you are wrong, because you are being reactionary and have no valid basis for criticism”.  This is not just a poor argument, it is actually an ad hominem fallacy.  It doesn’t matter whether you think someone is justified in making a criticism;  what matters is the actual content of said criticism.
    As was mentioned in a comment thread (news.ycombinator.com/item?id=3226223), if it is true that a person develops a preference for that which is familiar, then their first reaction to a change may well be the most unbiased opinion you are likely to get.  Particularly when changes are in beta, many show up to present the same ridiculous catch-22:  everyone should politely withold criticism until it’s too late to change anything.  In reality, “as soon as possible” is exactly when criticisms should be heard, before ideas become entrenched and the default presumption of acceptance goes even more mainstream.
    Unfortunately, groupthink usually wins.  In general, product changes backslide even when they do advance in some other area.  There is a perception that consumers always have their eye on “the next big thing”, but this is not merely due to novelty seeking;  it is also because consumers are seeking to escape the previous sinking ship.  Savor the early days of a product, because that is likely the best it will ever be;  it’s all downhill from there.
    Now someone will cleverly point out that this cynical position is being expressed over this amazing wonder called the Internet, so there.  Obviously, improvement does take place, but it normally happens only when a new contender arrives to displace an incumbent that has taken to resting on its laurels.  This is as true in the market as it is in politics, and, according to Max Planck, it’s even true in science.

    Matt |


    I appreciate the time and passion you put into your response. Obviously I take issue with your characterization of my post–of course negative initial reactions matter. Of course anyone can and should offer their opinion of the GMail redesign.

    The only point I’m trying to make is that broad claims about the usability of an interface as complex as GMail’s should be based on more than just personal experience with the UI. How the RWW author and I use GMail is not how most other people use it. I agree with some of the points the RWW author made, I disagree with his conclusion that the usability of the entire app is in “steep decline.” It may be, but I feel one needs more time and more data to make such a claim.

    That’s it. In no way did I mean to imply that only HCI experts should offer criticism about the usability of an app, I apologize if it came off that way.

    Thanks for reading.