Twitter Usability Test: “I don’t get it.”

The reported low 40% retention rate for Twitter really got me wondering. I’ve been on something of a mission to understand why it might be so low. As part of that mission, I recently conducted a usability study on six subjects. All these users either had never used Twitter at all and joined as part of my test, or they had set up an account but hadn’t been back since. In this post, I report my findings from this initial usability study.

First, a little about the subjects. They were all female and over 35, with most between the ages of 45 and 54. All but one of them was currently unemployed and actively looking for work. (I recruited for the test from a group for those seeking employment.)

I had each user fill out a short survey before the test, and then did a verbal protocol test with them. A verbal protocol involves simply observing the subjects as they use Twitter, and asking them to “think outloud” as they do so. This allows the tester to better understand why the user is taking certain actions, such as clicking on different items of the page.

Below are some key findings from this study:

• Users expect that Profile will let them create their profile. It takes them a while to figure out that Settings is where they do this.
• Users don’t have any difficulty figuring out how to search for and add followers.
• Users do have difficulty figuring out who they should follow and finding people they know on Twitter.
• Users who go to the Help (and in this test, it was a higher proportion than I’ve normally seen go to Help) tend click the Getting Started link.
• Some users think of the Twitter stream as an email inbox, and dislike the clutter.
• Users may be concerned about getting too much “junk” either via tweets in Twitter or sent to their email from Twitter.
• Many users seem reluctant to tweet, a tendency perhaps exacerbated by the “What are you doing?” prompt.
• Users are unsure of the value of Twitter and how it can be useful to them.

Assuming these users are typical new Twitter users, for any given problem one of them encountered, we can safely assume that a significant proportion of the general population of new Twitter users will also encounter the problem.

Creating a Profile and Entering Other Settings

Two of the six users clicked the Profile menu expecting to be able to create their profile there. One clicked it several times and thought Twitter was “broken.” Both of them eventually did stumble upon the profile definition under Settings.

One user encountered an error because she tried to enter “Greater Seattle area, Washington” into the location box. She then shortened it to “Greater Seattle area” and was fine.  Two of the six users discovered and changed their background designs.

“Settings contains the info I assumed would be under Profile.”

Finding and Adding People to Follow

All the users were able to find and add followers either through the screens that appear as part of the Join Twitter process or through the Find People menu.  Although one user did initially think Find People would tell her who the people were that were mysteriously following her.

Users did have difficulty figuring out who they should follow on Twitter. The users who joined during the test usually didn’t see why they would follow the people Twitter suggested to them. “Why do they think I’d want to follow these people?” Users often tried to search for people they knew, but in many cases couldn’t find anyone they knew on Twitter. In other cases, they found people that might have been their friends, but couldn’t tell for certain if those users were the people they knew.

Two out of the six users entered celebrities names in an attempt to find someone on Twitter. (I didn’t ask, but it’s quite likely they’ve heard in the media about the number of celebrities on Twitter.) These users were both put off, however, when they saw the number of different accounts with the celebrity’s name. They weren’t sure which one was the real deal and chose not to follow any of them when they couldn’t discern which was the true celebrity account.

Help and Getting Started

Four out of the six users clicked Help in Twitter. As someone who worked on and tested online Help at Microsoft for many years, I can tell you that this is an unusually high percentage. (When we were attempting to test Help at Microsoft, we often had to resort to actually asking users to please click the Help, as they would never go to it on their own.)

Three of the four users who went to Help clicked the Getting Started link under Using Help Resources. (None watched the video.)  One of these users then went the FAQ. That text has an apparently outdated reference to the Everyone tab, which caused this user to go back and hunt for the non-existent tab.

The fact that such a high percentage of users went to Help, along with some of the other observations from this test, suggests that users are confused about how to use Twitter and/or what they should use it for. Help is usually a last resort when self-exploration fails, and a large percentage of these users resorted to it.

Twitter Stream Confusion

Twitter is a bit of a different model. It’s something like a blog, something like email, something like IM. Users exhibited some confusion about the tweets they saw and exactly what they were. One user clicked on them expecting them to lead to the “full message” akin to clicking on an email snip in one’s inbox. Another user was confused about who was sending the tweets. Users easily found and clicked the user names of the tweeters, which precede each tweet. But it sometimes took them a while to realize this had taken them to the other Tweeter’s page, and that they were seeing that person’s stream.

Two of the six users wanted to remove the messages in the stream. “How do I get rid of 140 messages?” They expressed a desire to clean up their page, akin to cleaning up their inbox in a mail client. One person actually searched in Help for “Delete messages.”

Excess Tweets and Follower Anxiety

Several users expressed concern that they might get tweets via email and be inundated. They were somewhat reluctant to give Twitter their email information, perhaps for that reason. Only one that I observed used the import from email option during the join process.

Even within Twitter, several users expressed concern about being inundated with mundane tweets.  They professed not to be interested in what a stranger ate for lunch and that kind of thing, and worried that they would get a lot of that useless messages. Upon reading a portion of the help, one user remarked, “Good. That means I’m not going to get an email everytime somebody’s sneezing.”

“I don’t know exactly what follow means, but….” The second-time users who had set up their account a while back and not returned yet, also seemed perplexed at how they had gotten followers. They did not understand that people could simply choose to Follow them. “I have 9 followers. How did that happen?”  Several users seemed alarmed at these unsolicited and unapproved followers, and one expressed that she preferred Facebook’s model where you had to approve people. Another user figured out blocking and blocked many of the unsolicited followers.

Users also didn’t understand why these people would want to follow them. “I don’t know who these people are. Why would they want to follow me?” and “Why would he be following me?”

Nevertheless, once users got to another Tweeter’s profile page, many of them clicked on the Followers or Following list for that Tweeter, and often selected people to follow from that list. Though they often remained confused about who and why they were following people.  “What if I don’t want to follow anybody?”

A General Reluctance to Tweet

These users exhibited a general reluctance to tweet. This may have been partially related to their age and/or to the fact that they are active job-seekers and hence more concerned with their image. (I’ll be conducting another round of tests on a younger demographic who are not active job-seekers, and will report those results in a later blog entry.)

Out of six users, remarkably, only one actually tweeted during the study. The others were extremely hesitant. They expressed concern that they had nothing of interest to say. “Why would I tell them I’m sitting in the library. That’s totally boring.” and “I think I’d leave this because I don’t really want to say what I’m doing every ten minutes.”

Others expressed concern that they didn’t know Twitter that well or how to use it (perhaps they meant they didn’t know what kinds of things people tweeted about). “I don’t want to tweet before I know what I’m doing.”  and “If I put something in here, who cares? It assumes I know what I’m doing.”

It remains to be seen whether this feeling will abate after using Twitter for a while. I’ll be sending a follow-up survey to these users to see whether they have returned to Twitter and whether they are using it. I’ll post an update to the blog on the results.

I Don’t Get It

Every user in the test expressed frustration or confusion at the end of the test. The general feeling was, “I don’t get it.”

Some users searched topics or clicked a Trending Topic, but didn’t find anything of great interest to them and didn’t seem to know what to search on. As one user put it, she had “No knowledge of what to discover.”  Others noted they’d done all the steps Twitter suggested (well, except tweeting) but didn’t know where to go from there. “So, I’m following all the people who are following me. Maybe that’ll start something.” 

Other, similar comments were “What am I supposed to do?” “I’m not a blogger.” and “How can this help me?”

In general, users seemed confused about what they were supposed to do with or in Twitter and not inclined to spend much time playing around with it.

Recommendations

Based on this study, here are some changes Twitter might want to consider:

Change the “Profile” menu to “View Profile”, and change “Settings” to “Profile and Settings”. Or, get rid of “Profile” entirely and put a “View Profile” option in the Settings, as a lot of SNs do.

Do a pass on the Help to eliminate any outdated material or references, such as the Everyone tab reference. Given the users’ prediliction to go to Help and specifically select Getting Started, it’d probably be a good idea to streamline that experience. (Spoken as a long-time tech writer.) Perhaps go straight to a simple, short page with the basics about Twitter and link from there to additional, more detailed information.

Given that people have a tendency to look for celebrities to follow, especially when they don’t know many people on Twitter, it would behoove Twitter to highlight the “real” celebrity accounts in some way.  Presumably, if users see tweets of interest, they are more likley to come back to Twitter. And they’ve searched for celebrities because they are at least moderately interested in them.

Twitter might have more success with the “Look who else is here” page if they perhaps had made Look who’s here a tabbed box with a few categories likely to appeal to broad sets of people. For example, Sports, Entertainment, News, Politics. Then, list celebrities or well known figures within those categories.

Given people’s reluctance to make that first tweet and their general uncertainty about what to do, it might be helpful to apply the old “training wheels” concept and give new users a hand-holding interface. It or elements of it could disappear after the user has returned a certain number of times or tweeted a certain number of times.  This special, new user interface could perhaps have the reply and favorite icons always visible to start with. It could perhaps prohibit people from following the new user uninvited for the first few sessions. And it might even keep the user’s tweets off the public timeline for the first few sessions, to encourage people to tweet.

For new users, Twitter might also benefit by having not just trending topics, but a set of broad and popular topics displayed. Again, Sports, Entertainment, Politics, etc. Perhaps with a special interface that allows the user to drill down (Sports, Basketball, Seattle) to really ensure the new user finds topics of interest to draw him or her back to Twitter.

Conclusion

While Twitter has been quite a phenomenon and has a lot to offer people, it’s barriers to entry appear high. To keep broadening its userbase, Twitter will probably need to lower those barriers and find ways to make itself less intimidating to people, make it easier to learn the Twitter model, and easily enable people to see the benefits that can be gained from returning to Twitter on a regular basis.