I have just completed sessions of usability testing of Thunderbird.

This time, I had the pleasure of working with Andreas Nilsson, who came to London to observe the sessions. It was very useful to get his feedback and to work collaboratively with him on the analysis and implications of the findings. In addition to these benefits of our work together, there is an added one: since he observed participants struggling with certain aspects of the interface, he will no doubt be a very effective user experience advocate with his team.

Andreas, thanks for your time!

The Test

Twelve participants were recruited from the general public – one turned out to be a no-show. They represented a mix of gender and age. Special consideration was given to heavy email users. Of the 11 participants, 5 were exclusively Windows users, 3 were exclusively Mac users, and 3 used both Windows and Mac.

In preparation for the sessions, we set up 2 test email accounts. A few days prior to the sessions, I sent messages to these accounts and also subscribed them to mailing lists. When participants signed up, they had already received a sizable quantity of emails, allowing us to ask them to manage the messages the way they generally do in their own email boxes, to find specific messages, to create filters, and more.

Thunderbird was tested on Maverick and Unity.

Between sessions, Thunderbird was removed and all hidden files were deleted, so the next participant got to start from scratch.

The Methodology

Over the 60 minutes of each session, I went through as many features of Thunderbird as possible with each participant. Participants were asked to:

• Install Thunderbird from the Software Centre
• Create an account
• Sign up
• Create filters
• Set up alerts
• Manage emails in folders
• Create a signature
• Change the colour of the font
• Create a contact list
• Search for a specific email discussing a form (which I had sent prior to the session)
• Respond to an email that contained an attachment: in particular, open the attachment, modify it and send it back to the original sender

What participants liked

There were many aspects of Thunderbird that participants enjoyed, and many tasks at which they succeeded.

Participants commented positively on the tab system, which makes the navigation between messages easy and immediate, and which provides visibility on multitasking. The tagging of messages also got positive evaluation. Many participants commented on the simplicity and usefulness of the contacts. Filtering was perceived as effective, although, as we will see below, the majority of participants experienced some challenges here.

Participants found the activities which they carry out most often – opening, reading, responding to and deleting emails – easy and straightforward.

Where the trouble is

Critical issues

Participants encountered few critical usability issues – by ‘critical’, I mean issues that would make it difficult or even impossible to use the application on a regular basis. These issues need to be addressed if we are not to lose users to alternative products.

Install

After installation from the Ubuntu Software Centre, participants could not find Thunderbird to start using it. They did not see, in the product description, the bread crumb indicating the location of the download when provided.

Observation: After having installed a new application, users generally are excited about using their new software. The user experience would flow much better if, as the process of installation ends, the application opens automatically in the main window, allowing users to deal with their settings and messages right away. We need to keep users excited about Thunderbird. As it stands, it is a bit of a let down to not be able to find the new toy!

Create folders

One of the main challenges for participants was managing their many emails by creating folders.

Most participants did manage to create a folder by right clicking on the folder area. However, they could not find the folder once they’ve created it, and so couldn’t drop messages into it. This was because they had in fact not created a ‘folder’ (as promised by the menu label) but a ‘sub-folder’. The sub-folder was not visible, because it was hidden under a folder.

Those participants who did eventually find the sub-folder they had created wanted to make it into a folder, but were not able to do so. Users normally organise their folders in a way that facilitates their use of emails. They tried to drag their sub-folder out of the parent folder and relocate it.

There are 4 main folders they want visible: inbox, sent, junk and draft. It is worth noting that the ‘sent’ folder in Thunderbird is a sub-folder of gmail; this was confusing to participants. As a case in point, several participants failed at checking if a message they had sent me had really been sent because they couldn’t find the ‘sent’ folder at all.

Participants also expressed a preference for ordering their folders. In addition to the point mentioned just above, some indicated that they like to create a work folder and a personal folder. They place these folders next to each other. They were not able to do this in Thunderbird.

Observations: Users manage their mailbox by customising folders. The level of customisation they need goes beyond creating and naming sub-folders. They want to create their own hierarchy of folders and sub-folders as well as to order them for convenience and visibility.

One more thing on this topic: participants were not clear about some of the words used to describe folders. For example, they did not know the difference between a ‘folder’ and a ‘local folder’.

Create filters

Most participants failed at creating a filter.

First, they didn’t know where to look to set filters up. Most participants looked under preferences and account settings. After looking generally at the menus they gave up.

Second, participants were unsure of the meaning of the dialogue boxes and of what was expected of them. They found the process of setting a filter unduly complex and they needed more feedback to measure their progress.

After participants managed to create a filter in the filter rules dialogue box, they clicked OK but didn’t know if the filter was actually set up or not. Additionally, they couldn’t figure out how to run a filter they had created. The issue was that, once having created a filter, when participants came back to the message filters dialogue box, the last filter set up is not selected – thus the run now option is not enabled. At the same time the enabled check-box is selected indicating that the item has been selected.

Observation: After setting up a filter, users would like to run it to confirm that it works. Make the command ‘run now’ the next step in the process without users having to specifically select the filter to run it.

Find open and modify an attachment

None of our participants was able immediately to find the attachment in a message. They expected the attachment to be visible at the top of the message. While most participants eventually found the attachment, some didn’t, and consequently could not open and modify it.

When participants did not find the attachment, they consulted help, but were not provided correct information.

After some participants found the attachment, I asked them to edit it. They did not expect that the attachment would be in a read-only mode and tried to edit it without saving it first. The message warning them that the document is read-only only appeared after many attempts. It would have been friendlier for the message to be shown at the first attempt.

A few participants, after they attached a document, were not clear if the document was in fact attached to the message. They needed a stronger visual cue.

Observation: Sending, finding and reading attachments are fundamental activities on email. The user experience would be greatly improved if the attachments were located where users expect them, at the top of a message and/or if they would be more visible by changing the appearance of the link or using a colourful icon. Additionally, users would benefit from some immediate feedback on ‘read only’ documents as well as from a confirmation that a document has been successfully attached to an email.

In this case, for Thunderbird to be user-friendly, it would need to anticipate users’ needs, mainly need for visibility and for feedback at the first occurrence of an error. This anticipation of users’ needs would show the willingness of Thunderbird to collaborate with its users and to recognize their goals.

Search

Participants were unclear about the differences between the 2 search boxes at the top left of the screen.

Often they didn’t get results because the global search bar doesn’t suggest anything other than names.

Search doesn’t take into account misspellings – and so, when a word was misspelled, participants got no results.

Every time a participant performed a search, a tab opened automatically even if the search provided no results. As a result, participants opened many tabs that were not useful or wanted. They found that the tabs cluttered the interface and made it difficult to find such things as the inbox.

Observation: Users should know, before searching, what the fields will be actually searching. The area dedicated to filters is interpreted as a search and not a filter by participants. In part, the issue for users is that the boxes look virtually identical, and thus, from their point of view, should be interchangeable. A different visual treatment would greatly improve the usability of the different search boxes.

Less-than-critical issues

Participants also highlighted usability issues that were not critical, but that compromised their enjoyment of Thunderbird.

Mail account setup

Participants did not understand the message contained in the mail account setup dialogue box. They had to make a choice between:
IMAP – Access folders and messages from multiple computers
POP – Download all messages onto this computer, folders are local only

Uniformly, they did not understand the implications of this choice and went for the ‘recommended option’ – just because it was recommended. Most participants said that they would not read the message anyway and would just accept and move onto the next screen.

One participant chose the POP option, which caused her problems with search later.

In addition, the mail setup message has a button that says “create account”. This was confusing for some participants who thought they had already created an account and now were in doubt. Some wanted to go back to the signup page to check. There is no way to come back to the signup page, however.

Observation: While users are setting up their account, they are most eager to get the process over with. This is in part, because they want to see the application but also because they need to see what they will get, so to speak, before they can understand the pertinence of the various options proposed to them. In this case, it is good practice to make a recommendation – which simplifies the process. However, the choice should be clear, from the user needs perspective (so users don’t just choose what is recommended because it is simpler and don’t foresee the consequences of their choice).

Set up alerts

2 participants expected to be able to set up alerts in tools, but were not able to. Many participants were not able to find a way to set up alerts at all.

Create a signature

The majority of participants expected to be able to create a signature under ‘composition’. When that failed, they looked under tools, add-ons, preferences, insert and write. Most who wanted to did not succeed at creating a signature.

Some minor issues

‘Minor’ usability issues don’t compromise the main usage of an application or the integrity of the user experience. However, they can be annoying and irritating, particularly when the application is used on a regular basis.

Change the colour of the font of their message

Most participants either failed at changing the colour of the font for all their messages or were not sure they had succeeded after selecting a colour from the palette of ‘display’ in ‘preferences’.

In part, participants could not find the option to change colours. For those who found it, when they selected a new colour, the new selection was not reflected in the messages they wrote just afterwards. But also, after they selected a different colour, they were not sure if the chosen colour would appear in their message.

Observation: Many users like to personalise their communication. Playing with colours and lay-outs should be easier for them, with relevant options more visible. In addition, users need some feedback that their change will be immediately implemented. An ‘apply’ button or a confirmation that the new selection has been registered would reassure them.

Navigational issues

Participants did not know how to get back to their inboxes from either the address book or the ‘write’ screen. They didn’t understand that in these specific cases new windows were opened, instead of tabs, and that they needed to close them to go back to their in-boxes.

And more…

Participants had further suggestions for new features. They wished for:

• A calendar on the side so they can see messages and their commitments at the same time
• A way to compress large files directly from the email account
• Some social networking, at least so that they could see that their friends are online