Recently we hired an external consultant to compare the usability of 2 email clients: Thunderbird and Evolution. I have taken some highlights from the report to compose this blog.

Setting of the usability session

The sessions took place in early June at the Canonical Office in London. Thirty participants were recruited. All of them used at least 2 email clients.
Methodology

One email account was set up in preparation for the sessions; all users were asked to use this account’s details to set up the email package. Days prior to the testing, messages were sent to this account and it was also subscribed to a mailing list, in order to ensure a realistic influx of emails to this Inbox.

Half of the participants interacted with Thunderbird and the other half with Evolution; Thunderbird 5.0 build 1 and Evolution 3.1.2 were used on a desktop machine using Ubuntu 11.10.

During each 60 minute session, participants were asked to:

• set up the relevant email package for an existing account;
• create a signature;
• compose an email and change font colour;
• manage emails in folders;
• locate a specific email containing an attachment (sent prior to the session);
• respond to an email containing an attachment and send the latter back to the sender;
• create a contact list and send a message using contacts.

Highlights of Report

What Participants Liked

Thunderbird

• Straightforward and familiar set-up
• One-click Add to Address Book feature in email preview and window
• Window/message tabbing system
• Familiar, intuitive language
• Quick search that meets expectations

Evolution

• Useful guiding steps in mail configuration assistant
• Intuitive contextual menu option to Add Contact in email preview and window
• Menu items easily accessed as alternative to button shortcuts

Both

• Both were seen as having a familiar layout
• Both met expectations in terms of generally intuitive access to contextual menus
• Both provided intuitive access to search facility

Where Participants Struggled

Thunderbird

• Confusion over search use (severe)

Users were confused by the existence of two search fields, often opting for the All messages search box as they intuitively saw this as highest in the hierarchy. This choice often resulted in disappointment when users did not expect to be taken away from the folder they were searching in; in addition, they found the search results confusing and inefficient, reporting that they expected the item they were searching for to be more easily visible.

Participants were further frustrated by the fact that if they had misspelled an entry or their search returned no results, they would not be aware of this until taken to the search results tab, which they saw as a frustrating waste.

• Difficulty locating and managing folders (severe)

The majority of participants successfully created folders, either by right-clicking in the folder area or using the New functionality in the top menu. However, most of these users were unable to easily locate the folder created or move once they had located it. This was due to them not realising they were creating subfolders; once subfolders had been created, unless that folder already had folders within it and it was expanded, users did not notice the expand icon next to the folder and bypassed it. Finally, once they found the created folder, users attempted to relocate them to the desired place; majority of users failed in doing this successfully.

• Difficulty personalising message text (mild)

More than half of users struggled finding the function to change font colour; the majority looked for this in the message text toolbar, bypassing the colour functionality because they expected the icon to look differently. Users eventually found this with the use of tooltips, but not after looking through all toolbar and top menu options first. Participants voiced the issue for this to be the icon being in black and therefore too subtle; they mentioned preferring a more colourful icon or one resembling a palette.

• Unclear server options (mild)

Participants reported liking the apparent ease of setting up but most were confused by the server options provided in the second, and final, step. About half reported that they would navigate away from the window and research more into their options, with the rest either ignoring this message and go with the IMAP option already selected or choosing the POP option which caused them some issues finding emails later on. The majority of users reported preferring helpful information and guidance on the options provided in the set-up screen, in order to avoid navigating away or uncertainty.

• Difficulty finding and personalising signature (mild)

The majority of participants were unsure where to find the signature functionality, with the majority expecting it to be either in the main toolbar, message toolbar or Message menu section. Most participants were unable to find this feature on their own without looking up help or reporting that they would ask a friend for help.

Evolution

• Longwinded, unexpected set-up (severe)

Despite appreciating the guiding steps outlined in the mail configuration assistant, the majority of participants reported feeling that this process was unexpectedly too long and found the options provided very technical and confusing. For the majority of users, this culminated just at the second step, where they thought they were being asked to retrieve backed-up files, rather than being offered option to set up this feature. Some users failed at this point, reporting that they were confused by this and would revert to using the current email set-up they had.

• Locating account email (severe)

The majority of participants had difficulty initially locating account email due to the email folders displaying an Inbox and an account-specific email section. Most participants did not notice that the account email area was collapsed and were confused about the ‘Inbox’ shown at the top of the folder list not showing any messages. Users attempted to view the account Inbox by selecting Send/Receive and then clicking through all folders available. Eventually users noticed the email account folder with the expand icon next to it and accessed the account folders that way. This experience caused great alarm in these users, particularly as it was at the beginning of interaction with the system; as a result, many reported loss of trust in the package and considering ending its use.

• Unintuitive message search (severe)

As discussed, search was intuitively used by participants to quickly find a required message in a large Inbox. Many participants failed finding the required search results because they carried out a search unaware that they had selected an irrelevant folder. This resulted in no results being returned and users being confused because they had expected to be able to search all folders.

• Once email opened, difficulty getting back to Inbox (severe)

Half of participants naturally double-clicked to read email in more detail; however, in Evolution, this resulted in email opening up over the main Inbox window, hiding the email list. Participants were confused by this and struggled to get back to the message list; majority reported looking for a button or link to Inbox and were extremely weary of closing the window down (either via the buttons or menu items) because they were nervous about potentially closing down the entire email application.

• Inability to personalise email text (severe)

Almost all participants were unable to personalise message text in Evolution; they expected access to font colour to be available along with the other font toolbar options and entirely bypassed the HTML option. One participant selected HTML and still missed the font colour option. Participants were very disappointed by the lack of this feature and looked at all toolbar and top menu options for access to this.

• Despite long set-up, confusion over lack of password request (mild)

In addition to finding the Evolution email set-up longwinded, participants were confused why this had not asked them for account password details. The majority saw this as a frustrating time waster, particularly as they were asked for this separately, once their email had been set up.

• Difficulty locating and managing folders (mild)

As with Thunderbird, the majority of participants successfully created folders, but here mainly by using the Folder functionality in the top menu. All participants expected to be able to create a folder by right-clicking in the folder area and only a few right-clicked on a folder to look for this functionality. Despite being able to create the folders using the top menu, users were disappointed with the lack of quicker access to this feature in the folder area (either by right-clicking or with the use of a button) or a button in one of the top toolbars.

Usability Issues Common to Both

• Difficulty finding and personalising signature (mild)

As with Thunderbird, the majority of participants were unsure where to find the signature functionality, with the majority expecting it to be either in the main toolbar, message toolbar or Message menu section. When users found the Signature option in the message toolbar, they were very frustrated that this did not provide a shortcut to signature creation. Most participants were unable to find this feature on their own without looking up help or reporting that they would ask a friend for help.

When they were taken to the signature feature, users were again frustrated at the fact they could not find a font editing facility, despite the interface looking like it should allow for this.

Conclusions

As discussed, users gave both positive and negative feedback on their interactions with Thunderbird and Evolution, with Thunderbird consistently being perceived by users as easier to use and fit for purpose than Evolution.

Thunderbird was widely liked for the perceived straightforward set-up and facilitated access to contact save, search and open windows features. In addition, users commented on the familiar language used in the application.

However, participants encountered a few severe issues which tarred their image of the system. These consisted of extreme, at time show stopping, difficulty with:

• successfully understanding and choosing the relevant search field to use;
• locating and managing the preferred location of folders.

Finally, these users encountered some lack of clarity over server options in set-up and frustration at the inability to easily format email text.

Participants who interacted with Evolution liked the guiding steps in the mail configuration assistance, the intuitive contextual mention options to add contacts and the ability to easily access alternatives to button shortcuts in the menu.

Users reported multiple severe issues around the Evolution set-up, locating account email, message search use, formatting email text and navigating back to the Inbox. All of these issues were so major that users encountering them reported lack of trust in the Evolution package and a reluctance to continue its use.

One major fact to keep in mind is that, especially as the majority of participants were new to Ubuntu, they saw the email application they used as a representative of the operating system. This is particularly pertinent to the email system that is a system default and it should be ensured that, before either one of these products is chosen for this purpose, the severe issues reported here are addressed.