First and last impressions of Unity were that it was quite user-friendly, and pleasing in its design and ease of learning. The majority of participants left the session with very positive feelings and were looking forward to Unity’s release so they could download it. In short, participants in this testing session were considerably more positive about Unity than participants who tested the previous version in October.

This improvement, no doubt, is due to the significant changes we have made since the last testing, often in response to problems uncovered during that testing. Many of the serious issues discovered then have been resolved. Most significant, as it stands now, there are no longer any “show-stoppers”.

However, there are still a few interactions that were at odds with the product’s general ease of use.

Some important points to keep in mind

First, it appears that those of our participants who were Mac users seemed to have had more facility with the Unity interface than Windows users, especially those using anything previous to Windows 7. Generally, Windows users tended to rely on right click, and they sought menus from which they could find and launch applications as well as move and delete. They did not immediately take advantage of Unity’s visual assets. Accordingly, Windows users will need to be encouraged to manipulate icons and to develop a more physical relationship with Unity than the more text-heavy relationship they have with Windows.

Second, Unity’s concept of ‘Home’ (Nautilus file manager) is different for that of our users, even Mac users, and they did not immediately understand it. They had a tendency to go to the ‘Home’ icon, not only to find information about their computer, but for any programme or application they were looking for. Essentially, many navigated from one application to another using ‘Home’. For example, almost every participant first looked into ‘Home’ to find computer settings and to change their wallpaper.

Third, most participants were not able to figure out how to reveal the Launcher from the upper left corner. They immediately devised work-arounds, like closing windows or moving a window away from the left edge of the screen. They expected to be able to reveal the Launcher by approaching any point along the left side with their pointer. As the Launcher is one of the most important features of Unity, it should be either always visible or at least very easy to bring out.

Fourth, the Dash is hard to discover. The icon is too small and understated compared to the icons in the Launcher. By its size and placement, it is easily associated with the window management buttons. Participants who discovered the Dash found it very useful, but were more inclined to use Files and Applications Lenses at the bottom of the Launcher. This was, I’m convinced, partly due to the fact that there were no data, pictures, music or documents on the computer that they would want to access through the Dash at the time of the testing, whereas the Applications Lens, in early use, is more adapted to general exploration. The Dash needs to be more visible — it needs to be accorded its rightful place as a major feature of the interface.

Notwithstanding these small problems, it is fair to say that this test showed that we have made significant progress since the October testing.

Some Major Issues that Have Been Resolved since October

Visibility of icons at the bottom of the Launcher

During the April testing, participants experienced difficulty seeing the bottom of the Launcher when it was accordioned and then, when the Launcher expanded, it hid the bottom icons.  At the time of the testing, it was very difficult to reveal these bottom icons even by scrolling down. Recent updates have resolved this problem by making the Launcher automatically scroll down when users move the pointer down along it.  This way, the icons that were previously hidden are effortlessly revealed.

A related issue that has also been resolved is that, during testing, participants wanted to make the Launcher visible by touching any part of the left side border – whereas, in fact, the only way to reveal the Launcher was by reaching with the pointer to the upper left corner. With the updated version, users can now reveal the Launcher from any point on the left side of the screen.

Changing the order of icons in Launcher

During the October testing, when the interaction to move an icon in the Launcher was to select it and bring it outside of the Launcher before giving it a new position, many participants failed to do it.  The new interaction supports users’ natural way of moving an icon:  participants were able to move icons in the Launcher by selecting them and moving them vertically up and down.  It should also be noticed that the feedback provided when users select an icon they intend to move helps them understand that they have initiated an action.  Knowing that the icon has effectively been selected afforded them more freedom to move the icon around and to find a way to make it work.

Adding icons to the Launcher

Participants were able, even during the October testing, to drag and drop the icon of an application from the Applications Lens into the Launcher.  However, their first attempt, especially for Windows users, was to right click on the icon they intended to move and expect to be offered an option to attach to the Launcher in a drop down menu; and second, to look at the top of the Launcher for a ‘Launcher  menu’.

Identifying running applications

Most participants were able to see immediately which applications were running by means of the white arrows beside the icon in the Launcher.  However, they were not sure if they had made the right inference.  In short, although participants were unsure about the meaning of the white arrows and bars, they were able to figure them out which indicates that this is a feature easy to learn.

Changing the wallpaper

Most participants easily changed the wallpaper by right clicking on the desktop.

Deleting a document

Most participants easily deleted a document.

Detailed Summary of Benchmarking – Comparison of the October and April Test Results

The points above are the highlights of the findings. Let us now examine individually the differences in performance as revealed in the testing of last October and the one just completed in April.

Performance

October report:  “The level of performance in this regard significantly impaired the flow of use and the user experience.”
April testing:  Unity was quick and responsive.
Outcome:  This is fixed.

Multi-tasking: Having many items opened and accessing them

October report: “Thus, while working on a task, participants expected that Unity would provide them with a representation or visibility of what was available to them and how to easily access what they needed at any given point.”
April testing: No problems were observed with overlapping open applications and documents. Participants could easily move individual windows and reveal items placed underneath.
Outcome:  This is fixed.

General navigation

October report: “Overall, participants found the navigation to be cumbersome.”
April testing: Participants used Nautilus to find applications and documents as well as system settings. This is not, however, the most efficient way to do this.
The Files and Applications Lenses icons need to be more prominent in the Launcher.  However, participants found it easy to go from one document or window to another and to make them all visible to them.
Outcome:  In a recent update, the icon ‘Home’ (Nautilus) has been renamed ‘File Manager’ and the icon has been modified to downplay the home relationship.  This should help users recognise its role and lead them to look for an alternative place for system settings and other programmes and applications.

Minimising a window

October report:  “When participants minimised a document, the document seemed to have disappeared when they expected it to be shown at the bottom of the screen.”

April testing:  A few participants still expected to see a trace of their minimised document at the bottom of their screen.
Outcome:  Since the usability sessions this interaction has been updated to show the window minimised into its Launcher icon even when the Launcher is hidden.  This should help users to locate their minimised documents and windows.

Awareness of running applications

October report:  “Participants did not always see the white arrows that indicate a programme is running or documents are opened. Consequently, they were not aware of what was available to them.”
April testing:  Almost all our participants were able to tell which applications were running by looking at the white arrows. However, some were not sure at first and needed to ‘try it out’.  So they opened and closed windows and applications to check on the behaviour of the icon in the Launcher.
Outcome:  The white arrows seem to be working well once they have been discovered.  Although they are not obvious, users can figure them out.  This is easily learnable.

Displaying documents side by side

October report:  “No participant could find a way to resize his/her openoffice documents in such a way that they could be placed side by side while working on both at the same time.”
April testing:  Except for one, all participants were able to display two documents side by side. However,  as noted above, they were not able to discover the semi-maximised state.
Outcome:  The original problem is fixed.  In the new design of Unity, participants have a way to display their documents side by side and to work on them simultaneously.  The semi-maximised state is not readily discoverable.  Unfortunately, users are not yet taking full advantage of what Unity offers.

Overview of computer

October report:  “Many participants wished they could have an overview of what resides in various parts of their computer, as is facilitated by Windows’ ‘my computer’.”
April Testing:  This is still a problem. Participants in the April sessions were still looking for a place where they could do systems setting and have an overview of their computer.
Outcome:  None of the participants discovered the ‘system settings’ option in the top right indicators menu. Users need an icon either in the Launcher or in the indicator area, or a folder in Nautilus.

• Bug #764744 (“Add system setting icon to Launcher”)

Delete a document

October report: “Participants could not delete existing documents from their files and folders. “
April testing:  Everyone was able to delete a document that was no longer wanted.
Outcome:  This is fixed. One remaining problem is that many participants cannot see the Rubbish Bin at the bottom of the Launcher.  They used other ways to delete, like pressing the delete key.

• Bug #764751 (“Launcher – when Launcher contained folded icons, partcipants weren’t able to find the rubbish bin”)

Copy and paste

October report: “Copy and paste from one document to another didn’t always work for participants.”
April testing:  Everyone was able to copy and paste from one document to another.
Outcome:  This is fixed.

Lack of feedback

October report: “Unity is often slow, and as a result participants tended to be confused about what was going on.”
April testing:  Overall, and as noted earlier, the performance of Unity was much better and the system responded more readily to users’ commands. Some issues remain with feedback, however, for example, with the Rubbish Bin.  Participants wanted to be alerted, either with sound or a message that their document had been moved to the Bin.
Outcome:  Confirmative feedback is necessary whenever users complete an action, like deleting a file.

• Bug #750311 (“Launcher – When a item is deleted by dragging to Trash, the trash should pulse once before the Launcher disappears”)

Nautilus search

October report: “When searching, participants didn’t know what the field and scope were that were covered by the search engine they were using.”
April testing: Many participants searched for applications successfully. However, there are still problems with search. Participants made inappropriate searches, for instance in Nautilus, searching for Sudoku (search that pertained to Applications Lens) and they did not get the results they expected.
Outcome:  This is partially fixed.  Some issues with search are related to participants’ understanding of the structure of Unity. There should be some guidance hinting at the limitation of the search and thus, the kind of results that can be expected from the various search boxes in the various parts of Unity.

Adding an icon to Launcher

October report:  “Many participants were not able to add a short-cut of an application to the Launcher.”
April testing:  Most participants were able, this time, to add an icon to the Launcher. Windows users, however, had more difficulties than the others did; they tended to look for options in various menus or right clicking on the icon.
Outcome:  This is partially fixed.  The interaction is quite intuitive but, some users (particularly those using earlier versions of Windows) will require more guidance.

Reordering icons in Launcher

October report: “Most participants failed to reorganise the order of icons in the Launcher.“
April testing:  A few participants experienced some difficulty reordering icons in the Launcher because they did not have sufficient feedback to understand when the icon had actually been selected so that they could proceed vertically.  Consequently, they tried to move the icon too quickly after clicking on it and the icon did not respond.
Outcome:  This has been fixed in the latest update by providing feedback on selection – the interaction shows the icon as if it was detached from the Launcher – and by allowing users to move the icons vertically within the Launcher.

Finding the Dash

October report:  “The majority of participants who found the Dash found it by accident. They were not sure what it was, and didn’t know how they had gotten there if they accidentally had.”
April testing:  Participants still cannot readily find the Dash.
Outcome:  The Dash needs to be made more visible and promoted as a major feature of Unity, on a par at least with the icons of the Launcher.

• Bug #764771 (“The BFB is visually lost and his position does not communicate its importance”)

Ubuntu Software Centre

The same features of the Software Centre were not tested this time because everyone agrees on its need for redesign and its existing usability problems. Nevertheless, some issues emerged in the course of testing other interactions.
April testing:  The Software Centre is still not recognized and, during testing, was mistaken for ‘systems control’.
Outcome:  The Software Centre needs to have a different look and feel and general presentation. Needs redesign.

Changing the wallpaper

October report:  “Many participants did not succeed in changing their wallpaper because the default screen of appearance was open in full screen by default.”
April testing:  Almost all participants were able to change the wallpaper by right clicking on the desktop. Furthermore, one participant who was able to find ‘appearance’ had no problem changing the wallpaper because now, the screen opens in a way to provide visibility of the background.  The October usability problem was thus, fixed.  However, a new problem emerged.

In the April test, the target feature was, in fact, the ease of use of the Applications Lens by means of changing the wallpaper. Most participants were not able to change the wallpaper by finding ‘Appearance’ in the Applications Lens.  They were looking for ‘system settings’ to do that operation.
Outcome: The initial problem with the appearance screen covering up the immediate change of wallpaper and so, hiding the change from users, has been resolved.  Now, by default, the appearance screen does not open full screen.  In the April test, however, users could not find their ’system settings’, where they expected to make these changes.  Furthermore,  many participants did not think of system settings as an application and, thus, were not confident to find it in the Applications Lens.  Unity needs to provide obvious access to ‘system setting’ and make a distinction in the Application Lens between applications and other programmes.

Visibility of Files and Folders and Applications Lenses and  Rubbish Bin

October report:  “Participants thought that the grey icons at the bottom of the Launcher were inactive.”
April testing: These icons still have issues of visibility, especially when they are folded at the bottom. For example, most participants did not find the Rubbish Bin.  Another usability problem that arose from interacting with the Launcher is that some participants found it difficult to interact with the bottom part of the Launcher.  They found that it was ‘a long way to go’ to the Rubbish Bin or the Lenses when the Launcher was populated with many icons.
Outcome:  These icons still need more visibility. Changing the colour, and perhaps even changing their position in the Launcher, might help.

• Bug #764751 (“Launcher – when Launcher contained folded icons, partcipants weren’t able to find the rubbish bin”)

Some Usability Issues that Have Arisen from Some of Our New Design

Top Menu Bar

The top menu bar is actually a new design.  There was some confusion about the role of the top menu bar: Participants wondered if it pertained to ‘the computer’ or to the application they had open at the time. When participants had many windows opened, they did not understand that the bar corresponded to the selected window.

System Settings

During testing, I encouraged participants to change their wallpaper in another way than by right clicking on the desktop to see if they could find ‘appearance’ in the Applications Lens.  Finding system settings programmes in the Applications Lens is not intuitive. Most participants did not succeed in changing the wallpaper by going into the Applications Lens. They were looking for a ‘system settings’ icon in the Launcher or somewhere in the ‘Home’ at the top of the Launcher.  Those who went into the Applications Lens, did not expect to see ‘system settings’ in that area because they did not think of system settings as applications and accordingly they did not explore.  No one discovered the ‘system settings’ option in the drop down menu under the ‘turn off’ icon in the indicators menu bar.

Notification of message

This is also a new feature since the October testing.  The majority of participants did not see the notification that they had received a message. The change in colour of the icon was not noticed.  However, some noticed the change in the icon in the Launcher, in this case the Xchat, and they induced, by looking at the number that appeared on the icon, that they had received a message.  However, when the Launcher is invisible, participants were not aware that they had a message.

This said, a couple of participants saw the notification and the change in colour of the envelop in the notification area.  They had a strong positive impression of the feature.  It seems that in this case, it might be a question of making the change in the notification area more prominent.

Semi-maximised state

Again, semi-maximised state is another new feature.  Semi-maximised state is not readily discoverable. Only one participant discovered it. This participant was a Windows 7 user and said that there is the same feature in Windows 7. Two other participants interpreted the blue preview shadow as signalling that they were about to make a mistake or to do something not allowed by the system. The preview shadow was interpreted as a warning.  Users need both guidance and reassurance here.

We are doing better with the user’s experience and our users are closer to adoption

Overall, participants left with a strong positive impression of Unity after having tried it for 60 minutes.  Some of their closing comments:

“I like the layout and the screen (…) I want to customise it myself quite easily. It would be good to have a tutorial. (…) I like minimise and the fact that you can move things around. I like the casual font, aesthetically, it looks nice and it is easy to use. Nothing is really difficult. The important things are there and easy to use. It is nice.” [P1]
“The reason it was annoying today is because it is a new package. I like the design and layout. Design is important to me. It is quite clear. (…) “I would like more time to play around with it. It’s Ubuntu, I haven’t used it. This is new, the way I learn is by playing with it. (..) It’s good to use something that is a bit more independent. I like the idea that we can do things rather than being locked down in something more siloed like Windows or Mac. I would like to get it.” [P2]
“I prefer this set up to the start menu. I like the icons. We are a generation to see things with icons. I think there is a lot of significant gesture, like saving documents and I would not have any problem doing these activities. I really like the dragging format. I like to be able to order what I want. I think it is much easier than Windows. With Windows you have to go down menus. (…) I don’t think it’s complicated but it would take some time [to get use to it]. I’ve been working it out in an hour. It’s very user friendly. Even within the hour, I’ve learned a lot about how to do different things.” [P3]
“I really quite like it. I think it’s intuitive with the exception of the favourites, making an application a favourite. I would not be baffled to use it without a manual. I like the look of the desktop. It is modern. It looks like a Mac more than Windows. It’s quick.” [P5]
[About the Software Centre] “I didn’t anticipate to have access that easily to new apps. Also, I like the rating on the side. It’s quite helpful, I can see what I can trust. That’s quite nice.” [P5]
“It’s OK. Quite intuitive but I was going from what I know from Windows. I use the right click a lot, it’s nice to have it on the side. Generally this looks pretty good. It’s a bit more intuitive, for me, though, the right click is vital. It always brings up a good menu.” [P9]
“I think it’s very pretty, very pleasing as it were.” [P11]
“It’s quick and responsive. It’s very responsive, different from what I use, it would take a day or two to get acquainted. I wouldn’t be discouraged. I would rather spend time than pay money.” [P12]

In the summary of their experience post usability testing, participants also highlighted their main difficulties. It is meaningful that, at the end of the session, the following first came to mind:

“I don’t like the dragging in Launcher up and down. I mean I didn’t realise at first this is what I needed to do. It’s difficult to get to the Bin. It’s not easy to get to the top from the Bin, it is hard to drag things down a long way. I don’t like the dropping down.” [P1]
“My frustrations: I would like to know how to change the settings, I expect a button to change wallpaper clicking on a button right at the top. (…) The menu at the top bugged me.” [P2]
“I didn’t like when I have things minimised. There are many things I can’t do without maximising the screen.” [P3]
“It is hard to delete a file in this way. (…) You don’t find the menu bar and you don’t know what’s open.” [P4]
“I don’t know how to make the Launcher visible [when a window is opened]. I’m struggling a bit. This window [Dash] has a tendency to disappear.” [P5]
“I hated the Files and Folders, I didn’t know what it would do when I click on it, if it will open or just let me select it. I wasn’t able to select a document.” [P8]
[About the wallpaper] “I couldn’t find it. I wouldn’t have thought of it as an application for some reason.” [P10]
“I suppose my main thing is what I expected to have in terms of applications and control panel. I couldn’t find it. If I could have found this at the beginning life would have been a lot simple. I feel like I feel with Apple, I feel a bit stupid because I can’t do the things I normally do with my PC. I like things in words a lot, I like the drop down menu. This is interesting because this is generally shown with an icon.” [P11]
“I’m frustrated that I can’t find something like ‘my computer’. I want to find information about ‘my computer’ and what the hardware is, the driver versions, and I want to know if there are updates on Explorer. Here you need to go into ‘control panel’ to see if there are any updates. I still can’t figure it out.” [P12]

You  can also download a PDF of the full report by clicking on this link.