Usability Testing Script

The following is a script for doing usability testing on Kiwix - specifically browsing and searching. It is designed for use with a pre-installed copy Wikipedia 0.8 (which should already be indexed).

Contents 1 Introduction and setup 2 Explanation of roles 3 Background questions 4 Think aloud 5 Tasks 5.1 0 - Library 5.2 1 - Find an article 5.3 2 - Bookmarking 5.4 3 - In-page searching 5.5 4 - Tools 5.6 5 - Tabs 5.7 6 - Feedback 6 Final questions 7 Results 7.1 Hackathon 2011 7.1.1 Liam 7.1.2 Erik 7.1.3 Nemesh 7.1.4 Erik2 7.1.5 Atahurlpa 7.1.6 Brandon 7.1.7 Timo 7.1.8 Synthesis 8 See also

Introduction and setup

• Thanks for agreeing to help us test.
• Get the participant to sign the consent form.
• Explain what Kiwix is.
• We're collecting feedback on Kiwix and testing its usability. Basically I'll ask you to perform a task and then watch you use the software.
• This will take about 15 minutes.

Explanation of roles

• Your job is really easy, you just have to be yourself and act as you naturally would. As you interact with the program, please be honest with your positive and negative thoughts. There’s no right or wrong answers, and nothing you say will hurt my feelings.
• You are not being tested, the software is being tested.

Background questions

• Tell us a little about yourself, like where you're from and what kind of work you do.
• Have you ever used an offline Wikipedia reader before?

Think aloud

• Next I’d like to have you open up Kiwix to do a couple of tasks.
• There’s one thing that I would like you to do differently. Please think aloud as you use the program today. Feel free to say anything that comes to your mind - for example "That’s interesting" or "What were they thinking?" Talk out loud while you’re doing each task, so I can understand what you are thinking and doing - for example "Now I’m going to try to use the search bar".
• We're going to be focusing on using Kiwix for basic browsing and searching.
• Do you have any questions before we continue? Okay, let’s open the program.
• Launch Kiwix.

Tasks

0 - Library

• Open the library
• Find all content from Wikipedia in English
• Download a content
• This takes too long, stop the download an open a file (small ZIM file should be available) from the Hard disk

1 - Find an article

• Pull up the article for Berlin, Germany
• Find out who the current King of Thailand is.
• How much articles speak about "foobar" (need indexation)

2 - Bookmarking

• Bookmark the article on Bhumibol Adulyadej (King of Thailand).
• Close the bookmarking interface (if you haven't already).
• Go back to the Main Page.
• Go back to the Bhumibol Adulyadej article using the bookmark you created.
• What do you think about the bookmarking interface?

3 - In-page searching

• Within this article search for information on the "Finland Plot".
• What do you think about the in-page search interface?

4 - Tools

• Switch the interface to full-screen mode.
• Switch the interface back to regular-screen mode.
• Change the language to French

5 - Tabs

• Bring up three articles in separate tabs.
• What do you think of the tab interface?

6 - Feedback

• Give us a feedback
• Do you have a feature request, a bug? If yes, do it now.

Final questions

• Is there anything that you would want to see changed/done differently in Kiwix? Particularly related to the tasks you completed today? If so explain.
• Any other likes or dislikes that you’d like to mention?
• Anything else you’d like to share with me or ask me?
• Would it be okay for us to contact you if we have any follow up questions or information?

Results

Hackathon 2011

Liam

• search bar too long
• no bookmark bar which allows a simple drag&drop like in FF
• create new bookmark not so intuitive (too complicated)
• bookmarks: replace sets by folders
• bookmark sidebar croping the windows on the right (also on the left)
• search in place: position indicator in the vertical scrollbar (like in google chrome)
• fullscren button tooltip does not change at all
• On Mac this is only possible to open a new tab with CTRL+T and tabs header if visible (add context menu ofr ctrl+click)
• on Macos tabs are unfamiliar
• X button on tabbar is confusing because it does not close the tabbar, but only the current tab.
• closing tab is not buggy
• Interface does not follow mac app. standarts.

Erik

• wants to be able to rename bookmarks
• SHIFT+Command+F should make fullscreen on Mac
• Searchs a button in a corner for leaving the fullscreen
• Feature request: always open links in new tabs

Nemesh

• Needs a one click "bookmark this page" (maybe the bookmarkbar?)
• search in place: ctrl+g does not work, it goes to find previous
• search in place: F3 should go to find next
• this is not obvious that kiwix has tabs
• news tabs with CTRL+T is buggy

Erik2

• In the searbar: (1) should be go directly to the article independently of the case of the search pattern (2) suggestions should provide a "search in text" item.

Atahurlpa

• no contextual menu on right click to open link in new tab
• does not see the "tabs" menuitem (lika many testers)
• Want to have the tabs header displayed per default

Brandon

• He does not found/see the searchbar
• Did not really realized (made the difference) between the UI and the content (at the beginning)
• Mac tabs visual are weird
• tab close button effect not clear (which tab will be closed)
• tabs + on the other side (chrome and FF have it on the right)
• search in place on the top right corner

Timo

• He expects a bookmark bar and did not find the bookmark button
• It should be a one click bookmark method
• Bookmark sidebar is not handful
• selected bookmark does not means displayed (so nothing should be selected per default in the bookmark sidebar)
• Search button purpose was not clear (because it's near to the searcbar)
• Overlooked the "tabs" menuitem]
• drag&drop a ling to the tabs header should create a new tab
• want to have a feature to reformat the page (in columns). There an ipad app doing something like that.

Synthesis

PRIO1 - Should be fixed or implemented

1. Creating a new bookmark is too complicated,
2. Bookmark sidebar is cropping the Window on MacOSX ID: 3293986 FIXED
3. Fullscren button tooltip does not change at all FIXED Kelson 22:28, 20 June 2011 (CEST)
4. This is only possible to open a new tab with CTRL+T (right-click menu, drag&drop open, shortcut with command on macosx) IMPLEMENTED
5. Tabs headers are not displayed in any case by opening a new tab with CTRL+T FIXED
6. On MacOSX tabs are unfamiliar ID: 3304070 IMPLEMENTED
7. Tabs X button position (to close tabs) on Tabbar is confusing (Close the tabbar? Close the current tab). FIXED (like in FF now)
8. Tabs + on the rigth side (like chrome and FF) IMPLEMENTED
9. Closing tab is buggy FIXED
10. SHIFT+Command+F should make fullscreen on Mac ID: 3304022 IMPLEMENTED
11. Ctrl+g (search in place) does not work, it goes to find previous FIXED
12. With Search in place, F3 should go to find next (at least on Windows) FIXED
13. Search should always go directly to the article if possible, independently of the case or diacritics of the search pattern. FIXED, it tries now fullupercase, fulllowercase, ucfirst, lowercase + ucfirst
14. Suggestions should be provided, independently of the case or diacritics of the search pattern.
15. Suggestions should always provide a "search in text" item at the end of the list. FIXED
16. "tabs" menuitem seems invisible FIXED

PRIO2 - We have to discuss if we need to do something

1. No bookmarkbar
2. Revamp bookmark UI & Organisation
3. Interface does not follow Mac app. standarts.
4. Want to be able to rename bookmarks
5. Button in the corner for leaving the fullscreen
6. Feature request: always open links in new tabs
7. With Search in place: F3 should go to find next IMPLEMENTED
8. This is not obvious that Kiwix has tabs
9. Selected bookmark item does not means displayed (so nothing should be selected per default in the bookmark sidebar)

PRIO3 - Won't probably make something

1. Search in place should provide a position indicator in the vertical scrollbar (like in google chrome)
2. Visual difference between content and UI is too small
3. Search in place on the top right corner (like chrome)
4. Search button purpose was not clear (because it's near to the searchbar)
5. Want to have a feature to reformat the page (in columns). See http://itunes.apple.com/us/app/id384224429?mt=8

See also

• [1]
• [2]