In January of this year we deployed changes to our navigation labels and information architecture, which we covered in detail in a previous blog post. In April we re-visited the project with a remote online tree test to test the effectiveness of the changes.
Participants who visited library.unc.edu were given a pop-up notification to complete a survey that contained a short tree test. The tree test asked participants to complete tasks that were nearly identical to tasks from a similar test conducted before the navigation changes. By comparing results of both tests, we are able to assess what impact the changes had on our users.
The pop-up appeared only once for each user, our standard method of recruitment for online usability tests.
The tasks of the tree test, given in a random order, were
- Show where you would go if you wanted to borrow a book not available at UNC Libraries
- Show where someone not affiliated with UNC can find information on using the library
- You were told the library has information on how to cite sources. Show where you would find it
- You want to reserve a room in Davis Library for a group project. Show where you would find information on this
Participants then used an interactive tree reflecting the new navigation to locate the proper page to complete the task. Each task has specific pages that are deemed as the “correct” page to complete the task. The test was created with Treejack from Optimal Workshop. The test ran from April 15 to April 25, a 170 participants completed the test.
Results comparison & discussion
The navigation of library.unc.edu was changed because our old categories were often vague and too similar to each other. This turned the process of finding content into a scavenger hunt. Our research consistently showed little memorability in locating content — even library staff had difficulty remembering where certain content was located.
To fix the issue, we changed top-level labels into terms that are more semantically distinct. For example, Do Your Research became Search & Find. We also changed several page titles in order to improve clarity of their function. For another example, Tutorials became Tutorials for Research. We also removed a handful of pages from the navigation and added new pages when necessary like Book a Study Room.
Overall tree test results
| Before | After |
| Success rate: 63% Directness: 69% |
Success rate: 70% (+7) Directness: 74% (+5) |
We saw a slight overall increase in both success rates and directness. More participants were able to find the correct page to complete a given task, and fewer participants had to backtrack to find it while completing a task.
These results confirm an overall improvement in the ease-of-use and clarity of navigation titles.
Show where you would go if you wanted to borrow a book not available at UNC Libraries
| Before | After |
| Success rate: 79% Directness: 76% |
Success rate: 61% (-18) Directness: 62% (-14) |
The rationale for this task was to see if participants could locate an important page located under the Services category, a navigation label we did not change in this revision. The correct page, Services → Interlibrary Loan & Carolina BLU, is an important page as it is a popular service offered by UNC Libraries.
While the Services category did not change, changes to surrounding categories might have contributed to this task’s lowered rates. The category Search & Find (previously Do Your Research) received 46% of first clicks from participants compared to the correct category Services which received 42%.
Participants seemed relatively split then on thinking of ILL as a searching/finding engine versus thinking of it as a service form. In other words, if the user needs to locate a book not available from UNC Libraries, it logically follows for some that they would need to locate the book first. This might explain the higher first clicks for Search & Find. The most common incorrect answer, Search & Find → Catalog, supports this explanation, as those participants might have thought to search the catalog for the necessary book (even though our current catalog does not support searching for non-UNC books). Perhaps the previous Do Your Research label did not create this kind of confusion.
It should be noted that many users have either noted or been observed to access ILL during their search experience rather than as a destination, i.e. a user will request an item from WorldCat or Summon during their search process. We don’t currently know how many users access ILL directly to request an item they already know UNC does not have.
Show where someone not affiliated with UNC can find information on using the library
| Before | After |
| Success rate: 24% Directness: 67% |
Success rate: 46% (+22) Directness: 71% (+4) |
This task had slight improvement on directness and significant improvement on success rate. Still, the success rate for this task is markedly low.
The original rationale for this task was to ask participants to locate a page in the In the Library category, which has since been changed to Places & Spaces so that it’s purpose as a “physical space” category is clearer. As such, the correct page for this task has been moved from that category and into the Services category. Before the changes, this task had two correct answers: In the Library → Visitor Information as well as In the Library → Visiting Scholars & Postdocs. These two pages were combined in the recent changes, meaning there is currently only only one correct answer for this task — Services → Visitor Information.
This task was also reworded from the earlier test. The original test before navigation changes used the following phrasing for this task: “Show where you would find information about what people from other universities can borrow from UNC Libraries.” We determined that this phrasing might have felt too similar to ILL and thus contributed to the very low 24% success rate from the earlier test (our ILL page got 41% of answers). The new phrasing aimed to avoid confusion and ask participants for information about what non-affiliates may access at the Library.
Despite significant improvements on this task after these changes, this task is still considerably low. The most common incorrect answer on the more recent test was Libraries & Hours (30%). Like the previous task, it might be the case that surrounding categories contributed to participant confusion on the correct path. Participants who selected Libraries & Hours likely expected a library branch hub page to contain information on access, among other things. The participant’s mental model did not seem to match the “correct” path for the task.
You were told the library has information on how to cite sources. Show where you would find it
| Before | After |
| Success rate: 79% Directness: 84% |
Success rate: 84% (+5) Directness: 74% (-10) |
The increase in success rate for this task is likely due to correcting a procedural error in the previous test. In the previous test, the answer of Services → Tutorials was not included as a correct answer; the only correct answer within Treejack was Do Your Research → Citing & Writing. While the Citing & Writing page is truly the quickest page to complete the task, it is itself a tutorial, meaning that participants selecting Tutorials were not exactly wrong. With the newer test, this page was included as a correct answer, thus resulting in a slightly higher success rate (Tutorials, now renamed Tutorials for Research, received only 7% of answers).
It is unclear why directness dropped for this task. Perhaps the addition of a new category within the navigation created the need for more backtracking with some users. Both correct answers are found under the new category of Support & Guides. This new category might have created some unfamiliarity that lowered the directness of results.
You want to reserve a room in Davis Library for a group project. Show where you would find information on this
| Before | After |
| Success rate: 72% Directness: 52% |
Success rate: 88% (+16) Directness: 90% (+38) |
Results for this task show improvement for success rates and significant improvement for directness. In the previous tree test this task was found to have the lowest directness, and now it has the highest. This can be attributed to the renaming of categories to increase their semantic distinctiveness. Places & Spaces, previously named In the Library, is much clearer as a “physical space” category for users than before.
Additionally, a new page was added during navigation changes that likely further increased success rates. Places & Spaces → Book a Study Room is a new page that directly correlates with this task; it received 87% of answers. The other correct answer, Places & Spaces → Places to Study received 2%, while in the previous test is received 72%.
Conclusions
UNC Libraries offers a large variety of services in its digital products, so many that it’s often unclear if users are aware of everything at their disposal. Having a well-designed, logically organized navigation and information architecture scheme helps ensure learning, memorability, and discoverability of content. Tree testing with Treejack has proven itself as a useful tool for evaluating and assessing navigation systems.
This campaign of tree tests showed that we made some improvements in our navigation. Overall success rates and directness increased, and comments from the questionaire given to our participants with the tree test were generally positive.
Of course, there are still improvements to be made. At a basic level, participants too often assumed that certain content as located on a page where it wasn’t (as in the case of Task 2 and Libraries & Hours). But we’ve now established that we can make iterative changes to our navigation to improve overall usability, and can repeat these methods going forward.
Contributions
This project was completed by:
Daniel Pshock, User Experience CALA
Chad Haefele, Interim Head of User Experience