In a previous blog post about a NoTube demo for browsing an archive of BBC programmes using Linked Data techniques, I discussed the potential role of serendipity in bringing people’s attention to ‘surprisingly good’ content that they didn’t already know about.
At the time Libby and I were just about to start user testing the application. We wanted to find out if the experience of serendipitous content discovery could be supported by re-using the metadata created when BBC programmes are archived (in this case terms from the BBC subject classification system). We chose a technique which generates a similarity measure based on the number of categories in common between any pair of programmes, and then displayed these to the user as suggestions for related programmes.
NoTube Archive Browser
For the purposes of our evaluation, we identified two specific research questions. Does it help people find more interesting programmes if:
- they browse similar programmes rather than a random selection
- they see browsable subject category information about the programmes
Our hypothesis was that seeing both similar programmes and clickable subject category information (as shown in the screenshots below) would be optimal for finding interesting new programmes.
Clicking on "railway stations" links to a list of programmes classified with that term
During the trial the 96 participants had seven days during which to browse the Web-based on-demand video collection and add programmes of interest to a playlist. The participants were randomly allocated one of four experimental conditions:
- Random programmes, no subject categories displayed (our control)
- Random programmes, with subject categories displayed for each
- Similar programmes, with no categories displayed
- Similar programmes with subject categories displayed for each (our optimal condition?)
We measured the length of the playlists and the time spent browsing as indicators of ‘interestingness’, based on the assumption that people would create longer playlists and spend more time browsing when they found the links interesting. At the end of the trial 20 of the participants also filled out an online questionnaire about their experiences.
Statistical analysis of the data we collected (Libby created some box-whisker plots)
showed no clear effects of programme similarity or the display of subject categories on the length of participants’ playlists or the time they time spent browsing. Likewise, as the graphs below illustrate, responses to the questionnaire showed no significant variation in people’s experiences between the four experimental conditions. However, the questionnaire results suggest that, regardless of the experimental condition, people did find interesting new programmes and generally liked the application.
For example, one participant remarked: “I liked the way the app threw up programmes I had forgotten about. I liked the way that on selecting one programme I always found even more interesting programmes on the related programmes list.” In total, 15 out of the 20 respondents said they would recommend the application to others.
Comparison of questionnaire results between control and optimal groups
One possible explanation for the fact that the experimental condition had no significant effect on people’s enjoyment of the application was that our ‘random’ selection was actually too good, since it tended to display a fairly reasonable set of programmes taken from a variety of well-known TV series.
Overall, the results suggest to us that adopting this type of approach to navigating large video collections (including archives, on-demand content and EPGs) could help with the ‘cold start’ problem, whereby a recommendation system cannot provide useful recommendations until a substantial amount of user activity data has been gathered. Re-using existing programme metadata to create content-based links in this way could help large media organisations to provide users with a means of accessing niche video content that might not otherwise be found, in particular if no-one has yet watched it because it hasn’t appeared in a ‘What’s popular’ or ‘What’s new’ list.
This experiment complements the work of several of our NoTube colleagues, who are also exploring the role of serendipity in using Linked Data techniques to suggest programme recommendations that are diverse, unfamiliar and ‘surprisingly good’ - difficult as this is to measure.
Pingback: Preliminary findings of N-Screen user testing | NoTube