Responsive Web Design or why Jakob Nielsen is wrong

Introduction

About three years ago Ethan Marcotte published an article on “A List Apart”, called Responsive Web Design (Marcotte, 2010). Within this article he speaks about the web moving forward, specifically - the web moving to your mobile device. His article marks the start of Responsive Web Design, meaning websites that scale itself to the viewport of the device you’re accessing the web with.

Therefore you need a flexible grid, flexible media (pictures and videos) and Media Queries, (Viewport-specific styles) introduced with CSS3. With Media Queries you are able to optimize your design for various viewports.

To make your website accessible for any kind of device, it was very common for a long time that you create a static mobile site along with a site optimized for desktop computers. Many web designers recognized Responsive Web Design as a technique to bypass the extra amount of work for a static website and build a responsive site which just scales nicely to any type of device. Not only does this require to rethink the design process, furthermore the work of Information Architects and Usability-Experts becomes more and more important, because basic principles to design and build a responsive website instead of a normal desktop-use website are differing.

But - Responsive Web Design is not the answer for any web-related design problem we have and Responsive Web Design is not without critique. The usability-expert Jakob Nielsen commented on Responsive Web Design and said that RWD may not be useful if you try to achieve business-related goals with your website, speaking about money and commercial value. An optimal conversion rate as well as a good usability can only be achieved with a static desktop website and a static mobile site - not by combining them for a responsive website. The web community reacted with doubts about Nielsen’s statements but his statements were not academically evaluated. In one of his studies “Usability of Mobile Websites and Applications” (Budiu & Nielsen, 2012)

Nielsen speaks about usability principles for static mobile sites but he doesn’t say anything about responsive Websites. With a usability test I tried to evaluate Nielsen’s thoughts and to make the differences between static mobile sites and responsive sites a little bit more transparent.

Jakob Nielsen

Good mobile user experience requires a different design than what’s needed to satisfy desktop users. Two designs, two sites, and cross-linking to make it all work. (Nielsen, 2012a)

On April 10th 2012 Jakob Nielsen published an article in his blog, called “Alertbox” named “Mobile Site vs. Full Site” in which he listed the advantages of separate Mobile Sites. Therefore he had also done a usability test with hundreds of Websites within the Nielsen Norman Group. The test users came from America, Asia and also Europe. The rules, which Nielsen proclaimed as a result from the test are:

  1. If you have the money, create a separate Mobile Site. The measurable Usability of separate Mobile Sites are in excess of desktop websites.
  2. If a user comes from a mobile device, automatically forward the user to your Mobile Website, but keep in Mind to always have a visible Full Site Link on your mobile website.

In another post from Nielsen “Repurposing vs. Optimized Design” (Nielsen, 2012b) he is talking about Responsive Design. An outstanding User Experience could only be achieved, if you have a tight integration into the platform which concludes, that you build a separate mobile site. Differences between Desktop PC’s and Smartphones are very huge, so two separate designs are essential for success. Nielsen also argued that he had only seen responsive sites with poor designs, that are not matching the user’s need.

In an interview with .net Magazine Nielsen reinforced his arguments by specifically pointing out the disadvantages from Responsive Design.

  1. If all interface design elements are saved in one file, this concludes to longer loading times for the browser. Especially for mobile users who often don’t have a strong internet connection can this be a problem.
  2. The ideal design for mobile users means, not to show all the content from the desktop site also on a mobile device. This could be achieved with Responsive Design, but would be inconvenient.

Usability Testing

I had the assumption that Responsive Design is the answer to the digital problems of our time, meaning the constantly increasing number of mobile devices with various screen sizes. In order to verify if Nielsen’s testings or my assumptions are correct, I had to test it myself.

To evaluate if Nielsen’s assumptions and test results are wrong, I created two functional prototypes, to determine if the conversion rate is influenced by Responsive Design or not.

My plan for testing Nielsen’s statements was to - create two functional prototypes, a static mobile site and a responsive site - evaluate and compare the sites with real users, determine the conversion rate

My assumption for my usability test was Responsive Webdesign enables an optimal User Experience and Usability. Furthermore the conversion rate does not differ from a static mobile site.
Nielsen says, good, mobile Design which helps to sell Products and services and has a high conversion rate, can only be achieved by creating two separate experiences, meaning a desktop website and a static mobile site.

To evaluate if Nielsen’s assumptions and test results are wrong, I created two functional prototypes, to determine if the conversion rate is influenced by Responsive Design or not.

The goals of good Usability are first of all based on performance. That also requires efficiency and effectivity and how fast a user is able to fulfill certain tasks. Because I compare two designs, a comparison test is useful. Within the usability test statistic data and subjective data (Emotions, Feelings while using the prototype) is included and will be compared. The results lead to a hypothesis.

The conversion rate cannot be determined with this usability test, because I have to fake a purchase intention for the user. However, how successful a user can navigate through the site and how fast he is able to fulfill the given tasks we can make assumptions about the conversion rate.

Because I worked for trivago, a hotel price comparison portal, I had a perfect test environment. For the test I had 100 users who needed to fulfill two tasks within the context of booking a hotel. 50 users tested the prototype of the static mobile site, the other 50 users tested the responsive prototype. The two tasks I asked the user to fulfill are:

  1. You are looking for a cheap hotel on Mallorca, in the first week of August.
  2. You need a hotel for business reasons. You‘re travelling to Bostin, MA (USA) fora client meeting and you need a cheap hotel which is not more than 3 kilometers away from the center of town. It should have a minimum of three stars.

For the test I had 100 users who needed to fulfill two tasks within the context of booking a hotel.

The first task is obviously short, so the users get used to the prototype and have enough time to understand the design. The second task is a little more complex, so that all test users have to use certain filter techniques.

The usability test was done with an iPhone and iPod Touch. All tests were filmed by a video camera. All users also had to comment on what they are thinking and doing during the test. With that I could get additional feedback and user data.

The Prototypes

Because at the time of my usability test trivago had no static mobile site nor a responsive site I had to build two prototypes myself, using latest web technologies, especially HTML5, CSS3 and jQuery. The static mobile site prototype was build with the guidelines Jakob Nielsen deduced from his Usability testings. For the responsive prototype I ignored the guidelines by Nielsen and followed a common visual design language that is used all over the web as well as my own intentions. I was also very much influenced by the current design trivago is using to have a trivago look and feel and all the functionality that is neccessary to fulfill the given tasks.

All test users had a document with the tasks for the test written on it, as well as a small introduction. The users did not know, that the given sites were just prototypes. Because all tests were filmed by a videocamera I could determine the exact time they needed to complete the given tasks. This time shows how efficient and effective a user is able to complete the given tasks.

The 50 users of the static separate mobile site needed an average time of 402 seconds to complete the two given tasks. The 50 users of the responsive prototype needed an average time of 336 seconds. That shows that the completion of the tasks took longer time for the static prototype. So we can assume that users of the responsive prototype could complete the tasks faster and more efficient and effective.

After the usability test I had a little post-questionnaire, that should verfiy my assumptions. It contained six questions, like

  1. These three aspects of the site were easy to use.
  2. These three aspects of the site were hard to user.
  3. Did you miss anything, if so - what?
  4. Have you found everything you was looking for?
  5. Was the layout structured as you expected?
  6. If you could change one thing, what would it be?

The results showed that users of the responsive prototype had fewer problems with the site. Filter and Date functions were easier to use. Users of the static mobilesite generally missed more elements like filters, hotel informations and maps. Although the responsive site had the exact same functionality, users had fewer problems finding functionalities or filters to find the right hotel.

Summary

Jakob Nielsen says, content for mobile devices needs to be short and to the point. All content from a desktop website needs to shortend by half. The results of the questionnaire speak a different truth. Also because all users known the extent of content and filters on the trivago site, they wanted to use all of them. If a website is optimized for mobile devices, users want to get all content that is also on the desktop version of the site.

Because we cannot predict how users will reach websites on the future with what kind of devices, the future of the web is completely flexible and not static.

The video recordings of every test show that Nielsens statements are wrong. Users are faster on a responsive site that often comes with more functionality on the first page, meaning jQuery filters, slideshows, etc.. In Nielsens principles for mobile devices he says, that it is better to divide much content and functionality in various pages. All users of my responsive prototype did not need to go to a second page to filter something, it was done on the fly, all by jQuery.

Therefore we can say, that responsive websites are able to fulfill business- oriented goals without changing the design completely for a mobile device. Nielsens says „Good mobile user experience requires a different design than what‘s needed to satisfy desktop users. Two designs, two sites, and cross-linking to make it all work.“

But we better say: To create a good mobile experience, you need a scalable website which will be optimized for various viewports. Because we cannot predict how users will reach websites on the future with what kind of devices, the future of the web is completely flexible and not static.

References: