Eye Tracking Report

John-Paul Takats, Kacie Michaels, Pawee Kiratiya-Angul, Sree Anirudh J. Bhandaram, Terlene Romney

Introduction

Chain and franchise restaurants have shown steady expansion across the United States and have truly revolutionized the food industry. However, the marketplace in many areas is becoming saturated and the competition is becoming fierce. This is causing companies to develop new ideas and creative new ways of doing business. With the advent of the internet the possibility of online ordering has become a reality and is now being used by several restaurants. Improving the overall customer experience is what online ordering seems to address; it eliminates the need to wait in line, affords the ability to see the whole menu anywhere and store favorites, and provides several payment methods (which can also be stored). This eye-tracking study explored the concept of efficiency and user satisfaction of the online ordering website for the popular restaurant chain, Chipotle. We also analyzed whether the website is a suitable substitution for operations that take place in the physical location and whether visiting the store impacts the ease of executing orders via Chipotle’s website.

Research Questions and Hypotheses

Our team addressed the evaluation of viewing behavior for the Chipotle website, by identifying the different sections of the website. The Chipotle website contains five components that are observable in determining the the different viewing behaviors of the website pertaining to the particular task : identification markers (e.g. Chipotle logo), navigational headers (e.g. picture of orientation with the order placement process, subcategory headline), images, descriptive content (content interpretating of an image), and interactive components (e.g. entry bar that accepts keywords to store an order). The following questions relate to evaluating key interactions when viewing the website:

  • What is the first item that attracts users, when viewing the website?
  • Do users dwell from image-to-image, image-to-text, or text-to-text during different tasks?
  • How many images do users dwell, in comparison to text?
  • Do users pay attention to the identification and navigation markers of the page (i.e. identification markers, navigational headers)?
  • Is there a significant difference in viewing behavior of the Chipotle website users when engaging in different tasks?

Hypotheses

Below is are our hypotheses in original form Concerning Viewing Behavior Difference . As a result of a number of obstacles that our team met while completing this study, we were unable to have two subject groups equal in size to properly answer these questions. Below the preliminary hypotheses we have new hypotheses which can be used when studying our data.
Original Hypothesis
Null Hypothesis (H0): There is no significant difference in viewing behavior of the Chipotle website between users who have visited Chipotle and those people who have not visited the retail store location.
Alternative Hypothesis (HA): There is a significant difference in viewing behavior of the Chipotle website between users who have visited the physical store location of Chipotle and those people who have not visited the retail store location.

New Hypothesis

Null Hypothesis (H0): There is no significant difference in viewing behavior of the Chipotle website users when engaging in different tasks.
Alternative Hypothesis (HA): There is a significant difference in viewing behavior of the Chipotle website users when engaging in different tasks.

Participants

Participants were recruited by each team member in a variety of ways. Most team members asked friends or fellow students if they were able to participate. Also, in an effort to reach the desired number of participants (ie. 10), one of the team members found subjects by walking around the building the lab resides in and asking for help. There were a total of 11 participants that completed the study. All subjects that began the study, completed the study; we did not have to remove any participants due to faulty calibration. There were 6 females and 5 males, with an average age of 30 years. Most participants were 23 years of age (N = 5). Only 2 of the participants had not been to eat at Chipotle, which was unfortunate because we hoped to have a 50/50 ratio of those that had and hadn’t been to the restaurant in order to compare the two groups. Also, all but 3 participants had previously ordered food online (no participants had used the ordering system tested in this study prior to their visit to the lab). It is also worth mentioning that all subjects are regular internet users.

Methodology

This eye-tracking study examined the viewing behavior of users visiting Chipotle’s online ordering website using an SMI remote eye-tracker. Participants engaged in tasks on a local modified version of the website located at https://order.chipotle.com/. In addition to the eye-tracking data analysis information, we also was obtained information about their demographics in addition to internet usage, experience with online ordering, and eating at Chipotle.

Between-Subjects Design

A within subjects design experimental design was used, where every participant worked through a set of predetermined tasks on the local version of the Chipotle restaurant website created for this study. All participants completed the same tasks, all described below.

Task List

  1. Take a brief view of the order webpage for Chipotle so you could describe to your friend about the menu items. This is considered a free viewing task in this study and has a duration of 1 minute.
  2. Placing an Order for a Friend: “Mark just came by your office to carpool to the Chipotle restaurant. He would like for you to enter an order on her behalf. Mark gives you the following description of her order: a fruit juice drink, one main entrée with three ingredients, and one side item. Make sure his name is assigned to the order and store his order for future reference.” This is considered a navigational search task and ended after the participant indicated they had completed the task.

Procedures

Participants are first asked a few background questions prior to the beginning of the test session. One of the experimenters of the testing session briefly discussed the purpose of the study and the overall process. The participant was then calibrated on a 9-point image reference and validation process. The entire viewing process took place on a participant computer terminal next to the observation computer terminal. The first screen shot, the participant first saw the overall description of the study. The proceeding screen centers the user’s focal point on black point on the screen to direct the user’s visual attention to the specific target. The following screen displays showed the participant instructions for the task, proceeding with a screen shot of the Chipotle ordering website to complete the tasks. Once the participant completed the task, the user used a keyboard command to proceed to the next task.

Test Environment, Equipment, and Logistics

Our team conducted the study in the visual perception laboratory at Rochester Institute of Technology. We used the SMI iView X RED eye-tracker technology to capture the eye movements of the participants. The Experimental Center software was utilized to prepare the stimuli for the test sessions; BeGaze Software utilized the x- and y- coordinates of the eye movement for further data analysis in the form of fixations, scan paths, and heat maps.

Test Personnel Roles

Two test personnels were assigned to each participant tested. One assumed the role of the moderator while conducting the session and introduced the session, conducted a background interview, and then introduced tasks as appropriate. The second test personnel managed the eye-tracking equipment that captures the quantitative data of the testing sessions.

Data to be collected and evaluation measures

In answering our research questions, our team was interested in the following performance data measurements during the test sessions. The performance data evaluates the user’s visual interaction with the Chipotle website. Our team gathered metrics through the x- and y- coordinates supplied after each completion of a test. These coordinates will then be mapped to record the following metrics using the BeGaze Analysis software:

  • Area of Interest: total dwell time within specific area
  • Time to first fixation: The times being observed from the display of website (stimulus) are shown until the start of the first fixation within an AOI
  • Fixation Duration (Fixation Length): The length of fixation being observed in seconds within AOI
  • Gaze: The sum of all fixation duration within a prescribed area of the website and images. This can also be referred to as a “dwell”, “fixation cluster”, or “fixation cycle”
  • Fixation Count: The number of fixations counted within an AOI
  • Observation Length: The length of observation a participant looks at an AOI, starting with a fixation within the AOI until the fixation ends and leaves the AOI
  • Observation Count: The number of visits and re-visits to an AOI
  • Fixations Before: The number of fixations before the participant fixated within the AOI for the first time
  • Percentage of Participants: Number of participants (out of total participants) fixating at least once within an AOI
  • Scanpath: A complete series or ordering (sequence) of fixations and interconnecting saccades during fixation length
  • Time to complete task: duration of the task

Test Conditions

Independent Variables

  • Participants that visited the retail store
  • Participants that have not visited the retail store
  • Task 1 or Task 2 (The difference in visual attention and scanpaths between the free viewing and ordering tasks)

Dependent Variables – Ability to order efficiently
We will compare the results for different users groups (novice vs. experts). Novices are users that have not been to Chipotle and experts are users that have been to Chipotle. We will also look for influence on whether users have previously visited Chipotle.com.

Research Questions Answered* and Metrics Used in Analysis

(*see Results section for analysis)
What is the first item that attracts users when viewing the website for the first time?

  • Time to second fixation
  • Scan Path

Do users dwell from image-to-image, image-to-text, or text-totext during different tasks?

  • Scan Path (fixation sequence)
  • Fixation Duration

How many images do users dwell in comparison to text?

  • Fixation Count

How much time do users look at the navigation headers for each section of the web page? (Or do users pay attention to the navigation markers of the page (i.e. navigation headers, top sequence header)

  • Observation Length
  • Gaze percentage per AOI
  • Gaze duration mean per AOI
  • Number of fixations per AOI

What components of the website cause confusion for the user while executing the tasks (i.e. images, physical structure of webpage, navigation of website)?

  • Gaze duration mean per AOI
  • Scan paths (fixation sequence)
  • Observation Count

Is there a significant difference in viewing behavior between users who have visited the physical location vs. those who have not visited the physical location? Is there a significant difference in viewing behavior between those who have visited the restaurant website vs. those who have not visited the website?

  • Scan Path Similarity
  • Fixations Before (the number of fixations before the participant fixated within the AOI for the first time)
  • Time to complete task

General Comments on Process

During our design on the experiment to determine what the participants voluntarily looking at (free viewing) or looked at according to goal of the task (navigation task), we prepared many procedures to answer our research questions and hypotheses after we experienced and ran into several problems that did not work well. We had to fix it before having the experimental design so we could give the tasks and monitor our participants carefully through our procedures. Earlier, we had discussed the ideas of using the iPhone as one of our stimuli comparing to the online and offline web page for this experiment. However our concerns were that the iPhone did not answer our research questions so this determined that web pages would fare better in this kind of experiment. The tasks were our second problem earlier. We wanted our selected participants to follow the instructions carefully under our monitoring so we could watch their eye movement on our designed web page. Under our monitoring, the results were successful to answer our questions. The qualitative and quantitative of participants were designed very well. We had expected numbers in male and females, and so were their experiences. However, during our practical process we had an issue about participants. We got two groups which one group would already attend at the restaurant while the other has not. Both groups are not equal; we had a lot more participants in first group while we had 2 participants in other group who have not experienced the restaurant. Other issue is the schedule. We had started late for the experiment because we had difficult to find the volunteers to participate due to schedule conflicts. We did our best to keep the quantitative of participants which are 6 males and 5 females who volunteered throughout our eye tracking experiment.

Results

Results were compiled using the BeGaze Analysis software by SMI. Images and videos were exported and several of the images can be seen below. Unfortunately, the heat maps did not seem to be aligned correctly (they were cut off halfway down the middle), so are excluded from the results.

Figure A shows each participants second fixation. A colored coded plus sign corresponds to each participant. The reason second fixation was chosen is that the first fixation was at a predefined point in the middle screen. When this point was looked at the experiment would automatically start causing the first fixation point to be in a nearly identical location for all participants. The location of first fixation points appear to be grouped around the first two images corresponding to the menu items: burrito and bowl. This shows that the images may have been the most attention-grabbing upon first look at the page.


Figure A: The second fixations while free viewing (Task A).

Figure B displays all 11 participant scanpaths in the first 3 seconds of free viewing. It was interesting to note that participants tended to look up at the top navigation area right away. However only one participant actually looked at the logo. The left heading labeled “Meal For” was looked at by several participants, as well as the label below it: “Burritos, Tacos & More. We had originally expected people would look more at the logo and the ignore the side headings. Also, it can be seen that people tended to start looking in the middle of the page and look up to the top portion (largely disregarding the lower portion of the page).

Figure B: Scanpaths in the first 3 seconds of the free viewing (Task A)

Some areas of interest (AOI) are selected below in Figure C. The two areas of interest include the top navigational header that is colored purple and named “navigation” in the information boxes. The other AOI is in red and labeled “X”. This section groups the menu items (entrees) that can be customized with the ingredients below (shown in Figure B above). The entrees options on the menu are: burrito, bowl, soft tacos, crispy tacos and salad. Figure C and D show the first 15 seconds of the task. Studying the first 15 seconds revealed that the top navigational header was looked at 9 out of 11 times (as determined by a fixation time of 80 ms). This shows that the top header area was noticed and evidence from Figure B also support this trend.

Figure D is similar to C in that the same AOIs and time durations are examined. However, Figure D examines task B (ordering for a friend with specific requirements). The top navigational header is not looked at nearly as much in task B; only 4 out of 11 people looked at it. This is probably due to the nature of task b in which they are given specific ordering instructions, and already have some familiarity with the page after task A.

Figure C: AOIs key performance indicators (KPIs) for the first 15 seconds of free viewing (Task A)


Figure D: AOIs key performance indicators (KPIs) first the 15 seconds of ordering for friend (Task B)

Figure E below attempted to determine if people were looking at the images or the labeled radio button with text (for example: AOI 005 in Figure E). It is important to note that the radio button is selected regardless of if one clicks on the image of the food above, or within the radio button itself which is the normal usage of these.

Figure E: AOIs key performance indicators (KPIs) first the 15 seconds of ordering for friend (Task B)

Back to Usability Page