Concept, User Research, Persona Building, Journey Mapping, Wireframing, Prototyping, User Testing
For ten weeks, we explored the feasibility of a grocery shopping platform in Virtual Reality (VR). The exercise aimed to understand how users might interact within a virtual grocery shopping medium based on their experiences in the real world.
The past two years have accelerated our dependence on digital platforms, with consumers becoming accustomed to tech as a replacement for once vital in-person interactions. From the way we work, shop, or enjoy entertainment, consumers have developed an understanding that nearly everything can be done online.
Virtual Reality is no longer a nebulous technology found only in science fiction. It has become a tool available to the average consumer, but as an emerging technology, is still new enough that it poses challenges for users. It's intuitive yet clunky. It attempts to simulate how we interact in the physical world but utilizes new virtual tools and occurs within a virtual space. Though what we are capable of in VR environments is often an emulation of real human behavior, the concepts remain new to us.
Before we began user interviews, we had to consider two key questions:
We weighed the cost-benefit of outsourcing participants to a usability testing firm but ultimately chose to recruit via friends/family and cold calling. Given our project scope and early stage in generative research, it was faster, cheaper, and yielded areas of interest that we could continue to explore throughout our research.
We probed areas of interest from grocery shoppings by asking open-ended questions, giving us the benefit of allowing users to guide us towards issues that we may have overlooked while acquainting ourselves with the space. This was done with an eye towards how VR might be able to improve the grocery shopping experience.
Nearly all customers interviewed had poor experiences ordering fresh fruit/produce online.
Inventory disparity was reported as a major pain point for both the in-store and online shopping experience. For in-store shoppers, this meant being unable to find an item, while for online shoppers, inventory disparity meant online products didn’t reflect the actual quantity in the store.
Users noted that they often ask for help while shopping, usually to find an item that they can't locate on their own
Users reported increased usage of store-specific apps for coupons, discounts, and as an enhancement to the overall grocery shopping experience
We turned to the Reddit community, hoping to gain insight from those already familiar with the technology and its uses. We created a survey to gauge interest in a VR grocery shopping experience and administered it to relevant communities. Those that participated included r/HTC_Vive, r/Oculus, and r/VirtualReality.
These were the key insights that would go on to drive our ideation phase:
When asked "Why would you/wouldn't you be interested in VR grocery shopping?" the interaction with products and the ability to inspect from all angles was most noted response.
Those with less experience in headsets indicated greater hesitance to engaging with a VR grocery shopping experience.
Convenience is the primary motivating factor for determining where customers will choose to shop, and they expressed a willingness to pay for it.
Users reported varying systems of categorization/store layout as the main drivers of frustrations with finding different items.
We approached our designs from the framework of recontextualizing – rather than reinventing – the grocery shopping experience. Two questions anchored our approach:
In considering how a VR platform might improve areas of frustration in grocery shopping, we turned to those that our research highlighted as of interest:
We sketched various ideas that went beyond simply placing a grocery store into an immersive VR environment.
...enable movement for users within a VR context (if at all)?
...simulate the idea of a cart that holds users choices until ready to purchase?
...incorporate gestures that emulate behavior from a physical grocery shopping experience?
...personalize the experience unique to the individual?
Our vision was to introduce users to a new way of finding and interacting with products via the Grocery Carousel, a Lazy Susan-like display of columns rotating around a fixed, central point. These columns would aim to mimic the aisles of a grocery store as if they were vertical instead of horizontal. Our first iterations imagined these as grid-like displays that could be rotated, from which a user could grab items, inspect them, and add to their cart – just as they would do from shelves in real life.
We understood the need to distinguish between “levels” of the experience — allowing a user to progress through varying tiers of detail when viewing products. We aimed to draw parallels to the in-store setup by imagining the following, with each step bringing the user to a more detailed viewing and interactive ability:
All item categories (columns) represented by a selection of groceries from that category, likened to one’s view when first entering a store and seeing various aisles
As if the user had walked into a particular aisle or section at the store and now possessed the ability to view more closely the items within it
A detailed page of item-specific information that would appear if a particular item was selected for further inspection
A major consideration remained with regard to user interest in a VR grocery shopping platform. Recalling that almost half of polled users noted the "ability to inspect items at every angle and understand the scale of products" as the main draw for such a product, we considered the following:
We were inspired by Apple, particularly its use of touch gestures on its devices. Recall the process by which an iPhone app signals that it is ready for moving or deletion – it begins to shake when tapped and held.
We built on this idea – a defined area where an object could be held and "anchored” – to unlock an additional set of gestures available for that particular item. We would call this new mode the “Sandbox."
It would offer the most detailed level of object manipulation, such as understanding the size, the ability to change quantity/volume, a context to evaluate the product at a more detailed level, etc. – and truly leverage the benefits of VR technology.
More of a “mode” than a page, enable users to manipulate 3D assets and add these assets to one’s cart (ie. halving or quartering a watermelon, of which any or all parts may be purchased)
With an understanding of how this reimagination might work, we generated a sitemap to more clearly envision the architecture of the environment. This highlighted the difficulties of mapping out a 3D VR experience on a 2D plane; it also meant giving thought to user movement through Virtual Reality experiences. We honed in on the sense of presence, which Mina C. Johnson-Glenberg referred to as the “first profound affordance” of Virtual Reality in her 2018 study Immersive VR and Education: Embodied Design Principles That Include Gesture and Hand Controls.
Consider the following: You're immersed in headset on a product detail page or the detailed shelf view of the dairy section. On a web or mobile interface, it would make sense that you might navigate to the frozen foods by backing out of the current selection, finding the new category, and then working your way once again through the options. However, in a VR setting, this navigation entails lots of movement – and increases the likelihood of what is called "simulator sickness."
We felt that a user having to manually navigate their way through the aisles of a store and find specific grocery items would not only defeat the entire purpose of a convenient product, but also subject users to excess motion. We considered how we might ease navigation between levels of the experience.
We introduced the idea of a Smart Assistant, our alternative to the omni- yet optionally-present Siri or Alexa — willing and able to teleport one to a very particular part of the experience. We also added tabs of main categories (those represented by the columns of the carousel) that would allow users to reduce the steps needed to move around level-by-level.
As this was our first exploration into Virtual Reality, we evaluated the time it would take to create a prototyped experience, the constraints of free VR prototyping software, and remembered that our main priority was research. We decided on a low-fidelity prototype for exploratory purposes. This allowed us to focus on content rather than visual design while developing a more holistic understanding of the user journey.
We utilized the DraftXR plugin for AdobeXD, which allows users to leverage 2D assets into a 3D environment. Though it lacked the functionality and true feeling of presence within a VR environment, we felt comfortable mocking up most of what we had envisioned for the project. Paired with screenshots of our work in ShapesXR, we asked users to use a touch of imagination when progressing through a clickable prototype. Though we lacked the ability to utilize gesture control in conjunction with our VR headsets, we felt that a series of click-through screenshots demonstrating hand control and gestures worked well enough to communicate the functionality we were imagined.
Since we were using the Oculus Quest 2 VR headsets, we imported the native gestures that Oculus users were already accustomed to performing, such as point and pinch to select. With this basic framework for selection established, we turned to the VR gaming space to explore more engaging ways of interacting in headsets. We chose the space because video games are designed to be immersive, entertaining experiences. They must engage and hold onto the gamer: this means pushing the limits of either storytelling, graphics, mechanics, or a combination of all three.
Our approach to interactions was inspired by a game called SUPERHOT, wherein a player must destroy oncoming waves of enemies by punching, throwing objects, or shooting weapons.
For a SUPERHOT player, there is no pointing and clicking; instead, they must reach out with their arms and grab objects as though the latter were truly in front of them. We were intrigued by the gameplay mechanics of direct object interaction and intuited that it would translate well into our VR grocery shopping exploration.
Currently, VR leverages the use of dual controllers (one in each hand) for point and click interactions within the headset. Although these controllers utilize the basic principles of a computer mouse and are therefore precise and easy to learn, our findings indicated that this method will likely soon be a relic of the past – this echoes the sentiment of Meta, the parent company to Oculus, which released gesture controls in 2020 for the Quest headset.
Our ideation on hand gestures was informed further by the work of Guerra et al. (2019) in Applied Sciences, Hand Gestures in Virtual and Augmented 3D Environments for Down Syndrome Users, which sought to test commonly used interactive 3D gestures between both Down’s Syndrome (DS) and neurotypical participants. We chose this study to drive our next decisions with two key areas in mind:
Design in a way that utilizes gestures similar to those used in real life, with the goal being the simulation of actions performed in a physical setting
Although initially designed to benefit the disabled, accessible designs can produce a ripple effect of benefit for a much larger group.
The Guerra study suggests that there are differences in the successful execution of different gestures; more importantly, that due to limitations in motor skills, 3D gestures should involve the whole hand as opposed to fingers.
The impact of Guerra's work on our project was twofold:
We designed the direct interaction experience by repositioning the user closer to objects, about an arm’s length. By closing the distance between user and object, the former would be capable of clearing the distance to reach forward and directly grab hold of objects. In the screenshot below, a prototype demonstrates the grabbing of a yellow bag of candy. Just as easy as it is for a user to grab an object directly in front of her, she can also select objects further away. To select the can of chips located on the top shelf, the user would raise her hand to the object and perform the grab gesture, snapping the item into her palm.
We also integrated the following features, believing that these additions would create the most advantageous environment for users and minimize interaction costs:
Highlighted objects: a visual indication that certain objects have become interactive.
Voice command: to be used for activation of a smart assistant that mirrors the presence of a grocery store employee, fulfilling a role similar to that of Siri or Alexa.
As we worked to expand these accessibility principles into our interactive Sandbox Mode, we highlighted two areas of concern:
Prolonged raising of the arm could become tiresome, resulting in imperfect or accidental actions
Without guidance prompts, it would be difficult to know exactly how the object could be manipulated
We improved on these issues by again turning to the video game industry. In an online game of billiards, for example, a dotted line emerges to guide players on where the ball will travel based on the cue’s positioning. The dotted line is easily understood by players of all levels and conveys enough information for an accurate strike.
We believed that incorporating this dotted line functionality and the use of the hand (instead finger gestures) would help to resolve the shaky hand and imprecise tracking issues. The dotted line creates a more accurate, true-to-intent guideline, thereby reducing the attempts needed for the desired outcome. Furthermore, utilizing a large sweeping hand motion would allow users to reduce time spent with their arm raised as opposed to smaller finger gestures.
We created prototypes according to the key stages we identified in our user flow, incorporating the features we felt were vital for shipping an MVP.
The carousel view would act as the opening scene for users: a "main menu" from which one begins their experience, as though browsing through grocery aisles in a store.
We utilized the DraftXR plugin to prototype portions of the user journey and create photorealistic representations of our imagined 3D object fidelity. We created an array to represent a store shelf and the items on it. A user’s cart would always be right in front of them, affording product additions simply by grabbing items from the shelf and releasing them into it, much like an in-person experience.
Our main area of opportunity to explore new gestures and interactions. A study published in Applied Sciences, Design of 3d Microgestures for Commands in Virtual Reality or Augmented Reality, by Li et. al (2021) found that participants are willing to learn new, unfamiliar gestures for some commands instead of more familiar ones – a concept we wanted to test with users.
To make the experience accessible and available to someone in a stationary position and minimize any discomfort one might feel by needing to navigate around the test layout, we envisioned that users could employ voice commands for teleportation to various stages in the shopping experience.
We conducted a hybrid method of testing, guiding our participants through a user flow of the end-to-end shopping experience compiled in AdobeXD. Participants were encouraged to think aloud and gesture as though interacting with the 3D objects directly.
Usability tasks were developed with the following goals in mind:
We took a page out of the Oculus playbook and its First Steps with Hand Tracking to orient new users with the immersive experience. We created a tutorial video to familiarize users with the reimagined shopping environment, pointing out key features and gestures. There were two main considerations for this:
Most users interviewed didn't have extensive VR experience. The intent of the tutorial would be to prime them for what they could expect to encounter in the testing environment.
We wanted users to receive the same set of instructions and accompanying visuals. It would also reduce the number of variables that could lead to confusion about how to interact in the environment.
We tested our prototype on five users through a mixed-method format of VR (in headset) and 2D (on a computer screen). We moderated the usability testing and asked our users to perform a series of tasks covering the user journey from login to checkout in three phases:
Results nudged us to reconsider the signifiers we'd included to help with navigation around the carousel and into deeper levels. All users stated that they knew it could be rotated but only a few could determine how the apparatus moved. Progressing to a more detailed view presented challenges to users.
All users gestured with their arms forward as if they were grabbing the object and proceeding to drop it into the car in front of them.
Progressing to the Detailed Shelf View was not as intuitive, with users citing uncertainty of how to initiate – swiping, pointing, or grabbing and spinning – as the cause.
Users struggled with text since they were unsure whether the text was written to be read or clickable as a link for further info. Furthermore, some users failed to place the chips into their cart from the Product Detail Page when moments prior, they'd succeeded in performing the same gesture with a bag of candy. We learned that this was due to confusion among users in discerning whether or not the cart was still interactive.
We anticipated users would struggle with this sequence of tasks, yet to our surprise, all users completed both activities and required very little assistance.
Once users were familiarized with voice control, they routinely used or asked to use the feature for the remainder of the activities.
Testers appeared to act instinctively when cutting, all performing the full hand swipe motion with ease and minimal instruction.
All users successfully removed the chips from their cart on the first attempt. More importantly, they all motioned the same (if not identical) gesture: grabbing the item and tossing it out of view to remove.
All users progressed without issue through the checkout process. Some voiced concerns regarding security, given their payment method had their information pre-filled; however, this was more closely linked to our prototype fidelity rather than what we'd envisioned in a working product. We also realized that we'd not adequately built out the final order confirmation screen, which, interestingly, users have become accustomed to seeing at the end of a purchase and all noted as missing.
After the test, we had users fill out a System Usability Scale to gauge each of their subjective assessments on usability. The average score was 72, placing the experience at just about average for usability.
This project was a lesson in challenging decisions from prior steps and considering how to adjust moving forward. Our testing phase proved no different — due to imperfect testing methods and the inability to truly test the gestures we’d imagined within the VR environment, we found the space between success and failure on certain usability tasks was sometimes a bit too ambiguous, and lacked the type of insight that would drive new iterations. However, we found the feedback from our think-aloud testing to be immensely valuable. We’ll highlight some of these areas below and offer some reflections on how they might be improved.
We learned that our choice of Smart Assistant, cheekily named Broccoli Rob, might impose a type of pressure on users to make what are generally considered “smart” choices. Users were split on whether or not they wanted to know that the Assistant could be utilized at any point in the process, and thus have a constant presence somewhere in their field of view. Some suggested that a feature that checks in on shoppers frequently throughout their experience could be helpful, but we're wary that this might contribute to alert fatigue. However, the value of incorporating an Assistant tool to simplify steps in the process has a large potential upside and was regarded as a positive feature by participants in our testing. We can imagine a more active role in moving between categories of groceries, checking out, adding items to one’s cart, or comparing between products with the help of a Smart Assistant.
VR environments are a highly immersive and stimulating experience. There is a fine line between providing adequate levels of information for a user to progress and including too much. Some users noted the desire to be able to view prices from the Carousel View and Detailed Shelf View. We’re not sure this is the answer, as we iterated earlier on price inclusion and saw very quickly that it was a lot for a user to view at once. We can envision the ability to see prices when an item is picked up, or in the case of the cart, when products enter the field of view. We discovered a similar feeling among our participants when it came to Sandbox Mode — they were more confused than aided by the UI elements, so we removed them to return the focus purely back on the gestures.
It’s important to note that this project was by no means exhaustive — in fact, in light of its speculative nature and our limited resources in prototyping and testing in VR environments, it was never going to be. Our inability to test gestures among our subjects in the headset, paired with being forced to rely on multiple mediums to articulate our vision for what grocery shopping in VR might look like, posed challenges.
As it stands today, we don't believe that a product like this is feasible for companies to develop while hoping for a substantial return. Our research and testing indicated that while the interest for a VR-based grocery shopping platform exists, it is only found in niche market segments – namely those already accustomed to spending time in headsets and those who utilize the technology in other parts of their lives. The barriers to entry remain yet too high and the day-to-day use-cases too few, and the trend fails to extend to general audiences. The headsets are still too clunky, expensive, and seen as novelty when it comes to the general population, especially given our access to well-functioning delivery platforms like Postmates, Doordash, etc. which are native to the now-ubiquitous smartphone.
That said, as the technology develops and grows from novelty to a capable platform that affords a unique virtual immersion, we see the possibilities of where a project like this could lead. AmazonGo is a good example: a true grab-and-go alternative to traditional check-out processes driven by contactless convenience and “just walk out” technology. We don’t believe it is too far a leap to imagine a model that functions similarly, only purchased from the comfort of one’s headset. A quick browse through VR forums will show the leaps the technology has made just since the time of this writing – and its progress is not slowing.
An experience must be usable before it is enjoyable. To that end, we feel that we made strides towards usability: recontextualizing the grocery shopping experience and communicating those ideas in a VR environment. In the future, we imagine a platform capable of complete personalization and can't overstate the value of enabling users to tailor the platform to their own needs: perhaps incorporating the ability to set dietary preferences and restrictions, like setting alerts for foods with processed sugars or those that are high in sodium.
We’d also like to note the exploratory benefits of interactive VR assets, such as understanding size and weight differences between pieces of protein or the textural differences between a crumbly queso fresco and a more solid block of mozzarella. There is much to be gained from the immersive level of detail that Virtual Reality provides, and we believe that the ability to interact, understand, and customize products for one’s order at this level will restore many of the intangibles that are lost in the transition to online grocery shopping.