AUGMENTED

REALITY IN
RETAIL
Final Year Project

CAES9542 Progress Report 2

Anchit Som 3035244265 | Rachaika Verma 3035244320

Abstract

Online shopping provides convenience, minimal setup costs and limitless inventory for
retailers. While brick and mortar stores provide an interactive and intuitive environment for
customers to view products. This report explores the opportunity present in the two markets
and proposes a hybrid shopping experience enabled by Augmented Reality technology. The
vision would be manifested as an iOS based AR application for convenient viewing. The
project aims to create a working e-commerce platform within the time frame of ten months
that serves both customers and retailers. In terms of progress, the team is on schedule and has
complete the first phase of the project, Feasibility and ARKit Testing. We will be moving on
to the next stage of the project which is model creation of the cityscape and the shopfronts.
After they navigation interface has been created, the team will focus on making the product
demos and will start testing the application by March 2019.

pg. 2
Acknowledgement

I would like to thank our project supervisor Dr. HF Ting for his valuable feedback and
support during the course of the project. I would like to thank the Computer Science
Department for helping us procure iOS devices for testing. I would also like to thank
professor Cezar Cazan for providing us with guidelines on writing a good project report.

pg. 3
Table of Contents

List of Figures…………………………………………………………………………….5
List of Tables..…………………………………………………………………………….5
1 Introduction..................................................................................................................6
1.1 Background.............................................................................................................6
1.2 Scope........................................................................................................................6
1.3 Deliverables..............................................................................................................7
1.4 Outline of Report.....................................................................................................7
2 Related Works...............................................................................................................7
2.1 Ikea...........................................................................................................................8
2.2 Lacoste………………………………………………………….............................8
3 Features and Objectives……………………………………………..…………….....8
3.1 Cityscape..................................................................................................................9
3.2 Shopfronts..........................................................................................................…..9
3.2 Product Demos........................................................................................................ 9
3.2 User Interface.......................................................................................................... 9
4 Methodology............................................................................................................... ..9
4.1 Technologies used .......................................................................................... …….9
4.1.1 Apple’s ARKit 2……………………………………………….9
4.1.2 Unity 2018.1.1………………………………………………..10
4.1.3 Blender 3D…………………………………………………...10
4.1.4 Firebase Unity SDK………………………………………….10
4.2 Application Design ................................................................................................11
4.2.1 Navigation plane………………. ……………………………12
4.2.2 Shops Fronts………………………………………………….12
4.2.2.1 Shop Exterior……………………………………………………...12
4.2.2.2 Shop Interior……………………………………………………...12
4.2.3 Product Demo……………….……………………………………….13
4.3 System Design……………………………………………………………………..15
5 Current Status.............................................................................................................15
5.1 Feasibility Study: Completed.................................................................................15
5.2 ARKit Functionality Testing: Completed…………………………………...…...15
6 Results and Discussion...............................................................................................16
6.1 WRLD Unity API pivot……….............................................................................16
6.2 Environment Detection issues................................................................................17
6.3 Lighting Issues………………...............................................................................18
7 Next Steps..............................................................................................……………..19
8 Conclusion…………………………………………………………………………...19
9 References……………………………………………………………………………20

pg. 4
List of Figures
Figure 1: Impact of AR on Retail.............................................................7
Figure 2: Ikea application demo...............................................................8
Figure 3: Application Design Flowchart…………….............................11
Figure 5: Artwork on Frame....................................................................14
Figure 5: Hand Accessories .....................................................................14
Figure 6: Car demos..................................................................................14
Figure 7: Rendering Results using WRLD API......................................16
Figure 8: Tutorial screens.........................................................................17
Figure 9: Ambient Lighting Code Snippet..............................................18
Figure 10: Dull Rendering vs Ambient Light Rendering.........................18

List of Tables
Table 1 : Project Schedule………………………………………………19

Abbreviations
● AR: Augmented Reality
● SDKs: Software Development Kits
● API: Application Programming Interface
● UI: User Interface
● VIO: Visual Inertial Odometry
● IMU: Inertial Measurement Unit

pg. 5
1. Introduction
Augmented Reality today is a rather new technology with several implications in the
modern world. However, as it is still very fresh, not much is known about its potential
and implications. This section aims to implore several aspects of the technology with
respect to the project at hand.

1.1. Background

The current retail landscape can be broadly divided into two categories-offline shops
and online stores. Online stores have become a norm in the past few years due to
minimal setup costs and no inventory costs. However the product viewing experience
is not intuitive enough for buyers which often leads them to prefer Offline stores to
make purchase decisions. Unfortunately for retailers, Offline stores still have
infrastructure costs associated with them which makes them expensive to set up. Our
project aim to bring the advantages of physical product viewing to online shopping
using Augmented Reality. This hybrid shopping experience would enable users to
interact with the products in virtually created environments that they can view from
their phones.

1.2. Scope
The development of the retail application would be done on Apple’s ARKit2 platfrom
thus the scope will be limited to iOS devices. The reason for choosing ARKit as our
development platform has been discussed in the section 4.1.1 (kiran) The application
would be ready to deploy on the following ARKit Compatible devices.
- iPhone 6s and 6s Plus.
- iPhone 7 and 7 Plus.
- iPhone SE.
- iPad Pro (9.7, 10.5 or 12.9) – both first-gen and 2nd-gen.
- iPad (2017)
- iPhone 8 and 8 Plus.
- iPhone X, Xs and Xr

pg. 6
 1.3. Deliverables
Our end deliverable will be an iOS based application that will be ready to deploy on
the devices mentioned above. We will be using the Firebase SDK(refer to section
4.1.2) for database creation and would be handing over details for accessing the
database along with the application.

1.4. Outline
The report discusses the current trends in Augmented Reality and points out our
motivation to engage in this project. This is followed by a list of objectives we aim to
achieve and a detailed methodology for executing the project. The report then
explores the work we have done till now and the results obtained in the process. The
next steps to the project are discussed along with a project schedule in the final
section.

2. Related Works
The following section discusses a few use-cases Augmented Reality in Retail and furthers
our case for making an AR based application.
AR has been used in mobile shopping applications by various firms like IKEA and
Shopify to add competitive edge to their consumer-experience (Sheehan, 2018). Figure 1
below shows positive customer response on using AR in Retail settings.

[Figure 1: The Impact of AR on retail]

pg. 7
2.1 Ikea
Ikea Place is an application that helps the user visualise the furniture in their
home space before making a purchase decision. (Chang, 2018).” This is
particularly important in the home decor retail space as customers often spend
countless hours trying to decide whether something will look good in their
homes or not. Figure 2 how users can place objects in their homes before
making a purchase decision. The “try before you buy” concept by Ikea has
been very well received by its customers and serves as a motivation for us to
build our application.

[Figure 2 Ikea Application demo]

2.2 Lacoste
Lacoste has created an AR application that allows users to try on the lastest
pair of sneakers by pointing their phones at their feet (Mortimer, 2014). While
the Lacoste application is built for a store environment, our project aims to
build an application that can be used in a home environment with different
levels of lighting.

pg. 8
3. Features and Objectives
This section outlines the different features of the application that we aim on building in the
project

3.1 The Cityscape

The cityscape represents a navigable landscape on which shop fronts will be placed
for the user to interact with. Navigating in the cityscape would be similar to exploring
a street market in the real world with different shops to choose objects from.

3.2 Shop Fronts

The shop fronts will be used to emulate physical shops, they will have an exterior and
an interior like physical shops do and descriptions to identify the items they sell.

3.3 Product Demos

The items for sale will be placed inside the store as demos and this will form the basis
of our AR experience. Our project focuses on four types of products - Jewelery, Cars,
Furniture, Shoes and Handicrafts. The user will be able to interact with these objects
in AR in a manner similar to physical products before making a purchase.

3.3 User Interface

To guide the user through different modes of the application, an easy to use and
intuitive UI will be designed. This UI would integrate with the superimposed images
to provide an interactive shopping experience to the user.

4. Methodology
This section discusses the technologies used by the team to implement the project. It lays
out the application design as well as the system design in detail.

4.1. Technologies used

4.1.1 ARKit Unity SDK

pg. 9
ARKit is a Visual Inertial Odometry (VIO) system that tracks the user’s position by
matching a point in the real world to a pixel on the camera sensor at each frame
refresh. The Inertial Measurement Unit (IMU) also measures the user’s position using
inertial sensors - Gyroscope and accelerometer. The time measurements and
accelerations are integrated backwards to calculate velocity and distance travelled
between two frames. While error in such a system is upwards of 30% in most
systems, ARKit’s error correction reduces the error to single digit. Every tiny muscle
movement can be detected by the inertial system providing unparalleled accuracy.
The output of these systems are combined using a Kaiman Filter to provide the best
estimate of the actual position which is published to the ARKit SDK.

4.1.2 Unity 2018.1.1

Being a mature 3D game engine, Unity is the perfect choice for AR development. It
has well documented functions and large developer community, making it easier to
build applications. The Unity ARKit plugin wraps the SDK in C# scripts, this makes
the code object oriented and allows for a modular software design. Unity also has
tools like ARKit Remote that allow creating interactions directly in the Unity editor
saving time on building the application each time.

4.1.3 Blender 2.79

Blender is one of the most advanced 3D Modelling software as it provides a wide

range of templates for rapid prototyping. It has a wide range of textures, shaders and
tools for frame by frame animation. Blender integrates well with Unity as we can
directly load scenes from the platform to the Unity Project, along with importing .fbsx
files.

4.1.4 Firebase Unity SDK

The application design requires dynamically created 3D files for rendering and
display. Firebase is a great database platform that safely stores assets and user made
content in the cloud. It’s real-time database syncs content across all devices in a few
hundred milliseconds. With its Unity SDK it becomes easy to query the database

pg. 10
 directly from the application itself and allows for the backend logic to be
implemented in C# scripts.

4.2. Application Design

The application design will incorporate a navigation plane, the shop fronts and payment
portal system as shown in figure 2. When the user opens the application, he would be told to
point the camera at a flat textured surface to create the navigation plane. On top of this
navigation plane, the shop fronts will be loaded for the user to interact with and explore.
Once the user selects a shop front, he/she will be directed to in-store mode where products
will be displayed in different product lines. Each product line will act as a product shelf
where a user can select a particular item and view the product demo. If the user likes the
item, he/she can add it to the cart from this view and proceed to the the payment portal.

[Figure 3 Application Design Flowchart]

pg. 11
The payment portal will be implemented as an abstract class for future development and will
not be a part of this iteration of the project.

The following section discusses in detail how each of the components works in ARKit and
how we will be designing them.

4.2.1 Navigation Plane

ARKit detects planes using the ARAnchorManager class. When feature points are
detected, a bounding box is created inside while a horizontal plane is initialised. The
following function takes in a prefab object and inserts it into the scene as the
horizontal plane.

4.2.2 Shop Fronts

Each of the shop front will be a Gameobject in Unity and these would be stored in a
Gameobject List belonging to the specific shopkeeper (see figure 2). When the
navigation plane has been set, ARKit would perform Raycasting in order to find
anchor points where it can place these shop objects from the list.

4.2.2.1 Shop Exterior

The shop fronts will have exterior elements that will be visible as soon as the
objects are spawned such that the user understands what each shop is about
before entering it.
The shop exterior can be divided into 2D elements describing the shop and the
3D architecture that would emulate a physical storefront. The 2D elements
include a hoarding/label to identify the shop, a short store description to
provide more detail (see fig 2), and an average rating given by users for that
storefront.
The 3D model will include basic elements like roofs, walls and doors, we will
design a base model in Blender to import to Unity for testing. After the Base
model is tested, we will move onto designing details of the store exterior and
create a range of options for the shopkeeper to select from when designing
their own stores.

pg. 12
 4.2.2.2 Shop Interior

Once the user has selected the store, the application will shift from the general
navigation to the in-store mode. The In-store mode will be used to emulate the
inside of a retail shop with different product lines and the option to view
product demos. When this mode is initialised, the navigation plane prefab will
be replaced by the In-Store floor prefab and the Plane Function will be called
again. Hitcasting will be performed to create the store interiors and place
product lines on Anchor Points that the camera detects.
The interior decor aims to emulate physical stores as closely as possible, it
will include elements like walls, doors and windows. Similar to the approach
with the exterior decor, we will design a base model in Blender for testing. We
would then be adding different options that the shopkeeper can use to
customise their shop and make it unique.
The product lines would appear like aisles in the grocery store, with a label
describing the products inside this shelf. They will be designed in Blender as
basic shelves for testing; after a few iterations we would include customisation
options.

4.2.3 Product Demos

Each product will be a .usdz file that would be uploaded by the shopkeeper to view in
Augmented Reality. Some product demos like the ones for furniture will require a
change of plane, thus on entering this mode, the user will be asked to perform Plane
detection again so that the object can be spawned accordingly. Once Plane detection
is done, the object will be pulled in from the Product Gameobject List (refer fig 2) and
the user will be able to perform basic functions like rotating the object and scaling the
object.

A small catalog with the products from the product line will be displayed at the
bottom to easily select products for demp. The product demo is an integral part of the
application as it allows the user to make a purchasing decision based on the look of
the product, its dimensions and other physical factors. For the purpose of this project,
three different categories of product demos will be created.

pg. 13
 List of Product Demos

[Figure 4 Artwork on frames]

The user can point to an empty frame or a wall mount
and see a painting appear on the view

[Figure 5 Hand Accessories]

The user can keep their hands under the camera screen
and view accessories like jewelry, watches.

[Figure 6 Car Demos]

The user can point to a tabletop and view a car model in

3D appear on their device screen.

A small cart button will be displayed on the bottom right to add the current product to the cart
while an exit button would take the user back to the In-Store mode. It is important to note that
when the user comes back into the In-store mode, he would be asked to perform Plane
detection again such that objects can be spawned accurately.

pg. 14
4.3 System Design
The current system consists of two users primarily, the shopkeepers and the customer. They
will have access to a different set of classes and objects. The shopkeeper will setup his shop,
design the exterior of the shop and add labels for shop identification. In the In-Store mode,
the shopkeeper will add Product lines and 3D models for product demos. These models
would be converted to the .usdz format that Apple has developed specifically for ARKit
objects. The shopkeeper will have access to the customer view to see how their shopfronts
appear in the application. The Customer will be able to perform basic functions for
interacting with the application. In the navigation screen, the customer can view the
shopfronts and select a shop to visit. Inside the shop the customer can view different product
lines and select a product line to view the products on offer.

5. Current Status
This chapter will discuss and conclude the various milestones achieved by the team across
the various development phases.

5.1 Feasibility Study: Completed

This study was carried out to understand the structure and components of the ARKit SDK
along with its functioning. We performed a thorough analysis of the Apple Developer
Documentation pertaining to ARKit. We explored tutorials created by Unity and Udacity to
understand the workflow for creating an AR application.

It was essential for the team to be well-versed with C# before jumping into full scale
development. Thus, the team spent considerable amount of time shifting from Java based
development to C#. The feasibility study helped the team define the objectives and scope for
the project more clearly.

5.2 ARKit Functionality Testing: Completed

pg. 15
The team performed preliminary plane detection tests in different environments to assess the
platform’s reliability. These plane detection tests involve identifying feature points and
loading prefabs to into the plane function for rendering. After rendering, we tweaked texture
values and created shaders to make the plane appear better.

After plane detection tests were completed, the team tried to spawn objects using Raycasting
functions. These objects were sample 3D models downloaded from https://free3d.com/ and
stored as assets in Unity. The objects were placed in a Gameobject list and were spawned
randomly to check the accuracy of the Raycasting.

6. Results and Discussion

During our testing we came across several issues ranging from missing dependencies to
imperfect object renderings.

6.1 WRLD API Unity SDK Issues

For creating the cityscape of the application, we decided to use the WRLD API as it
provided in built city maps and a number of templates with pre-created buildings. We
ran a number of experiments with the API and as seen in figure 5, we had really good
rendering results

[Figure 7 Rendering Results using WRLD API]

pg. 16
However, on further exploration of the documentation, we found that each building in
the cityscape required licensing permissions from the building owners. Considering
the scale of our application and the large number of building, it would have been
extremely difficult to get all these permissions. Thus we decided to pivot and create
our own cityscapes from scratch, instead of relying on a third-party API.

6.2 Environment Detection Issues

The plane detection performed well in textured areas however performed poorly in
areas with smooth surfaces. In most instances the plane detection was good however
at certain points the plane started shifting when the camera moved instead of staying
in one consistent position. This limitation cannot be solved by Software Engineering
alone as it is heavily dependent on the camera’s sensing quality and the hardware
Apple builds. Thus, despite being the most advanced VIO system, ARKit has
problems in performing well in certain environments.

Thus the team aims to introduce a small tutorial screen guiding the user to select a
well textured environment for running the application (similar to Ikea).

[Figure 8 Tutorial Screen]

pg. 17
6.2 Lighting Issues
In the object spawn test, many a times objects weren’t visible or appeared dull leading
to a bad overall rendering. After investigation, we found that the lighting estimate was
heavily dependent on the light available in the surrounding. In environments with
bright light our objects performed well, however in low light conditions the rendering
performed poorly(refer to figure 9 Dull Rendering)

This problem was fixed by the team by introducing a function(figure 8) that adjusted
the object lighting according to the ambient lighting of the room. The difference after
the function is applied can be seen in figure 9.

[Figure 9 Ambient Lighting Code Snippet]

[Figure 10 Dull Rendering vs Ambient Light Rendering]

pg. 18
 7. Next Steps
Having completed the feasibility study and the ARKit functionality testing, the team is
currently building 3D models of shops and planes in Blender. These models would be
imported into Unity for further testing and development of the application.

Schedule Task Status

September-October 2018 Feasibility Study and ARKit Testing Completed

November-December 2018 Shopfront and Plane Design In Progress

January-February 2019 UI and Backend Development Planned

March-April 2019 Deployment and Testing Planned

[ Table 1 Project Schedule ]

8. Conclusion
Augmented Reality is still in its infancy and has a lot of potential to grow in the future. With
companies like Apple and Google developing dedicated software bundles for AR, developers
are creating world-class AR applications. Our aim to bridge the gap between physical and
online stores is a humble attempt at building a strong use case for AR for mobile devices.
Though our shops are limited in the number of items they sell, we only aim to create a proof-
of-concept that can serve as a guide for future development. We hope to implement the
payment portals and other features once the project objectives are achieved and a product is
created. Our project isn’t just a one-off application but a new kind of user experience that
uses visually imposed images as interaction points. We believe that the application has the
potential of changing the retail industry and how people perceive AR applications.

pg. 19
References

1. Blender Foundation. (2018). Free Software Never Looked This Awesome. Retrieved
from https://www.blender.org/features/
2. Blum, A. (2018). ARCore vs. ARKit: Which Is Better for Building Augmented Reality
Apps? IFlexion. Retrieved from https://www.iflexion.com/blog/arcore-vs-arkit-better-
building-augmented-reality-apps/
3. Chang, L. (2018). See how that couch would look in your living room in AR with Ikea
Place. Digital Trends. Retrieved from https://www.digitaltrends.com/home/ikea-
place-ar/
4. Mortimer, N. (2014). Lacoste boosts in-store shopping experience with AR app for
new trainer range. The Drum. Retrieved from
https://www.thedrum.com/news/2014/07/21/lacoste-boosts-store-shopping-
experience-ar-app-new-trainer-range
5. Sensor Tower. (2017). Games Lead ARKit App Downloads, Which Top 3 Million
Since Launch. Sensor Tower Inc. Retrieved from https://sensortower.com/blog/top-
arkit-apps
6. Sheehan, A. (2018). How These Retailers Use Augmented Reality to Enhance the
Customer Experience. Shopify POS. Retrieved from
https://www.shopify.com/retail/how-these-retailers-are-using-augmented-reality-to-
enhance-the-customer-experience

pg. 20