Fruit Freshness Detection Using Android-Based Transfer Learning MobileNetV2
Fruit Freshness Detection Using Android-Based Transfer Learning MobileNetV2
Abstract. Fruit is an important part of the source of food nutrition in humans. Fruit freshness is one of the most
important factors in selecting fruit that is suitable for consumption. Fruit freshness is also an important factor in
determining the price of fruit in the market. So it is very necessary to detect fruit freshness which can be done by
machine. Take apples, bananas, and oranges as samples. The machine learning algorithm used in this study uses
MobileNetV2 with transfer learning techniques. MobileNetV2 introduces many new ideas aimed at reducing the
number of parameters to make it more efficient to run on mobile devices and achieve high classification accuracy.
Transfer learning is used so that data does not need training from the start, so it only takes several networks from
MobileNetV2 that have previously been trained and then retrained with a different purpose to improve accuracy results.
Then the models that have been created are inserted into the application using Android Studio. Software testing is done
through black box testing.
Purpose: The purpose of this research is to design a machine-learning model to detect fruit freshness and then apply it
to application Android smartphones.
Methods/Study design/approach: The algorithm used in this study uses MobileNetV2 with transfer learning
techniques. Models that have been created are inserted into the application using Android Studio.
Result/Findings: The training results using MobileNetV2 transfer learning obtained an accuracy of 99.62% and the
loss results obtained were 0.34%. The results of the application after testing using the black box testing method required
improvements to the application and the machine learning model so that it can run optimally.
Novelty/Originality/Value: Machine learning models that have been created using transfer learning MobileNetV2 are
applied to Android applications so that they can be used by the public.
This work is licensed under a Creative Commons Attribution 4.0 International License.
INTRODUCTION
Computer Vision is one of the newest technological developments today that utilizes computers to obtain
various information from an image or video. In contrast to image processing, which is primarily used to
process an image into another image, in Computer Vision, when a computer obtains information from an
image or video, the computer is also capable of processing and analyzing information so that the
information can be used for user needs [1]. One example of the application of Computer Vision is the
detection of fruit freshness.
Fruits are an important part of the human diet because they are the main source of dietary nutrients [2].
Apples, bananas, and oranges are one of several types of fruit that exist in Indonesia and are very popular
with the general public, both young to old people who like to consume these fruits. This shows that apples,
bananas, and oranges are widely consumed by the public and have competitiveness as well [3]. So it is very
necessary to detect fruit freshness which can be done by machine. Since the first appearance of machine
learning algorithms that have been implemented, various techniques have been proposed [4]. Classification
of fresh and rotten fruit is among other studies using a variety of different techniques, such as Regression
Tree [5], Convolutional Neural Network [6], and MobileNet [7].
1
*Corresponding author.
Email addresses: fajarmirfan@students.unnes.ac.id (Muttaqin)
DOI: 10.15294/rji.v2i1.70845
Android is an open-source operating system that runs on the Linux kernel. Android applications are
developed using the Java language [10]. Android is currently the most widely used smart mobile device
platform in the world, occupying 85% of the world market [11]. Machine learning and artificial intelligence
(AI) are starting to revolutionize mobile app development, especially on Android [12]. Machine learning
applications can now identify speech, photos, and gestures, and translate speech with high accuracy.
Machine learning is an important innovation in mobile applications that mobile developers must fully
understand because it changes the way people visualize and experience mobile applications [12]. The
programming language that will be used to create Android applications using Kotlin.
In May 2017, Google announced via official support for Kotlin on the Android platform. Kotlin is a
statically typed programming language that runs on the Java Virtual Machine (JVM) and fully interoperates
with Java. Immediately after the announcement, the language grew in popularity, appearing in rankings
such as the 2018 and 2019 Stack Overflow Annual Developer Surveys [13]. Kotlin doesn't impact
maintainability with respect to Java when working on two small applications. At the same time, Kotlin is
moving toward more concise code when asked to develop new features for ongoing software projects [14].
METHODS
As research conducted by [15], datasets are important in the field of deep learning. Therefore, before the
classification process, it is necessary to carry out data preprocessing to clean and balance the dataset. After
that, the classification process is carried out, and the model that is successfully created will be tested using
the evaluation method. The following are the steps in the method used.
Data Collection
At this stage, the author performs a dataset search for the research object. The datasets used in this study
are Fruits fresh and rotten for classification and Fresh and Stale Images of Fruits and Vegetables [16] [17].
This dataset is taken from Kaggle by downloading the dataset. The Fruits fresh and rotten dataset for
classification dataset has been divided into train data and test data, containing fresh and rotten fruit which
include apples, bananas, and oranges. The Fresh and Stale Images of Fruits and Vegetables dataset contains
images of six fruits and vegetables: apple, banana, bitter gourd, capsicum, orange, and tomato. The pictures
taken are simply apples, bananas, and oranges.
Data Preprocessing
At this stage, data processing is carried out from the results of the previous stage. The purpose of this stage
is to obtain the best model which will later be applied to Android applications. Several stages are arranged
sequentially, namely dividing the dataset into data train, data validation, and data test, creating data
augmentation, training, and evaluation.
Split Dataset
In order to be able to evaluate the model, it is necessary to divide the dataset [18]. The dataset is divided
into train data, validation data, and test data. The training data is used for learning (model-based), i.e.
adjusting the parameters to the machine learning model. Data validation is used to provide an unbiased
evaluation of the model fitted to the training data set when setting the model hyperparameters [19]. Test
data is unseen data or data that has not been seen and is not included in the process of making a classification
model [18]. In this study the method used to split the dataset using a random sampling technique. Random
sampling or random sampling is a data sampling method that protects the data modeling process from being
biased towards the possibility of different data characteristics [20]. However, random separation can cause
problems if there is an uneven distribution of data.
Training Model
The training process begins with creating a basic model using the transfer learning method. The model used
is MobileNetV2. The MobileNetV2 layer used for feature extraction is the last layer before the flatten
operation (bottleneck layer). MobileNetV2 is loaded using 3 input channels with ImageNet weights.
Include_top = False is used because the layer used for feature extraction is the last layer before the flatten
operation (bottleneck layer) [22]. All layers on MobileNetV2 will be frozen to prevent specific layer
weights from updating during training. Then the feature extraction process is carried out by compiling the
model, GlobalAveragePooling2D, and dense into sequential. The dense layer uses the Softmax activation
function which will classify images into many classes [22]. The Softmax activation function can be seen in
Equation 1.
𝐞𝐱𝐩 (𝒙 )
𝒔𝒐𝒇𝒕𝒎𝒂𝒙(𝒙) = ∑𝒏 𝒙𝒊 (1)
𝒋=𝟏 𝒋
Information
𝒙 = input vector
𝒙𝒊 = exponential function for the input vector
𝒙𝒋 = exponential function for the output vector
𝒏 = jumlah class
The model is compiled using the adam optimizer with a learning rate of 0.01, 0.001, or 0.0001. The model
training process uses an initial epoch value of 100 [22].
Evaluation
Evaluation is the process during model development to check whether the model is the best fit for a given
problem and the appropriate data. The training result model is tested using accuracy and loss evaluation
metrics. Accuracy is defined as the level of truth of a model to classify data correctly. Loss is the value of
not knowing the model recognizes the data [23].
𝑻𝑷+𝑻𝑵
𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚 = (4)
𝑻𝑷+𝑻𝑵+𝑭𝑷+𝑭𝑵
𝑭𝑷+𝑭𝑵
𝑳𝒐𝒔𝒔 = (5)
𝑻𝑷+𝑻𝑵+𝑭𝑷+𝑭𝑵
Information
TP = True Positive, the right data is classified by the model as a positive or true value.
TN = True Negative, the right data is classified by the model as a negative or wrong value.
FP = False Positive, the data is incorrect but classified as correct data.
FN = False Negative, data that is correct but classified as wrong as wrong data.
In the Android application scheme in Figure 1, the first user will open the application and go directly to the
welcome page, then go to the fruit selection page which contains a list of bananas, apples, and oranges. If
you click on a fruit, it will go to the fruit detail page which contains a fruit description page. To detect fruit
using a smartphone camera, you can directly from the fruit selection page. After performing the detection,
it will go directly to the detection results page to display the detection results using the smartphone camera.
Testing
After all the coding of the program has been completed, the next step is to carry out the testing phase. The
testing phase is carried out to ensure that the functions contained in the application are running according
to design [24]. Testing was carried out using the blackbox testing method.
Data Preprocessing
Split Dataset
The first stage of dividing the dataset or split dataset is defining the path of each training data, data
validation, and test data. Path creation is based on the class to be created, namely fresh orange, rotten
orange, fresh banana, rotten banana, fresh apple, and rotten apple. The division of the dataset is done with
a ratio of 90:10 for training data and validation data, and 90:10 for test data and validation data. To split
the dataset into train data, validation data, and test data using the Random Sample technique. The dataset
is divided based on several classes, namely fresh orange, rotten orange, fresh banana, rotten banana, fresh
apple and rotten apple. The following results of the split dataset can be seen in Table 1.
Data Augmentation
The results of the split dataset into train data, validation data, and test data will be further processed using
augmentation techniques using the Image Data Generator. An Image Data Generator will be created so that
the model receives new image variations at each epoch during the training process such as zooming,
rotating, and inverting the image. The author applies several transformation techniques to each data train,
data validation, and test data such as rescale, rotation, width shift, height shift, shear range, zoom range,
horizontal flip, vertical flip, and fill mode. The division of augmentation techniques for training data,
validation data, and test data is shown in Table 2.
Training Model
The data is trained using the Adam optimizer, the loss function uses categorical cross-entropy and accuracy
metrics. The author uses the Adam optimizer which has many advantages such as more efficient
computational processes, uses less memory, and is easy to implement. The loss function exists to calculate
the loss in the model so that the weights of the neurons can be updated so that in the next evaluation it is
hoped that the loss can be reduced. The categorical cross-entropy loss function is used for binary and
multiclass assignments. The accuracy metric is used to get model accuracy. The data is trained on 100
epochs. The training process stops at epoch 91 when the metrics are monitored to prevent overfitting. The
training and validation process lasted 14,105 seconds overnight. The best epoch is the 80th epoch with a
validation accuracy value of 99%, a loss value of 0,04, and an accuracy of 98%. The graphical training and
validation processes are shown in Figure 3.
Result
The following are the results of the model training process using MobileNetV2 transfer learning.
1. Accuracy : 98,91%
2. Loss : 0,47%
3. Validation Accuracy : 98,86%
4. Validation Loss : 0,55%
5. Accuracy Test : 99,62%
6. Loss Test : 0,34 %
System Planning
In implementing it into the Android system using the Kotlin programming language. After creating a project
in Android Studio, new settings are needed in the AndroidManifest.xml section by adding a new function,
namely uses-permission. The uses-permission function functions to add new permissions to the created
application. New permission added permission to use smartphone cameras, read data storage, and write
data storage. The new permission was added to detect fruit, upload images or fruit images to the application,
and download images that have been detected on the results page. The Gradle section requires a new
dependency so that Android Studio can read the tflite model. Next, add a new function to the build gradle
(module) file, namely the view binding function. View binding functionality is a feature that makes it easy
to write code that interacts with views. to display a fruit list using the recycler view, while transferring data
between other activities using parcelable.
Testing
Testing or testing of fruit freshness detection applications using the blackbox method. Blackbox testing
focuses on testing by looking at the functions in the application. In this application, testing is carried out on
the functions that are owned. Then compare with the expected results. If the results of the tests carried out
are the expected results, the application that has been made is by what has been previously determined. If
the results of the tests carried out are not following the expected results, it is necessary to check and improve
the application. The following are the results of tests carried out on the fruit freshness detection application
using the blackbox method, which can be seen in Table 3.
According to
1. Fresh Apple
expectations
According to
2. Fresh Banana
expectations
According to
4. Rotten Banana
expectations
According to
5. Fresh Banana
expectations
According to
6. Rotten Orange
expectations
According to
7. Fresh Apple
expectations
According to
8. Fresh Banana
expectations
According to
9. Rotten Apple
expectations
Discussion
The application of MobileNetV2 transfer learning begins with first creating a model from the first, namely
collecting datasets from Kaggle and then processing the data in the form of dividing the dataset by
separating train data, validation data, and test data. Furthermore, data augmentation is carried out to increase
the dataset using various transformation techniques. Finally, training was carried out using transfer learning
by taking certain networks from MobileNetV2 and then training again with a different purpose. After
training, the model is created in tflite format. Models that have been created using MobileNetV2 transfer
learning are implemented on Android by incorporating the tflite model into the Android Studio application
project. The project in an Android Studio is rearranged, especially in the Gradle section so that it can apply
models to Android Studio. The model entered must be the tflite model, because Android Studio only
recognizes the tflite format instead of h5. Coding on the uploaded tflite model is copied to one of the classes
in the project. After that, create a label that contains the classification result class which is made sequentially
based on the model made from the first, namely Rotten Apple, Fresh Apple, Rotten Banana, Fresh Banana,
Rotten Orange, and Fresh Orange. MobileNetV2 transfer learning can be applied in detecting fruit freshness
on the Android system so that the classification of fruit freshness images can be done properly.
Conclusion
The accuracy results obtained from the MobileNetV2 transfer learning algorithm for detecting fruit
freshness is 99.62% and the loss results obtained are 0.34% of the total dataset of 14,000 fruit images,
namely apples, bananas, and oranges. Based on the test results, it is expected to add a new dataset containing
real-time fruit images in Indonesia, especially in the fruit market area so that the results obtained are in
accordance with the fruits currently available in Indonesia. This Android application can be further
developed by adding other features such as fruit duration, detailed fruit description, or other features that
can help to choose fresh fruit.
REFERENCES
[1] A. O. P. Dewi, “Kecerdasan Buatan sebagai Konsep Baru pada Perpustakaan,” Anuva J. Kaji.
Budaya, Perpustakaan, dan Inf., vol. 4, no. 4, pp. 453–460, 2020, doi: 10.14710/anuva.4.4.453-
460.
[2] A. Ren et al., “Machine Learning Driven Approach towards the Quality Assessment of Fresh Fruits
Using Non-Invasive Sensing,” IEEE Sens. J., vol. 20, no. 4, pp. 2075–2083, 2020, doi:
10.1109/JSEN.2019.2949528.
[3] F. N. Cahya, R. Pebrianto, and T. A. M, “Klasifikasi Buah Segar dan Busuk Menggunakan
Ekstraksi Fitur Hu-Moment , Haralick dan Histogram,” IJCIT (Indonesian J. Comput. Inf. Technol.,
vol. 6, no. 1, pp. 57–62, 2021, doi: 10.31294/ijcit.v6i1.10052.
[4] N. Özkurt, T. Yıldırım, Yaşar Üniversitesi, Institute of Electrical and Electronics Engineers. Turkey
Section., and Institute of Electrical and Electronics Engineers, 2019 Innovations in Intelligent
Systems and Applications Conference (ASYU) : proceedings : 31 October-2 November 2019, Izmir,
Turkey.