An Experience in Collecting Requirements for Mobile, Energy Efficient
Applications from End Customers in the Bank Sector
Vladimir Ivanov1 , Pavel Kolychev2 , Sergey Masyagin1 , Giancarlo Succi1 , Rafael Valeev2
and Vasilii Zorin3
1 Innopolis
University, Russia
Bars Digital Technologies, Russia
3 Schaffhausen Institute of Technology, Switzerland
2 Ak
Keywords:
Empirical Methods, Software Experimentation.
Abstract:
Several development processes recommend strongly user participation and involvement in requirement acquisition. However, there are very few studies detailing the empirical results of direct user involvement in large
industrial software development products. In this paper we report the outcomes of a novel approach taken
by the Software House of one of major Russian banks (Ak Bars Bank) on how to improve the development
process by directly involving end customers in the requirement elicitation phase of mobile, energy efficient applications. We observe that such involvement in a form of a workshop has led to improvement of requirements
collection and higher levels of user satisfaction.
1 INTRODUCTION
One of the tenets of agile methods is to directly involve the users in the development of the product and,
in converse, one of the alleged difficulties in agile
methods is to involve customers in the development
of products.
User participation and involvement (UPI) is a
well-known and widely studied concept in software
engineering. Despite the presence of several studies
that claim the benefits of UPI, it is a challenge to understand quantitatively the profits arising from user
involvement into development. The specificity of the
domain, the software development process, and other
covariates may affect the results dramatically.
In the present paper we report results of application of an original tools and methods applied in mobile banking domain and discuss the lessons learned.
Moreover, we use a formal methods to measure the results and show the significance of the effects of UPI.
In this paper we discuss a novel approach taken
by the Software House of a Russian Bank, Ak Bars
Digital Technologies (ABDT), to increase its product
quality and the associated preliminary results. ABDT
holds at every sprint requirement focused elicitation
workshops with end users that have been recruited
using social media tools. The application domain is
mobile banking with specific attention for energy ef-
ficiency a primary concern for its users.
Ak Bars Digital Technologies employees about
300 people, organized in 40 agile teams managing
about 300 interconnected IT systems. The approach
is centered in an adaptation of Scrum to take into account the multiplicity of the desires of the end customers, which are in high number.
In the discussion we provide (links to) the concrete tools and methods used during the experiment.
Moreover, we supply also the screenshots of the ads
used to recruit end users and of the various alternative product design – they are in Russian and the most
essential features are translated in the caption, but we
hope in this way to supply the reader the most realistic
description of work, to facilitate is replica in other settings, and, if needed, its generalization in other contexts.
The results of the application of the approach have
been quite positive, since it has been evidenced that
such workshops appear to predict reliably the desires
of the end users, that results in products with higher
ratings in the app store, and, somehow, make the development teams more effective in managing and executing tasks.
The think that the value of this work lies on the
relevance of the subject by itself and of its application
domain, of the size of the experimentation and on the
quality of the analysis of the data, on the results that
522
Ivanov, V., Kolychev, P., Masyagin, S., Succi, G., Valeev, R. and Zorin, V.
An Experience in Collecting Requirements for Mobile, Energy Efficient Applications from End Customers in the Bank Sector.
DOI: 10.5220/0009574805220530
In Proceedings of the 15th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2020), pages 522-530
ISBN: 978-989-758-421-3
Copyright c 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
An Experience in Collecting Requirements for Mobile, Energy Efficient Applications from End Customers in the Bank Sector
have been achieved, and on the plenitudes of details
that have been provided that should facilitate replicas
and extensions.
This paper is organized as follows. Section 2 provides a background to the problem. Section 3 details
the overall experimental setting of the study. Section
4 describes the results that have been obtained and
Section 5 analyses them. Finally, Section 6 draws
some conclusions and outlines the direction of the future research.
2 BACKGROUND ON THE
PROBLEM
2.1
Collecting Requirements from Users
Development of information system usually involves
user to for improve system quality and to ensure successful system implementation. The importance of
user-centered design was recognized in 1980s with
works of E. Mumford (Mumford, 1983) where he
distinguishes direct and indirect user participation;
and D. Norman’s seminal book (Norman and Draper,
1986) in which he urged designers to take users’
needs and interests into account. It is obvious that
user involvement in system development and design
is one of the easiest ways to achieve product usefulness and usability. First empirical studies related
to the topic in software engineering appeared around
that years. In (Baroudi et al., 1986) a survey of 200
production managers was performed. Their results
demonstrate that user involvement in the development
of information systems may enhance both system usage and the user’s satisfaction with the system. Also,
the study has provided an evidence that high system usage is the result of user’s satisfaction with the
system. In (Damodaran, 1996), L. Darmodan has
explored the types of user involvement in the systems design processes as well as the roles of users
in such processes. She claimed that knowledge and
experience of end users are required at different steps
throughout the software development process. In later
works (Simonsen and Robertson, 2012; Kanstrup and
Bertelsen, 2011) the research focus has moved from
user-centered design to the concept of so-called user
driven innovation.
Nowadays, involvement of users in software engineering becomes a common practice intended to increase the usage of software systems. It is true, especially, in those applications domains where users
of software are main consumers of the corresponding services provided by business or government.
It is typical in health-care domain (Marcilly et al.,
2016; Kushniruk and Nøhr, 2016), where there is
a lack of user satisfaction with health-care information systems became a serious issue. In (Kushniruk
and Nøhr, 2016) discussed several methods to include
users in development and evaluation of software products. They describe challenges in IT evaluation during participatory using several case studies. Thus, in
modern software engineering, there is no doubt that
user should be involved in development processes,
however questions “How?” and “When?” are emerging and demand a scientific exploration. This drives
the need of additional case studies in specific domains
related to services such as heath-care, financial domain and e-government.
2.2
Requirements for Development of
Energy-efficient Mobile
Applications
In this section we review approaches to formulation
of requirements for energy-efficient mobile applications. Development of energy-efficient applications
needs measuring of energy consumption related to an
application. Therefore, a prerequisite for any energyefficient approach to development is establishing a
way to measure the overall power consumption of
a mobile device: using either some external instruments or by means of self-metering. The collected
data form parts of the input for training power consumption models and hence can be used for energyefficient development (Hoque et al., 2016).
The first category includes external instruments,
external power monitors and provides the ground
truth for mobile device power analysis due to their
high precision and accuracy. However, their usage is
limited by a laboratory; it is not feasible for largescale testing and development.
The second category, self-metering, assumes that
a device has its own capabilities to measure the
system-level power consumption without the help of
external instruments. This category of measurement
methods can be divided into three types: (i) Battery
models, voltage, and state of charge; (ii) State of
charge estimation (Hoque and Tarkoma, 2016); (iii)
Fuel gauge and battery APIs1 . From the point of view
of an energy profiling application, the self-metered
data is provided through the device’s battery API. The
API exposes to OS information about the battery status (e.g. the Android’s BatteryManager). So, the data
1 For
instance,
Andriod’s
Battery
Historian:
https://developer.android.com/
about/versions/android-5.0.html.
Android Power Profiles: https://source.android.com/ devices/tech/power.html
523
ENASE 2020 - 15th International Conference on Evaluation of Novel Approaches to Software Engineering
can be used during application development. The API
updates status periodically and whenever there is a
change in the charge and/or the temperature of a battery or not (typical frequency of the updates ranges
between 0.25Hz to 4Hz (Zhang et al., 2010)).
Hence, the measurement type and the data coming
via API about energy consumption require from mobile application developers to follow one of the following three major approaches: utilization-based energy requirements (Zhang et al., 2010), event-based
energy requirements (Pathak et al., 2011), and the
ones based on code analysis. The first and the second approaches to requirement specification needs a
real device to run an application. Thus, for our study
the third approach is more relevant. This approach
relies on the analysis of the program code to be executed. The approach is exemplified, for instance, as
an instruction-level model, which works by relating
the power consumption of a piece of code with each
instruction executed. This requires the evaluation of
power consumption for each of the instructions of the
software considered. Clearly, this can be applied at a
level of functions, procedures, or subroutines.
Selecting a proper energy profiler forms another
requirement for mobile development. The models and
approaches described above have been implemented
in the on-device profilers such as Sesame(Dong and
Zhong, 2011), PowerBooter(Zhang et al., 2010), VEdge(Xu et al., 2013) etc. The choice of a profiler
for a typical developer is not straightforward. Along
with their accuracy, other requirements are important,
including availability, hardware support, ease of use
and installation, and expertise of the developer.
Additionally, there are requirements related to the
privacy issues. As it is discussed in (Diamond et al.,
2018) collecting data about energy consumption (either of the device as a whole or in an application)
can clearly cause privacy concerns. Thus, a developer of the mobile energy-efficient application should
consider such requirements as transparency, appropriation and security in the usage of the energy consumption data.
3 DESCRIPTION OF THE
EXPERIMENTAL SETTING
3.1
Situation That Induced the Change
During several years traditional Banking sector in
Russia has been suffering from aggressive competition with FinTech companies providing gamechanging easy-to-use financial products to customers.
524
Ak Bars Digital Technologies (ABDT) was founded
in October 2016 in response to this process.
First product teams were created as soon as the
company was established. The development process
in the teams was based on Scrum framework described in Scrum Guide2 . All teams in the company
incorporates all mandatory events and artifacts prescribed by Scrum.
There were several reasons to use Scrum as a process framework. Firstly, developing digital product is
a complex task due to high level of uncertainty and
a lot of dependencies between systems and components (20 main systems and 2 mission-critical developed in parallel by different teams inside and outside the organization). Secondly, there are no stable
requirements: they constantly change in response to
new requests from customers, implementation of new
features by competitors and claims of government institutions. Finally, the product should be developed in
a competitive environment, thus fast delivery of new
versions of a product and adaptability are extremely
important. Taking into account these issues choosing
Scrum is looking quite reasonable: Scrum is based on
the empirical process, which allows to identify and
tackle emerging problems in a rapid pace. It creates
framework, inside which teams could establish their
own process that satisfies their needs.
In this article we analyze the process in one product group that develops Ak Bars Online system which
provides banking and financial services for customers
among various digital platforms. The whole product
group consists of 8 cross-functional agile teams and
Product management team. Two of these agile teams
includes vendor developers who works by out-staffing
agreement. Agile teams use Scrum framework and include from 5 to 10 developers. By “developers” we
mean all people who analyze, design, implement and
test the system. In total the product team includes 65
people, among them: 11 back-end developers, 5 iOS
developers, 5 Android developers, 3 web developers,
12 QA-engineers, 4 user interface designers, 2 system analysts, 2 site reliability engineers, a system architect, a data scientist, an UX expert, an application
security expert, 5 outstaffing specialists and product
management. The product management includes a financial expert, a web analyst, marketing and content
experts. Also we have 5 Product Owners and three
Scrum Masters. Due to lack of Some Scrum Masters
and Product Owners work with several agile teams
due to lack of the competences on the market.
The product group develops Android, iOS, web
applications and back-end using SOA (service2 https://www.scrumguides.org/docs/scrumguide/v2017/
2017-Scrum-Guide-US.pdf
An Experience in Collecting Requirements for Mobile, Energy Efficient Applications from End Customers in the Bank Sector
oriented architecture), adapters and interfaces to communicate with other bank’s middleware and adjacent
systems. All front-end applications are deployed on
production environment and available for end-users.
Android and iOS applications are published at PlayMarket and AppleStore accordingly. For iOS development the team used Objective C and Swift. To develop Android application the team used Java, Kotlin,
RxJava2. Ak Bars Online platform is developed by
using C#, .Net Framework, and .Net Core. The team
also includes 4 outstaffing developers who makes necessary implementation on CRM that based on Oracle Siebel by using eScript programming language.
One developer implements procedures and scripts in
processing system. Autotests are developed by using
Python. While the detailed breakdown is confidential,
we can say that overall the codebase for the project
spanned several hundreds of thousands of lines of
code.
As it has been stated before, all teams use Scrum
framework as it is described in Scrum Guide. The
core idea of the framework is to constantly inspect and
adapt. It incorporates several feedback loops, and the
most important one is a review of product increment
at the end of every iteration. Sprint review is a multicomponent event, where development team, product
owner and stakeholders observe implemented product increment, the market, product aims and discuss
which features should be done next.
For the first year of working, teams invited to
the event in-house stakeholders and did not take into
consideration direct feedback from users. However,
at some time, during one of Sprint Retrospectives,
teams understood that except in-house stakeholders it
is necessary to ask other type of stakeholders, endcustomers. Yes, product owner could represent users
being a voice of customers, but this usually leads to a
longer distance between product teams and real customers and raises several issues including:
• people describe the world based on their own persuasions and views, thus this business analysts
or product owner always (unconsciously) transform real requirements into somehow other requirements,
• developers often do not see how end-customers
work with the products and do not understand
their real pains.
However, the competition is very stiff and the key
stakeholders have been putting always higher goals
for the ABDT, so a radical change has been implemented to attempt to target such issues.
Figure 1: Customers trying the products and team members
observing without speaking.
3.2
Implemented Change
Substantially, the change was to involve directly end
customers in requirement elicitation; if not enough
customers were available, direct proxies, were used:
the business employees of the bank that operate on a
daily basis with customers using such products.
Specifically, volunteers users were recruited via
the social network channels of the bank:
• on Vkontakte at https://vk.com/akbars ru
• on Facebook:
akbarsbank/
https://www.facebook.com/
The product owner also tried to define properly the
target group of customers invited at each session attempting to balance among ages, gender, background.
1. From 1 to 10 estimate, how easily was for
you to completed the task. Comment on what
was the most confusing part.
2. From 1 to 10 estimate how much time it took
you to complete the task. In particular, comment on which actions took the most time
3. Which functionality did you miss in the app?
4. Do you understand all instructions and comments? If not, please comment the ones that
are not clear for you
5. Are all the developed functionality really
needed? What could be removed?
Figure 2: Structured questionnaire administered to the customers during the second phase of the workshop.
Each interaction with the customers was organized in the format of a workshop lasting approximately one hour and it occurred at every sprint, that
525
ENASE 2020 - 15th International Conference on Evaluation of Novel Approaches to Software Engineering
is, every two weeks. Such workshop was organized in
two phases:
• in the first phase, the customers were presented
the applications and asked to perform some use
cases with them. The members of the development team did not speak, comment, or communicate in any way with the customers: they just observed the interaction process, as shown in Figure
1;
• in the second phase, the team members sat with
the customers and asked them questions about
their experience with the applications; after the
first informal and unstructured comments from
the customers, a very simple structured questionnaire was employed (Figure 2). All the discussion
was recorded in written format.
The simple structured questionnaire was developed by the product group and structured as follows.
Please note that while this questionnaire is quite primitive and could be improved according to the best
practices in the field, the teams have always resisted
any such suggestion and kept it as is because they are
accustomed to it, know it by heart, so it is very simple
for them to administer it. Anyway, it is not the purpose of this study to determine how to do better the
questionnaire, but to describe in details the approach
and its effects.
All the comments coming from the customers
were then elaborated by the teams and user stories
were created based on such feedback. Such user stories were clustered in three classes:
• “quick win:” simple modifications or bug fixes
that required an effort that could be condensed in
a sprint;
• “major rework:” major required changes in the
apps spanning multiple iterations;
• “to ignore:” comments not required at the current
stage.
The comments were then passed to the product owner, with the understanding that quick wins
should be executed as early as possible, major reworks should be planned within a timeframe of 6
months, and items to ignore were simply removed,
rather than kept in the backlog, with the idea that
they would have never be targeted anyway, and if
the represented some important feature that got missclassified, they would have emerged again in a following customer workshop.
4 EMPIRICAL RESULTS
Overall, we are considering 21 “frequent” releases of
the software to the customer, which spanned from the
beginning of March 2018 to the end of March 2019.
In total there were 22 releases, 8 with the “traditional”
approach of AkBars (named T1 . . . T8 in the following) and 14 following the new approach (named N1
. . . N14 in the following).
4.1
Evaluation of the Effects of the
Changes
To evaluate the results of the new approach we have
selected the following criteria:
• Ability of user sessions to predict preferences of
end users
• Customer satisfaction measured in ratings in the
appstore
• Ability to predict the amount of work to complete
The first criteria relates to the practical usefulness of
the data collected by customers selected as per the
process, and on how representative such customers
are with respect to the overall populations of customers of the bank. The second criteria is a quite obvious choice, since it measures the satisfaction of the
customers as a whole. The third criteria that we used
were the ability of the team to predict the amount of
work to complete, measured as a percentage between
the planned and the accomplished tasks, and it is connected to the cohesiveness of the work done, which
should reflect in more effective work.
Needless to say, the reader may imagine several
other metrics and criteria that could have been collected during the development process or on the final
product and then used in our analysis. Being this a
study on a real company with real datasets, we are
indeed constrained with the information that is concretely available to us. Nonetheless, we think that the
size of the experiment in terms of both time duration
and people involved, the metrics collected, and the
importance for Ak Bars of the product being developed make this study a very valuable, original, and
unique contribution to the research in this area.
4.2
Prediction of Preferences of the End
Users
At the beginning of the application of the methodology the company has performed a preliminary assessment on the effectiveness of the sessions with customers to predict the preferences of the end users. To
526
An Experience in Collecting Requirements for Mobile, Energy Efficient Applications from End Customers in the Bank Sector
this end a formal experiment was run using a post-test
only control group design with randomisation (Campbell and Stanley, 1963).
To this end, two experiments were conducted. In
the first, a problematic feature was identified in the
app related to the discussion with customers and a
variation of such feature was discussed during the sessions with customers. Specifically, the desire of the
bank was to have users interact one another via the
app. Originally such feature was called “friends” and
after the discussion was called “chat”.
After the discussion, two concurrent versions of
the app were deployed, one with the old label (about
50.4% of the all deployments) and one with the new
(about 49.6% of the all deployments) – the concrete
numbers are of the order of the tens of thousands,
but are omitted for confidentiality. About 3.8% of the
users of the app with the old label used the feature vs.
about 4.5% of the users of the app with the new label
(an increase of about 18.6%). To determine the significance of the variation we have used the chi-squared
test, which provides an excellent approximation of
the real value, given the large sample size, since the
size of such sample makes impossible to adoption
of the Fisher’s exact test (Fisher, 1922; Wasserman,
2006). The result identifies a statistically significant
improvement with α = 0.05 (p is 5.8 × 10−5 , giving
a very high confidence in the robustness of the improvement against possible statistical fluctuations).
The second experiment aimed at determining the
best way to promote new products of the bank to the
users of the app. The original version was not resulting in users exploring such opportunities, a proposal
was made in the focus group.
In this case, three concurrent versions of the app
were deployed, one with the old label (about 33.2% of
the all deployments), one with the large button (about
33.5% of the all deployments), one with the opportunity added to the list of features (about 33.3% of the
all deployments) – the concrete numbers are of the
order of the hundred of thousands, but are omitted for
confidentiality. About 16.6% of the users of the original app explored the new products of the bank vs.
about 27.1% of the users of the first alternative design
(an increase of 63.1%) and 24.1% of the users with
the alternative design (an increase of 44.6%).
Also in this case, to determine the significance of
the variation we have used the chi-squared test (again
the size of such sample has not allowed the use of
the Fisher’s exact test). The result identifies a statistically significant improvement with α = 0.05 (p is
5.9 × 10−94 , giving a very high confidence in the robustness of the improvement against possible statistical fluctuations).
The two experiments have been conducted using
a quite standard experimental design of treatment, using a post-test only control group design with randomisation (Campbell and Stanley, 1963). In both
cases we have achieve a statistically significant effect,
which, in the second case, is also quite remarkable in
terms of size. Altogether, It appears that the new approach appears quite effective in predicting the preferences of the end users. Indeed, in all these situations there could be several confounding factors, even
if in this case we follow a such a solid experimental design and we obtain very comfortable statistical
evaluations.
4.3
Ratings of Users in the Appstore
As mentioned, we consider the ratings in the Appstore
one of the criteria that we use to evaluate the results of
the introduction of the new approach to elicit requirements. We concentrate our attention to the Google
Play Store, because we did not have easy access to
the ratings in the Apple App Store.
Figure 3: Overall results of the application of the new approach.
4.3.1 Overall Analysis
The first analysis we have made concerns whether
the introduction of the new approach has resulted in
higher ratings. To gather an overall intuitive understanding of the situation we have constructed cumulative beanplots of the ratings before and after such
introduction (Kampstra, 2008) (Figure 3). The thin
(black) bars represent the discrete values of the stars,
the thicker (blue) bar the mean, and the (red) plus sign
the median. It is clear that the introduction of the
new approach has significantly improved the ratings
by the end users. The mean rating has moved from
3,29 to 4.60 – statistically significant at α = 0.05 as
evaluated using the non parametric Mann-Whitney U
test (Mann and Whitney, 1947). The median has been
bumped from 4 to 5 – statistically significant at α =
527
ENASE 2020 - 15th International Conference on Evaluation of Novel Approaches to Software Engineering
0.05 as evaluated using the non parametric KruskalWallis test (Kruskal and Wallis, 1952). We have
then analysed the data with the Kolmogorov-Smirnov
statistics (Kolmogorov, 1933), which assumes a value
of 0.37 (rounded to the second decimal), statistically
significant at the 0.05 level (actually, with a p value
less than 1070 ). It is important to mention that the
Mann-Whitney U test, the Kruskal-Wallis test, and
Kolmogorov-Smirnov statistics are a very conservative non parametric tests, which make no assumption
on the data, simply considering them on an ordinal
scale (Wasserman, 2006; Wasserman, 2010).
variation of the means or the medians of the customer
evaluations (Figure 4); to gather an understanding of
the trend, we have run a non parametric correlation on
the values of the means (the Spearman’s rank correlation), obtaining a positive correlation (0.60) that is
not statistically significant at the 0.05 level (p=0.12),
meaning that such positive variation may just come
from some random fluctuation.
4.3.3 Discussion of the Results of the Customer
Evaluations
Considering the data that we have analyse we can hypothesize that the introduction of the new approach to
collect requirements has resulted in:
• a statistically significantly higher level of customer ratings overall after the introduction of the
new approach than with the traditional approach
• a statistically significantly improvement pattern in
customer ratings after the introduction of the new
approach, while before the situation was stagnant.
Figure 4: Situation before the application of the new approach.
Figure 5: Evolution of the improvement after the application of the new approach.
4.3.2 Detailed Analysis of the Evolution of the
Versions
To gather a further understanding of the introduction
of the new approach in terms of customer complaints
we have first analysed version by version the situation
before and after the introduction of the new approach
in terms of analysis of beanplots (Figure 4 represents
the versions before the introduction of the new approach and Figure 5 after).
It is evident that before the introduction of the new
approach the situation was quite stable, with no real
528
We can therefore hypothesize that the new approach has improved the quality of the requirement
collection process as measured by customer ratings
and that such approach is still in its learning phase,
so we could expect further improvements. We would
like to mention that this also includes fully the energy concerns. However, we need to mention that the
sensitivity of this metrics to further improvement is
limited, since we are approaching the end of the scale
in the customer ratings, so further increases could remain below the level of statistical significance.
We would like to emphasize again that we have
obtained all these results using very robust non parametric techniques, which do not make any assumption on the structure of data - the Mann-Whitney U
test (Mann and Whitney, 1947), the Kruskal-Wallis
test (Kruskal and Wallis, 1952), the KolmogorovSmirnov test (Kolmogorov, 1933) and the Spearman’s
rank correlation coefficient (Myers et al., 2010). This
makes us confident that the effects that we have notices are not simple random perturbation of data due
to noise.
4.4
Percentage of Tasks Completed in
Each Iteration
We have then turned our attention to the tasks that
have been completed in each iteration as per plan by
each team. We have considered 5 out of the 8 teams
working on the project and a total of 23 iterations, 10
performed with the traditional approach and 13 with
the new approach. 3 teams could not be inserted in
An Experience in Collecting Requirements for Mobile, Energy Efficient Applications from End Customers in the Bank Sector
higher customer ratings, evidencing that also the
energy concerns have been resolved
• the overall process appears somehow better managed, even if such improvement is not easily noticeable.
We think that such findings are generalizable to
larger settings, since:
• the size of the study is quite large and represents a
problem that is quite typical in the banking sector,
Figure 6: Completion rate of tasks during sprints using the
traditional approach.
• the structure of the development process and of
the teams is quite typical for the Russian industry
and also for the banking industry worldwide,
• the number of evaluations that we have collected
for customer ratings is large, the effect size is remarkable, and the p-value is very low.
As already mentioned, additional studies are
needed to confirm and extend these results. To this
end, a plenitude of information has been provided to
facilitate such followups.
6 CONCLUSIONS
Figure 7: Completion rate of tasks during sprints using the
new approach.
this analysis, since their data was considered confidential at all levels.
While the specific number of tasks is obviously
confidential, the beanplots of percentage of completion are presented in Figure 6 (before the introduction
of the new approach) and in Figure 7 (after).
We have finally analysed the results as a whole
taking a meta analytical approach (Cook, 1994) and
performing a paired value sign test on the direction
of the variation of the means. In this case we have
obtained a result that is statistically significant, as we
could expect given that in 5 cases out of 5 the mean
percentage of accomplished tasks have improved.
5 DISCUSSION
In this work we have evidenced three major outcomes
of the proposed approach to involve customers in mobile, energy efficient applications using directly end
customer workshops:
• such workshops predict fairly well the desires of
the overall population of end users, an evidence
that thus it also targets the energy concerns
• the resulting app appears to have significantly
In this paper we evaluate a novel approach taken by
the IT department of a Russian Bank, Ak Bars Digital Technologies. The approach is centered on techniques and tools for taking into account effectively
the multiplicity of the desires of the end customers.
The results have been a higher predictability of the
desires of the customers, significantly higher ratings
of the app in the appstore, and somehow better managed product.
The contribution of the work is concrete results of
application of an original tools and methods applied
in mobile development for banking domain.
Indeed, replicas of this study are advocated, and,
to this end, the work has been discussed quite in details. Moreover, future works will be connected to application of the proposed approach in other software
products for the bank sector.
ACKNOWLEDGMENTS
The work presented in this paper was supported by the
grant of Russian Science Foundation # 19-19-00623.
REFERENCES
Baroudi, J. J., Olson, M. H., and Ives, B. (1986). An empirical study of the impact of user involvement on system
529
ENASE 2020 - 15th International Conference on Evaluation of Novel Approaches to Software Engineering
usage and information satisfaction. Commun. ACM,
29(3):232–238.
Campbell, D. T. and Stanley, J. C. (1963). Experimental and
quasi-experimental designs for research. Houghton,
Mifflin and Company.
Cook, T. (1994). Meta-analysis for explanation : a casebook. Russell Sage Foundation, New York.
Damodaran, L. (1996). User involvement in the systems
design process-a practical guide for users. Behaviour
& information technology, 15(6):363–377.
Diamond, L., Schrammel, J., Fröhlich, P., Regal, G., and
Tscheligi, M. (2018). Privacy in the smart grid: enduser concerns and requirements. In Proceedings of the
20th International Conference on Human-Computer
Interaction with Mobile Devices and Services Adjunct,
pages 189–196. ACM.
Dong, M. and Zhong, L. (2011). Self-constructive high-rate
system energy modeling for battery-powered mobile
systems. In Proceedings of the 9th international conference on Mobile systems, applications, and services,
pages 335–348. ACM.
Fisher, R. A. (1922). On the interpretation of χ 2 from contingency tables, and the calculation of p. Journal of
the Royal Statistical Society, 85(1):87.
Hoque, M. A., Siekkinen, M., Khan, K. N., Xiao, Y., and
Tarkoma, S. (2016). Modeling, profiling, and debugging the energy consumption of mobile devices. ACM
Computing Surveys (CSUR), 48(3):39.
Hoque, M. A. and Tarkoma, S. (2016). Sudden drop in
the battery level?: understanding smartphone state of
charge anomaly. ACM SIGOPS Operating Systems
Review, 49(2):70–74.
Kampstra, P. (2008). Beanplot: A boxplot alternative for visual comparison of distributions. Journal of Statistical
Software, Code Snippets, 28(1):1–9.
Kanstrup, A. M. and Bertelsen, P. (2011). User innovation
management: A handbook. Aalborg Universitetsforlag.
Kolmogorov, A. N. (1933). Sulla determinazione empirica
di una legge di distribuzione. Giornale dell’Istituto
Italiano degli Attuari, 4:83–91.
Kruskal, W. H. and Wallis, W. A. (1952). Use of ranks in
one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583–621.
Kushniruk, A. and Nøhr, C. (2016). Participatory design,
user involvement and health it evaluation. Stud Health
Technol Inform, 222:139–151.
Mann, H. B. and Whitney, D. R. (1947). On a test of
whether one of two random variables is stochastically
larger than the other. The Annals of Mathematical
Statistics, 18(1):50–60.
Marcilly, R., Peute, L., and Beuscart-Zephir, M.-C. (2016).
From usability engineering to evidence-based usability in health it. Stud Health Technol Inform, 222:126–
38.
Mumford, E. (1983). Designing human systems for new
technology: the ETHICS method. Manchester Business School.
530
Myers, J., Well, A., and Lorch, R. (2010). Research Design
and Statistical Analysis. Research Design and Statistical Analysis. Routledge.
Norman, D. A. and Draper, S. W. (1986). User centered
system design: New perspectives on human-computer
interaction. CRC Press.
Pathak, A., Hu, Y. C., and Zhang, M. (2011). Bootstrapping energy debugging on smartphones: a first look
at energy bugs in mobile devices. In Proceedings of
the 10th ACM Workshop on Hot Topics in Networks,
page 5. ACM.
Simonsen, J. and Robertson, T. (2012). Routledge international handbook of participatory design. Routledge.
Wasserman, L. (2006). All of Nonparametric Statistics
(Springer Texts in Statistics). Springer-Verlag, Berlin,
Heidelberg.
Wasserman, L. (2010). All of Statistics: A Concise Course
in Statistical Inference. Springer Publishing Company, Incorporated.
Xu, F., Liu, Y., Li, Q., and Zhang, Y. (2013). V-edge:
Fast self-constructive power modeling of smartphones
based on battery voltage dynamics. In Presented
as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI}
13), pages 43–55.
Zhang, L., Tiwana, B., Qian, Z., Wang, Z., Dick, R. P., Mao,
Z. M., and Yang, L. (2010). Accurate online power estimation and automatic battery behavior based power
model generation for smartphones. In Proceedings of
the eighth IEEE/ACM/IFIP international conference
on Hardware/software codesign and system synthesis,
pages 105–114. ACM.