Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living

UNIVERSIDADE DA BEIRA INTERIOR Engenharia Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Ivan Miguel Serrano Pires Tese para obtenção do Grau de Doutor em Engenharia Informática (3º ciclo de estudos) Orientador: Prof. Dr. Nuno M. Garcia (Universidade da Beira Interior, Portugal) Co-orientador: Prof. Dr. Francisco Flórez-Revuelta (Universidad de Alicante, Spain) Covilhã, Novembro de 2018 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living ii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Thesis prepared at the Ambient Assisted Living Computing and Telecommunications Laboratory (ALLab), Instituto de Telecomunicações, Universidade da Beira Interior, and submitted to the Universidade da Beira Interior for defence in a public examination session. Research partially financed by the Portuguese Fundação para a Ciência e a Tecnologia through the project identified by UID/EEA/50008/2013 and partially supported by the COST Action IC1303 – AAPELE – Architectures, Algorithms and Protocols for Enhanced Living Environments. Work was also partially supported by Altran Portugal at different stages of the research. iii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living iv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Dedicatory “Do the difficult things while they are easy and do the great things while they are small. A journey of a thousand miles must begin with a single step.” Lao Tzu To my parents, José e Helena, who often gave themselves and gave up their dreams, so that I could accomplish mine. I want to say that this achievement is not only mine, but ours. All I got was only possible thanks to the love, support and dedication you have always had for me. I have always been taught to act with respect, simplicity, dignity, honesty and love for others. With the union of all, the obstacles were overcome, the victories were achieved and the happiness was shared. Thank you for your patience and understanding during this long journey. To the rest of my family, supervisors, professors and friends for their continuous friendship, affection, companionship, unconditional support and encouragement. Thanks you for cheering my achievements along this journey. “The ultimate measure of a man is not where he stands in moments of comfort and convenience, but where he stands at times of challenges and controversy” Martin Luther King v Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living vi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Acknowledgements This thesis would not have been possible without the help of many people. Firstly, I would like to thank to my parents for having been by my side in the difficult moments and helped me with their knowledge, supporting me in all moments and encouraging me to finish this PhD Thesis. I also would like to thank them for their unconditional love, because I came from a humble family and I have this achievement. During all life, my parents had several dreams that cannot realize, but one of their dreams was now achieved with my success. Secondly, I would also like to thank my family for all support in the good and bad moments during the last four years. They are very important to this achievement. Friends are the support of our life, I would also like to thank them, specially those who have always believed in my success. In our life, sometimes we find people that can help us with their good viewpoints and friendship. However, sometimes we find false friends, that are the rocks in our travel, but they always teach something to us. I also want to thank my supervisor, Prof. Dr. Nuno M. Garcia, which is the person responsible for my success. He had a lot of patience to give me the help, guidance, trust and motivation needed, and to teach some important things for the future, but the adventure always continue and we will together in a lot of things in the future. When I started the PhD research, I didn’t know more than Portugal, and my English was not good. He found the best solutions and encouraged me to go to different countries in order to do some work with people with different cultures. Related to this PhD research, I also need to thank my co-supervisor, Prof. Dr. Francisco FlórezRevuelta, for the availability and help in my travel to Kingston University, Kingston upon Thames, United Kingdom, and for their support in the different activities of my PhD research. During this PhD research, as I did some parts of the work at Universidad de Alicante, Alicante, Spain, I need thank to all people for their support and availability in my visit, where it was very useful for the success of this PhD research. Other people are important to support my knowledge in different cultures, including Prof. Dr. Susanna Spinsante, who helped me in my visit to the Marche Polytechnic University, Ancona, Italy, with the support for my PhD research, new ideas for future research in cooperation with her, and other support in the scientific publications of the last years. vii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living During my Bachelor studies in Computer Science and Engineering at the Polytechnic Institute of Castelo Branco, I was a student of Prof. Dr. Cristina Canavarro, who was very important in the success of the statistical analysis of the results obtained with the framework for the recognition of Activities of Daily Living (ADL) and environments developed during this PhD research. I would also like to thank other professors, especially Nuno Pombo, Eftim Zdravevski, Rossitza Goleva and Natalia Diaz Rodriguez, for their help and support in the last activities and publications of my PhD research. For the financial support of all activities, I also would like to thank the Instituto de Telecomunicações, that through the Portuguese Fundação para a Ciência e a Tecnologia through the project identified by UID/EEA/50008/2013 supported several activities of this PhD research. I would also like to acknowledge the contribution of the COST Action IC1303 – AAPELE – Architectures, Algorithms and Protocols for Enhanced Living Environments, which helped to know some people and supported financially some activities of my PhD research. Other important thing during my PhD research was my work for Altran Portugal, that especially with some people I have found the important trust for this achievement, including Pedro Furtado, João Minhota, André Freitas and remaining colleagues and managers. The Ambient Assisted Living Computing and Telecommunications Laboratory (ALLab), at Universidade da Beira Interior, Covilhã, Portugal, was very important for the development and success of this PhD research with the support of several colleagues, including Virginie Felizardo, Henriques Zacarias, Rui Santos, Dmytro Vasyanovych, among others. Thank you for all friendship and help during my PhD research. In my travels to Kingston University, Kingston upon Thames, United Kingdom, and Marche Polytechnic University, Ancona, Italy, I also need to thank the people who give me good experiences and knowledge in the research about Ambient Assisted Living (AAL) systems. One of the difficulties of my PhD research were focused in the finding of people that agreed to perform the experiments needed for the acquisition of data from the sensors available in a commodity smartphone. Thus, I would also to thank the people that agreed to help with their efforts and availability. In addition, I would also like thank to the people that works in physiotherapy that helped to know the physical signals acquired from the different sensors available in the mobile devices. During my PhD research, I had some health problems and I would also thank the healthcare professionals of Centro Hospitalar Universitário de Coimbra, Coimbra, Portugal, who did all efforts to improve my health status. viii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living During our path, at the end but not the less important, we are always looking for opportunities, but the best things magically appear out of nowhere, as new projects, new findings and people that change our manner to view the life, including Tânia Valente, Diogo Marques, António Santos, and other people that we can cooperate in the future, and I need to thank them for all availability, encouragement and support at the different stages of my PhD research. ix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living x Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Foreword This thesis describes the research work performed in the scope of the 4-year doctoral research programme and presents its conclusions and contributions. The research activities were mainly carried out at the Ambient Assisted Living Computing and Telecommunications Laboratory (ALLab), Universidade da Beira Interior and also Instituto de Telecomunicações, Covilhã, Portugal. Additionally, some research activities during the 4-year doctoral research programme were performed in other universities in different European countries, including Kingston University, Kingston upon Thames, United Kingdom, Universidad de Alicante, Alicante, Spain, and Marche Polytechnic University, Ancona, Italy. The research work was supervised by Dr. Nuno M. Garcia, from the Universidade da Beira Interior, and co-supervised by Dr. Francisco Flórez-Revuelta, from Universidad de Alicante, Spain. Dr. Francisco Flórez-Revuelta was at Kingston University at the time of the research activities and a supervised the research work carried out there. During the time of the research activities in the Marche Polytechnic University, the author was under supervision of Dr. Susanna Spinsante. The work performed out of Portugal was mainly performed at the Universidad de Alicante with supervision of Dr. Francisco Flórez-Revuelta, which contributed to the success of this PhD research. This work was financially supported in part by the Fundação para a Ciência e a Tecnologia, Portugal, through the project identified by UID/EEA/50008/2013 and some activities were supported by Short Term Scientific Meeting funds, granted by the COST Action IC1303 – AAPELE – Architectures, Algorithms and Protocols for Enhanced Living Environments, funded by The COST Association, through the European Science Foundation and the H2020 program. The research resulted in the definition and development of a framework for the recognition of Activities of Daily Living (ADL) and its environments based on the data acquired from the sensors available in off-the-shelf mobile devices. The architecture of the framework relies on several concepts, including data acquisition, data processing, data cleaning, data imputation, feature extraction, data fusion and classification with machine learning methods. At Kingston University, the review of the state of the art was carried out, resulting in a review article. The research about the state of the art was completed with other three articles mainly related to the validation techniques and classification of the data acquired from sensors, and a review about audio fingerprinting techniques. These literature reviews were fundamental in the definition of the architecture of the framework for the recognition of ADL and environments. Therefore, during the development of the framework, it was verified that the step related to the data imputation may be done in the future in order to increase the reliability of the xi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living framework, but this research is out of the scope of this PhD thesis. The main development of the framework is related to the data fusion and classification of the data acquired from offthe-shelf mobile devices in free living activities. The research work developed during the doctoral programme and described in this Thesis is the consequence of the activities performed in the different institutions visited, which allowed the author to benefit from the different cultures and experiences of the researchers at these different academies. xii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living List of Publications List of articles included in the thesis resulting from this 4-year doctoral research programme 1. From Data Acquisition to Data Fusion a Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living using Mobile Devices Ivan Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, published in Sensors, MDPI, Sensors 2016, 16(2), 184; doi:10.3390/s16020184, January 2016 (IF 2017: 2.677, Q1 Electrical and Electronic Engineering); 38 citations. 2. Validation Techniques for Sensor Data in Mobile Health Applications Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta and Natalia Rodríguez, published in the special issue Integration of Sensors in Control and Automation Systems, Journal of Sensors, Hindawi publishers, September 2016 (IF: 2017: 2.057, Q2 Engineering, Electrical & Electronic); 3 citations. 3. Recognition of Activities of Daily Living Based on Environmental Analyses Using Audio Fingerprinting Techniques: A Systematic Review Ivan Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante, Rossitza Goleva, Eftim Zdravevski, published in Sensors, MDPI, Sensors 2018, 18(1), 160; doi: 10.3390/s18010160, January 2018 (IF 2017: 2.677, Q1 Electrical and Electronic Engineering); 0 citations. 4. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices Ivan Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta and Susanna Spinsante, published in Sensors, MDPI, Sensors 2018, 18(2), 640; doi: 10.3390/s18020640, February 2018 (IF 2017: 2.677, Q1 Electrical and Electronic Engineering); 1 citations. 5. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante and Maria Canavarro Teixeira, published in Pervasive and Mobile Computing, Elsevier, 47, pp. 78-93; doi: 10.1016/j.pmcj.2018.05.005, July 2018 (IF 2017: 2.349, Q1 Computer Science (miscellaneous)); 0 citations. xiii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 6. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions Ivan Miguel Pires, Maria Canavarro Teixeira, Nuno Pombo, Nuno M. Garcia, Francisco Flórez-Revuelta, Susanna Spinsante, Rossitza Goleva and Eftim Zdravevski, published in Open Bioinformatics Journal, Bentham Science Publishers B.V., 11, pp. 61-88; doi: 10.2174/1875036201811010061, May 2018 (CiteScore 2016: 4.86, D1 Computer Science (miscellaneous)); 0 citations. Other publications resulting from the doctoral research programme not included in the thesis 1. Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices Ivan Miguel Pires, Nuno M. Garcia and Francisco Flórez-Revuelta, published in ECMLPKDD 2015 Doctoral Consortium, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, 7-11 September 2015; 2 citations. 2. Identification of Activities of Daily Living using Sensors Available in Off-the-shelf Mobile Devices: Research and Hypothesis Ivan Pires, Nuno M. Garcia, Nuno Pombo and Francisco Flórez-Revuelta, published in 7th International Conference on Ambient Intelligence, Seville, Spain, 1-3 June 2016; 2 citations. 3. Limitations of energy expenditure calculation based on a mobile phone accelerometer Ivan Pires, Virginie Felizardo, Nuno Pombo, Nuno M. Garcia, published in the International Conference on High Performance Computing & Simulation (HPCS 2017), Genova, Italy, 17-21 July 2017; 0 citations. 4. Limitations of the Use of Mobile Devices and Smart Environments for the Monitoring of Ageing People Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo and Francisco Flórez-Revuelta, published in ICT4AWE 2018 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health, Madeira, Portugal, 22-23 March 2018; 0 citations. xiv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 5. User Environment Detection with Acoustic Sensors Embedded on Mobile Devices for the Recognition of Activities of Daily Living Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante, Maria Canavarro Teixeira and Eftim Zdravevski, Accepted for publication in Statistic, Optimization & Information Computing, iapress, 2018; 0 citations. 6. A Multiple Source Framework for the Identification of Activities of Daily Living Based on Mobile Device Data Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante, Maria Canavarro Teixeira and Eftim Zdravevski, submitted for publication in an international journal, March 2018. 7. Measurement of the Reaction Time in the 30-S Chair Stand Test using the Accelerometer Sensor Available in off-the-Shelf Mobile Devices Ivan Miguel Pires, Diogo Marques, Nuno Pombo, Nuno M. Garcia, Mário C. Marques and Francisco Flórez-Revuelta, published in ICT4AWE 2018 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health, Madeira, Portugal, 22-23 March 2018; 0 citations. 8. A review on the artificial intelligence algorithms for the recognition of Activities of Daily Living using sensors in mobile devices Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Eftim Zdravevski and Susanna Spinsante, submitted for publication in an international journal, March 2018. 9. Pattern Recognition Techniques for the Identification of Activities of Daily Living using Mobile Device Accelerometer Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante, Maria Canavarro Teixeira and Eftim Zdravevski, submitted for publication in an international journal, April 2018. 10. Mobile Platform for the Recognition of Activities of Daily Living and their Environments based on Artificial Neural Networks Ivan Miguel Pires, Nuno Pombo, Nuno M. Garcia and Francisco Flórez-Revuelta, accepted for presentation in IJCAI-ECAI 2018, the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence, Stockholm, Sweden, 9-19 July 2018; 0 citations. xv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 11. Framework for the Recognition of Activities of Daily Living and their Environments in the Development of a Personal Digital Life Coach Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo and Francisco Flórez-Revuelta, accepted for presentation in DATA 2018, the 7th International Conference on Data Science, Technologies and Applications, Porto, Portugal, 26-28 July 2018; 0 citations. xvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Resumo Após os recentes avanços tecnológicos e o crescente uso dos dispositivos móveis, como por exemplo os smartphones, várias soluções podem ser desenvolvidas para melhorar a qualidade de vida dos utilizadores no contexto de Ambientes de Vida Assistida (AVA) ou Ambient Assisted Living (AAL). Os dispositivos móveis integram vários sensores, tais como acelerómetro, giroscópio, magnetómetro, microfone e recetor de Sistema de Posicionamento Global (GPS), que permitem a aquisição de vários parâmetros físicos e fisiológicos para o reconhecimento de diferentes Atividades da Vida Diária (AVD) e os seus ambientes. A definição de AVD inclui um conjunto bem conhecido de tarefas que são tarefas básicas de autocuidado, baseadas nos tipos de habilidades que as pessoas geralmente aprendem na infância. Essas tarefas incluem alimentar-se, tomar banho, vestir-se, fazer os cuidados pessoais, caminhar, correr, pular, subir escadas, dormir, ver televisão, trabalhar, ouvir música, cozinhar, comer, entre outras. No contexto de AVA, alguns indivíduos (comumente chamados de utilizadores) precisam de assistência particular, seja porque o utilizador tem algum tipo de deficiência, seja porque é idoso, ou simplesmente porque o utilizador precisa/quer monitorizar e treinar o seu estilo de vida. A investigação e desenvolvimento de sistemas que fornecem algum tipo de assistência particular está em crescente em muitas áreas de aplicação. Em particular, no futuro, o reconhecimento das AVD é uma parte importante para o desenvolvimento de um assistente pessoal digital, fornecendo uma assistência pessoal de baixo custo aos diferentes tipos de pessoas. pessoas. Para ajudar no reconhecimento das AVD, os ambientes em que estas se desenrolam devem ser reconhecidos para aumentar a fiabilidade destes sistemas. O foco principal desta Tese é o desenvolvimento de métodos para a fusão e classificação dos dados adquiridos a partir dos sensores disponíveis nos dispositivos móveis, para o reconhecimento quase em tempo real das AVD, tendo em consideração a grande diversidade das características dos dispositivos móveis disponíveis no mercado. Para atingir este objetivo, esta Tese iniciou-se com a revisão dos métodos e tecnologias existentes para definir a arquitetura e os módulos do novo método de identificação das AVD. Com esta revisão da literatura e com base no conhecimento adquirido sobre os sensores disponíveis nos dispositivos móveis disponíveis no mercado, um conjunto de tarefas que podem ser identificadas foi definido para as pesquisas e desenvolvimentos desta Tese. Esta revisão também identifica os principais conceitos para o desenvolvimento do novo método de identificação das AVD, utilizando os sensores, são eles: aquisição de dados, processamento de dados, correção de dados, imputação de dados, extração de características, fusão de dados e extração de resultados recorrendo a métodos de inteligência artificial. Um dos desafios está relacionado aos diferentes tipos de dados adquiridos pelos diferentes sensores, mas outros desafios foram encontrados, sendo os mais relevantes o ruído ambiental, o posicionamento do dispositivo xvii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living durante a realização das atividades diárias, as capacidades limitadas dos dispositivos móveis. As diferentes características das pessoas podem igualmente influenciar a criação dos métodos, escolhendo pessoas com diferentes estilos de vida e características físicas para a aquisição e identificação dos dados adquiridos a partir de sensores. Com base nos dados adquiridos, realizou-se o processamento dos dados, implementando-se métodos de correção dos dados e a extração de características, para iniciar a criação do novo método para o reconhecimento das AVD. Os métodos de imputação de dados foram excluídos da implementação, pois não iriam influenciar os resultados da identificação das AVD e dos ambientes, na medida em que são utilizadas as características extraídas de um conjunto de dados adquiridos durante um intervalo de tempo definido. A seleção dos sensores utilizáveis, bem como das AVD identificáveis, permitirá o desenvolvimento de um método que, considerando o uso de tecnologias para a fusão de dados adquiridos com múltiplos sensores em coordenação com outras informações relativas ao contexto do utilizador, tais como a agenda do utilizador, permitindo estabelecer um perfil de tarefas que o utilizador realiza diariamente. Com base nos resultados obtidos na literatura, o método escolhido para o reconhecimento das AVD são as diferentes variantes das Redes Neuronais Artificiais (RNA), incluindo Multilayer Perceptron (MLP), Feedforward Neural Networks (FNN) with Backpropagation and Deep Neural Networks (DNN). No final, após a criação dos métodos para cada fase do método para o reconhecimento das AVD e ambientes, a implementação sequencial dos diferentes métodos foi realizada num dispositivo móvel para testes adicionais. Após a definição da estrutura do método para o reconhecimento de AVD e ambientes usando dispositivos móveis, verificou-se que a aquisição de dados pode ser realizada com os métodos comuns. Após a aquisição de dados, os mesmos devem ser processados no módulo de processamento de dados, que inclui os métodos de correção de dados e de extração de características. O método de correção de dados utilizado para sensores de movimento e magnéticos é o filtro passa-baixo de modo a reduzir o ruído, mas para os dados acústicos, a Transformada Rápida de Fourier (FFT) foi aplicada para extrair as diferentes frequências. Após a correção dos dados, as diferentes características foram extraídas com base nos tipos de sensores usados, sendo a média, desvio padrão, variância, valor máximo, valor mínimo e mediana de dados adquiridos pelos sensores magnéticos e de movimento, a média, desvio padrão, variância e mediana dos picos máximos calculados com base nos dados adquiridos pelos sensores magnéticos e de movimento, as cinco maiores distâncias entre os picos máximos calculados com os dados adquiridos dos sensores de movimento e magnéticos, a média, desvio padrão, variância e 26 Mel-Frequency Cepstral Coefficients (MFCC) das frequências obtidas com FFT com base nos dados obtidos a partir do microfone, e a distância calculada com os dados adquiridos pelo recetor de GPS. xviii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Após a extração das características, as mesmas são agrupadas em diferentes conjuntos de dados para a aplicação dos métodos de RNA de modo a descobrir o método e o conjunto de características que reporta melhores resultados. O módulo de classificação de dados foi incrementalmente desenvolvido, começando com a identificação das AVD comuns com sensores magnéticos e de movimento, i.e., andar, correr, subir escadas, descer escadas e parado. Em seguida, os ambientes são identificados com dados de sensores acústicos, i.e., quarto, bar, sala de aula, ginásio, cozinha, sala de estar, hall, rua e biblioteca. Com base nos ambientes reconhecidos e os restantes sensores disponíveis nos dispositivos móveis, os dados adquiridos dos sensores magnéticos e de movimento foram combinados com o ambiente reconhecido para diferenciar algumas atividades sem movimento (i.e., dormir e ver televisão), onde o número de atividades reconhecidas nesta fase aumenta com a fusão da distância percorrida, extraída a partir dos dados do recetor GPS, permitindo também reconhecer a atividade de conduzir. Após a implementação dos três métodos de classificação com diferentes números de iterações, conjuntos de dados e configurações numa máquina com alta capacidade de processamento, os resultados relatados provaram que o melhor método para o reconhecimento das atividades comuns de AVD e atividades sem movimento é o método DNN, mas o melhor método para o reconhecimento de ambientes é o método FNN with Backpropagation. Dependendo do número de sensores utilizados, esta implementação reporta uma exatidão média entre 85,89% e 89,51% para o reconhecimento das AVD comuns, igual a 86,50% para o reconhecimento de ambientes, e igual a 100% para o reconhecimento de atividades sem movimento, reportando uma exatidão global entre 85,89% e 92,00%. A última etapa desta Tese foi a implementação do método nos dispositivos móveis, verificando que o método FNN requer um alto poder de processamento para o reconhecimento de ambientes e os resultados reportados com estes dispositivos são inferiores aos resultados reportados com a máquina com alta capacidade de processamento utilizada no desenvolvimento do método. Assim, o método DNN foi igualmente implementado para o reconhecimento dos ambientes com os dispositivos móveis. Finalmente, os resultados relatados com os dispositivos móveis reportam uma exatidão entre 86,39% e 89,15% para o reconhecimento das AVD comuns, igual a 45,68% para o reconhecimento de ambientes, e igual a 100% para o reconhecimento de atividades sem movimento, reportando uma exatidão geral entre 58,02% e 89,15%. Com base nos resultados relatados na literatura, os resultados do método desenvolvido mostram uma melhoria residual, mas os resultados desta Tese identificam mais AVD que os demais estudos disponíveis na literatura. A melhoria no reconhecimento das AVD com base na média das exatidões é igual a 2,93%, mas o número máximo de AVD e ambientes reconhecidos pelos estudos disponíveis na literatura é 13, enquanto o número de AVD e ambientes reconhecidos com o método implementado é 16. Assim, o método desenvolvido tem uma melhoria de 2,93% na exatidão do reconhecimento num maior número de AVD e ambientes. xix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Como trabalho futuro, os resultados reportados nesta Tese podem ser considerados um ponto de partida para o desenvolvimento de um assistente digital pessoal, mas o número de ADL e ambientes reconhecidos pelo método deve ser aumentado e as experiências devem ser repetidas com diferentes tipos de dispositivos móveis (i.e., smartphones e smartwatches), e os métodos de imputação e outros métodos de classificação de dados devem ser explorados de modo a tentar aumentar a confiabilidade do método para o reconhecimento das AVD e ambientes. Palavras-Chave Fusão de dados de sensores, aquisição de dados, acelerómetro, giroscópio, magnetómetro, microfone, recetor de Sistema de Posicionamento Global (GPS), processamento de sinais, sensores, atividades da vida diária, dispositivos móveis, reconhecimento de padrões, processamento de dados, correção de dados, imputação de dados, extração de características, inteligência artificial, redes neuronais artificiais. xx Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Resumo Alargado Introdução Este capítulo resume, de forma alargada e em Língua Portuguesa, o trabalho de investigação descrito na Tese de doutoramento intitulada de “Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living”. Inicialmente, este capítulo descreve o enquadramento da Tese, o problema abordado e os objetivos desta Tese de doutoramento, bem como o enquadramento da mesma e as principais contribuições. Seguidamente, será apresentado um resumo de cada um dos capítulos desta Tese, que correspondem às principais contribuições desta Tese. O capítulo termina com a apresentação das principais conclusões desta Tese, bem como a apresentação de algumas linhas de investigação para o futuro. Enquadramento da Tese Com base na literatura, o reconhecimento das Atividades de Vida Diária (AVD) concentra-se na identificação de um conjunto bem conhecido de tarefas que são tarefas básicas de autocuidado, baseadas nos tipos de habilidades que as pessoas geralmente aprendem na infância. Essas tarefas incluem alimentar-se, tomar banho, vestir-se, fazer os cuidados pessoais, caminhar, correr, pular, subir escadas, dormir, ver televisão, trabalhar, ouvir música, cozinhar, comer, entre outras. No contexto dos Ambientes de Vida Assistida (AVA) [1, 2], alguns indivíduos (comumente chamados de utilizadores) precisam de assistência particular, seja porque o utilizador tem algum tipo de deficiência, seja porque é idoso, ou simplesmente porque o utilizador precisa/quer monitorizar e treinar o seu estilo de vida. No entanto, as AVD incluem um conjunto mais amplo de tarefas, não limitado a tarefas relacionadas com cuidados pessoais e higiene, incluindo também as tarefas de socialização e profissionais. O desenvolvimento de métodos para o reconhecimento das AVD é importante para apoiar a autonomia de pessoas idosas e/ou com doenças crónicas e utilizadores que tenham algum problema de saúde [3, 4]. No entanto, pode ser útil para todos, incluindo atletas e jovens, já que essas soluções podem ser integradas num método de monitorização e treino de estilos de vida [5]. xxi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Atualmente, o uso de sensores para o reconhecimento das AVD e ambientes é um tema muito estudado [6-14]. No entanto, não existem estudos que definam um método estruturado para o reconhecimento das AVD e ambientes, usando um grande número de sensores disponíveis em dispositivos móveis e para o reconhecimento de um grande número de AVD. O foco desta Tese inclui o desenho de um novo método através da seleção e desenvolvimento de métodos adequados para cada módulo do método proposto para o reconhecimento de AVD e ambientes, sendo estes: aquisição, fusão, processamento e filtragem dos dados, e extração de características e classificação dos dados; com um especial destaque para os métodos relacionados com a fusão e classificação dos dados, descobrindo-se quais são os aspetos mais importantes a serem abordados para obter resultados com maior exatidão. A definição de um conjunto de tarefas que podem ser corretamente identificadas inclui a deteção e a análise de um grande número de tarefas, bem como a identificação das tarefas que podem ser identificadas pelos sensores disponíveis nos dispositivos móveis. A seleção do conjunto de sensores utilizáveis e de tarefas identificáveis permitem o desenvolvimento de um método que, considerando tecnologias de fusão de dados de múltiplos sensores e tendo em conta o contexto, em coordenação com outras informações do utilizador, como agenda e a hora do dia, permite estabelecer um perfil das tarefas que o utilizador realiza diariamente. Existem vários desafios envolvidos no desenvolvimento de um novo método para o reconhecimento das AVD. Por exemplo, com a aquisição de dados, os desafios incluem o posicionamento incorreto do dispositivo móvel, a taxa de amostragem de dados, a indisponibilidade dos sensores necessários para reconhecer uma determinada AVD e as diferentes condições ambientais [15]. Existem vários métodos que podem ajudar a minimizar esses problemas, incluindo os referidos no artigo [16] apresentado no capítulo 2. Em relação ao processamento de dados, os desafios estão ligados às diferentes arquiteturas e plataformas nas quais a solução pode ser implementada, como, por exemplo, utilização de processamento remoto ou localmente. Para fornecer um resultado aceitável em vários ambientes de aquisição de dados, o método deve funcionar sem uma ligação de rede e ser adaptada aos baixos recursos de processamento, memória e armazenamento de alguns dispositivos móveis, sendo que a implementação dos métodos é objeto de estudo desta Tese [15, 17-19]. O âmbito desta Tese consiste no uso de métodos de fusão de dados e Redes Neurais Artificiais (RNA) para o reconhecimento das AVD e ambientes com base nos dados adquiridos pelos sensores disponíveis num dispositivo móvel, como um smartphone. As limitações dos smartphones também foram exploradas, criando um método que adquire 5 segundos de dados dos sensores a cada 5 minutos, permitindo o mapeamento dos estilos de vida e o desenvolvimento de novas soluções para AVA. A pesquisa relacionada aos métodos de classificação desta Tese está relacionada com a comparação de diferentes implementações de métodos de RNA adaptados aos recursos restritos de um dispositivo móvel, utilizando técnicas xxii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living de processamento local, de modo a fornecer o feedback ao utilizador relacionado com as suas AVD e os seus ambientes quase em tempo real. Descrição do Problema e Objetivos da Investigação O problema abordado nesta Tese consiste em segmentar e identificar corretamente um conjunto significativo de AVD que os utilizadores executam nas suas rotinas diárias ou semanais, usando os sensores e as informações do utilizador disponíveis num smartphone. A identificação de AVD é um desafio complexo, devido à existência de um grande conjunto de AVD e à complexidade da identificação do método correto para identificar cada uma delas. Para detetar e identificar corretamente uma AVD, existem vários fatores que influenciam na aquisição dos dados e, consequentemente, afetam a identificação de vários tipos de atividades, que devem ser considerados, como as restrições ambientais, os movimentos involuntários do utilizador, a posição do dispositivo móvel, os movimentos do dispositivo móvel durante a aquisição de dados e a frequência da aquisição de dados dos sensores. O reconhecimento de AVD e ambientes é, até ao momento, um tópico importante para a criação de sistemas para diversas finalidades, como assistentes pessoais, monitorização das condições de saúde, monitorização do estilo de vida e outros. Atualmente, esse reconhecimento é comumente realizado por sistemas complexos e caros, com alto poder de processamento e capacidade de memória [1, 20-24]. No entanto, essa tarefa pode ser executada com recurso aos dispositivos móveis utilizados diariamente utilizando técnicas de processamento local para um rápido feedback. Hoje em dia, e com base na literatura, existem vários estudos usando diferentes subconjuntos dos sensores disponíveis nos dispositivos móveis. No entanto, os estudos analisados reportam a inexistência de métodos que utilizem um grande número de sensores disponíveis em dispositivos móveis para o reconhecimento das AVD e ambientes. Atualmente, a literatura existente reporta que as Redes Neurais Artificiais (RNA) e as suas variantes são os métodos mais utilizados, apresentando uma exatidão relevante para a identificação de algumas AVD, conforme apresentado em [16]. O principal objetivo desta Tese é a criação de um novo método para o reconhecimento de AVD e ambientes com a fusão e classificação dos dados adquiridos por um grande conjunto de sensores disponíveis nos dispositivos móveis. O método desenvolvido tem em conta o diferente número de sensores disponíveis nos diferentes dispositivos móveis e as características limitadas dos mesmos, de modo a permitir a sua execução localmente [25]. O método desenvolvido inclui xxiii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living várias etapas, tais como aquisição de dados, processamento de dados, filtragem de dados, imputação de dados, extração de características e reconhecimento de padrões. As AVD identificadas durante esta Tese são andar, correr, subir escadas, estar parado/ver televisão, descer escadas, conduzir e dormir, e os ambientes que foram identificados durante esta Tese foram quarto, bar, sala de aula, ginásio, cozinha, sala de estar, hall, rua e biblioteca. Os seguintes objetivos intermédios foram definidos a fim de dividir e organizar o trabalho de pesquisa necessário para atingir o objetivo principal desta Tese: 1. De modo a compreender as soluções atualmente existentes e possíveis métodos, esta Tese começou com a revisão da literatura relacionada com os diferentes conceitos incluídos nesta Tese. Além disso, devido aos tipos de dados utilizados neste trabalho, esta revisão também inclui o estudo relativo a métodos de validação de dados e métodos de impressão digital de áudio. Finalmente, foram descritos os sensores, as AVD e ambientes que podem ser identificados com os dados adquiridos a partir dos dispositivos móveis. 2. Como segundo objetivo intermédio, apresentou-se a definição da arquitetura proposta para o método de reconhecimento de AVD e ambientes, incluindo os diferentes conceitos disponíveis na literatura. 3. Como terceiro objetivo intermédio, foi realizado o desenvolvimento de uma aplicação móvel para aquisição dos dados dos sensores disponíveis nos dispositivos móveis. Alguns utilizadores foram selecionados para executar a AVD e etiquetá-las. Durante a aquisição de dados, o dispositivo móvel foi colocado no bolso frontal das calças do utilizador. Esta aplicação móvel executa o processo de aquisição de dados. 4. Após a aquisição dos dados, o quarto objetivo intermédio consiste na implementação de métodos de processamento de dados, incluindo filtragem de dados e extração de características. Os métodos de imputação de dados foram evitados, pois sua implementação não tem influência nos resultados da identificação das AVD e dos ambientes, pois as características são extraídas com um conjunto de dados adquiridos durante um intervalo de tempo definido. 5. Após o processamento dos dados, o quinto objetivo intermédio está relacionado com a implementação e comparação de três configurações diferentes de RNA, de modo a descobrir a melhor configuração para o reconhecimento das AVD e ambientes. Além disso, esta análise incluiu a fusão das diferentes combinações de características extraídas dos dados dos sensores para a implementação de diferentes RNA. 6. O sexto objetivo intermédio consiste na implementação de métodos que abordam os diferentes módulos do método proposto para o reconhecimento de AVD e ambientes, implementando-a numa aplicação móvel para a realização de testes adicionais. O reconhecimento das ADL e ambientes implementados como resultado desta Tese é importante para projetar o desenvolvimento de um assistente pessoal digital [5]. O tema desta xxiv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Tese é também importante para apoiar a autonomia de idosos, pessoas com doenças crónicas e utilizadores que apresentem algum tipo de deficiência [3, 4]. No entanto, pode ser útil para todos, incluindo atletas e jovens, já que o método proposto pode ser utilizado para monitorização e treino de estilos de vida [5]. Argumento da Tese Esta Tese propõe um novo método para o reconhecimento de AVD e ambientes, tendo em conta a fusão dos dados adquiridos com os sensores disponíveis nos dispositivos móveis. Especificamente, o argumento da Tese é: A identificação automática das Atividades de Vida Diária (AVD) e ambientes é um desafio complexo amplamente estudado na literatura, mas não há métodos estruturados para o reconhecimento de AVD. Existem diferentes tipos de dados que podem ser adquiridos com dispositivos móveis, de onde podem ser extraídas diversas características, sendo aplicadas técnicas de fusão de dados a confiabilidade do reconhecimento de AVD e ambientes. A fusão e classificação de dados são os passos mais importantes no desenvolvimento de um método para o reconhecimento de AVD e ambientes. O posicionamento do dispositivo móvel e a frequência da aquisição de dados também são fatores importantes que devem ser tidos em conta. Para suportar o argumento desta Tese, foi realizada a revisão bibliográfica sobre os diferentes conceitos envolvidos neste tema, incluindo aquisição, processamento, validação, filtragem, fusão e classificação de dados, bem como extração de características e métodos de impressão digital de áudio, propondo uma arquitetura para um método para o reconhecimento de AVD e ambientes. Depois da revisão da literatura, a aquisição de dados foi realizada para as AVD e ambientes selecionados. Após a aquisição de dados, a implementação dos diferentes métodos para cada módulo do método proposto foi realizada, comparando três implementações diferentes de RNA para a criação dos diferentes métodos, tendo em conta diferentes subconjuntos de sensores. No final, o método foi implementado numa aplicação móvel para testes adicionais. xxv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Principais Contribuições Esta secção descreve resumidamente as principais contribuições científicas resultantes do trabalho de pesquisa apresentado nesta Tese. A primeira contribuição está relacionada com a proposta de uma estrutura para o método de reconhecimento de AVD, descrevendo também os métodos existentes na literatura referentes a cada conceito relacionado com esta Tese, incluindo métodos de classificação, métodos de aquisição, processamento, imputação e fusão de dados adquiridos com os sensores de dispositivos móveis, bem como a aplicabilidade destes métodos no reconhecimento de AVD [16]. A segunda contribuição desta Tese é a classificação e aplicabilidade dos métodos existentes para a validação dos dados adquiridos pelos sensores, tendo em conta os diferentes tipos de dados em falta e/ou inválidos [26]. A terceira contribuição desta Tese é a comparação da exatidão das características e métodos utilizados para a definição de uma impressão digital de áudio e já reportados na literatura, utilizados aqui para o reconhecimento dos ambientes com base nos diferentes tipos de sinais acústicos [27]. A quarta contribuição desta Tese é a definição da arquitetura do método proposto para o reconhecimento de AVD e seus ambientes, propondo possíveis métodos para cada módulo [28]. A quinta contribuição desta Tese consiste na apresentação e análise dos resultados da investigação relacionada com o reconhecimento das atividades de andar, correr, subir e descer escadas, e estar parado com base na utilização dos dados adquiridos pelo acelerómetro, magnetómetro e giroscópio disponíveis num dispositivo móvel, comparando três implementações diferentes de RNA com dados normalizados e não normalizados, de modo a descobrir a melhor implementação para o reconhecimento destas AVD, concluindo que o melhor método é o DNN com dados normalizados [29]. A sexta contribuição desta Tese consiste na construção, implementação e testes de uma biblioteca para dispositivos com o sistema operativo Android com o método para o reconhecimento de AVD e ambientes com base nos resultados obtidos em estudos anteriores, apresentando uma validação dos resultados obtidos pelo método proposto [30]. xxvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Estado da arte Inicialmente, esta Tese iniciou com a pesquisa do estado da arte relativa aos diversos conceitos, tais como aquisição, processamento, validação, imputação, filtragem, fusão e classificação de dados, criação de impressões digitais de áudio e outros conceitos relacionados com dispositivos móveis e sensores. 1. Da aquisição até à fusão de dados: Uma revisão de literatura para a identificação de atividades de vida diária usando dispositivos móveis A identificação das AVD é possível com o uso dos sensores disponíveis nos dispositivos móveis utilizados diariamente, tais como smartphones, smartwatches ou outros dispositivos, pois estes incluem diversos sensores, e.g., acelerómetro, magnetómetro, giroscópio, microfone e recetor de Sistema de Posicionamento Global (GPS) [31, 32]. Para tal, começou por se estudar um determinado conjunto de conceitos, tais como sensores de dispositivos móveis, aquisição, processamento, imputação e fusão de dados, de modo a, baseado na literatura, aferir quais são os melhores métodos e sensores que se adequam à identificação das AVD. Os sensores disponíveis nos dispositivos móveis são de diversos tipos, tendo sido proposta a classificação que inclui sensores magnéticos, e.g., magnetómetro e compasso, sensores de ambiente, e.g., temperatura ambiente, pressão atmosférica e humidade, sensores de localização, e.g., recetor de GPS e serviços de localização por Wi-Fi, sensores de movimento, e.g., acelerómetro, sensor de gravidade, giroscópio e sensor de orientação, sensores de imagem, e.g., câmara fotográfica e/ou de vídeo, sensores acústicos, e.g., microfone, sensores óticos, e.g., sensor infravermelhos e sensor de Radio Frequency Identifier (RFID), e sensores de pressão, e.g., sensor de impressão digital [16]. Relativamente à aquisição de dados, na literatura tem vindo a ser criadas várias metodologias diferentes, de entre as quais uma das mais utilizadas é a Acquisition Cost-Aware QUery Adaptation (ACQUA) [33], embora no decorrer desta Tese se tivesse optado por utilizar os métodos comuns para a aquisição de dados. xxvii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Relativamente ao processamento de dados, as metodologias apresentadas na literatura dividem-se em duas vertentes, são elas [34]: processamento local e processamento num servidor remoto. Os métodos utilizados em incluem a posteriori o uso de Redes Neuronais Artificial (RNA), Support Vector Machine (SVM), k-Nearest Neighbour (k-NN), entre outros [3542]. O tema seguinte foi o da imputação de dados, tendo sido concluído que os métodos utilizados variam de acordo com o tipo de dados em falta, podendo os mesmos serem classificados como dados em falta completamente aleatórios, dados em falta distribuídos por diferentes conjuntos de dados e dados em falta não aleatórios [43, 44]. Relativamente à fusão de dados, os métodos utilizados dividem-se em três grandes grupos, são eles: métodos probabilísticos, métodos estatísticos, métodos baseados na teoria do conhecimento e métodos baseados nas evidências de raciocínio [45]. Os métodos probabilísticos são métodos que permitem uma aprendizagem não supervisionada, estimando modelos para as diversas tarefas, contudo necessitam ter um conhecimento prévio do sistema. Os métodos estatísticos têm melhores resultados com a redução do erro, mas exigem uma grande capacidade computacional. Os métodos baseados na teoria do conhecimento permitem a inclusão da incerteza, são fáceis de implementar e robustos para dados com ruido, embora exija intervenção humana para a aprendizagem. Os métodos baseados nas evidências de raciocínio requerem um determinado grau de incerteza para cada uma das entrados dos métodos. Contudo, a utilização destes métodos depende dos sensores utilizados, sendo que o método mais usado para a fusão de dados é o filtro de Kalman [46], que é um algoritmo que utiliza os mínimos quadrados recursivos ponderados dinamicamente, e as RNA [35-42], que são também utilizados para a classificação dos dados. Os conceitos até aqui apresentados nesta Tese podem ser utilizados para a identificação das AVD e os seus ambientes, servindo de prova de conceito para o desenvolvimento de um assistente pessoal digital. Nesta secção foram somente apresentados alguns exemplos, sendo que serão apresentados os métodos de aquisição, processamento, imputação e fusão de dados com mais detalhe no capítulo 2. 2. Técnicas de validação de dados de sensores utilizados em aplicações móveis para a saúde Aquando da captura dos dados dos diversos sensores disponíveis nos dipositivos móveis são vários os problemas que podem acontecer, causando falhas na aquisição dos dados que pode xxviii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living incluir a aquisição de dados inválidos ou a inexistência dos mesmos em alguns instantes durante a aquisição dos dados. A validação dos dados permite-nos reduzir os dados em falta ou incorretos, aplicando mecanismos de filtragem e/ou imputação de dados. Assim, durante o processamento devem ser aplicados filtros de novo a reduzir os valores inválidos presentes no sinal capturado, sendo posteriormente aplicados mecanismos de imputação de dado no caso da existência de valores em falta. Os mecanismos de filtragem dos dados dependem do tipo de dados a serem adquiridos, isto é, para os dados de sensores de movimento ou magnéticos, os mecanismos de filtragem são normalmente constituídos por filtro de passa-baixo [47], enquanto para dados acústicos é comummente aplicada a transformada de Fourier para extrair as frequências mais relevantes [48]. A validação de dados, bem como a correção dos mesmos, permite a obtenção de resultados com maior exatidão e em diferentes condições de aquisição de dados. Nesta secção foram somente apresentados alguns exemplos, sendo que serão apresentados os métodos de validação, filtragem e imputação de dados com mais detalhe no capítulo 2. 3. Técnicas de criação de impressões digitais de áudio para a deteção do ambiente das atividades de vida diária A impressão digital de áudio é aplicada para processar dados de áudio e permitir a identificação das AVD. De acordo com os autores de [49], a implementação de um sistema que gera impressões digitais de áudio inclui a definição de uma plataforma ou método, das representações de impressões digitais, das estruturas de dados, das técnicas para a medição de similaridades e dos métodos estatísticos para o processamento dos mesmo. Diversos autores apresentaram diferentes métodos de criação d impressões digitais de áudio para diferentes tipos de utilização [49-54]. Os resultados obtidos com as impressões digitais de áudio dependem da granularidade dos dados armazenados na base de dados, bem como dos dados a serem capturados para a comparação. Com base nos vários estudos, a Google, Inc. implementou um método chamado Musicg [55], que é uma API (Application Programming Interface) Java para análise de áudio, compatível com o sistema operativo Android. Os recursos implementados incluem a deteção da entrada de áudio que corresponde a um aplauso e um assobio, a leitura dos cabeçalhos WAVE do PCM xxix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living (Pulse-code modulation) e a leitura e seleção dos dados de áudio. Todas as diferentes características consideradas para a impressão digital de áudio são processadas pelo teorema de Nyquist para definir o número de amostras, a definição dos valores mínimos e máximos das amplitudes da frequência, intensidade, desvio padrão, número de zeros e robustez do sinal de áudio, bem como a definição de filtros passa-alta e passa-baixo para maximizar os recursos importantes do sinal de áudio. Nesta secção foram somente apresentados alguns exemplos, sendo que serão apresentados os métodos de impressão digital de áudio com mais detalhe no capítulo 2. Proposta de um método para identificação das atividades de vida diária Após a revisão de literatura relativa aos diversos conceitos, a arquitetura do método proposto nesta Tese para a identificação das AVD e ambientes foi definida e apresentada, incluindo os diversos conceitos abordados na revisão de literatura. 1. Abordagem para o desenvolvimento de um método para a identificação de atividades de vida diária utilizando os sensores de dispositivos móveis O método proposto tem por objetivo a utilização de vários sensores disponíveis nos dispositivos móveis, tais como acelerómetro, magnetómetro, giroscópio, microfone e recetor de GPS, sendo que o método adaptar ao máximo número de sensores disponível em cada dispositivo móvel no mercado. Assim, o método deve implementar quatro grandes módulos, são eles: aquisição de dados, processamento de dados, fusão de dados e classificação de dados. Relativamente ao módulo de aquisição de dados, o mesmo será implementado com os métodos comuns de aquisição de dados, verificando-se desnecessária a utilização de métodos complexos dados a sua simplicidade. Assim, serão adquiridos 5 segundos de dados dos diversos sensores disponíveis para a identificação da AVD que está a ser realizada a cada 5 minutos, bem como o ambiente a ser frequentado. xxx Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living No caso do módulo de processamento de dados, o mesmo será composto por filtragem de dados, imputação de dados e extração de características. A filtragem de dados depende dos sensores utilizados, sendo que para o dados capturados com o acelerómetro, o magnetómetro e o giroscópio devem ser implementados filtros de passa-baixo [47], para dados capturados com o microfone deve ser aplicada a Transformada Rápida de Fourier (FFT) para extrair as frequências mais relevantes [48], e, por fim, para os dados capturados com recurso ao recetor de GPS verificou-se que não seria necessária a aplicação de qualquer método de filtragem. A imputação de dados verificou-se desnecessária devido às características selecionadas para a classificação de dados, sento sido implementada de seguida os métodos de extração de características. Como explicado com mais detalhe no capitulo 2, para os dados capturados com o acelerómetro, o magnetómetro e o giroscópio devem ser extraídas as cinco maiores distâncias entre os picos, a médio, desvio padrão, variância e mediana dos picos máximos, e a média, desvio padrão, máximo, mínimo, variância e mediana do sinal em bruto, para os dados adquiridos com o microfone devem ser extraídos 26 coeficientes de MFCC, e a média, desvio padrão, máximo, mínimo, variância e mediana do sinal em bruto, e para os dados adquiridos com o recetor de GPS deve ser extraída a distância percorrida. Relativamente aos módulos de fusão e classificação de dados são normalmente executados em conjunto, tendo sido escolhido o uso de RNA, pois são os métodos que reportam melhores resultados na literatura e são dos mais utilizados. Assim, no módulo de classificação começouse por identificar as AVD mais comuns como andar, correr, estar parado, subir escadas e descer escadas, com o recurso aos dados adquiridos com o acelerómetro, magnetómetro e/ou giroscópio. Numa fase posterior foram identificados os ambientes frequentados, como sala de aula, ginásio, rua, hall, cozinha, bar, biblioteca, sala de estar e quarto, com recurso aos dados adquiridos com o microfone. Finalmente, tendo em conta o ambiente identificado e os dados extraídos com o recetor de GPS tornou-se possível a diferenciação de atividades que são consideradas sem movimento com dados adquiridos com o acelerómetro, magnetómetro e giroscópio, optando-se por diferenciar entre dormir, ver televisão e conduzir. Para a classificação, propôs-se a comparação entre três diferentes bibliotecas, implementando diversas configurações que serão apresentadas com mais detalhe no capitulo 4, incluindo Multilayer Perceptron (MLP) with Backpropagation [56], Feedforward Neural Networks (FNN) with Backpropagation [57] e Deep Neural Networks (DNN) [58]. Em conclusão, o método proposto inclui a classificação de 9 ambientes e 7 AVD que poderão ser posteriormente incrementadas com novos desenvolvimentos. xxxi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Fusão de dados para a identificação de atividades de vida diária Por fim, os detalhes da implementação do método para identificação das AVD e ambientes foram apresentados, utilizando os diversos sensores disponíveis nos dispositivos móveis e com foco nos métodos de fusão e classificação de dados. 1. Identificação de atividades de vida diária através a fusão dos dados dos sensores magnéticos e de movimento dos dispositivos móveis Tendo em conta os dados previamente adquiridos e categorizados nas diferentes AVD (i.e., andar, correr, subir escadas, descer escadas e estar parado), disponíveis em [59], e a estrutura do método apresentado no capítulo 3, os dados foram processados aplicando o filtro de passabaixo e foram extraídas as características dos dados adquiridos com os diferentes sensores, tais como acelerómetro, giroscópio e magnetómetro, sendo exploradas as diferentes combinações dos mesmos, isto é, foram explorados os resultados obtidos com o acelerómetro, com a combinação do acelerómetro com o magnetómetro e, por fim, com a combinação entre acelerómetro, magnetómetro e giroscópio. Antes da aplicação dos métodos de classificação, os dados foram normalizados, sendo que para as configurações de Multilayer Perceptron (MLP) with Backpropagation e Feedforward Neural Networks (FNN) with Backpropagation os dados foram normalizados com recurso aos máximos e os mínimos e a normalização dos dados para o método Deep Neural Networks (DNN) foi realizada com recurso à média e desvio padrão. Assim, no decorrer deste estudo foram testados diferentes conjuntos de características extraídas dos diferentes sensores, são elas: • Cinco maiores distâncias entre os picos máximos combinados com a média, desvio padrão, variância e mediana dos picos máximos, e desvio padrão, média, máximo, mínimo, variância e mediana do sinal em bruto; • Média, desvio padrão, variância e mediana dos picos máximos, combinados com o desvio padrão, média, máximo, mínimo, variância e mediana do sinal em bruto; • xxxii Desvio padrão, média, máximo, mínimo, variância e mediana do sinal em bruto; Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living • Desvio padrão, média, variância e mediana do sinal em bruto; • Desvio padrão e média do sinal em bruto. Os diferentes conjuntos de características foram utilizados para as diferentes combinações de sensores, obtendo-se melhores resultados com a utilização de todas as características extraídas nas diferentes combinações e com as diferentes configurações de RNA apresentadas no capítulo 4. Assim, obtiveram-se os seguintes resultados com as diferentes configurações de redes neuronais implementadas com dados normalizados e não normalizados: • Acelerómetro: o Dados não normalizados: § Neste caso o método MLP reportou uma exatidão de 34.75%, enquanto o método FNN reportou uma exatidão de 74.45% e o método DNN reportou uma exatidão de 80.35%; o Dados normalizados: § Neste caso o método MLP reportou uma exatidão de 24.03%, enquanto o método FNN reportou uma exatidão de 37.07% e o método DNN reportou uma exatidão de 85.89%. • Acelerómetro e magnetómetro: o Dados não normalizados: § Neste caso o método MLP reportou uma exatidão de 35.15%, enquanto o método FNN reportou uma exatidão de 42.75% e o método DNN reportou uma exatidão de 70.43%; o Dados normalizados: § Neste caso o método MLP reportou uma exatidão de 24.93%, enquanto o método FNN reportou uma exatidão de 64.94% e o método DNN reportou uma exatidão de 86.49%. • Acelerómetro, magnetómetro e giroscópio: o Dados não normalizados: § Neste caso o método MLP reportou uma exatidão de 38.32%, enquanto o método FNN reportou uma exatidão de 76.13% e o método DNN reportou uma exatidão de 74.47%; o Dados normalizados: § Neste caso o método MLP reportou uma exatidão de 37.13%, enquanto o método FNN reportou uma exatidão de 29.54% e o método DNN reportou uma exatidão de 89.51%. Em conclusão verificou-se que com os resultados obtidos é possível identificar um determinado conjunto de AVD, sendo que o método que reporta melhores resultados é o método DNN com dados normalizados em qualquer uma das combinações de sensores. xxxiii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 2. Desenvolvimento de uma biblioteca para o reconhecimento de atividades de vida diária: considerações de implementação, desafios e soluções Com base nos resultados obtidos com os estudos anteriores relacionados com a identificação das AVD comuns, identificação de ambientes e identificação de atividades sem movimento, foi implementada a estrutura do método anteriormente apresentado numa biblioteca desenvolvida para dispositivos móveis com o sistema operativo Android. Durante a implementação verificaram-se alguns desafios relacionados com a exatidão apresentada pelos métodos anteriormente criados. A aplicação móvel desenvolvida implementa métodos de aquisição de dados, implementa os métodos de processamento e os métodos de fusão e classificação de dados anteriormente apresentados. Contudo, especialmente na implementação do método para identificação de ambientes que anteriormente tinha sido verificado que os melhores resultados eram obtidos com o método de Feedforward Neural Networks (FNN) with Backpropagation com dados não normalizados, na aplicação móvel foi notado um decréscimo da exatidão e do desempenho no processamento de dados de áudio com este método, tendo sido testada a implementação deste método em combinação com o método Deep Neural Networks (DNN) com dados normalizados, verificando-se que os resultados melhoravam, mas os melhores resultados foram obtidos somente com o utilização do método DNN com dados normalizados, melhorando o desempenho e a exatidão dos resultados. Para as restantes fases, foi implementado o método DNN com dados normalizados, tendo-se verificado os resultados esperados. Assim, os resultados obtidos pela aplicação móvel desenvolvida são os seguintes: • Identificação de AVD comuns: o Acelerómetro: § o Acelerómetro e magnetómetro: § o Microfone § Os resultados reportados têm uma exatidão de 45,68% Identificação de atividades sem movimento o Inclusão do recetor de GPS e/ou ambiente reconhecido: § xxxiv Os resultados reportados têm uma exatidão de 89,15%. Identificação de ambientes o • Os resultados reportados têm uma exatidão de 86,49%; Acelerómetro, magnetómetro e giroscópio: § • Os resultados reportados têm uma exatidão de 86,39%; Os resultados reportados têm uma exatidão de 100%; Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Em conclusão, os resultados reportam uma exatidão global entre 58,02% e 89.15%, que se verificou por comparação com os resultados disponíveis na literatura que se obteve uma melhoria de 2.93% com base nos resultados médios, tendo-se obtido resultados similares em termos máximos (sem melhoria), embora o número máximo de eventos detetado pelos estudos anteriores seja, no máximo, de 13, enquanto os resultados apresentados nesta Tese realizam a identificação de 16 eventos. Principais Conclusões O foco desta Tese está relacionado ao desenvolvimento de um novo método para o reconhecimento de AVD e seus ambientes, incluindo a seleção dos melhores métodos para os diferentes módulos, como métodos de aquisição, fusão, processamento e filtragem de dados, extração de características e classificação dos dados, com base nos sensores disponíveis num smartphone. Os módulos mais importantes são os que incluem os métodos de fusão e classificação de dados que influenciam diretamente os resultados obtidos pelo método resultante da investigação desta Tese. Os dispositivos móveis integram vários sensores, e.g., acelerómetro, giroscópio, magnetómetro, microfone e recetor de GPS, que permitem a aquisição de vários tipos de parâmetros físicos e fisiológicos. Os dispositivos móveis podem ser considerados dispositivos com múltiplos sensores, que consiste na primeira parte do título desta Tese. Existe uma grande variedade de dispositivos móveis, mas o âmbito desta Tese foca-se apenas no uso de smartphones. A aquisição dos diferentes parâmetros permite o reconhecimento das AVD, onde o trabalho desta Tese é implementar métodos de fusão e classificação de dados para o seu reconhecimento e reconhecimento de ambientes. Existem várias AVD que podem ser reconhecidas, mas esta Tese foca-se no reconhecimento de sete AVD (i.e., andar, correr, subir, descer, estar parado/assistir TV, dormir e conduzir) e nove ambientes (i.e., quarto, bar, sala de aula, academia, cozinha, sala de estar, hall, rua e biblioteca) com recurso aos dados recolhidos com os sensores disponíveis nos dispositivos móveis. No entanto, esses dispositivos têm várias limitações ao nível da capacidade da bateria, de processamento e de memória, limitações estas cujo impacto foi minimizado com a implementação de diferentes métodos. No final, o método desenvolvido pode ser utilizado para o mapeamento de estilos de vida tendo em conta o reconhecimento de AVD e seus ambientes. O principal objetivo do trabalho desta Tese é a criação de um novo método para o reconhecimento de AVD e ambientes com a fusão e classificação dos dados adquiridos dos sensores disponíveis nos dispositivos móveis disponíveis. O método desenvolvido é deve ter em xxxv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living conta a variedade e número de sensores disponíveis nos diferentes dispositivos móveis. Deve ser implementado com algoritmos adaptados às características destes dispositivos, para permitir a sua execução localmente. O objetivo principal foi dividido em seis objetivos secundários, que começaram com a revisão da literatura, verificando que não existem estudos que utilizem um grande número de sensores disponíveis nos dispositivos móveis e apliquem técnicas de fusão de dados adquiridos pelos mesmos, mas é um tópico que está crescendo amplamente com diferentes trabalhos de investigação. A revisão da literatura também revela que as Redes Neurais Artificiais (RNA) são amplamente utilizadas e reportam melhores resultados que outros métodos. Por fim, a revisão da literatura ajudou na definição das AVD e ambientes a serem reconhecidos pelo método desenvolvido, bem como a definição da sua arquitetura que inclui aquisição, fusão, processamento e filtragem dos dados, e extração de características e classificação dos mesmos. A aquisição de dados dos sensores dos dispositivos móveis é realizada por métodos básicos. Os métodos de processamento e filtragem de dados, bem como de extração de características, dependem dos tipos de sensores utilizados, conforme apresentado no capítulo 3, que é focado na definição da arquitetura do método para o reconhecimento de AVD e ambientes. Seguidamente, a aquisição de dados foi realizada com recurso a uma aplicação móvel que permite a captura e etiquetagem das AVD e dos ambientes para análise e criação dos métodos. Após a aquisição dos dados, os mesmos os foram analisados e os diferentes métodos foram implementados, incluindo filtros passa-baixo, Transformada Rápida de Fourier (FFT), entre outros. Após a extração das características foram analisadas três diferentes implementações de métodos de RNA, incluindo Multilayer Perceptron (MLP) with Backpropagation, Feedforward Neural Networks (FNN) with Backpropagation e Deep Neural Networks (DNN), apresentando mais detalhes no capítulo 4. As diferentes implementações reportaram que o método que obtém os melhores resultados é o método DNN com dados normalizados. A primeira parte desta Tese, apresentada no capítulo 2, consiste na revisão da literatura sobre os diferentes conceitos relacionados à criação de um novo método para o reconhecimento das AVD e ambientes. Inicialmente, os conceitos estudados consistem na aquisição, processamento, imputação e fusão de dados foram explorados, analisando os vários métodos anteriormente utilizados para cada conceito, bem como a sua aplicabilidade. No entanto, as soluções encontradas são difíceis de ser adaptadas aos dispositivos móveis, porque o desenvolvimento de soluções para esses dispositivos deve ter em conta as diferentes restrições de hardware que dependem dos fabricantes. Além disso, durante a primeira parte da revisão da literatura, verificou-se que a imputação de dados faz parte do processamento de dados, no entanto esta etapa foi excluída da investigação desta Tese, pois os métodos de classificação utilizam as características extraídas dos dados dos sensores, não sendo relevante a inexistência de dados em alguns instantes. A segunda parte da revisão de literatura consiste na pesquisa sobre os métodos de validação de dados, incluindo métodos de filtragem e imputação de dados no módulo de processamento de dados. Os métodos de filtragem de dados incluem os filtros de xxxvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living passa-baixo, filtros de passa-alto e outros; e os métodos de imputação de dados analisados consistem principalmente no K-Nearest Neighbor (KNN) e as suas variantes, verificando-se que um dos métodos mais utilizados com melhores resultados são as RNA. Como os dados acústicos também foram utilizados para o reconhecimento dos ambientes, os métodos de impressão digital de áudio também foram analisados, verificando-se que a FFT deve ser implementada para a extração das frequências, concluindo-se que os métodos de classificação mais utilizados com melhores resultados são os métodos estatísticos. A segunda parte desta Tese, apresentada no capítulo 3, consiste na definição da arquitetura do método proposto para o reconhecimento das AVD, mencionando-se que a arquitetura começa com a aquisição de dados. Após a aquisição dos dados, os dados são processados, incluindo filtragem, imputação e extração de características. A filtragem dos dados tem em conta os tipos de sensores utilizados, implementando o método de filtro de passa-baixo para os sensores de movimento e magnéticos e a FFT para os sensores acústicos. Primeiramente, para sensores magnéticos e de movimento, as características extraídas são as cinco maiores distâncias entre os picos máximos combinados com a média, desvio padrão, variância e mediana dos picos máximos, e desvio padrão, média, máximo, mínimo, variância e mediana do sinal em bruto. Seguidamente, para os sensores acústicos, as características extraídas são os 26 Mel-Frequency Cepstral Coefficients (MFCC) combinados com o desvio padrão, média, máximo, mínimo, variância e mediana do sinal. Finalmente, para os sensores de localização, a característica extraída é a distância percorrida. Após a extração de características e com base nas RNA, o método de classificação foi implementado. Os resultados da classificação devem ser combinados com a agenda dos utilizadores para desenvolvimentos futuros. A terceira parte desta Tese, apresentada no capítulo 4, consiste na apresentação dos resultados principais desta Tese, que consistem na implementação dos diferentes métodos para o reconhecimento das AVD e ambientes. A implementação apresentada nesta Tese foi incremental, começando com a implementação com apenas um sensor, i.e., acelerómetro, seguindo-se a implementação com sensores mais comuns nos smartphones, i.e., acelerómetro, magnetómetro, giroscópio, microfone e recetor GPS. As implementações são realizadas com três diferentes configurações de RNA, tais como o método MLP with Backpropagation, método FNN with Backpropagation e método DNN com dados normalizados e não normalizados. Todas as implementações são baseadas em vários grupos de características extraídas a partir dos dados dos diferentes sensores. A normalização dos dados para os métodos MLP e FNN foi realizada com recurso à normalização com os máximos e os mínimos e a normalização dos dados para o método DNN foi realizada com recurso à média e desvio padrão. Os detalhes sobre as configurações dos métodos de RNA estão disponíveis no capítulo 4. O método proposto realiza o reconhecimento de ADL e ambientes em 3 fases, tais como o reconhecimento de AVD comuns com sensores magnéticos e de movimento, o reconhecimento de ambientes com sensores acústicos, e o reconhecimento de atividades sem movimento com sensores de movimento, xxxvii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living magnéticos e de localização combinados com o ambiente reconhecido. Para os resultados obtidos com os dados do acelerómetro, os melhores resultados foram obtidos com o método DNN, reportando uma exatidão de 85,89% com dados normalizados e 80,35% com dados não normalizados. Para os resultados obtidos com a fusão dos dados do acelerómetro e do magnetómetro, os melhores resultados também foram obtidos com o método DNN, reportando uma exatidão de 86,49% com dados normalizados e 70,43% com dados não normalizados. Para os resultados obtidos com a fusão dos dados do acelerómetro, do magnetómetro e do giroscópio, os melhores resultados foram também obtidos com o método DNN, reportando uma exatidão de 89,51% com dados normalizados e 74,47% com dados não normalizados. Para os resultados obtidos com os dados acústicos, os melhores resultados foram obtidos com o método FNN with Backpropagation, reportando uma exatidão de 82,75% com dados normalizados e 86,50% com dados não normalizados. Para os resultados obtidos com a fusão dos sensores utilizados anteriormente com os sensores de localização e/ou ambiente reconhecido, os melhores resultados foram obtidos com o método DNN, relatando uma precisão de 100% com as diferentes o método proposto foi implementado e validado num smartphone, reportando uma exatidão geral entre 58,02% e 89,15%. O objetivo principal desta Tese foi atingido com o desenvolvimento de um novo método para o reconhecimento de AVD e seus ambientes, reconhecendo um maior número de AVD/ambientes que os trabalhos anteriores que foram analisados, relatando uma exatidão geral superior à média dos resultados, e exatidão semelhante aos melhores resultados reportados pelos trabalhos existentes na literatura. Direções para trabalho futuro O reconhecimento das AVD e ambientes pode ser realizado com outros métodos, e os resultados obtidos pelo método desenvolvido podem ser melhorados com pesquisa e implementação de métodos de Imputação de dados a serem incorporados no módulo de processamento de dados. O módulo de processamento de dados também pode ser melhorado com outras abordagens relacionadas aos métodos de validação e filtragem de dados. No entanto, o módulo mais importante do método para o reconhecimento de ADL e ambientes é o módulo de classificação, no qual outros tipos de métodos de inteligência artificial podem ser implementados e testados com os conjuntos de dados adquiridos durante esta Tese, incluindo Support Vector Machines (SVM), Árvores de Decisão e outros métodos de aprendizagem automática (e.g., Adaboost) para verificar se os resultados são melhores que com outros métodos. xxxviii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Tendo em conta os recursos limitados de bateria, processamento e memória dos smartphones, o desafio mais importante no desenvolvimento do método é descobrir uma forma de adquirir e processar os dados continuamente, sem efeitos no desempenho dos smartphones, a fim de poder reconhecer situações de risco, e.g., detecção de quedas. Atualmente, o método desenvolvido é preparado para smartphones. A implementação do método para execução noutros tipos de dispositivos (e.g., smartwatches) exigiria mais experimentação, adquirindo novos dados e desenvolvendo novos métodos de classificação adaptados a esses dispositivos. Consequentemente, o método deve ser adaptado às características específicas do dispositivo móvel em uso. Uma das diferenças mais importantes na aquisição de dados entre o smartphone e o smartwatch é a posição deles, porque o smartphone pode ser colocado no bolso frontal das calças do utilizador, mas o smartwatch é comumente colocado no pulso. Como trabalho futuro, o número de AVD e ambientes reconhecidos pelo método podem ser incrementados, mas isso também requereria mais experiências. Como objetivo futuro, este método pode ser usado no desenvolvimento de um assistente pessoal digital. Referências [1] C. Dobre, C. x. Mavromoustakis, N. Garcia, R. I. Goleva, and G. Mastorakis, Ambient Assisted Living and Enhanced Living Environments: Principles, Technologies and Control: Butterworth-Heinemann, 2016 [2] N. M. Garcia and J. J. P. Rodrigues, Ambient assisted living: CRC Press, 2015 [3] M. Kara, O. Lamouchi, and A. Ramdane-Cherif, "A Quality Model for the Evaluation AAL Systems," Procedia Computer Science, vol. 113, pp. 392-399, 2017/01/01/ 2017. doi: https://doi.org/10.1016/j.procs.2017.08.354 [4] S. Spinsante, E. Gambi, L. Raffaeli, L. Montanini, L. Paciello, R. Bevilacqua, et al., "Technology-based assistance of people with dementia: state of the art, open challenges, and future developments," Human Monitoring, Smart Health and Assisted Living: Techniques and Technologies, vol. 9, p. 55, 2017 [5] N. M. Garcia, "A Roadmap to the Design of a Personal Digital Life Coach," in ICT Innovations 2015, ed: Springer, 2016. xxxix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [6] A. D’Ambrosio, M. Aria, and R. Siciliano, "Accurate Tree-based Missing Data Imputation and Data Fusion within the Statistical Learning Paradigm," Journal of Classification, vol. 29, pp. 227-258, 2012. doi: 10.1007/s00357-012-9108-1 [7] J. Dong, D. Zhuang, Y. Huang, and J. Fu, "Advances in multi-sensor data fusion: algorithms and applications," Sensors (Basel), vol. 9, pp. 7771-84, 2009. doi: 10.3390/s91007771 [8] R. C. King, E. Villeneuve, R. J. White, R. S. Sherratt, W. Holderbaum, and W. S. Harwin, "Application of data fusion techniques and technologies for wearable health monitoring," Med Eng Phys, vol. 42, pp. 1-12, Apr 2017. doi: 10.1016/j.medengphy.2016.12.011 [9] O. Banos, M. Damas, H. Pomares, and I. Rojas, "On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition," Sensors (Basel), vol. 12, pp. 8039-54, 2012. doi: 10.3390/s120608039 [10] M. A. A. Akhoundi and E. Valavi, "Multi-Sensor Fuzzy Data Fusion Using Sensors with Different Characteristics," arXiv preprint arXiv:1010.6096, 2010 [11] P. Paul and T. George, "An Effective Approach for Human Activity Recognition on Smartphone," 2015 Ieee International Conference on Engineering and Technology (Icetech), pp. 45-47, 2015. doi: 10.1109/icetech.2015.7275024 [12] Y.-W. Hsu, K.-H. Chen, J.-J. Yang, and F.-S. Jaw, "Smartphone-based fall detection algorithm using feature extraction," in 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Datong, China, 2016, pp. 1535-1540. [13] S. Dernbach, B. Das, N. C. Krishnan, B. L. Thomas, and D. J. Cook, "Simple and Complex Activity Recognition through Smart Phones," in 2012 8th International Conference on Intelligent Environments (IE), Guanajuato, Mexico, 2012, pp. 214-221. [14] C. Shen, Y. F. Chen, and G. S. Yang, "On Motion-Sensor Behavior Analysis for HumanActivity Recognition via Smartphones," in 2016 Ieee International Conference on Identity, Security and Behavior Analysis (Isba), Sendai, Japan, 2016, pp. 1-6. [15] S. D. Bersch, D. Azzi, R. Khusainov, I. E. Achumba, and J. Ries, "Sensor data acquisition and processing parameters for human activity classification," Sensors (Basel), vol. 14, pp. 4239-70, 2014. doi: 10.3390/s140304239 xl Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [16] I. Pires, N. Garcia, N. Pombo, and F. Flórez-Revuelta, "From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices," Sensors, vol. 16, p. 184, 2016 [17] V. Pejovic and M. Musolesi, "Anticipatory Mobile Computing," ACM Computing Surveys, vol. 47, pp. 1-29, 2015. doi: 10.1145/2693843 [18] F. X. Lin, A. Rahmati, and L. Zhong, "Dandelion: a framework for transparently programming phone-centered wireless body sensor applications for health," in Wireless Health 2010, 2010, pp. 74-83. [19] O. Postolache, P. S. Girao, M. Ribeiro, M. Guerra, J. Pincho, F. Santiago, et al., "Enabling telecare assessment with pervasive sensing and Android OS smartphone," in 2011 IEEE International Workshop on Medical Measurements and Applications Proceedings (MeMeA), 2011, pp. 288-293. [20] C. Siegel, A. Hochgatterer, and T. E. Dorner, "Contributions of ambient assisted living for health and quality of life in the elderly and care services--a qualitative analysis from the experts' perspective of care service professionals," BMC Geriatr, vol. 14, p. 112, 2014. doi: 10.1186/1471-2318-14-112 [21] J. A. Botia, A. Villa, and J. Palma, "Ambient Assisted Living system for in-home monitoring of healthy independent elders," Expert Systems with Applications, vol. 39, pp. 8136-8148, 2012. doi: 10.1016/j.eswa.2012.01.153 [22] N. M. Garcia, J. J. P. C. Rodrigues, D. C. Elias, and M. S. Dias, Ambient Assisted Living: Taylor & Francis, 2014 [23] M. Huch, A. Kameas, J. Maitland, P. J. McCullagh, J. Roberts, A. Sixsmith, et al., Handbook of Ambient Assisted Living: Technology for Healthcare, Rehabilitation and Well-being - Volume 11 of Ambient Intelligence and Smart Environments: IOS Press, 2012 [24] R. I. Goleva, N. M. Garcia, C. X. Mavromoustakis, C. Dobre, G. Mastorakis, R. Stainov, et al., "AAL and ELE Platform Architecture," 2017 [25] I. M. Pires, N. M. Garcia, N. Pombo, and F. Flórez-Revuelta, "Limitations of the Use of Mobile Devices and Smart Environments for the Monitoring of Ageing People," in ICT4AWE 2018 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health, Madeira, Portugal, 2018. xli Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [26] I. M. Pires, N. M. Garcia, N. Pombo, F. Flórez-Revuelta, and N. D. Rodríguez, "Validation Techniques for Sensor Data in Mobile Health Applications," Journal of Sensors, vol. 2016, pp. 1687-725, 2016 [27] I. M. Pires, R. Santos, N. Pombo, N. M. Garcia, F. Florez-Revuelta, S. Spinsante, et al., "Recognition of Activities of Daily Living Based on Environmental Analyses Using Audio Fingerprinting Techniques: A Systematic Review," Sensors (Basel), vol. 18, Jan 9 2018. doi: 10.3390/s18010160 [28] I. M. Pires, N. M. Garcia, and F. Flórez-Revuelta, "Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices," in Proceedings of the ECMLPKDD 2015 Doctoral Consortium, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, 2015. [29] I. M. Pires, N. M. Garcia, N. Pombo, F. Flórez-Revuelta, and S. Spinsante, "Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices," in Pervasive and Mobile Computing, vol. 47, pp. 78-93, 2018. doi: 10.1016/j.pmcj.2018.05.005 [30] I. M. Pires, M. C. Teixeira, N. Pombo, N. M. Garcia, F. Flórez-Revuelta, S. Spinsante, et al., "Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions," in Open Bioinformatics Journal, vol. 11, pp. 61-88, Bentham Science Publishers B.V.. doi: 10.2174/1875036201811010061 [31] L. H. A. Salazar, T. Lacerda, J. V. Nunes, and C. Gresse von Wangenheim, "A Systematic Literature Review on Usability Heuristics for Mobile Phones," International Journal of Mobile Human Computer Interaction, vol. 5, pp. 50-61, 2013. doi: 10.4018/jmhci.2013040103 [32] D. Foti and J. S. Koketsu, "Activities of daily living," Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction, vol. 7, pp. 157-232, 2013 [33] L. Lim, A. Misra, and T. Mo, "Adaptive data acquisition strategies for energy-efficient, smartphone-based, continuous processing of sensor streams," Distributed and Parallel Databases, vol. 31, pp. 321-351, 2012. doi: 10.1007/s10619-012-7093-3 [34] K. Dolui, S. Mukherjee, and S. K. Datta, "Smart Device Sensing Architectures and Applications," 2013 International Computer Science and Engineering Conference (Icsec), pp. 91-96, 2013. doi: 10.1109/icsec.2013.6694759 xlii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [35] A. Bujari, B. Licar, and C. E. Palazzi, "Movement pattern recognition through smartphone's accelerometer," in Consumer Communications and Networking Conference (CCNC), 2012 IEEE, Las Vegas, NV, 2012, pp. 502-506. [36] B. Chikhaoui, S. Wang, and H. Pigot, "A Frequent Pattern Mining Approach for ADLs Recognition in Smart Environments," in Advanced Information Networking and Applications (AINA), 2011 IEEE International Conference on, Biopolis, 2011, pp. 248255. [37] I. Kouris and D. Koutsouris, "A comparative study of pattern recognition classifiers to predict physical activities using smartphones and wearable body sensors," Technol Health Care, vol. 20, pp. 263-75, 2012. doi: 10.3233/THC-2012-0674 [38] C. Zhu and W. Sheng, "Realtime recognition of complex human daily activities using human motion and location data," IEEE Trans Biomed Eng, vol. 59, pp. 2422-30, Sep 2012. doi: 10.1109/TBME.2012.2190602 [39] K. Farrahi and D. Gatica-Perez, "Daily Routine Classification from Mobile Phone Data," in Machine Learning for Multimodal Interaction. vol. 5237, ed: Springer Berlin Heidelberg, 2008, pp. 173-184. [40] J.-H. Hong, J. Ramos, C. Shin, and A. K. Dey, "An Activity Recognition System for Ambient Assisted Living Environments," in Evaluating AAL Systems Through Competitive Benchmarking. vol. 362, ed: Springer Berlin Heidelberg, 2013, pp. 148158. [41] S. Phithakkitnukoon, T. Horanont, G. Lorenzo, R. Shibasaki, and C. Ratti, "ActivityAware Map: Identifying Human Daily Activity Pattern Using Mobile Phone Data," in Human Behavior Understanding. vol. 6219, ed: Springer Berlin Heidelberg, 2010, pp. 14-25. [42] Y. Nam, S. Rho, and C. Lee, "Physical activity recognition using multiple sensors embedded in a wearable device," ACM Transactions on Embedded Computing Systems, vol. 12, pp. 1-14, 2013. doi: 10.1145/2423636.2423644 [43] P. Vateekul and K. Sarinnapakorn, "Tree-Based Approach to Missing Data Imputation," presented at the Data Mining Workshops, 2009. ICDMW '09. IEEE International Conference on Miami, FL, 2009. [44] W. Ling and F. Dong-Mei, "Estimation of Missing Values Using a Weighted K-Nearest Neighbors Algorithm," presented at the Environmental Science and Information Application Technology, 2009. ESIAT 2009. International Conference on, Wuhan, 2009. xliii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [45] N. Pombo, K. Bousson, P. Araujo, and J. Viana, "Medical decision-making inspired from aerospace multisensor data fusion concepts," Inform Health Soc Care, vol. 40, pp. 185-97, 2015. doi: 10.3109/17538157.2013.872113 [46] F. Tanveer, O. T. Waheed, and Atiq-ur-Rehman, "Design and Development of a Sensor Fusion based Low Cost Attitude Estimator," Journal of Space Technology,, vol. 1, pp. 45-50, Jun 2011 [47] V. Graizer, "Effect of low-pass filtering and re-sampling on spectral and peak ground acceleration in strong-motion records," in Proc. 15th World Conference of Earthquake Engineering, Lisbon, Portugal, 2012, pp. 24-28. [48] C. Rader and N. Brenner, "A new principle for fast Fourier transformation," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 24, pp. 264-266, 1976. doi: 10.1109/tassp.1976.1162805 [49] P. J. O. Doets, Modeling Audio Fingerprints: Structure, Distortion, Capacity: TU Delft, Delft University of Technology, 2010 [50] P. Cano, E. Batle, T. Kalker, and J. Haitsma, "A review of algorithms for audio fingerprinting," presented at the 2002 IEEE Workshop on Multimedia Signal Processing, 2002. [51] P. Cano, E. Batlle, E. Gómez, L. de C.T.Gomes, and M. Bonnet, "Audio Fingerprinting: Concepts And Applications," vol. 2, pp. 233-245, 2005. doi: 10.1007/10966518_17 [52] P. Cano, E. Batlle, T. Kalker, and J. Haitsma, "A Review of Audio Fingerprinting," Journal of VLSI signal processing systems for signal, image and video technology, vol. 41, pp. 271-284, 2005. doi: 10.1007/s11265-005-4151-3 [53] P. Doets, M. M. Gisbert, and R. Lagendijk, "On the comparison of audio fingerprints for extracting quality parameters of compressed audio," in Electronic Imaging 2006, 2006, pp. 60720L-60720L-12. [54] H. Kekre, N. Bhandari, N. Nair, P. Padmanabhan, and S. Bhandari, "A Review of Audio Fingerprinting and Comparison of Algorithms," International Journal of Computer Applications, vol. 70, 2013 [55] Google. (2012, 2nd February). Google Code Archive - Long-term storage for Google Code Project Hosting. Available: https://code.google.com/archive/p/musicg/ [56] Neuroph. (2017, 2 Sep. 2017). Java Neural Network Framework Neuroph. Available: http://neuroph.sourceforge.net/ xliv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [57] H. Research. (2017, 2 Sep. 2017). Encog Machine Learning Framework. Available: http://www.heatonresearch.com/encog/ [58] A. Chris Nicholson. (2017, 2 Sep. 2017). Deeplearning4j: Open-source, Distributed Deep Learning for the JVM. Available: https://deeplearning4j.org/ [59] ALLab. (2017, September 2nd). August 2017- Multi-sensor data fusion in mobile devices for the identification of activities of daily living - ALLab Signals. Available: https://allab.di.ubi.pt/mediawiki/index.php/August_2017-_Multisensor_data_fusion_in_mobile_devices_for_the_identification_of_activities_of_daily_l iving xlv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living xlvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Abstract Following the recent advances in technology and the growing use of mobile devices such as smartphones, several solutions may be developed to improve the quality of life of users in the context of Ambient Assisted Living (AAL). Mobile devices have different available sensors, e.g., accelerometer, gyroscope, magnetometer, microphone and Global Positioning System (GPS) receiver, which allow the acquisition of physical and physiological parameters for the recognition of different Activities of Daily Living (ADL) and the environments in which they are performed. The definition of ADL includes a well-known set of tasks, which include basic selfcare tasks, based on the types of skills that people usually learn in early childhood, including feeding, bathing, dressing, grooming, walking, running, jumping, climbing stairs, sleeping, watching TV, working, listening to music, cooking, eating and others. On the context of AAL, some individuals (henceforth called user or users) need particular assistance, either because the user has some sort of impairment, or because the user is old, or simply because users need/want to monitor their lifestyle. The research and development of systems that provide a particular assistance to people is increasing in many areas of application. In particular, in the future, the recognition of ADL will be an important element for the development of a personal digital life coach, providing assistance to different types of users. To support the recognition of ADL, the surrounding environments should be also recognized to increase the reliability of these systems. The main focus of this Thesis is the research on methods for the fusion and classification of the data acquired by the sensors available in off-the-shelf mobile devices in order to recognize ADL in almost real-time, taking into account the large diversity of the capabilities and characteristics of the mobile devices available in the market. In order to achieve this objective, this Thesis started with the review of the existing methods and technologies to define the architecture and modules of the method for the identification of ADL. With this review and based on the knowledge acquired about the sensors available in off-the-shelf mobile devices, a set of tasks that may be reliably identified was defined as a basis for the remaining research and development to be carried out in this Thesis. This review also identified the main stages for the development of a new method for the identification of the ADL using the sensors available in off-the-shelf mobile devices; these stages are data acquisition, data processing, data cleaning, data imputation, feature extraction, data fusion and artificial intelligence. One of the challenges is related to the different types of data acquired from the different sensors, but other challenges were found, including the presence of environmental noise, the positioning of the mobile device during the daily activities, the limited capabilities of the mobile devices and others. Based on the acquired data, the processing was performed, implementing data cleaning and feature extraction methods, in order to define a new framework for the xlvii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living recognition of ADL. The data imputation methods were not applied, because at this stage of the research their implementation does not have influence in the results of the identification of the ADL and environments, as the features are extracted from a set of data acquired during a defined time interval and there are no missing values during this stage. The joint selection of the set of usable sensors and the identifiable set of tasks will then allow the development of a framework that, considering multi-sensor data fusion technologies and context awareness, in coordination with other information available from the user context, such as his/her agenda and the time of the day, will allow to establish a profile of the tasks that the user performs in a regular activity day. The classification method and the algorithm for the fusion of the features for the recognition of ADL and its environments needs to be deployed in a machine with some computational power, while the mobile device that will use the created framework, can perform the identification of the ADL using a much less computational power. Based on the results reported in the literature, the method chosen for the recognition of the ADL is composed by three variants of Artificial Neural Networks (ANN), including simple Multilayer Perceptron (MLP) networks, Feedforward Neural Networks (FNN) with Backpropagation, and Deep Neural Networks (DNN). Data acquisition can be performed with standard methods. After the acquisition, the data must be processed at the data processing stage, which includes data cleaning and feature extraction methods. The data cleaning method used for motion and magnetic sensors is the low pass filter, in order to reduce the noise acquired; but for the acoustic data, the Fast Fourier Transform (FFT) was applied to extract the different frequencies. When the data is clean, several features are then extracted based on the types of sensors used, including the mean, standard deviation, variance, maximum value, minimum value and median of raw data acquired from the motion and magnetic sensors; the mean, standard deviation, variance and median of the maximum peaks calculated with the raw data acquired from the motion and magnetic sensors; the five greatest distances between the maximum peaks calculated with the raw data acquired from the motion and magnetic sensors; the mean, standard deviation, variance, median and 26 MelFrequency Cepstral Coefficients (MFCC) of the frequencies obtained with FFT based on the raw data acquired from the microphone data; and the distance travelled calculated with the data acquired from the GPS receiver. After the extraction of the features, these will be grouped in different datasets for the application of the ANN methods and to discover the method and dataset that reports better results. The classification stage was incrementally developed, starting with the identification of the most common ADL (i.e., walking, running, going upstairs, going downstairs and standing activities) with motion and magnetic sensors. Next, the environments were identified with acoustic data, i.e., bedroom, bar, classroom, gym, kitchen, living room, hall, street and library. After the environments are recognized, and based on the different sets of sensors commonly available in the mobile devices, the data acquired from the motion and magnetic sensors were combined with the recognized environment in order to differentiate some activities without motion, i.e., sleeping and watching TV. The number of xlviii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living recognized activities in this stage was increased with the use of the distance travelled, extracted from the GPS receiver data, allowing also to recognize the driving activity. After the implementation of the three classification methods with different numbers of iterations, datasets and remaining configurations in a machine with high processing capabilities, the reported results proved that the best method for the recognition of the most common ADL and activities without motion is the DNN method, but the best method for the recognition of environments is the FNN method with Backpropagation. Depending on the number of sensors used, this implementation reports a mean accuracy between 85.89% and 89.51% for the recognition of the most common ADL, equals to 86.50% for the recognition of environments, and equals to 100% for the recognition of activities without motion, reporting an overall accuracy between 85.89% and 92.00%. The last stage of this research work was the implementation of the structured framework for the mobile devices, verifying that the FNN method requires a high processing power for the recognition of environments and the results reported with the mobile application are lower than the results reported with the machine with high processing capabilities used. Thus, the DNN method was also implemented for the recognition of the environments with the mobile devices. Finally, the results reported with the mobile devices show an accuracy between 86.39% and 89.15% for the recognition of the most common ADL, equal to 45.68% for the recognition of environments, and equal to 100% for the recognition of activities without motion, reporting an overall accuracy between 58.02% and 89.15%. Compared with the literature, the results returned by the implemented framework show only a residual improvement. However, the results reported in this research work comprehend the identification of more ADL than the ones described in other studies. The improvement in the recognition of ADL based on the mean of the accuracies is equal to 2.93%, but the maximum number of ADL and environments previously recognized was 13, while the number of ADL and environments recognized with the framework resulting from this research is 16. In conclusion, the framework developed has a mean improvement of 2.93% in the accuracy of the recognition for a larger number of ADL and environments than previously reported. In the future, the achievements reported by this PhD research may be considered as a start point of the development of a personal digital life coach, but the number of ADL and environments recognized by the framework should be increased and the experiments should be performed with different types of devices (i.e., smartphones and smartwatches), and the data imputation and other machine learning methods should be explored in order to attempt to increase the reliability of the framework for the recognition of ADL and its environments. xlix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Keywords Sensor data fusion, data collection, accelerometer, gyroscope, magnetometer, microphone, Global Positioning System (GPS) receiver, signal processing, sensors signal, activities of daily living, mobile devices, pattern recognition, data processing, data cleaning, data imputation, feature extraction, artificial intelligence, machine learning, artificial neural networks. l Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Contents Dedicatory .................................................................................................... v Acknowledgements ....................................................................................... vii Foreword .................................................................................................... xi List of Publications ...................................................................................... xiii List of articles included in the thesis resulting from this 4-year doctoral research programme ............................................................................................. xiii Other publications resulting from the doctoral research programme not included in the thesis ..................................................................................................... xiv Resumo .................................................................................................... xvii Resumo Alargado ......................................................................................... xxi Abstract ................................................................................................... xlvii Contents ..................................................................................................... li List of Figures .............................................................................................. lv List of Tables.............................................................................................. lvii Acronyms ................................................................................................... lxi Chapter 1 Introduction ................................................................................................. 1 1. Thesis Focus and Scope ........................................................................... 1 2. Problem Definition and Research Objectives .................................................. 3 3. Thesis Statement ................................................................................... 5 4. Main Contributions ................................................................................. 5 5. Thesis Organization ................................................................................ 6 References ................................................................................................. 7 Chapter 2 State-of-the-art ........................................................................................... 11 1. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices ........................... 11 Abstract ............................................................................................... 13 li Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 1. Introduction ................................................................................... 13 2. Sensors ......................................................................................... 15 3. Sensor Data Fusion in Mobile Devices ...................................................... 23 4. Application in Ambient Assisted Living .................................................... 30 5. Conclusions .................................................................................... 30 References ........................................................................................... 32 2. Validation Techniques for Sensor Data in Mobile Health Applications ................... 41 Abstract ............................................................................................... 43 1. Introduction ................................................................................... 43 2. Data Validation Methods ..................................................................... 44 3. Classification of Data Validation Methods ................................................ 46 4. Applicability of the Sensor Data Validation Methods .................................... 47 5. Conclusion ..................................................................................... 49 References ........................................................................................... 49 3. Audio Fingerprinting Techniques for detection of the environment of Activities of Daily Living (ADL): A Systematic Review ............................................................ 53 Abstract ............................................................................................... 55 1. Introduction ................................................................................... 56 2. Methodology ................................................................................... 56 3. Results .......................................................................................... 63 4. Discussion ...................................................................................... 70 5. Conclusions .................................................................................... 74 References ........................................................................................... 75 Chapter 3 Framework for the Identification of Activities of Daily Living ................................... 79 1. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices ....................................................... 79 Abstract ............................................................................................... 81 1. Introduction ................................................................................... 81 2. Related Work .................................................................................. 83 3. Methods and Expected Results.............................................................. 92 4. Discussion and Conclusions .................................................................. 95 References ........................................................................................... 96 Chapter 4 Data fusion for the Identification of Activities of Daily Living ................................. 103 1. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices .............................................................. 103 lii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Abstract .............................................................................................. 105 1. Introduction .................................................................................. 105 2. Related Work ................................................................................. 106 3. Methods ....................................................................................... 108 4. Results ......................................................................................... 112 5. Discussion ..................................................................................... 115 6. Conclusions ................................................................................... 118 References .......................................................................................... 118 2. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions ......................................................... 121 Abstract .............................................................................................. 123 1. Introduction .................................................................................. 124 2. Methods ....................................................................................... 130 3. Results ......................................................................................... 133 4. Discussion ..................................................................................... 137 5. Conclusions ................................................................................... 145 References .......................................................................................... 146 Chapter 5 Conclusions and Future Work ........................................................................ 151 1. Final Conclusions ................................................................................ 151 2. Future Work ...................................................................................... 154 liii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living liv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living List of Figures Chapter 2 State-of-the-art 1. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices Figure 1. Schema of a multi-sensor mobile system to recognize activities of daily living .............................................................................. 15 2. Validation Techniques for Sensor Data in Mobile Health Applications Figure 1. Sequence of activities performed during the data validation process . 45 Figure 2. Different categories of data validation methods .......................... 48 3. Audio Fingerprinting Techniques for detection of the environment of Activities of Daily Living (ADL): A Systematic Review Figure 1. Flow diagram of the identification and inclusion papers ................. 64 Chapter 3 Framework for the Identification of Activities of Daily Living 1. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices Figure 1. Schema for the framework for the recognition of Activities of Daily Living (ADL) ...................................................................... 90 Figure 2. Sensors used for the recognition of Activities of Daily Living (ADL) and environments for each phase of development ............................. 94 Chapter 4 Data fusion of the Identification of Activities of Daily Living 1. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices Figure 1. Simplified diagram for the framework for the identification of ADL .. 111 Figure 2. Schema of the architecture of the different implementations of ANN used in this study .............................................................. 113 Figure 3. Results obtained with Neuroph framework for the different datasets of accelerometer and magnetometer sensors ................................ 114 Figure 4. Results obtained with Encog framework for the different datasets of accelerometer and magnetometer sensors ................................ 114 Figure 5. Results obtained with DeepLearning4j framework for the different datasets of accelerometer and magnetometer sensors. ................. 114 Figure 6. Results obtained with Neuroph framework for the different datasets of accelerometer, magnetometer and gyroscope sensors .................. 115 lv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Figure 7. Results obtained with Encog framework for the different datasets of accelerometer, magnetometer and gyroscope sensors .................. 115 Figure 8. Results obtained with DeepLearning4j framework for the different datasets of accelerometer, magnetometer and gyroscope sensors .... 115 2. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions Figure 1. Schema of the classification stage of the framework for the recognition of ADL and their environments .............................................. 132 Figure 2. Design of the interface of the Android application developed ......... 133 Figure 3. Comparison between the minimum, maximum and average of the accuracies obtained in the literature review with the common ADL and environments recognized by the Android library developed. ........... 142 lvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living List of Tables Chapter 2 State-of-the-art 1. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices Table 1. Services classified by categories .............................................. 17 Table 2. Examples of data acquisition methods. ...................................... 19 Table 3. Data processing: architectures and methods................................ 21 Table 4. Examples of data imputation methods. ...................................... 22 Table 5. Advantages and disadvantages of the sensor data fusion methods. ..... 24 Table 6. Examples of sensor data fusion methods. ................................... 29 2. Validation Techniques for Sensor Data in Mobile Health Applications Table 1. Classification of the data validation methods by functionality .......... 47 3. Audio Fingerprinting Techniques for detection of the environment of Activities of Daily Living (ADL): A Systematic Review Table 1. Study analysis .................................................................... 57 Table 2. Study summaries ................................................................. 61 Table 3. Distribution of the features extracted in the studies ...................... 70 Table 4. Distribution of the methods implemented in the studies ................. 71 Table 5. Potential accuracies for the top most accurate methods vs top most mean accurate features ....................................................... 73 Chapter 3 Framework for the Identification of Activities of Daily Living 1. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices Table 1. List of sensors available in mobile devices .................................. 83 Table 2. Summary of data acquisition methods ....................................... 84 Table 3. Relation between the types of sensors and the data cleaning techniques allowed. .......................................................................... 86 Table 4. Relation between the sensors and the extracted features ............... 88 Table 5. Relation between the different types of sensor and some data fusion methods .......................................................................... 89 Table 6. Relation between the different types of sensors and some pattern recognition methods ........................................................... 91 Table 7. Sensors, Activities of Daily Living (ADL) and environments for recognition with the framework proposed ................................................ 94 lvii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 4 Data fusion of the Identification of Activities of Daily Living 1. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices Table 1. Distribution of the ADL extracted in the studies analysed. .............. 108 Table 2. Distribution of the features extracted in the studies analysed. ........ 109 Table 3. Distribution of the classification methods used in the studies analysed. ................................................................................... 109 Table 4. Configurations of the ANN methods implemented. ....................... 112 Table 5. Best accuracies obtained with the different frameworks, datasets and number of iterations. ......................................................... 116 Table 6. Best accuracies obtained with the different frameworks, datasets and number of iterations .......................................................... 116 Table 7. Best accuracies achieved by the method using only the accelerometer sensor ............................................................................ 116 Table 8. Comparison between the best results achieved only using the accelerometer sensors, and using the accelerometer and magnetometer sensors ........................................................................... 117 Table 9. Comparison between the best results achieved only using the accelerometer sensors, and using the accelerometer, magnetometer and gyroscope sensors ........................................................ 117 Table 10. Comparison between the best results achieved only using the accelerometer and magnetometer sensors, and using the accelerometer and magnetometer sensors ................................................... 117 2. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions Table 1. Summary of the literature studies related to ADL recognition based on the ANN method. .............................................................. 126 Table 2. Summary of the literature studies related to ADL recognition based on the ANN method (cont.) ...................................................... 127 Table 3. Summary of the literature studies related to environment recognition based on the ANN method .................................................... 127 Table 4. Summary of the literature studies related to environment recognition based on the ANN method (cont.)........................................... 128 Table 5. Summary of the literature studies related to environment recognition based on the ANN method (cont.)........................................... 128 Table 6. Average accuracy of the ADL recognized ................................... 128 Table 7. Average accuracy of the environments recognized ....................... 129 Table 8. Configurations of the classification methods .............................. 132 Table 9. Analysis of the results of the stage 1 with accelerometer sensor ...... 134 lviii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Table 10. Analysis of the results of the stage 1 with accelerometer and magnetometer sensors ........................................................ 134 Table 11. Analysis of the results of the stage 1 with accelerometer, magnetometer and gyroscope sensors ...................................... 135 Table 12. Analysis of the results of the stage 2 using FNN method with Backpropagation ............................................................... 135 Table 13. Analysis of the results of the stage 2 using FNN method with Backpropagation and DNN method .......................................... 135 Table 14. Analysis of the results of the stage 2 using DNN method ............... 136 Table 15. Analysis of the overall recognition of ADL with accelerometer and environment recognized ...................................................... 136 Table 16. Analysis of the overall recognition of ADL with accelerometer, magnetometer and environment recognized .............................. 136 Table 17. Analysis of the overall recognition of ADL with accelerometer, magnetometer, gyroscope and environment recognized ................ 136 Table 18. Analysis of the overall recognition of ADL with accelerometer, GPS receiver and environment recognized ...................................... 137 Table 19. Analysis of the overall recognition of ADL with accelerometer, magnetometer, GPS receiver and environment recognized ............. 137 Table 20. Analysis of the overall recognition of ADL with accelerometer, magnetometer, gyroscope, GPS receiver and environment recognized ................................................................................... 137 Table 21. Analysis of the accuracies reported in studies [65-68] using the accelerometer sensor ......................................................... 137 Table 22. Analysis of the accuracies reported in studies [65-68] using the accelerometer and the magnetometer sensors ........................... 138 Table 23. Analysis of the accuracies reported in studies [65-68] using the accelerometer, the magnetometer and the gyroscope sensors ........ 138 Table 24. Analysis of the accuracies reported by the Android library using the accelerometer sensor ......................................................... 138 Table 25. Analysis of the accuracies reported by the Android library using the accelerometer and the magnetometer sensors ........................... 138 Table 26. Analysis of the accuracies reported by the Android library using the accelerometer, the magnetometer and the gyroscope sensors ........ 138 Table 27. Recognition of each ADL in stage 1 ........................................ 139 Table 28. Recognition of each environment in stage 2 .............................. 139 Table 29. Statistical analysis of the recognition of each ADL in the stage 3 ..... 140 Table 30. Statistical analysis of the recognition of each ADL in the stage 3 (cont.) ................................................................................... 140 Table 31. Statistical analysis of the recognition of each ADL in the framework 140 lix Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Table 32. Statistical analysis of the recognition of each ADL in the framework (cont.) ........................................................................... 141 Table 33. Comparison between the minimum, maximum and average of the accuracies obtained in the literature review and the accuracy of our results in the recognition of the selected ADL and environments. .... 142 Table 34. Normality tests ................................................................ 143 Table 35. Values of the Student's t-test for a sample mean for comparing our results and average accuracy ................................................ 143 Table 36. Normality tests ................................................................ 143 Table 37. Values of the Student's t-test for a sample mean for comparing our results with minimum, and our results with maximum accuracy....... 144 Table 38. Number of ADL/Environments recognized by the studies, where the minimum, maximum and average of the accuracies obtained in the literature review were verified .............................................. 144 Table 39. Normality tests ................................................................ 144 Table 40. Values of the Student's t-test for a sample mean for comparing our number of ADL/Environments with the average number of ADL/Environments recognized in the literature .......................... 144 lx Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Acronyms AAL Ambient Assisted Living AANN Auto-Associative Neural Network ACQUA Acquisition Cost-Aware Query Adaptation ADL Activities Of Daily Living AGO Accumulated Generating Operation ALHMM Adaptive Learning Hidden Markov Model ALLab Assisted Living Computing And Telecommunications Laboratory ANN Artificial Neural Networks AR Autoregressive ARMA Autoregressive Moving Averages AS Audio Signature ASF Audio Spectrum Flatness ASRS Automated Storage And Retrieval System ASV Algorithmic Sensor Validation AVI Automatic Vehicle Identification BBQ Barbie-Q BMW Balanced Multiwavelets BP Back Propagate BSW Binary Spray-And-Wait BSWF Binary Spray-And-Wait With Fusion C-SPINE Collaborative-Signal Processing In Node Environment CAALYX Complete Ambient Assisting Living Experiment CAN Controller Area Network CBSN Collaborative Body Sensor Networks CHG Continuous Hand Gestures CkWAS Correlation Based K-Weighted Angular Similarity CNT Carbon Nanotube COUPON Cooperative Framework For Building Sensing Maps In Mobile Opportunistic Networks CQT Constant Q Transform CRF Conditional Random Field CS Compressive Sampling D2FAS Decentralized Data Fusion And Active Sensing DAL Divide-And-Locate DBN Dynamic Bayesian Network lxi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living DCT Discrete Cosine Transform DD Diverse Density DFF Depth From Focus DFT Discrete Fourier Transform DNN Deep Neural Networks DNRF Drift And Noise Removal Filter DR Dead-Reckoning DSC Distributed Sparsity Classifier DTW Dynamic Time Warping ECG Electrocardiography EDA Electrodermal Activity EEG Electroencephalography EKF Extended Kalman Filter EM Expectation-Maximization EMD Empirical Mode Decomposition EMG Electromiography EOG Electrooculography EQ Estimation Quantization ER Epidemic Routing ERF Epidemic Routing With Fusion FCMimpute Fuzzy C-Means Clustering Imputation FDI Fault Detection And Isolation FFT Fast Fourier Transform FFTW Fastest Fourier Transform In The West FKL Fisher Kernel Learning FLk-NN Fourier And Lagged K-NN Combined System FNN Feedforward Neural Networks FR Frequency Response GBM Grey Bootstrap Method GDA Gaussian Discriminant Analysis GMM Gaussian Mixture Model GPS Global Positioning System GPU Graphic Processing Unit GTD Generalized Trend Diffusion HAS Human Auditory System HHMM Two-Level Hierarchical Hidden Markov Model HMM Hidden Markov Models HNR Harmonic-To-Noise Ratio HPF High Pass Filter HSV Heuristic Sensor Validation lxii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living IB Information Bottleneck IBL Instance-Based Learning IBV Index Bit Vectors IDFT Inverse Discrete Fourier Transform IEPE Integrated Electronic Piezoelectric IPTV Internet Protocol Television IR Infrared IRTR Improved Real-Time TV-Channel Recognition IT Instituto de Telecomunicações ITree Imputation Tree JDL Joint Directors Of Laboratories KD K-Dimensional KMI K-Means-Based Imputation KNN K-Nearest Neighbor KNNimpute K-Nearest Neighbour Imputation KPCA Kernel Principal Component Analysis LDA Linear Description Analysis LDB Local Discriminant Bases LEC Local Energy Centroid LMCE Local Maximum Chroma Energy LMCLT Logarithmic Module Complex Lapped Transform LMT Logistic Model Trees LPC Linear Predictive Coding LPCC Linear Prediction Coefficient Derived Cepstral Coefficients LPF Low-Pass Filter LSM Least Squares Method LSTM Long Short Term Memory MA Moving Average MAR Missing At Random MASK Masked Audio Spectral Keypoints MCAR Missing Completely At Random MDCT Modified Discrete Cosine Transform MEI Mean Imputation MEMS Micro-Electro-Mechanical Systems MFCC Mel-Frequency Cepstral Coefficients MICE Multivariate Imputation By Chained Equations MKNNimpute K-Nearest Neighbour Imputation Method Based On Mahalanobis Distance MLP Multilayer Perceptron MMF Multi-Matrices Factorization Model MNAR Missing Not At Random lxiii Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living MP Matching Pursuit MPT Missing Pattern Tree MSDF Mobile-Sensor Data Fusion MSE Mean Squared Error MV Magnitude Of Vector NFC New Field Communication NTP Network Time Protocol OSGi Open Services Gateway Initiative PARs Personal Audio Recordings PAT Phone Acceleration Threshold PCA Principal Component Analysis PCM Pulse Code Modulation PDF Probability Density Function PDLC Personal Digital Life Coach PIR Passive Infra-Red PNN Probabilistic Neural Networks PPR Phone Pattern Recognition PRH Philips Robust Hash PSN Personal Navigation System QDA Quadratic Discriminant Analysis QIFFT Quadratically Interpolated Fast Fourier Transform QoS Quality Of Service QUC Query Context RBA Rule-Based Algorithms RBF Radial Basis Function RBFNN Radial Basis Function Neural Network RBUKF Rao-Blackwellization Unscented Kalman Filter RFID Radio-Frequency Identifier RFS Rough And Fuzzy Sets RMS Root-Mean-Square RNN Recurrent Neural Networks RRP Random Recursive Partitioning RSE Relative Spectral Entropy RTLS Real-Time Location Systems RVM Relevance Vector Machine SAD Speech Activity Detection SAF Streaming Audio Fingerprinting SEVA Self-Validating SKNNimpute Sequential K-Nearest Neighbour Method-Based Imputation SLA Statistical Learning Algorithms lxiv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living SMO Sequential Minimal Optimization SMV Signal Magnitude Vector SNR Signal-To-Noise Ratio SOM Self-Organizing Maps SPB Symmetric Pairwise Boosting SPR Surface Plasmon Resonance SPRT Sequential Probability Ratio Test SRR Search By Range Reduction SSC Spectral Subband Centroid STFT Short-Time Fourier Transform SVD Singular Value Decomposition SVDD Support Vector Data Description SVM Support Vector Machines SWNC Sensor Weighted Network Classifier TBA Threshold Based Algorithm TO-Combo- Threshold Optimized Combo Speech Activity Detection SAD TSM Time-Scale Modification TSS Two-Stage Search UBI Universidade da Beira Interior UKF Unscented Kalman Filter VLSI Very Large-Scale Integration VRFV Validated Random Fuzzy Variable VRS Video Retrieval System VU Validated Uncertainty WIP Walking-In-Place WMA Weighted Moving Average WSN Wireless Sensor Network WTPR Watch Threshold And Pattern Recognition ZCR Zero-Crossing-Rate lxv Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living lxvi Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 1 Introduction This Thesis addresses the problem of the automatic recognition of Activities of Daily Living (ADL) [1] and the environments in which they are performed based on the fusion of the data acquired by the sensors available in off-the-shelf mobile devices, such as the accelerometer, the gyroscope, the magnetometer, the microphone and the Global Positioning System (GPS) receiver [2]. As a result, this Thesis proposes an architecture for a framework to perform the recognition of ADL and its environments. The focus, scope and research objectives of the Thesis are described in this chapter, followed by the Thesis statement, the main contributions and the Thesis organization. 1. Thesis Focus and Scope Following a recurrent definition of the literature, the identification of ADL focuses on the identification of a well-known set of tasks that are basic self-care tasks, based on the kinds of skills that people usually learn in early childhood. These activities include feeding, bathing, dressing, grooming, moving without danger, and other simple tasks related to personal care and hygiene. On the context of Ambient Assisted Living (AAL) [3, 4], some individuals (henceforth called user or users) need particular assistance, either because the user has some sort of disability, or because the user is an older adult, or simply because the user needs or wants to monitor and train his/her lifestyle. In this later context, ADL include a wider range of tasks, not only related to personal care and hygiene, but also extended to social and professional tasks. The development of methods for the recognition of ADL is important to support the autonomy of older people, patients with chronic diseases, and users that may have some type of disability [5, 6]. However, it may be useful for everyone, including athletes and young users, as these solutions can be integrated in a tool for the monitoring and training of lifestyles [7]. To date, the use of sensors for the recognition of ADL and environments is a well studied topic [8-16]. However, there are no studies that define a structured architecture for the recognition of ADL and environments using a large set of sensors available in off-the-shelf mobile devices and for the recognition of a large set of ADL. The focus of this Thesis includes the design of a novel framework through the selection and development of the most adequate methods for each stage of the recognition of ADL and its environments, being these: Data Acquisition, Data 1 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Fusion, Data Processing, Data Cleaning, Feature Extraction and Classification; with a special focus in the methods related to Data Fusion and Classification, concluding on what are the most important aspects to address in order to obtain results with reliable accuracy. The definition of a set of tasks that may be reliably identified, includes sensing and measuring a large set of tasks and identifying those tasks that are reliably identified with the sensors available in the mobile devices. The joint selection of the set of usable sensors and the identifiable set of tasks allow the development of a tool that, considering multi-sensor data fusion technologies and context awareness, in coordination with other information available from the user context, such as his/her agenda and the time of the day, will further allow to establish a profile of the tasks that the user performs in a regular activity day. There are several challenges involved in the development of a novel framework for the recognition of ADL. For instance, related to Data Acquisition, the challenges include the incorrect positioning of the mobile device, the uncontrolled data sampling rate, the unavailability of all sensors needed to recognize a specific ADL, and the different environmental conditions [17]. There are several frameworks that may help to minimize these problems, including the ones referred in the article presented in chapter 2 [18]. In relation to Data Processing, the challenges are linked to the different architectures and platforms in which the solution may be implemented, such as for example, relying on server-side processing or on local processing. In order to provide a reliable result in several data collection environments where a network connection may not be available and to be adapted to the low processing, storage and memory resources of some mobile devices, the implementation of lightweight methods is also object of study of this Thesis [17, 19-21]. The scope of this Thesis consists in the use of data fusion and Artificial Neural Networks (ANN) methods for the recognition of ADL and its environments based on the data collected from sensors available in an off-the-shelf mobile device, such as a smartphone. The limitations of smartphones were also explored, creating a method that acquires 5 seconds of sensors’ data every 5 minutes, allowing the mapping of the lifestyles and handling the development of new solutions for AAL. The research part related to the classification methods of this Thesis is related to the comparison of different implementations of ANN methods adapted to the constrained resources of a mobile device, using local processing techniques, in order to provide feedback to the user related to the ADL performed and its environment in an almost real-time manner. 2 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 2. Problem Definition and Research Objectives The problem addressed in this Thesis is to segment and identify correctly a significant set of ADL that users perform as their daily or weekly routines, using the sensors and user information available on a commodity smartphone. The identification of ADL is a complex challenge, due to the existence of a large set of ADL and the complexity of the identification of the correct method to identify each one of them. To correctly detect and identify an ADL, various factors that influence the acquisition of the data and, consequently, affect the identification of several types of activities, have to be taken into consideration, such as the environmental problems, the involuntary movements that the user performs, the position of the mobile device, the movements of the mobile device during the data acquisition, and the frequency of the acquisition of data from the sensors. The recognition of ADL and the environment where the subject is at a given moment is important for the creation of systems aimed at several purposes, such as personal coaching, health monitoring, lifestyle monitoring and others. Nowadays, such recognition is commonly performed by complex and often expensive systems with high processing power and memory capabilities [3, 22-26]. However, to a reasonable extent, this task can be performed with offthe-shelf mobile devices, supported by local processing techniques. To date, and based on the available literature, there are several studies using different subsets of sensors available in off-the-shelf mobile devices. However, studies reporting methods that rely on a larger set of sensors available in mobile devices for the recognition of ADL and its environments were not found. Currently, existing literature reports the use of Artificial Neural Networks (ANN) and their variants as the most used methods, claiming relevant accuracies for the identification of some ADL, as presented in [18]. The main objective of this Thesis is the creation of a new method and framework for the recognition of ADL and its environments with the fusion and classification of the data acquired from a large set of sensors available in the off-the-shelf mobile devices. The developed method is a function of the number of sensors available on the different mobile devices that should be executed with lightweight algorithms in order to allow its execution locally in the mobile devices [27]. The developed method includes several stages, including data acquisition, data processing, data cleaning, feature extraction and pattern recognition. The ADL that were identified during this Thesis are walking, running, going upstairs, standing/watching TV, going downstairs, driving and sleeping, and environments that were identified during this Thesis were bedroom, bar, classroom, gym, kitchen, living room, hall, street and library. The following intermediate objectives were defined in order to divide and organize the research work required to accomplish the main objective of this thesis: 3 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 1. In order to understand the existing solutions and possible methods, the Thesis started with the review of the state of the art related to the different concepts included in this Thesis. Additionally, due to the types of data used in this work, this review also included the study of data validation and audio fingerprinting methods. Finally, the useful sensors, and the ADL and their environments that can be identified with the data acquired from off-the-shelf mobile devices were described. 2. As second intermediate objective, the proposed architecture of the framework for the recognition of ADL and environments was defined, including the different concepts available in the literature. 3. As third intermediate objective, the development of a mobile application that acquires the data from the sensors available in mobile devices has been performed. Some users were selected to perform ADL and label them. For this data set the mobile device was worn in the front pocket of the user’s trousers. This mobile application performs the data acquisition process. 4. After data acquisition, the fourth intermediate objective consists in the implementation of data processing methods, including data cleaning and feature extraction. Data imputation methods were avoided, because their implementation does not have influence in the results of the identification of the ADL and environments, because the features are measured with a data set acquired during a defined time interval. 5. After the processing of the data, the fifth intermediate objective is related to the implementation and comparison of three different configurations of ANN in order to discover the best configuration for the recognition of ADL and environments. Additionally, this analysis included the fusion of the different combinations of sensors’ data for the implementation of different methodologies. 6. The sixth intermediate objective consists in the implementation of methods addressing the different stages of the framework for the recognition of ADL and environments, implementing the framework in a mobile application for further testing of its performance. The recognition of ADL and environments implemented as results of this Thesis is important to design a personal digital life coach [7]. It is also important to support the autonomy of older users, patients with chronic diseases and users that may have some type of disability [5, 6]. Therefore, it may be useful for everyone, including athletes and young users, as the proposed framework can be integrated in a tool for the monitoring and training of lifestyles [7]. 4 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 3. Thesis Statement This Thesis proposed a new framework for the recognition of ADL and environments, fusing the data acquired with the sensors available in off-the-shelf mobile devices. Specifically, the Thesis statement is: The automatic identification of the Activities of Daily Living (ADL) and environments is a complex challenge widely studied in the literature, but there are no structured frameworks for the recognition of ADL. There are different types of data that can be acquired with mobile devices, where the features should be defined and fused for the increasing reliability of the recognition of ADL and environments. Data fusion and classification are the most important steps in the development of a method for the recognition of ADL and environments. The positioning of the mobile device and the timeframe of the data acquisition are also important factors that should be taken in account. To support this Thesis statement, the literature review about the different concepts involved in this research was done, including data acquisition, data processing, data validation, data cleaning, data imputation, feature extraction, audio fingerprinting, data fusion and classification methods, proposing an architecture for a framework to recognize the ADL and environments. After that, the data acquisition have been performed for the ADL and environments selected. When all data sets were available, the implementation of the different methods for each stage of the framework was performed, comparing three different types of ANN for the creation of the different methods with the different subsets of sensors. At the end, the framework was implemented in a mobile application for further testing. 4. Main Contributions This section briefly describes the main scientific contributions resulting from the research work presented in this Thesis. The first contribution is the proposal of a framework for the recognition of ADL and it also describes the existing methods related to each concept related to this Thesis, including sensor classification, data acquisition, data processing, data imputation and sensor data fusion in mobile devices, and the applicability of the recognition of ADL [18]. The second contribution of this Thesis presents the existing methods for the validation of the data acquired from sensors, their classification and their applicability based on the different types of missing and/or invalid data [28]. 5 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living The third contribution of this Thesis presents the existing audio fingerprinting methods used for the recognition of the environments based on the different acoustic signals, discussing the accuracy of the features and methods implemented [29]. The fourth contribution of this Thesis details the architecture of the proposed framework for the recognition of ADL and their environments, proposing possible methods for each module of the framework [30]. The fifth contribution of this Thesis presents the research about the recognition of walking, standing, running, going upstairs and going downstairs using the data acquired from the accelerometer, magnetometer and gyroscope sensors, comparing three different implementations of ANN with normalized and non-normalized data in order to discover the best implementation for the recognition of these ADL. The conclusion is that the best method is the DNN method with normalized data [31]. The sixth contribution of this Thesis presents the results of the implementation of an Android library with the framework for the recognition of ADL and environments based on the results reported by previous studies, verifying the reliability of the framework proposed [32]. 5. Thesis Organization This Thesis is organized as follows: Chapter 1: Introduction A brief introduction to the Thesis is presented including the focus and scope, Thesis objectives, Thesis statement, and main contributions of the work carried out. Chapter 2: State-of-the-art The main concepts involved in this PhD Thesis are presented and discussed, including data acquisition, data processing, data validation, data cleaning, data imputation, feature extraction, data fusion and classification methods. Additionally, the methods for the processing of the acoustic data were also presented and discussed. These concepts are presented in three papers. Chapter 3: Framework for the Identification of Activities of Daily Living This chapter presents the proposed architecture and possible methods for the development of the framework for the recognition of Activities of Daily Living. The proposed architecture of the framework is presented in one paper. 6 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 4: Data fusion of the Identification of Activities of Daily Living This chapter presents and discusses the use of Artificial Neural Networks (ANN) and the comparison between three different implementations for the recognition of ADL and its environments. This chapter also presents the implementation of the different stages of the framework for these recognitions. The implementation of the framework is presented in two papers, as a small overview of the implementation of the framework for the recognition of ADL and its environments. Chapter 5: Conclusion and Future Work The results presented throughout the Thesis are discussed and the main achievements are summarized, pointing directions for the future. References [1] D. Foti and J. S. Koketsu, "Activities of daily living," Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction, vol. 7, pp. 157-232, 2013 [2] L. H. A. Salazar, T. Lacerda, J. V. Nunes, and C. Gresse von Wangenheim, "A Systematic Literature Review on Usability Heuristics for Mobile Phones," International Journal of Mobile Human Computer Interaction, vol. 5, pp. 50-61, 2013. doi: 10.4018/jmhci.2013040103 [3] C. Dobre, C. x. Mavromoustakis, N. Garcia, R. I. Goleva, and G. Mastorakis, Ambient Assisted Living and Enhanced Living Environments: Principles, Technologies and Control: Butterworth-Heinemann, 2016 [4] N. M. Garcia and J. J. P. Rodrigues, Ambient assisted living: CRC Press, 2015 [5] M. Kara, O. Lamouchi, and A. Ramdane-Cherif, "A Quality Model for the Evaluation AAL Systems," Procedia Computer Science, vol. 113, pp. 392-399, 2017/01/01/ 2017. doi: https://doi.org/10.1016/j.procs.2017.08.354 [6] S. Spinsante, E. Gambi, L. Raffaeli, L. Montanini, L. Paciello, R. Bevilacqua, et al., "Technology-based assistance of people with dementia: state of the art, open challenges, and future developments," Human Monitoring, Smart Health and Assisted Living: Techniques and Technologies, vol. 9, p. 55, 2017 [7] N. M. Garcia, "A Roadmap to the Design of a Personal Digital Life Coach," in ICT Innovations 2015, pp. 21-27, Springer, 2016. 7 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [8] A. D’Ambrosio, M. Aria, and R. Siciliano, "Accurate Tree-based Missing Data Imputation and Data Fusion within the Statistical Learning Paradigm," Journal of Classification, vol. 29, pp. 227-258, 2012. doi: 10.1007/s00357-012-9108-1 [9] J. Dong, D. Zhuang, Y. Huang, and J. Fu, "Advances in multi-sensor data fusion: algorithms and applications," Sensors (Basel), vol. 9, pp. 7771-84, 2009. doi: 10.3390/s91007771 [10] R. C. King, E. Villeneuve, R. J. White, R. S. Sherratt, W. Holderbaum, and W. S. Harwin, "Application of data fusion techniques and technologies for wearable health monitoring," Med Eng Phys, vol. 42, pp. 1-12, Apr 2017. doi: 10.1016/j.medengphy.2016.12.011 [11] O. Banos, M. Damas, H. Pomares, and I. Rojas, "On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition," Sensors (Basel), vol. 12, pp. 8039-54, 2012. doi: 10.3390/s120608039 [12] M. A. A. Akhoundi and E. Valavi, "Multi-Sensor Fuzzy Data Fusion Using Sensors with Different Characteristics," arXiv preprint arXiv:1010.6096, 2010 [13] P. Paul and T. George, "An Effective Approach for Human Activity Recognition on Smartphone," 2015 Ieee International Conference on Engineering and Technology (Icetech), pp. 45-47, 2015. doi: 10.1109/icetech.2015.7275024 [14] Y.-W. Hsu, K.-H. Chen, J.-J. Yang, and F.-S. Jaw, "Smartphone-based fall detection algorithm using feature extraction," in 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Datong, China, 2016, pp. 1535-1540. [15] S. Dernbach, B. Das, N. C. Krishnan, B. L. Thomas, and D. J. Cook, "Simple and Complex Activity Recognition through Smart Phones," in 2012 8th International Conference on Intelligent Environments (IE), Guanajuato, Mexico, 2012, pp. 214-221. [16] C. Shen, Y. F. Chen, and G. S. Yang, "On Motion-Sensor Behavior Analysis for HumanActivity Recognition via Smartphones," in 2016 Ieee International Conference on Identity, Security and Behavior Analysis (Isba), Sendai, Japan, 2016, pp. 1-6. [17] S. D. Bersch, D. Azzi, R. Khusainov, I. E. Achumba, and J. Ries, "Sensor data acquisition and processing parameters for human activity classification," Sensors (Basel), vol. 14, pp. 4239-70, 2014. doi: 10.3390/s140304239 8 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [18] I. Pires, N. Garcia, N. Pombo, and F. Flórez-Revuelta, "From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices," Sensors, vol. 16, p. 184, 2016 [19] V. Pejovic and M. Musolesi, "Anticipatory Mobile Computing," ACM Computing Surveys, vol. 47, pp. 1-29, 2015. doi: 10.1145/2693843 [20] F. X. Lin, A. Rahmati, and L. Zhong, "Dandelion: a framework for transparently programming phone-centered wireless body sensor applications for health," in Wireless Health 2010, 2010, pp. 74-83. [21] O. Postolache, P. S. Girao, M. Ribeiro, M. Guerra, J. Pincho, F. Santiago, et al., "Enabling telecare assessment with pervasive sensing and Android OS smartphone," in 2011 IEEE International Workshop on Medical Measurements and Applications Proceedings (MeMeA), 2011, pp. 288-293. [22] C. Siegel, A. Hochgatterer, and T. E. Dorner, "Contributions of ambient assisted living for health and quality of life in the elderly and care services--a qualitative analysis from the experts' perspective of care service professionals," BMC Geriatr, vol. 14, p. 112, 2014. doi: 10.1186/1471-2318-14-112 [23] J. A. Botia, A. Villa, and J. Palma, "Ambient Assisted Living system for in-home monitoring of healthy independent elders," Expert Systems with Applications, vol. 39, pp. 8136-8148, 2012. doi: 10.1016/j.eswa.2012.01.153 [24] N. M. Garcia, J. J. P. C. Rodrigues, D. C. Elias, and M. S. Dias, Ambient Assisted Living: Taylor & Francis, 2014 [25] M. Huch, A. Kameas, J. Maitland, P. J. McCullagh, J. Roberts, A. Sixsmith, et al., Handbook of Ambient Assisted Living: Technology for Healthcare, Rehabilitation and Well-being - Volume 11 of Ambient Intelligence and Smart Environments: IOS Press, 2012 [26] R. I. Goleva, N. M. Garcia, C. X. Mavromoustakis, C. Dobre, G. Mastorakis, R. Stainov, et al., "AAL and ELE Platform Architecture," Ambient Assisted Living and Enhanced Living Environments, pp. 171-209, Elsevier, 2017 [27] I. M. Pires, N. M. Garcia, N. Pombo, and F. Flórez-Revuelta, "Limitations of the Use of Mobile Devices and Smart Environments for the Monitoring of Ageing People," in ICT4AWE 2018 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health, Madeira, Portugal, 2018. 9 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living [28] I. M. Pires, N. M. Garcia, N. Pombo, F. Flórez-Revuelta, and N. D. Rodríguez, "Validation Techniques for Sensor Data in Mobile Health Applications," Journal of Sensors, vol. 2016, pp. 1687-725, 2016 [29] I. M. Pires, R. Santos, N. Pombo, N. M. Garcia, F. Florez-Revuelta, S. Spinsante, et al., "Recognition of Activities of Daily Living Based on Environmental Analyses Using Audio Fingerprinting Techniques: A Systematic Review," Sensors (Basel), vol. 18, Jan 9 2018. doi: 10.3390/s18010160 [30] I. M. Pires, N. M. Garcia, N. Pombo, F. Florez-Revuelta, and S. Spinsante, "Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices," Sensors (Basel), vol. 18, Feb 21 2018. doi: 10.3390/s18020640 [31] I. M. Pires, N. M. Garcia, N. Pombo, F. Flórez-Revuelta, and S. Spinsante, "Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices," in Pervasive and Mobile Computing, vol. 47, pp. 78-93, 2018. doi: 10.1016/j.pmcj.2018.05.005 [32] I. M. Pires, M. C. Teixeira, N. Pombo, N. M. Garcia, F. Flórez-Revuelta, S. Spinsante, et al., "Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions," in Open Bioinformatics Journal, vol. 11, pp. 61-88, Bentham Science Publishers B.V.. doi: 10.2174/1875036201811010061 10 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 2 State-of-the-art This chapter is related to the state-of-the-art and it is composed by three articles, each presented in its section. These three articles provide a comprehensive review of the different areas addressed in this research. 1. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices The following article is the first part of chapter 2. From Data Acquisition to Data Fusion A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo and Francisco Flórez-Revuelta Sensors (MDPI), published, 2016. According to 2016 Journal Citation Reports published by Thomson Reuters in 2017, this journal has the following performance metrics: ISI Impact Factor (2016): 2.677 ISI Article Influence Score (2016): 0.6 Journal Ranking (2016): 104/642 (Electrical and Electronic Engineering) 11 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 12 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living sensors Article From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices Ivan Miguel Pires 1,2,3, *, Nuno M. Garcia 1,3,4 , Nuno Pombo 1,3 and Francisco Flórez-Revuelta 5 1 2 3 4 5 * Instituto de Telecomunicações, University of Beira Interior, 6201-001 Covilhã, Portugal; ngarcia@it.ubi.pt or ngarcia@di.ubi.pt (N.M.G.); ngpombo@di.ubi.pt or ngpombo@ubi.pt (N.P.) Altranportugal, 1990-096 Lisbon, Portugal ALLab - Assisted Living Computing and Telecommunications Laboratory, Department of Informatics, University of Beira Interior, 6201-001 Covilhã, Portugal ECATI, Universidade Lusófona de Humanidades e Tecnologias, 1749-024 Lisbon, Portugal Faculty of Science, Engineering and Computing, Kingston University, Kingston upon Thames KT1 2EE, UK; F.Florez@kingston.ac.uk Correspondence: impires@it.ubi.pt or ivan.pires@altran.com or impires@ubi.pt; Tel.: +351-966-379-785 Academic Editor: Vittorio M. N. Passaro Received: 21 November 2015; Accepted: 26 January 2016; Published: 2 February 2016 Abstract: This paper focuses on the research on the state of the art for sensor fusion techniques, applied to the sensors embedded in mobile devices, as a means to help identify the mobile device user’s daily activities. Sensor data fusion techniques are used to consolidate the data collected from several sensors, increasing the reliability of the algorithms for the identification of the different activities. However, mobile devices have several constraints, e.g., low memory, low battery life and low processing power, and some data fusion techniques are not suited to this scenario. The main purpose of this paper is to present an overview of the state of the art to identify examples of sensor data fusion techniques that can be applied to the sensors available in mobile devices aiming to identify activities of daily living (ADLs). Keywords: sensor data fusion; accelerometer; data collection; signal processing; sensors signal; activities of daily living 1. Introduction Sensors are present in much of the equipment used in everyday life by everyone, including mobile devices, which are currently applied in ambient assisted living (AAL) systems, such as smartphones, smartwatches, smart wristbands, tablets and medical sensors. In these devices, sensors are commonly used to improve and support peoples’ activities or experiences. There are a variety of sensors that allow the acquisition of the various types of data, which can then be used for different types of tasks. While sensors may be classified according to the type of data they manage and their application purposes, the data acquisition is a task that is highly dependent on the user’s environment and application purpose. The main objective of this paper is to present a comprehensive review of sensor data fusion techniques that may be employed with off-the-shelf mobile devices for the recognition of ADLs. We present a classification of the sensors available in mobile devices and review multi-sensor devices and data fusion techniques. Sensors 2016, 16, 184; doi:10.3390/s16020184 www.mdpi.com/journal/sensors 13 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 2 of 27 The identification of ADLs using sensors available in off-the-shelf mobile devices is one of the most interesting goals for AAL solutions, as this can be used for the monitoring and learning of a user’s lifestyle. Focusing on off-the-shelf mobile devices, these solutions may improve the user’s quality of life and health, achieving behavioural changes, such as to reduce smoking or control other addictive habits. This paper does not comprehend the identification of ADLs in personal health or well-being, as this application ecosystem is far wider and deserves a more focused research, therefore being addressed in future work. AAL has been an important area for research and development due to population ageing and to the need to solve societal and economic problems that arise with an ageing society. Among other areas, AAL systems employ technologies for supporting personal health and social care solutions. These systems mainly focus on elderly people and persons with some type of disability to improve their quality of life and manage the degree of independent living [1,2]. The pervasive use of mobile devices that incorporate different sensors, allowing the acquisition of data related to physiological processes, makes these devices a common choice as AAL systems, not only because the mobile devices can combine data captured with their sensors with personal information, such as, e.g., the user’s texting habits or browsing history, but also with other information, such as the user’s location and environment. These data may be processed either in the device or sent to a server using communication technologies for later processing [3], requiring a high level of quality of service (QoS) to be needed to achieve interoperability, usability, accuracy and security [2]. The concept of AAL also includes the use of sophisticated intelligent sensor networks combined with ubiquitous computing applications with new concepts, products and services for P4-medicine (preventive, participatory, predictive and personalized). Holzinger et al. [4] present a new approach using big data to work from smart health towards the smart hospital concept, with the goal of providing support to health assistants to facilitate a healthier life, wellness and wellbeing for the overall population. Sensors are classified into several categories, taking into account different criteria, which include the environmental analysis and the type of data acquired. The number and type of sensors available in an off-the-shelf mobile device are limited due to a number of factors, which include the reduced processing capacity and battery life, size and form design and placement of the mobile device during the data acquisition. The number and type of available sensors depend on the selected mobile platform, with variants imposed by the manufacturer, operating system and model. Furthermore, the number and type of sensors available are different between Android platforms [5] and iOS platforms [6]. Off-the-shelf mobile devices may include an accelerometer, a magnetometer, an ambient air temperature sensor, a pressure sensor, a light sensor (e.g., photometer), a humidity sensor, an air pressure sensor (e.g., hygrometer and barometer), a Global Positioning System (GPS) receiver, a gravity sensor, a gyroscope, a fingerprint sensor, a rotational sensor, an orientation sensor, a microphone, a digital camera and a proximity sensor. These sensors may be organized into different categories, which we present in Section 2, defining a new classification of these sensors. Jointly with this classification, the suitability of the use of these sensors in mobile systems for the recognition of ADLs is also evaluated. The recognition of ADLs includes a number of different stages, namely data acquisition, data processing, data imputation, sensor data fusion and data mining, which consist of the application of machine learning or pattern recognition techniques (Figure 1). As shown in Figure 1, the process for recognizing ADLs is executed at different stages, which starts with the data acquisition by the sensors. Afterwards, the data processing is executed, which includes the validation and/or correction of the acquired data. When data acquisition fails, data correction procedures should be performed. The correction of the data consists of the estimation of the missing or inconsistent values with sensor data imputation techniques. Valid data may be sent to the data fusion module, which consolidates the data collected from several of the available sensors. After the data fusion, ADLs can be identified using several methods, such as data mining, pattern recognition or machine learning techniques. Eventually, the system may require the user’s validation 14 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 3 of 27 or feedback, either at an initial stage, or randomly in time, and this validation may be used to improve, to train and to fine-tune the algorithms that handle the consolidated data. Figure 1. Schema of a multi-sensor mobile system to recognize activities of daily living. Due to the processing, memory and battery constraints of mobile devices, the selection of the algorithms has to be done in a particular way, or otherwise, the inappropriate use of resource-greedy algorithms will render the solution unusable and non-adoptable by the users. The remaining sections of this paper are organized as follows: Section 2 proposes a new classification for the sensors embedded in off-the-shelf mobile devices and identifies different techniques related to data acquisition, data processing and data imputation; Section 3 is focused on the review of data fusion techniques for off-the-shelf mobile devices; some applications of mobile sensor fusion techniques are presented in Section 4; in Section 5, the conclusions of this review are presented. 2. Sensors Sensors are hardware components that have the capability to capture different types of signals. Sensors are widely available in equipment used daily, including in smartphones, smartwatches, tablets and specific devices, including medical and industrial devices, and may be used to collect data in a plethora of situations. Examples of the use of sensors’ data include measuring some property of the sensor’s environment, such as chemical sensing, the measurement of motion, touch and proximity data or acoustic or imaging detection. 2.1. Sensor Classification In AAL systems, sensors are mainly used to measure data from the user and his/her environment. For example, the sensors available in most mobile devices allow the detection of movements and activities of the user, which may lead to identifying the ADLs, with a high accuracy. The research about sensor data fusion techniques should include the identification of the different types of sensors available for use. The analysis of the different types of sensors is commonly named sensor classification, consisting of the definition of different classes to which sensors may be assigned. Examples of features that define the different sensor classes include the purpose of the sensor, its working environment and the type of data they acquired [7,8]. The environment itself can be classified as controlled, uncontrolled, static, dynamic, uncertain and undefined [7,8]. The classification of sensors presented in this paper has a special focus on the identification of the ADLs. According to [9], the sensors used for sleep detection include the measurement of different activities and physical phenomena, such as electro-encephalography (EEG), electro-cardiography (ECG), blood pressure, photoplethysmography, non-cardiac electropotentials, oxygen saturation, respiration, body movement, arterial tonometry and temperature. These measurements include the s 15 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 4 of 27 use of different sensors, such as accelerometer, gyroscope, heart band, microphones and video cameras, to assess the body’s acceleration and its vital signs while recording audio and video. For the detection of physical activities, the authors in [10] propose the use of inertial sensors, i.e., accelerometers and gyroscopes, complemented with orientation measurement using magnetic sensors, e.g., a compass and a magnetometer, and location measurement using location sensors, e.g., a GPS receiver. However, location measurement can also be achieved using inertial sensors and acoustic sensors, e.g., a microphone [11]. Sensors can be used to perform the measurement of physical activities in controlled or uncontrolled environments. When used in controlled environments, these sensors, e.g., the video sensors, may be distributed in different places to measure the movements in a defined and often a static spatial area [12]. Gentili and Mirchandani [13] make use of counting sensors, image sensors and automatic vehicle identification (AVI) readers to identify the origin and the location of an object. The use of static or dynamic images relies heavily on image processing algorithms to extract the relevant features. Due to the large variety of data that can be extracted from still images or videos, it is an autonomous research topic. On the other hand, Lee et al. [14] make use of a wearable electrogoniometer composed of a knee angular and a three-axis accelerometer sensor to detect the ADLs in uncontrolled environments, presenting also four different machine learning techniques for detecting occurrences of walking. Other studies were performed in uncontrolled environments using different mobile technologies. The accuracy of the results is influenced by several constraints also presented in this paper. Sensors may be grouped into different categories related to the identification of the movements and ADLs. One of the usual categories is related to acoustic sensors, which can be used in almost all environmental scenarios. This category includes simple microphones, silicon microphones [15] and other acoustic wave devices [16], which are usually used for physical sound wave sensing. Another proposed category integrates chemical sensors, especially useful for detecting the presence of specific molecules in the air and/or the environment. In [17], the classification of these sensors is presented as metal-oxide, semiconductive polymers, conductive electroactive polymers, optical, surface acoustic wave and electrochemical gas sensors. These types of sensors are also used for medical purposes, e.g., for the detection of the correct dosage in a patient’s treatment [18]. Another class of sensors includes mechanical sensors, such as mass sensors, strain sensors [19], pressure sensors, contact sensors, mechanical switches and others. These sensors may be used to detect movements of the user’s body. Magnetic sensors may be used to complement the sensing of the different movements. These include search-coil magnetometers, fluxgate magnetometers, superconductor magnetometers, hall effect sensors, magnetoresistive magnetometers, spin-valve transistors, giant magnetoimpedance magnetic sensors, magnetodiodes, magnetotransistors, magnetostrictive magnetometers, magneto-optical sensors and micro-electro-mechanical systems (MEMS)-based magnetometers [20]. Optical sensors may also be used to measure the parameters of ADLs [21], which include photoplethysmography, fiber optic sensors, infrared sensors and radio frequency sensors. These sensors are able to work with adverse temperatures, corrosion/erosion surroundings, high-vibration, voltage and pressure environments [22], but these are not usual on smartphones. The detection of physical activities can also be achieved by sensors used in medical applications. The majority of these sensors can be included in the categories previously mentioned, such as inertial sensors, e.g., accelerometers, gyroscopes, inclinometers or pedometers, mechanical sensors, e.g., ball/tilt/foot switches, sole switches and force-sensitive resistors, acoustic sensors, e.g., microphones, optical sensors, e.g., infrared sensors, radio frequency sensors, e.g., radio frequency identifiers (RFID), New Field Communication (NFC) and Ubisense Real-Time Location Systems (RTLS), atmospheric sensors, e.g., barometers and hygrometers, electrical sensors, e.g., electromyogram (EMG), ECG, electrodermal activity (EDA) and electrooculography (EOG) sensors, magnetic sensors, e.g., magnetometers, photoelectric sensors, e.g., oximeters, and chemical sensors, e.g., actinometers [23]. 16 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 5 of 27 Another category of sensors includes the location sensors, such as GPS receiver or WiFi location methods. Zhang et al. [24] presented a comparison between the accuracy of the use of WiFi location and the use of a GPS receiver, in which the GPS signal is limited in indoor environments, and in this case, the use of a WiFi location method is advised. In [25], wearable sensors are considered. These include the sensors embedded in mobile devices, such as the GPS receiver, accelerometer, proximity sensor, camera, microphone, gyroscope, light sensor, gravity sensor and other sensors connected by, for example, Bluetooth (e.g., heart rate sensors, chemical sensors and others) [26]. Wearable devices of this type are mainly used in the medical area to capture data that will be used for diagnosis [27,28], e.g., EEG, optical sensors, thermal sensors, acoustic sensors, magnetic sensors or mechanical sensors. A variety of sensors is commonly available in off-the-shelf mobile devices, improving the ability to detect some physiological parameters anywhere and at anytime. Taking previous research into consideration and having as a goal the selection of sensors that may be used in off-the-shelf devices to allow the identification of ADLs, a new classification of sensors is proposed with the following categories: magnetic/mechanical sensors, environmental sensors, motion sensors, imaging/video sensors, proximity sensors, acoustic sensors, medical sensors, chemical sensors and force sensors. Table 1 presents a non-exhaustive list for the sensors embedded in mobile devices and external sensors for each category. In addition, other sensors may be connected to the off-the-shelf mobile devices using over-the-air technologies, improving the capabilities of the data collection, but as these are not integrated into the mobile device itself, they are not considered in this review. Table 1. Sensors classified by categories. Category External Sensors Mobile Sensors Magnetic/Mechanical Compass; Magnetometer; Strain sensors; Search-coil magnetometer; sensors Fluxgate magnetometer; Superconductor magnetometers; Hall effect sensor; Magnetoresistive magnetometers; Spin-valve transistors; Giant magnetoimpedance magnetic sensors; Magnetodiode; Magnetotransistor; Magnetostrictive magnetometers; Magneto-optical sensor; MEMS Based Magnetometers; Ball/tilt/foot switch; Sole pressure switch; Pressure sensors; Contact sensors; Mechanical switches Magnetometer; Compass Environmental sensors Barometer; Humidity; Light sensor; Thermal sensor Ambient air temperature and pressure; Light Sensor; Humidity Barometer; Photometer; Thermal sensor Location sensors GPS receiver; Automatic Vehicle Identification (AVI) readers; Ubisense Real-Time Location Systems (RTLS); Wi-Fi Location-Based Services GPS receiver; Wi-Fi Location-Based Services Motion sensors Accelerometer; Gyroscope; Pressure sensor; Gravity sensor; Inclinometer; Pedometer; Rotational sensor Accelerometer; Gravity sensor; Gyroscope; Rotational vector sensor; Orientation sensor Imaging/Video sensors Digital camera; 3D camera; Optical sensor; Infrared sensor Digital camera; Infrared sensor Proximity sensors Proximity sensor; Touch sensor; RFID; Tactile sensor; NFC Proximity sensor; Touch sensor; RFID; NFC Acoustic sensors Microphone; Silicon microphones; Acoustic wave devices; Surface acoustic wave Microphone Medical sensors EEG; ECG; EMG; EOG; EDA; Photoplethysmogram; Blood pressure and arterial tonometry; Respiration; Dosage control/detection; Stress sensors; Heart rate sensors; electrooculography; electrodermal activity sensors None Chemical sensors Oxygen saturation; Aroma sensors; Metal-oxide; Semi conductive polymers; Conductive electro active polymers; Electrochemical gas sensors; Actinometer None Optical sensors Photoplethysmography sensors; Fiber optic sensors; Infrared sensors; Radio frequency sensors Infrared sensors; Radio frequency sensors Force sensors Force sensitive resistor; Mass sensor; Fingerprint sensor Fingerprint sensor Photoelectric sensors Oximeter None 2.2. Data Acquisition Data acquisition depends on the particular characteristics of the sensors selected to perform the recognition of ADLs, on the environments where data are captured and also on the architecture of each 17 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 6 of 27 system. The data acquisition process is accomplished by a module embedded in the mobile device and consists of the measurement and conversion of the electrical signals received by each sensor into a readable format [29]. Several challenges are associated with the data acquisition process when recognizing ADLs, including the positioning of the mobile device, the actual data sampling rate and the number of sensors to be managed [30]. The problems associated with data acquisition in an off-the-shelf mobile device influence the correct extraction of meaningful features of the ADLs. As the sensors are embedded in the mobile device, they cannot be located separately in different parts of the body; rather, the mobile device needs to be situated in an usual and comfortable position. Another issue related to mobile devices is the power consumption of the data acquisition tasks. Therefore, the limited battery capacity constraint needs to be taken into account to ensure a successful continuous operation of mobile applications [31] by developing and applying lightweight acquisition methods. The main advantage of using mobile devices for data acquisition is related to the possibility to acquire data anywhere and at anytime [32]. However, it has some limitations in the performance of the data acquisition for real-time applications. Multitasking execution patterns differ among mobile devices, because these depend on their processing ability, memory and power capabilities, which are very limited, and on the operating system and on the number and type of mobile applications currently installed and/or running. The Acquisition Cost-Aware QUery Adaptation (ACQUA) framework, presented in [31], consists of a query processing engine implemented in an off-the-shelf mobile device that dynamically modifies both the order and the segments of data streams requested from different data sources (sensors). The ACQUA framework starts by learning the selectivity properties of several sensor streams and then utilizes such estimated selectivity values to modify the sequence in which the mobile device acquires data from the sensors. The authors said that it is supported by some basic automated storage and retrieval system (ASRS) algorithms for acquisition cost-aware query processing. Query specification can be made using ASRS query evaluation algorithms, the disjunctive normal form or optimizing for multiple predicates on the same stream. The main features of ACQUA are the accommodation of the heterogeneity in sensors’ data rates, the determination of the packet sizes and radio characteristics, the adaptation of the dynamic changes in query selective properties and the support of multiple queries and heterogeneous time window semantics. The authors claim that it can result in a 70% reduction in the energy overhead of continuous query processing, by reducing the volume of sensor data that is sent over-the-air to a server between a mobile device and its attached sensors, without affecting the fidelity of the processing logic [33]. Lim et al. [31] reviewed different frameworks to optimize data acquisition. The Orchestrator framework focuses on resource-sharing between multiple context-aware applications executing queries independently [34,35]. The ErdOS framework views sensors available in off-the-shelf mobile devices as a shared resource and seeks to rationally distribute the consumption for all resources [36]. The LittleRock prototype uses a special low-energy coprocessor to decrease the computational energy spent in embedded processing of on-board sensor data streams [37]. The Jigsaw continuous sensing engine implements a pipelined stream processing architecture that adaptively triggers different sensors at different sampling rates to fit the context accuracy required by different applications [38]. The SociableSense framework combines the cloud-based computation and adaptive sensor sampling to reduce the computational and sensing overheads during continuous mobile sensing [39]. The BBQ approach builds a multi-dimensional Gaussian probability density function of the sensors’ likely data values and then uses conditional probabilities to determine, in an iterative manner, the next sensor whose value is most likely to resolve a given query [40]. These frameworks try to improve data acquisition and support the preparation of data for their later processing. Table 2 presents a summary of the data acquisition frameworks, including their features and limitations. The data can be acquired by different methods, and the correct definition of the position of the sensors is the main factor that influences the data acquisition process. The definition of the best positioning for the sensors improves the reliability of the data acquisition methods implemented 18 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 7 of 27 by the mobile operating systems, improving the data acquisition and data processing techniques. The selection of the best data acquisition methods depends on the purpose of use, the type of data acquired and their environment [41–44]. Table 2. Examples of data acquisition methods. ACQUA, Acquisition Cost-Aware QUery Adaptation; ASRS, automated storage and retrieval system. Method Features Limitations ACQUA framework It is a query processing engine implemented on an off-the-shelf mobile device that dynamically modifies both the order and the segments of data streams requested from different data sources; supported by some basic ASRS algorithms for acquisition-cost-aware query processing; seeks to additionally reduce the communication energy overheads involved in acquiring the data wirelessly from additional external sensors; it is complementary to the Jigsaw and LittleRock frameworks. It does not exploit correlations, which means that it lacks the predictive power of representations based on probabilistic models. Orchestrator framework Used for resource-sharing between multiple context-aware applications executing queries independently; enables the platform to host multiple applications stably, exploiting its full resource capacity in a holistic manner. Timing relations are not known. ErdOS framework Distributes the consumption for all resources of the off-the-shelf mobile device; restarts the jobs in case of failure. Statically configured and non-extensible; difficult to adapt for each case. LittleRock prototype Uses a special low-energy coprocessor to decrease the computational energy spent in embedded processing of on-board sensor data streams; loads the interactions with sensors and gives the phone’s main processor and associated circuitry more time to go into sleep mode; flexible storage. Limited resources of the mobile devices. Jigsaw Balances the performance needs of the application and the resource demands of continuous sensing on the phone; comprises a set of sensing pipelines for the accelerometer, microphone and GPS sensors. Robustness inferences. SociableSense framework Combines the cloud-based computation and adaptive sensor sampling to reduce the computational and sensing overheads during continuous mobile sensing. Limited resources of the mobile devices. BBQ approach Builds a multi-dimensional Gaussian probability density function of the sensors’ likely data values and then uses conditional probabilities to determine, in an iterative manner, the next sensor whose value is most likely to resolve a given query; similar to ACQUA. Similar to ACQUA. 2.3. Data Processing Data processing is the next step of a multi-sensor mobile system used to process the sensor data. Data processing is a complex process, which also depends on environmental conditions, the types of sensor, the capabilities of the mobile device used and the types of data collected. The type of application also influences the use of data processing methods. Data processing may be executed locally, using the capabilities of the mobile device, or at the server side, sending the collected data to a remote server, where the data processing methods are executed [42,45,46]. For server-side processing, the mobile device is only required to acquire the sensors data and to present the results to the user. As frequent sensor sampling operations and further data processing can significantly reduce the battery lifetime and the capacities of the mobile device [47,48], several methods may be used to process different types of data. For example, while for audio processing, the time-scale modification (TSM) algorithm and, for medical imaging, remote processing methods are highly recommended, for other sensors, data can be processed locally in the mobile device without significant consumption of local resources [49–51]. Data processing methods may include a segmentation method, which divides a larger data stream into smaller chunks appropriate for processing, and a definition of the window size [52]. According to Pejovic and Musolesi [53], the challenges of data processing in mobile devices lie in the adaptation, context-driven operation, computation, storage and communication. The adaptation and the context-driven operation have several solutions that include the adaptive sampling, the hierarchical modality switching, the harnessing domain structure and the use of cloud offloading. Possible solutions for computation, storage and communication are hierarchical processing, cloud offloading and hardware co-processing. Several systems and frameworks to classify and to process sensors’ data have been developed during the last few years. An option is, after contexts are extracted from the collected data, to discard 19 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 8 of 27 part of the data to reduce memory usage [54]. Methods such as support vector machines (SVM), artificial neural networks (ANN), Bayes classifiers and k-nearest neighbour (KNN) algorithms, among others, may be used for data processing. In several research studies, the analysis and processing of sensor data include complex tasks that need high processing and memory capabilities [55]. In such cases, these activities need to be performed at the server-side. In [54], Imai et al. present a rule-based data processing engine for sensor nodes to minimize the amount of energy needed to process the tasks. The data processing may be performed on a computer located in the neighbourhood of the sensor nodes, which processes the data and provides the results, obtaining a good balance between reducing the network load and advanced sensor data analysis and processing. Yamada et al. [56] present a location-based information delivery method using StreamSpinner, which achieves efficient stream data processing based on novel multiple continuous query optimization techniques. In [57], the Dandelion system is presented, achieving good results with senselet, a smartphone-style, platform-agnostic programming abstraction for in-sensor data processing. Dandelion provides a unified programming model for heterogeneous systems that span diverse execution contexts, supporting data-parallel applications. Dolui et al. [58] defined two architectures: the Device Data Processing Architecture and the Server Data Processing Architecture. The Device Data Processing Architecture is designed to acquire the data from the sensors embedded in an off-the-shelf mobile device and to process the data locally. This architecture is useful when the processing methods require low resources, such as processing the accelerometer data, proximity sensor data and others. On the contrary, the Server Data Processing Architecture consists of the dispatch of the data collected to a remote server allowing the computation of a large amount of data, as well as computations of a complex nature. This architecture is employed for instance with data acquired with the GPS receiver and the imaging sensors. Imai et al. [59] defined a method for data processing using the similarity of motions between observed persons. The method presented implements sensor data processing using a neighbour host, executed in two phases: a basic action phase and a changing sensor node settings phase [59]. In the basic action phase, a neighbouring host receives sensor data from the sensor nodes. Then, this host analyses and processes the data and sends the results to a different host. In the changing sensor node settings phase, when analytic results of sensor data fulfill the conditions determined in advance, the neighbouring host instructs sensor nodes to change the settings. Next, the sensor data processing method based on similarity is defined, where the system acquires and accumulates a newly-observed person’s acceleration data, while it estimates the related motions using a similar observed person’s profile. When the system acquires sufficient acceleration data of a newly-observed person, the system creates a profile that is added to the knowledge base. The ACQUA framework [31,33] optimizes the data processing, using ASRS algorithms. The ACQUA framework can also be implemented in systems with a remote processing architecture or in systems that process the tasks locally. Reilent et al. [41] developed an open software architecture for patient monitoring, which supports semantic data processing and (soft) real-time reasoning. Usually, in medical environments, the context awareness and decision making are performed in a remote server, which returns the results to the mobile device [42]. Another option used for processing the collected data is the use of a cloud-based server that processes the data in real time. However, it makes the mobile device and data processing dependent on a constant Internet connection [60]. In order to avoid this constraint, some tele-medicine systems have implemented data processing locally on the mobile device. For instance, Postolache et al. [61] implemented advanced data processing, data management, human computing interfacing and data communication using a smartphone running the Android operating system. Augmented reality is also an application relevant for AAL. Paucher and Turk [32] implemented a system for image processing that used a remote server. The system implements a nearest 20 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 9 of 27 neighbour search of descriptors using a k-dimensional (KD)-Tree structure, applying it for each image. Other algorithms for image processing using the camera of a mobile device are presented in [62]. Table 3 summarizes the different architectures for data processing, presenting the most relevant methods and achievements. It is often described in the literature that the data that can be processed locally on the mobile device include the motion or positioning sensors. However, the processing of images or videos is computationally expensive, and due to the low memory and processing capabilities of mobile devices, it is highly recommended that these tasks use external processing. Table 3. Data processing: architectures and methods. Architectures Methods Achievements Device Data Processing Architecture Dandelion system; SVM; ANN; Bayes classifiers; KNN algorithms; location-based information delivery method using StreamSpinner Acquisition of the data from the sensors embedded in an off-the-shelf mobile device; process the data locally; the results are rapidly presented to the user; processing methods should require low resources; using segmentation methods, a larger data stream is divided into smaller chunks improving the methods; the correct definition of the window size is important for achieve good results. Server Data Processing Architecture SVM; ANN; Bayes classifiers; KNN algorithms; nearest neighbour search of descriptors using a KD-Tree structure. Dispatching of the data collected to a remote server allowing the computation of a large amount of data, as well as computations of complex nature; in some cases, the data processing may be performed on a computer located in the neighbourhood of the sensor nodes; in server-side processing, the mobile device and data processing are dependent on a constant Internet connection. 2.4. Data Imputation Data acquisition with sensors embedded in off-the-shelf mobile devices for real-time recognition of ADLs may fail in many circumstances. Firstly, the acquisition process fails when the sensors report unexpected values. Secondly, due to the multitasking nature of many of the mobile devices, this can happen anytime and for any given collection scenario. Depending on the timing of the missing data included in a subset of sensors’ data, the missing data are classified as missing completely at random (MCAR) when the missing values are randomly distributed by all time instants [63,64]. The missing data are classified as missing at random (MAR) when the missing values are randomly distributed by subsets of the collected data [63,64]. Additionally, the missing data are classified as missing not at random (MNAR) when the missing values are not randomly distributed [63,64]. Several methods to minimize the effects of missing data have been developed, estimating the missing values based either on other values correctly obtained or on external factors. In [63], the Imputation Tree (ITree) method is presented, which is a tree-based algorithm for missing values imputation. This method constructs a missing pattern tree (MPT), which is a binary classification tree for identifying the absence of each observation. It uses clustering techniques, e.g., K-means clustering, to impute missing values and linear regression analysis to improve data imputation. In [65], a multi-matrices factorization model (MMF) for the missing sensor data estimation problem is formulated, which uses statistical methods to estimate the missing values. There are other methods for sensor data imputation that employ KNN-based imputation [66], multiple imputation [67], hot/cold imputation [68], maximum likelihood and Bayesian estimation [69] and expectation maximization [70]. In general, these methods are used to verify and increase the consistency of sensor data. The KNN method and its variants, such as KNNimpute (K-nearest neighbour imputation), SKNNimpute (sequential K-nearest neighbour method-based imputation) and MKNNimpute (K-nearest neighbour imputation method based on Mahalanobis distance), are the most used methods [71]. Other cluster-based imputation methods exists, such as KMI (K-means-based imputation) [72] and FCMimpute (fuzzy C-means clustering imputation) [73]. For audio signal imputation purposes, Smaragdis et al. [74] used nearest neighbours and singular value decomposition (SVD) algorithms. The SVD algorithm is executed after nearest neighbour clustering, replacing all of the missing values with an initial value, computing the SVD of the resulting matrix and 21 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 10 of 27 replacing the missing values with their prediction according to the SVD decomposition, repeating the process until the change in the imputed missing data falls below some user-defined threshold. In [75], a generalized trend diffusion (GTD) method is used to create small datasets. It applies the so-called shadow data and membership functions for increasing the knowledge when using back propagate-based (BP) neural networks to predict the missing values. In [76], random recursive partitioning (RRP) is used to generate a proximity matrix used in non-parametric matching problems, such as hot-deck missing data imputation and average treatment effect estimation. In [77], a discrete model is presented to improve data imputation at the time of collection, reducing the errors and repairing the data if possible. Other approaches [64] have also been studied to improve data imputation, such as ignoring and deleting data (e.g., listwise deletion and discarding instances), available-case methods (e.g., pairwise deletion), non-model based imputation procedures (e.g., unconditional mean imputation, conditional mean imputation) and model-based imputation procedures. Other implicit and explicit models exist for data imputation. The implicit models are based on implicit assumptions on the proximity between individuals belonging to the dataset (e.g., hot deck imputation, cold deck imputation, substitution method, composite methods), and the explicit models are based on a statistical model to describe the predictive distribution of missing data (e.g., linear regression, logistic regression, multiple imputation methods, the expectation-maximization (EM) algorithm, probabilistic neural networks, fuzzy min–max neural networks, general regression auto associative neural network and distribution free methods, such as non-parametric regression and tree-based methods). The use of big data brings other challenges in data processing that consist of the extraction of relevant information to enable the correct analysis, discovery and interpretation of the data. Pombo et al. [55] presented a predictive model using the radial basis function neural network (RBFNN) combined with a filtering technique aiming at the estimation of the electrocardiogram (ECG) waveform, which supports healthcare professionals on clinical decisions and practices. The study [55] is related to the PhD thesis, presented in [78], that starts with the analysis of several methods for machine learning prediction and finalizes with the creation of a clinical decision support system. The methods studied are the rule-based algorithms (RBA), the ANN, the rough and fuzzy sets (RFS) and the statistical learning algorithms (SLA). The data imputation methods can be applied in off-the-shelf mobile devices with some technical limitations, due to the capacities of these devices. For instance, in [79], neural networks are used to perform data imputation for a visual system. Table 4 presents a summary about the data imputation methods analysed on this paper, presenting the types of data acquired and achievements obtained. Table 4. Examples of data imputation methods. 22 Types of Data Models Achievements MCAR listwise deletion; pairwise deletion; ITree method; KNN method and their variants; KMI; FCMimpute; SVD; GTD method; BP neural networks; RRP; MMF; MPT; hot/cold imputation; expectation maximization; Bayesian estimation; unconditional mean imputation; conditional mean imputation; ANN. MAR maximum likelihood; multiple imputation; ITree method; KNN method and their variants; KMI; FCMimpute; SVD; GTD method; BP neural networks; RRP; MMF; MPT; hot/cold imputation; expectation maximization; Bayesian estimation; unconditional mean imputation; conditional mean imputation; ANN. These methods improve the identification of the absence of each observation; the use of clustering techniques and linear regression analysis improves the data imputation; the data imputation also increases the consistency of the data; other improvements of data imputation are ignoring and deleting the unusable data; another measured variable can be indirectly predicted with the probability of missingness. MNAR selection model; pattern mixture models; maximum likelihood; multiple imputation; ITree method; KNN method and their variants; KMI; FCMimpute; SVD; GTD method; BP neural networks; RRP; MMF; MPT; hot/cold imputation; expectation maximization; Bayesian estimation; unconditional mean imputation; conditional mean imputation; ANN. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 11 of 27 However, some devices do not have enough capabilities to perform complex data imputation methods. In order to solve this, Kim and Prabhakaran [80] presented a new method, named canonical correlation based k-weighted angular similarity (CkWAS), that maps the missing data with a reference pattern dataset to assign values to the missing data. 3. Sensor Data Fusion in Mobile Devices Data fusion is a critical step in the integration of the data collected by multiple sensors. The main objective of the data fusion process is to increase the reliability of the decision that needs to be made using the data collected from the sensors, e.g., to increase the reliability of the identification of the ADL algorithm running in an off-the-shelf mobile device. If a single stream of data cannot eliminate uncertainty from the output, data fusion will use data from several sources with the goal of decreasing the uncertainty level of the output. Consequently, the data fusion increases the level of robustness of a system for the recognition of ADLs, reducing the effects of incorrect data captured by the sensors [81] or helping to compute solutions when the collected data are not usable for a specific task. A mobile application implemented by Ma et al. [82] was tested in a Google Nexus 4 and uses the accelerometer, gyroscope, magnetometer and GPS receiver to evaluate the sensors’ accuracy, precision, maximum sampling frequency, sampling period, jitter and energy consumption in all of the sensors. The test results show that the built-in accelerometer and gyroscope sensor data have a standard deviation of approximately 0.1 to 0.8 units between the measured value and the real value, the compass sensor data deviate approximately three degrees in the normal sampling rate, and the GPS receiver data have a deviation lower than 10 meters. Thus, one of the working hypotheses of the research is the data collected by mobile sensors may be fused to work with more precision towards a common goal. The data fusion may be performed with mobile applications, accessing the sensors data as a background process, processing the data and showing the results in a readable format or passing the results or the data to a central repository or central processing machine for further processing. Durrant-Whyte et al. [83] described a decentralized data fusion system, which consists of a network of sensor nodes, each one with its own processing facility. This is a distributed system that does not require any central fusion or central communication facility, using Kalman filters to perform the data fusion. Other decentralized systems for data fusion have also been developed, improving some techniques and the sensors used [84]. The definition of the categories of the data fusion methods has already been been discussed by several authors [85–87]. According to these authors, the data fusion methods may be categorized as probabilistic, statistic, knowledge base theory and evidence reasoning methods. Firstly, probabilistic methods include Bayesian analysis of sensor values with Bayesian networks, state-space models, maximum likelihood methods, possibility theory, evidential reasoning and, more specifically, evidence theory, KNN and least square-based estimation methods, e.g., Kalman filtering, optimal theory, regularization and uncertainty ellipsoids. Secondly, statistic methods include the cross-covariance, covariance intersection and other robust statistics. Thirdly, knowledge base theory methods include intelligent aggregation methods, such as ANN, genetic algorithms and fuzzy logic. Finally, the evidence reasoning methods include Dempster-Shafer, evidence theory and recursive operators. Depending on the research purpose of the data fusion, these methods have advantages and disadvantages presented in Table 5. The data fusion methods are influenced with the constraints in the previous execution of data acquisition, data processing and data imputation. The advantages and disadvantages also depend on the environmental scenarios and the choice of the correct method to apply for each research scenario. 23 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 12 of 27 Table 5. Advantages and disadvantages of the sensor data fusion methods. Methods Advantages Disadvantages Probabilistic methods Provide methods for model estimation; allows unsupervised classification; estimate the state of variables; reduce errors in the fused location estimate; increase the amount of data without changing its structure or the algorithm; produce a fused covariance matrix that better reflects the expected location error. Require a priori probabilistic knowledge of information that is not always available or realistic; classification depends on the starting point; unsuitable for large-scale systems; requires a priori knowledge of the uncertainties’ co-variance matrices related to the system model and its measurements. Statistic methods Accuracy improves from the reduction of the prediction error; high accuracy compared with other local estimators; robust with respect to unknown cross-covariance. Complex and difficult computation is required to obtain the cross-variance; complexity and larger computational burden. Knowledge base theory methods Allows the inclusion of uncertainty and imprecision; easy to implement; learning ability; robust to noisy data and able to represent complex functions. The knowledge extraction requires the intervention of human expertise (e.g., physicians), which takes time and/or may give rise to interpretation bias; difficulty in determining the adequate size of the hidden layer; inability to explain decisions; lack of transparency of data. Evidence reasoning methods Assign a degree of uncertainty to each source. Require assigning a degree of evidence to all concepts. In [88], data fusion methods are distributed in six categories. The categories of the data fusion methods include data in–data out, data in–feature out, feature in–feature out, feature in–decision out, decision in–decision out and data in–decision out. Performing the data fusion process in real time can be difficult because of the large amount of data that may need to be fused. Ko et al. [88] proposed a framework, which used dynamic time warping (DTW), as the core recognizer to perform online temporal fusion on either the raw data or the features. DTW is a general time alignment and similarity measure for two temporal sequences. When compared to hidden Markov models (HMMs), the training and recognition procedures in DTW are potentially much simpler and faster, having a capability to perform online temporal fusion efficiently and accurately in real time. The most used method for data fusion is the Kalman filter, developed for linear systems and then improved to a dynamically-weighted recursive least-squares algorithm [89]. However, as the sensor data are not linear, the authors in [89] used the extended Kalman filter to linearize the system dynamics and the measurement function around the expected state and then applied the Kalman filter as usual. A three-axis magnetometer and a three-axis accelerometer are used for the estimation of several movements [89]. Other systems employ variants of the Kalman filter to reduce the noise and improve the detection of movements. Zhao et al. [90] use the Rao-Blackwellization unscented Kalman filter (RBUKF) to fuse the sensor data of a GPS receiver, one gyroscope and one compass to improve the precision of the localization. The authors compare the RBUKF algorithm to the extended Kalman filter (EKF) and unscented Kalman filter (UKF), stating that the RBUKF algorithm improves the tracking accuracy and reduces computational complexity. Walter et al. [91] created a system for car navigation by fusing sensor data on an Android smartphone, using the embedded sensors (i.e., gyroscope) and data from the car (i.e., speed information) to support navigation via GPS. The developed system employs a controller area network (CAN)-bus-to-Bluetooth adapter to establish a wireless connection between the smartphone and the CAN-bus of the car. The mobile application fuses the sensors’ data and implements a strap down algorithm and an error-state Kalman filter with good accuracy, according to the authors of [91]. Anther application for location inference was built by using the CASanDRA mobile OSGi (Open Services Gateway Initiative) framework using a LocationFusion enabler that fused the data acquired by all of the available sensors (i.e., GPS, Bluetooth and WiFi) [92]. Mobile devices allow the development of context-aware applications, and these applications after use a framework for context information management. In [93], a mobile device-oriented framework for context information management to solve the problems on context shortage and communication 24 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 13 of 27 inefficiency is proposed. The main functional components of this framework are the data collector, the context processor, the context manager and the local context consumer. The data collector acquires the data from internal or external sensors. Then, the context processor and the context manager extract and manage context. Finally, the local context consumer develops context-aware applications and provides appropriate services to the mobile users. The authors claim that this framework is able to run real-time applications with quick data access, power efficiency, personal privacy protection, data fusion of internal and external sensors and simplicity in usage. Blum et al. [94] use the GPS receiver, compass and gyroscope embedded in Apple iPhone 4 (iOS) (Apple, Cupertino, CA, USA), iPhone 4s (iOS) (Apple, Cupertino, CA, USA) and Samsung Galaxy Nexus (Android) (Samsung, Seul, Korea), for the measurement of the location and augmented reality situations. Blum et al. analysed the position of the smartphone during the data collection, testing in three different orientation/body position combinations and in varying environmental conditions, obtaining results with location errors of 10 to 30 m (with a GPS receiver) and compass errors around 10 to 30°, with high standard deviations for both. In [95], the problems of data fusion, placement and positioning of fixed and mobile sensors are focused on, presenting a two-tiered model. The algorithm combines the data collected by fixed sensors and mobile sensors, obtaining good results. The positioning of the sensors is very important, and the study implements various optimization models, ending with the creation of a new model for the positioning of sensors depending on their types. Other models are also studied by the authors, which consider the simultaneous deployment of different types of sensors, making use of more detailed sensor readings and allowing for dependencies among sensor readings. Haala and Böhm [96] created a low-cost system using an off-the-shelf mobile device with several sensors embedded. The data collected by a GPS receiver, a digital compass and a 3D CAD model of a region are used for provisioning data related to urban environments, detecting the exact location of a building in a captured image and the orientation of the image. A different application where data fusion is required is the recognition of physical activity. The most commonly-used sensors for the recognition of physical activity include the accelerometer, the gyroscope and the magnetic sensor. In [26], data acquired with these sensors are fused with a combined algorithm, composed of Fisher’s discriminant ratio criterion and the J3 criterion for feature selection [97]. The collection of data related to the physical activity performed is very relevant to analyse the lifestyle and physiological characteristics of people [98]. Some other applications analyse the user’s lifestyle by fusing the data collected by the sensors embedded in mobile devices. Yi et al. [99] presented a system architecture and a design flow for remote user physiological data and movement detection using wearable sensor data fusion. In the medical scope, Stopczynski et al. [27] combined low-cost wireless EEG sensors with smartphone sensors, creating 3D EEG imaging, describing the activity of the brain. Glenn and Monteith [100] implemented several algorithms for the analysis of mental status, by using smartphone sensors. These algorithms fuse the data acquired from all of the sensors with other data obtained over Internet, to minimize the constraints of mobile sensors data. The system proposed in [101] makes use of biosensors for different measurements, such as a surface plasmon resonance (SPR) biosensor, a smart implantable biosensor, a carbon nanotube (CNT)-based biosensor, textile sensors and enzyme-based biosensors, combined with smartphone-embedded sensors and applying various techniques for the fusion of the data and to filter the inputs to reduce the effects of the position of the device. Social networks are widely used and promote context-aware applications, as data can be collected with the user’s mobile device sensors to promote the adaptation of the mobile applications to the user-specific lifestyle [102]. Context-aware applications are an important research topic; Andò [103] proposed an architecture to adapt context-aware applications to a real environment, using position, inertial and environmental sensors (e.g., temperature, humidity, gases leakage or smoke). The authors 25 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 14 of 27 claim this system is very useful for the management of hazardous situations and also to supervise various physical activities, fusing the data in a specific architecture. Other authors [104] have been studying this topic with mobile web browsers, using the accelerometer data and the positional data, implementing techniques to identify different ADLs. These systems capture the data while the user is accessing the Internet from a mobile device, analysing the movements and the distance travelled during that time and classifying the ADLs performed. Steed and Julier [105] created the concept of behaviour–aware sensor fusion that uses the redirected pointing technique and the yaw fix technique to increase the usability and speed of interaction in exemplar mixed-reality interaction tasks. In [106], vision sensors (e.g., a camera) are used and combined with accelerometer data to apply the depth from focus (DFF) method, which was limited to high precision camera systems for the detection of movements in augmented reality systems. The vision systems are relevant, as the obtained information can identify more accurately a moving object. Some mobile devices integrate RFID readers or cameras to fetch related information about objects and initiate further actions. Rahman et al. [107] use a spatial-geometric approach for interacting with indoor physical objects and artefacts instead of RFID-based solutions. It uses a fusion between the data captured by an infrared (IR) camera and accelerometer data, where the IR cameras are used to calculate the 3D position of the mobile phone users, and the accelerometer in the phone provides its tilting and orientation information. For the detection of movement, they used geometrical methods, improving the detection of objects in a defined space. Grunerbl et al. [108] report fusing data from acoustic, accelerometer and GPS sensors. The extraction of features from acoustic sensors uses low-level descriptors, such as root-mean-square (RMS) frame energy, mel-frequency cepstral coefficients (MFCC), pitch frequency, harmonic-to-noise ratio (HNR) and zero-crossing-rate (ZCR). They apply the naive Bayes classifier and other pattern recognition methods and report good results in several situations of daily life. Gil et al. [109] present other systems to perform sensor fusion, such as LifeMap, which is a smartphone-based context provider for location-based services, as well as the Joint Directors of Laboratories (JDL) model and waterfall IF model, which define various levels of abstraction for the sensor fusion techniques. Gil et al. also present the inContexto system, which makes use of embedded sensors, such as the accelerometer, digital compass, gyroscope, GPS, microphone and camera, applying several filters and techniques to recognize physical actions performed by users, such as walking, running, standing and sitting, and also to retrieve context information from the user. In [110], a sensor fusion-based wireless walking-in-place (WIP) interaction technique is presented, creating a human walking detection algorithm based on fusing data from both the acceleration and magnetic sensors integrated in a smartphone. The proposed algorithm handles a possible data loss and random delay in the wireless communication environment, resulting in reduced wireless communication load and computation overhead. The algorithm was implemented for mobile devices equipped with magnetic, accelerometer and rotation (gyroscope) sensors. During the tests, the smartphone was adapted to the user’s leg. After some tests, the authors decided to implement the algorithm with two smartphones and a magnetometer positioned on the user’s body, combining the magnetic sensor-based walking-in-place and acceleration-based walking in place, in order to discard the use of a specific model (e.g., gait model) and a historical data accumulated. However, the acceleration-based technique does not support the correct use in slow-speed walking, and the magnetic sensor-based technique does not support both normal and fast walking speeds. Related to the analysis of walking, activities and movements, several studies have employed the GPS receiver, combined with other sensors, such as the accelerometry, magnetometer, rotation sensors and others, with good accuracy using some fusion algorithms, such as the naive and oracle methods [111], machine learning methods and kinematic models [112] and other models adapted to mobile devices [91]. The framework implemented by Tsai et al. [113] detects the physical activity fusing data from several sensors embedded in the mobile device and using crowdsourcing, based on the 26 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 15 of 27 methods’ result of the merging of classical statistical detection and estimation theory and uses value fusion and decision fusion of human sensor data and physical sensor data. In [114], Lee and Chung created a system with a data fusion approach based on several discrete data types (e.g., eye features, bio-signal variation, in-vehicle temperature or vehicle speed) implemented in an Android smartphone, allowing high resolution and flexibility. The study involves different sensors, including video, ECG, photoplethysmography, temperature and a three-axis accelerometer, which are assigned as input sources to an inference analysis framework. A fuzzy Bayesian network includes the eyes feature, bio-signal extraction and the feature measurement method. In [81], a sensor weighted network classifier (SWNC) model is proposed, which is composed of three classification levels. A set of binary activity classifiers consists of the first classification level of the model proposed. The second classification level is defined by node classifiers, which are decision making models. The decisions of the model are combined through a class-dependent weighting fusion scheme structure with a structure defined through several base classifiers. The weighted decisions obtained by each node classifier are fused on the model proposed. Finally, the last classification level has a similar constitution of the second classification level. Independently of the level of noise imposed, when the mobile device stays in a static position during the data acquisition process, a performance up to 60% can be achieved with the proposed method. Chen et al. [115] and Sashima et al. [116] created client-server platforms that monitor the activities in a space with a constant connection with a server. This causes major energy consumption and decreases the capabilities of the mobile devices in the detection of the activities performed. Another important measurement with smart phone sensors is related to the orientation of the mobile device. Ayub et al. [117] have already implemented the DNRF (drift and noise removal filter) with a sensor fusion of gyroscope, magnetometer and accelerometer data to minimize the drift and noise in the output orientation. Other systems have been implemented for gesture recognition. Zhu et al. [118] proposed a high-accuracy human gesture recognition system based on multiple motion sensor fusion. The method reduces the energy overhead resulting from frequent sensor sampling and data processing with a high energy-efficient very large-scale integration (VLSI) architecture. The results obtained have an average accuracy for 10 gestures of 93.98% for the user-independent case and 96.14% for the user-dependent case. As presented in Section 2.4, decentralized systems, cloud-based systems or server-side systems are used for data processing. As the data fusion is the next stage, van de Ven et al. [119] presented the Complete ambient assisting living experiment (CAALYX) system that provides continuous monitoring of people’s health. It has software installed on the mobile phone that uses data fusion for decision support to trigger additional measurements, classify health conditions or schedule future observations. In [120], Chen et al. presented a decentralized data fusion and active sensing (D2 FAS) algorithm for mobile sensors to actively explore the road network to gather and assimilate the most informative data for predicting the traffic phenomenon. Zhao et al. [121,122] developed the COUPON (Cooperative Framework for Building Sensing Maps in Mobile Opportunistic Networks) framework, a novel cooperative sensing and data forwarding framework to build sensing maps satisfying specific sensing quality with low delay and energy consumption. This framework implements two cooperative forwarding schemes by leveraging data fusion; these are epidemic routing with fusion (ERF) and binary spray-and-wait with fusion (BSWF). It considers that packets are spatial-temporally correlated in the forwarding process and derives the dissemination law of correlated packets. The work demonstrates that the cooperative sensing scheme can reduce the number of samplings by 93% compared to the non-cooperative scheme; ERF can reduce the transmission overhead by 78% compared to epidemic routing (ER); BSWF can increase the delivery ratio by 16% and reduce the delivery delay and transmission overhead by 5% and 32%, respectively, compared to binary spray-and-wait (BSW). 27 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 16 of 27 In [123], a new type of sensor node is described with modular and reconfigurable characteristics, composed of a main board, with a processor, FR (Frequency Response) circuits and a power supply, as well as an expansion board. A software component was created to join all sensor data, proposing an adaptive processing mechanism. In [124], the C-SPINE (Collaborative-Signal Processing in Node Environment) framework is proposed, which uses multi-sensor data fusion among CBSNs (Collaborative Body Sensor Networks) to enable joint data analysis, such as filtering, time-dependent data integration and classification, based on a multi-sensor data fusion schema to perform automatic detection of handshakes between two individuals and to capture possible heart rate-based emotional reactions due to individuals meeting. A wireless, wearable, multi-sensor system for locomotion mode recognition is described in [125], with three inertial measurement units and eight force sensors, measuring both kinematic and dynamic signals of the human gait. The system uses a linear discriminant analysis classifier, obtaining good results for motion mode recognition during the stance phase, during the swing phase and for sit-to-stand transition recognition. Chen [126] developed an algorithm for data fusion to track both non-manoeuvring and manoeuvring targets with mobile sensors deployed in a wireless sensor network (WSN). It applies the GATING technique to solve the problem of mobile-sensor data fusion tracking (MSDF) for targets. In WSNs, an adaptive filter (Kalman filter) is also used, consisting of a data association technique denoted as one-step conditional maximum likelihood. Other studies have focused on sensor fusion for indoor navigation [127,128]. Saeedi et al. [127] proposes a context-aware personal navigation system (PNS) for outdoor personal navigation using a smartphone. It uses low-cost sensors in a multi-level fusion scheme to improve the accuracy and robustness of the context-aware navigation system [127]. The system developed has several challenges, such as context acquisition, context understanding and context-aware application adaptation, and it is mainly used for the recognition of the people’s activities [127]. It uses the accelerometer, gyroscope and magnetometer sensors and the GPS receiver available on the off-the-shelf mobile devices to detect and recognize the motion of the mobile device, the orientation of the mobile device and the location and context of the data acquisition [127]. The system includes a feature-level fusion scheme to recognize context information, which is applied after the data are processed and the signal’s features are extracted [127]. Bhuiyan et al. [128] evaluate the performance of several methods for multi-sensor data fusion focused on a Bayesian framework. A Bayesian framework consists of two steps [128], such as prediction and correction. In the prediction stage, the current state is updated based on the previous state and the system dynamics [128]. In the correction stage, the prediction is updated with the new measurements [128]. In [128], the authors studied different combinations of methods for data fusion, such as a linear system and Kalman filter and a non-linear system and extended Kalman filter, implementing some sensor data fusion systems with good results. In [129], a light, high-level fusion algorithm to detect the daily activities that an individual performs is presented. The proposed algorithm is designed to allow the implementation of a context-aware application installed on a mobile device, working with minimum computational cost. The quality of the estimation of the ADLs depends on the presence of biometric information and the position and number of available inertial sensors. The best estimation for continuous physical activities obtained, with the proposed algorithm, is approximately 90%. The CHRONIOUS system is developed for smart devices as a decision support system, integrating a classification system with two parallel classifiers. It combines an expert system (rule-based system) and a supervised classifier, such as SVM, random forests, ANN (e.g., the multi-layer perceptron), decision trees and naive Bayes [130]. Other systems for the recognition of ADLs have also been implemented using other classifiers during the data fusion process, e.g., decision tables. Martín et al. [131] evaluated the accuracy, computational costs and memory fingerprints in the classifiers mentioned above working with different sensor data and different optimization. The system 28 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 17 of 27 that implements these classifiers encompasses different sensors, such as acceleration, gravity, linear acceleration, magnetometer and gyroscope, with good results, as presented in [131]. Other systems were implemented using lightweight methods for sensor data fusion, e.g., the KNOWME system [132], which implements the autoregressive-correlated Gaussian model for data classification and data fusion. Other research studies focused on the compensation of the ego-motion of the camera, carried out with the data fusion of Viola and Jones face detector and inertial sensors, reporting good results [133]. In [134], sensor fusion is used for error detection, implementing the fusion of ECG with the blood pressure signal, blood pressure with body temperature and acceleration data with ECG. In [135], Jin et al. proposed a robust dead-reckoning (DR) pedestrian tracking system to be used with off-the-shelf sensors. It implements a robust tracking task as a generalized maximum a posteriori sensor fusion problem, and then, they narrow it to a simple computation procedure with certain assumptions, with a reduction in average tracking error up to 73.7%, compared to traditional DR tracking methods. Table 6. Examples of sensor data fusion methods. Sensors Methods Achievements Accelerometer; Gyroscope; Magnetometer; Compass; GPS receiver; Bluetooth; Wi-Fi; digital camera; microphone; RFID readers; IR camera. DR pedestrian tracking system; Autoregressive-Correlated Gaussian Model; CASanDRA mobile OSGi framework; Genetic Algorithms; Fuzzy Logic; Dempster-Shafer; Evidence Theory; Recursive Operators; DTW framework; CHRONIOUS system; SVM; Random Forests; ANN; Decision Trees; Naive Bayes; Decision Tables; Bayesian analysis. The use of several sensors reduces the noise effects; these methods also evaluated the accuracy of sensor data fusion; the data fusion may be performed with mobile applications, accessing the sensors data as a background process, processing the data and showing the results in a readable format. Accelerometer; Gyroscope; Magnetometer; Compass; GPS receiver; Bluetooth; WiFi; digital camera; microphone; low-cost wireless EEG sensors; RFID readers; IR camera Kalman Filtering; C-SPINE framework; DNRF method; SWNC model; GATING technique; COUPON framework; CAALYX system; high energy-efficient very large-scale integration (VLSI) architecture; sensor-fusion-based wireless walking-in-place (WIP) interaction technique; J3 criterion; DFF method; ERF method; BSWF method; inContexto system; RMS frame energy; MFCC method; pitch frequency; HNR method; ZCR method; KNN; Least squares-based estimation methods; Optimal Theory, Regularization; Uncertainty Ellipsoids. These methods allow a complex processing of amount of data acquired, because it is central processed in a server-side system; using data from several sources decreases the uncertainty level of the output; performing the data fusion process in real time can be difficult because of the large amount of data that may need to be fused; the data fusion may be performed with mobile applications, accessing the sensors data as a background process, processing the data and showing the results in a readable format or passing the results or the data to a central repository or central processing machine for further processing. Gyroscope; Compass; Magnetometer; GPS receiver. Kalman Filtering; Bayesian analysis. It is mainly useful for the context-aware localization systems; defined several recognizer algorithms to perform online temporal fusion on either the raw data or the features. ECG and others Kalman Filtering. Using data from several sources decreases the uncertainty level of the output; defined several recognizer algorithms to perform online temporal fusion on either the raw data or the features. In [136], Grunerbl et al. developed a smartphone-based recognition of states and state changes in bipolar disorder patients, implemented an optimized state change detection, developing various fusion methods with different strategies, such as logical AND, OR, and their own weighted fusion, obtaining results with good accuracy. In Table 6, a summary of sensor data fusion techniques, their achievements and sensors used is presented. This table also helps make clear that different methods can be applied to different types of data collected by different sensors. In conclusion, sensor fusion techniques for mobile devices are similar to those employed with other external sensors, because the applications usually involve embedded and external sensors at the same time. The Kalman filter is the most commonly used, but on these devices, there is a need to use low processing techniques. Some research applications need a high processing capacity, and in this case, the mobile device is only used to capture the data. After capturing the data, the captured data will be sent to a server for later processing. The most important criterion for choosing a data fusion 29 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 18 of 27 method should be based on the limited capacities of the mobile devices, as these devices increase the possibility to collect physiological data anywhere and at anytime with low cost. 4. Application in Ambient Assisted Living As described in this paper, the use of data fusion techniques with sensors available in off-the-shelf mobile devices may be shaped to form different types of systems and architectures. These techniques are implementable in systems based on mobile technologies, with local processing, server-side processing or a mix of these, or as distributed systems that work as a cooperative system collecting the data with different mobile devices. Mobile systems for the recognition of ADLs have two possible architectures, are where all processing activities are performed locally on the mobile device or another where the data processing will be executed totally, or in part, on a remote machine. Several authors have developed algorithms related to context-aware sensing (e.g., environment recognition, gestural recognition and browsing history) [88,93,104,109,115], positioning of objects [107] and localization sensing [82,90,91,94,137]. Several works have focused on the development of different methods to recognize some ADLs [92,111,112,129,132,138,139], such as sitting, walking, running, cycling, standing still, climbing stairs, using the lift, getting up, falling and having a meeting. In addition, off-the-shelf mobile devices have several sensors embedded that are capable of recognizing psychological/emotional states, including the recognition of mental stress [140]. The recognition of the ADLs [81,110,131,141,142] is very important in systems developed to support and monitor elderly people or people with some disease or impairment [143,144]. When combined with the user’s agenda, they can be employed for monitoring and learning lifestyles and physical exercise, helping people in emergency situations [116,136] and other situations where the sensors can improve life quality, e.g., driving monitoring [114]. Healthcare is one of the most important purposes of the recognition of the ADLs, and commonly, it is performed in smart health environments [4,145], with several technologies that include the relation between several mobile devices capturing the sensors’ data connected by WSNs and using sensor webs to aggregate the data collected by several sensors and identify the ADLs with more accuracy [146]. Distributed systems for sensor data fusion with off-the-shelf mobile devices have been used for tracking [120–122,126,147], location detection [83,84,123,124,148,149], health monitoring [99,103,119], user monitoring [113,150], automotive systems [151] and other purposes. In [152], several types of sensors (e.g., complementing the detection of objects using a camera and inertial sensors) are used to classify objects and detect changes. Using mobile devices to capture sensors’ signals, a problem with the involuntary movements of the mobile device during the performance of some ADLs is very high. However, the use of data fusion technologies allows one to minimize the effects of the possible involuntary movements of the mobile device, commonly considered noise [117]. In conclusion, the sensor fusion techniques are very useful to improve the recognition of the ADLs, which is the main goal of this review. Moreover, the identification of a wide range of ADLs by a mobile device is a major milestone on the development of a personal digital life coach [153]. The sensor fusion techniques increase the reliability of these systems, allowing, e.g., to learn the different peoples’ lifestyles. Many mobile applications send data to a remote computer that carries out the processing and sends back the results to the mobile device. However, the local processing is faster than server-side processing, reducing the bandwidth usage during all processes. 5. Conclusions Data acquisition, data processing, data imputation on one sensor data stream and, finally, multiple sensor data fusion together are the proposed roadmap to achieve a concrete task, and as discussed in this paper, the task at hand is the identification of activities of daily living. While these steps pose little challenge when performed in fixed computational environments, where resources are virtually 30 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 19 of 27 illimited, when their execution is intended in mobile off-the-shelf devices, a new type of challenges arises with the restrictions of the computational environment. The use of mobile sensors requires a set of techniques to classify and to process the data acquired to make them usable by software components and to automate the execution of specific tasks. The data pre-processing and data cleaning tasks are performed at the start of the sensors’ data characterization. The collected data may have inconsistent and/or unusable data, normally called environmental noise. However, the application of filters helps with removing the unusable data. The most commonly-used filters are the Kalman filter and its variants, as these methods are often reported to have good accuracy. Nevertheless, it must be noted that the correct definition of the important features of the collected data influences the correct measurement of the accuracy of the Kalman filter. Therefore, and because it is difficult to assess the accuracy of one particular type of Kalman filter, especially when used regarding the identification of activities of daily living, it is advisable to explore different sensor data fusion technologies. The main methods for the identification of the different features of the sensor signal are machine learning and pattern recognition techniques. Sensor fusion methods are normally classified into four large groups: probabilistic methods, statistical methods, knowledge-based theory methods and evidence reasoning methods. Although several methods have been studied and discussed in this paper, the choice of the best method for each purpose depends on the quantity and types of sensors used, on the diversity in the representation of the data, on the calibration of the sensors, on the limited interoperability of the sensors, on the constraints in the statistical models and, finally, on the limitations of the implemented algorithms. Sensors available in off-the-shelf mobile devices can support the implementation of sensor fusion techniques and improve the reliability of the algorithms created for these devices. However, mobile devices have limited processing capacity, memory and autonomy. Nevertheless, the fusion of the sensors’ data may be performed with mobile applications, which access the sensors’ data as a background task, processing the data collected and showing the results in a readable format to the user or sending the results or the data to a central repository or central processing machine. The techniques related to the concepts of sensor and multi-sensor data fusion presented in this paper have different purposes, including the detection/identification of activities of daily living and other medical applications. For off-the-shelf mobile devices, the positioning of the device during the data acquisition process is in itself an additional challenge, as the accuracy, precision and usability of obtained data are also a function of the sensors’ location. Therefore, it has to be expected that these systems can fail only because of poor mobile device positioning. Several research studies have been carried out regarding sensor fusion techniques applied to the sensors available in off-the-shelf mobile devices, but most of them only detect basic activities, such as walking. Due to the large market of mobile devices, e.g., smartphones, tablets or smartwatches, ambient assisted living applications on these platforms become relevant for a variety of purposes, including tele-medicine, monitoring of elderly people, monitoring of sport performance and other medical, recreational, fitness or leisure activities. Moreover, the identification of a wide range of activities of daily living is a milestone in the process of building a personal digital life coach. The applicability of sensor data fusion techniques for mobile platforms is therefore dependent on the variable characteristics of the mobile platform itself, as these are very diverse in nature and features, from local storage capability to local processing power, battery life or types of communication protocols. Nevertheless, experimenting with the algorithms and techniques described previously, so as to adapt them to a set of usage scenarios and a class of mobile devices, will render these techniques usable in most mobile platforms without major impairments. Acknowledgments: This work was supported by the FCT project UID/EEA/50008/2013 (Este trabalho foi suportado pelo projecto FCT UID/EEA/50008/2013). The authors would also like to acknowledge the contribution of the COST Action IC1303–AAPELE - Architectures, Algorithms and Protocols for Enhanced Living Environments. Author Contributions: All of the authors have contributed to the structure, content and writing of the paper. 31 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 20 of 27 Conflicts of Interest: The authors declare no conflict of interest. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 32 Yin, G.; Bruckner, D. Daily activity learning from motion detector data for Ambient Assisted Living. In Technological Innovation for Sustainability; Springer Berlin Heidelberg: Berlin, Germany; Heidelberg, Germany, 2010. pp. 89–94. Memon, M.; Wagner, S.R.; Pedersen, C.F.; Beevi, F.H.; Hansen, F.O. Ambient assisted living healthcare frameworks, platforms, standards, and quality attributes. Sensors 2014, 14, 4312–4341. Siegel, C.; Hochgatterer, A.; Dorner, T.E. Contributions of ambient assisted living for health and quality of life in the elderly and care services–A qualitative analysis from the experts’ perspective of care service professionals. BMC Geriatr. 2014, 14, 112. Holzinger, A.; Röcker, C.; Ziefle, M., From Smart Health to Smart Hospitals. In Smart Health: State-of-the-Art and Beyond, Springer Lecture Notes in Computer Science, LNCS 8700; Springer: Berlin, Germany, 2015. Developers, A. Sensors Overview | Android Developers 2015. Available online: http://developer.android. com/guide/topics/sensors/sensors_overview.html (accessed on 1 February 2016). Apple. Apple - iPhone 6 - Technical Specifications 2015. Available online: http://www.apple.com/iphone-6/ specs/ (accessed on 1 February 2016). Santochi, M.; Dini, G. Sensor Technology in Assembly Systems. CIRP Ann. Manuf. Technol. 1998, 47, 503–524. White, R. A Sensor Classification Scheme. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 1987, 34, 124–126. Roebuck, A.; Monasterio, V.; Gederi, E.; Osipov, M.; Behar, J.; Malhotra, A.; Penzel, T.; Clifford, G.D. A review of signals used in sleep analysis. Physiol. Meas. 2014, 35, R1–R57. Preece, S.J.; Goulermas, J.Y.; Kenney, L.P.; Howard, D.; Meijer, K.; Crompton, R. Activity identification using body mounted sensors: a review of classification techniques. Physiol. Meas. 2009, 30, R1–R33. Mumolo, E.; Nolich, M.; Vercelli, G. Algorithms for acoustic localization based on microphone array in service robotics. Robot. Auton. Syst. 2003, 42, 69–88. Guan, D.; Yuan, W.; Jehad Sarkar, A.M.; Ma, T.; Lee, Y.K. Review of Sensor-based Activity Recognition Systems. IETE Techn. Rev. 2011, 28, 418. Gentili, M.; Mirchandani, P.B. Locating sensors on traffic networks: Models, challenges and research opportunities. Transp. Res. Part C: Emerg. Technol. 2012, 24, 227–255. Lee, S.; Ozsecen, M.; Della Toffola, L.; Daneault, J.F.; Puiatti, A.; Patel, S.; Bonato, P. Activity detection in uncontrolled free-living conditions using a single accelerometer. In Proceedings of the 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN), Cambridge, MA, USA, 9–12 June 2015; pp. 1-6. Scheeper, P.R.; van der Donk, A.G.H.; Olthuis, W.; Bergveld, P. A review of silicon microphones. Sens. Actuators A Phys. 1994, 44, 1–11. Hoummady, M.; Campitelli, A.; Wlodarski, W. Acoustic wave sensors: Design, sensing mechanisms and applications. Smart Mater. Struct. 1997, 6, 647. Wilson, A.D.; Baietto, M. Applications and advances in electronic nose technologies. Sensors 2009, 9, 5099–5148. Suzuki, T.; Nakauchi, Y. Intelligent medicine case that monitors correct dosing. In Proceedings of the 2010 7th International Symposium on Mechatronics and Its Applications (ISMA), Sharjah, UAE, 20–22 April 2010. Hill, E.W.; Vijayaragahvan, A.; Novoselov, K. Graphene Sensors. IEEE Sens. J. 2011, 11, 3161–3170. Lenz, J.; Edelstein, S. Magnetic sensors and their applications. IEEE Sens. J. 2006, 6, 631–649. Allen, J. Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 2007, 28, R1–R39. Deng, K.-L. Next generation fiber optic sensing and its applications. In Proceedings of the International Conference on Wireless and Optical Communications, Newark, NJ, USA, 22–23 April 2005. Khusainov, R.; Azzi, D.; Achumba, I.E.; Bersch, S.D. Real-time human ambulation, activity, and physiological monitoring: taxonomy of issues, techniques, applications, challenges and limitations. Sensors 2013, 13, 12852–12902. Zhang, L.; Liu, J.; Jiang, H.; Guan, Y. SensTrack: Energy-Efficient Location Tracking With Smartphone Sensors. IEEE Sens. J. 2013, 13, 3775–3784. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 40. 41. 42. 43. 44. 21 of 27 Novak, D.; Rebersek, P.; De Rossi, S.M.; Donati, M.; Podobnik, J.; Beravs, T.; Lenzi, T.; Vitiello, N.; Carrozza, M.C.; Munih, M. Automated detection of gait initiation and termination using wearable sensors. Med. Eng. Phys. 2013, 35, 1713–1720. He, Y.; Li, Y. Physical Activity Recognition Utilizing the Built-In Kinematic Sensors of a Smartphone. Int. J. Distrib. Sens. Netw. 2013, 2013, 1–10. Stopczynski, A.; Stahlhut, C.; Larsen, J.E.; Petersen, M.K.; Hansen, L.K. The smartphone brain scanner: A portable real time neuroimaging system. PLoS ONE 2014, 9, e86733. Daponte, P.; de Vito, L.; Picariello, F.; Riccio, M. State of the art and future developments of measurement applications on smartphones. Measurement 2013, 46, 3291–3307. Scalvini, S.; Baratti, D.; Assoni, G.; Zanardini, M.; Comini, L.; Bernocchi, P. Information and communication technology in chronic diseases: A patient’s opportunity. J. Med. Person 2013, 12, 91–95. Bersch, S.D.; Azzi, D.; Khusainov, R.; Achumba, I.E.; Ries, J. Sensor data acquisition and processing parameters for human activity classification. Sensors 2014, 14, 4239–4270. Lim, L.; Misra, A.; Mo, T. Adaptive data acquisition strategies for energy-efficient, smartphone-based, continuous processing of sensor streams. Distrib. Parallel Databases 2012, 31, 321–351. Paucher, R.; Turk, M. Location-based augmented reality on mobile phones. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), San Francisco, CA, USA, 13–18 June 2010. Misra, A.; Lim, L. Optimizing Sensor Data Acquisition for Energy-Efficient Smartphone-Based Continuous Event Processing. In Proceedings of the 2011 12th IEEE International Conference on Mobile Data Management (MDM), Lulea, Sweden, 6–9 June 2011. Kang, S.; Lee, Y.; Min, C.; Ju, Y.; Park, T.; Lee, J.; Rhee, Y.; Song, J. Orchestrator: An active resource orchestration framework for mobile context monitoring in sensor-rich mobile environments. In Proceedings of the 2010 IEEE International Conference on Pervasive Computing and Communications (PerCom), Mannheim, Germany, 29 March–2 April 2010; pp. 135–144. Agrawal, S.; Kamal, R. Computational Orchestrator: A Super Class for Matrix, Robotics and Control System Orchestration. Int. J. Comput. Appl. 2015, 117, 12–19. Vallina-Rodriguez, N.; Crowcroft, J. ErdOS: Achieving Energy Savings in Mobile OS. In Proceedings of the Sixth International Workshop on MobiArch (MobiArch’11), Bethesda, MD, USA, 28 June 2011; pp. 37–42. Priyantha, B.; Lymberopoulos, D.; Liu, J. LittleRock: Enabling Energy-Efficient Continuous Sensing on Mobile Phones. IEEE Pervasive Comput. 2011, 10, 12–15. Lu, H.; Yang, J.; Liu, Z.; Lane, N.D.; Choudhury, T.; Campbell, A.T. The Jigsaw Continuous Sensing Engine for Mobile Phone Applications. In Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems (SenSys’10), Zurich, Switzerland, 3–5 November 2010; pp. 71–84. Rachuri, K.K.; Mascolo, C.; Musolesi, M.; Rentfrow, P.J. SociableSense: Exploring the Trade-offs of Adaptive Sampling and Computation Offloading for Social Sensing. In Proceedings of the 17th Annual International Conference on Mobile Computing and Networking (MobiCom’11), Las Vegas, NV, USA, 19–23 September 2011; pp. 73–84. Deshpande, A.; Guestrin, C.; Madden, S.R.; Hellerstein, J.M.; Hong, W. Model-driven Data Acquisition in Sensor Networks. In Proceedings of the Thirtieth International Conference on Very Large Data Bases-Volume 30. VLDB Endowment (VLDB’04), Toronto, ON, Canada, 31 August–3 September 2004; pp. 588–599. Reilent, E.; Loobas, I.; Pahtma, R.; Kuusik, A. Medical and context data acquisition system for patient home monitoring. In Proceedings of the 2010 12th Biennial Baltic Electronics Conference (BEC), Tallinn, Republic of Estonia, 4–6 October 2010. Marzencki, M.; Hung, B.; Lin, P.; Huang, Y.; Cho, T.; Chuo, Y.; Kaminska, B. Context-aware physiological data acquisition and processing with wireless sensor networks. In Proceedings of the 2010 IEEE International Workshop on Medical Measurements and Applications Proceedings (MeMeA), Ottawa, ON, Canada, 30 April–1 May 2010. Agoston, K.; Nagy, C. Data acquisition for angular measuring and positioning system. In Proceedings of the 2012 IEEE International Conference on Automation Quality and Testing Robotics (AQTR), Cluj-Napoca, Romania, 24–27 May 2012 . Harvey, P.; Woodward, B.; Datta, S.; Mulvaney, D. Data acquisition in a wireless diabetic and cardiac monitoring system. IEEE Eng. Med. Biol. Soc. Conf. Proc. 2011, 2011, 3154–3157. 33 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 59. 60. 61. 62. 63. 64. 34 22 of 27 Bieber, G.; Haescher, M.; Vahl, M. Sensor requirements for activity recognition on smart watches. In Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments, Island of Rhodes, Greece, 29–31 May 2013; pp. 1–6. Sorber, J.M.; Shin, M.; Peterson, R.; Kotz, D. Plug-n-trust: Practical trusted sensing for mhealth. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services, Lake District, UK, 25–29 June 2012; pp. 309–322. Zhu, W.; Liu, L.; Yin, S.; Hu, S.; Tang, E.Y.; Wei, S. Motion-sensor fusion-based gesture recognition and its VLSI architecture design for mobile devices. Int. J. Electron. 2013, 101, 621–635. Lane, N.; Miluzzo, E.; Lu, H.; Peebles, D.; Choudhury, T.; Campbell, A. A survey of mobile phone sensing. IEEE Commun. Mag. 2010, 48, 140–150. Kim, T.h.; Adeli, H.; Robles, R.J.; Balitanas, M. Ubiquitous Computing and Multimedia Applications; Springer: New York, NY, USA, 2011. Hwang, Y.C.; Oh, R.D.; Ji, G.H. A Sensor Data Processing System for Mobile Application Based Wetland Environment Context-aware. In Ubiquitous Computing and Multimedia Applications; Springer: Berlin, Germany; Heidelberg, Germany, 2011. Choi, M. A Platform-Independent Smartphone Application Development Framework. In Computer Science and Convergence; Springer Netherlands: Springer: Dordrecht, The Netherlands, 2012. Bersch, S.D.; Azzi, D.; Khusainov, R.; Achumba, I.E.; Ries, J. Sensor data acquisition and processing parameters for human activity classification. Sensors 2014, 14, 4239–4270. Pejovic, V.; Musolesi, M. Anticipatory Mobile Computing. ACM Comput. Surv. 2015, 47, 1–29. Imai, S.; Miyamoto, M.; Arai, Y.; Inomata, T. A data processing method for motion estimation considering network and sensor node loads. In Proceedings of the 2012 IEEE 11th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), Kyoto, Japan, 22–24 August 2012; pp. 356–362. Pombo, N.; Garcia, N.; Felizardo, V.; Bousson, K. Big data reduction using RBFNN: A predictive model for ECG waveform for eHealth platform integration. In Proceedings of the 2014 IEEE 16th International Conference on e-Health Networking, Applications and Services (Healthcom), Natal, Brazil, 15–18 October 2014. Yamada, S.; Watanabe, Y.; Kitagawa, H.; Amagasa, T. Location-Based Information Delivery Using Stream Processing Engine. In Proceedings of the 7th International Conference on Mobile Data Management (2006 MDM), Nara, Japan, 10–12 May 2006. Lin, F.X.; Rahmati, A.; Zhong, L. Dandelion: A framework for transparently programming phone-centered wireless body sensor applications for health. In Proceedings of the WH’10 Wireless Health 2010, San Diego, CA, USA, 5–7 October 2010. Dolui, K.; Mukherjee, S.; Datta, S.K. Smart Device Sensing Architectures and Applications. In Proceedings of the 2013 International Computer Science and Engineering Conference (Icsec), Bangkok, Thailand, 4–7 September 2013; pp. 91–96. Imai, S.; Miyamoto, M.; Arai, Y.; Inomata, T. Sensor Data Processing Method Based on Observed Person’s Similarity for Motion Estimation. In Proceedings of the 2013 IEEE 27th International Conference on Advanced Information Networking and Applications Workshops (Waina), Barcelona, Spain, 25–28 March 2013; pp. 601–606. Gurrin, C.; Qiu, Z.; Hughes, M.; Caprani, N.; Doherty, A.R.; Hodges, S.E.; Smeaton, A.F. The smartphone as a platform for wearable cameras in health research. Am. J. Prev. Med. 2013, 44, 308–313. Postolache, O.; Girao, P.S.; Ribeiro, M.; Guerra, M.; Pincho, J.; Santiago, F.; Pena, A. Enabling telecare assessment with pervasive sensing and Android OS smartphone. In Proceedings of the 2011 IEEE International Workshop on Medical Measurements and Applications Proceedings (MeMeA), Bari, Italy, 30–31 May 2011 . Wang, G.; Zimmermann, R. Spatial sensor data processing and analysis for mobile media applications. In Proceedings of the 1st ACM SIGSPATIAL PhD Workshop, Dallas, TX, USA, 4–7 November 2014; pp. 1–5. Vateekul, P.; Sarinnapakorn, K. Tree-Based Approach to Missing Data Imputation. In Proceedings of the IEEE International Conference on Data Mining Workshops (ICDMW’09), Miami, FL, USA, 6 December 2009. D’Ambrosio, A.; Aria, M.; Siciliano, R. Accurate Tree-based Missing Data Imputation and Data Fusion within the Statistical Learning Paradigm. J. Classif. 2012, 29, 227–258. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 78. 79. 80. 81. 82. 83. 84. 85. 86. 87. 23 of 27 Huang, X.Y.; Li, W.; Chen, K.; Xiang, X.H.; Pan, R.; Li, L.; Cai, W.X. Multi-matrices factorization with application to missing sensor data imputation. Sensors 2013, 13, 15172–15186. García-Laencina, P.J.; Sancho-Gómez, J.L.; Figueiras-Vidal, A.R.; Verleysen, M. K nearest neighbours with mutual information for simultaneous classification and missing data imputation. Neurocomputing 2009, 72, 1483–1493. Ni, D.; Leonard, J.D.; Guin, A.; Feng, C. Multiple Imputation Scheme for Overcoming the Missing Values and Variability Issues in ITS Data. J. Transp. Eng. 2005, 131, 931–938. Smith, B.; Scherer, W.; Conklin, J. Exploring Imputation Techniques for Missing Data in Transportation Management Systems. Transp. Res. Rec. 2003, 1836, 132–142. Qu, L.; Zhang, Y.; Hu, J.; Jia, L.; Li, L. A BPCA based missing value imputing method for traffic flow volume data. In Proceedings of the 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, The Netherlands, 4–6 June 2008 . Jiang, N.; Gruenwald, L. Estimating Missing Data in Data Streams. In Proceedings of the 12th International Conference on Database Systems for Advanced Applications (DASFAA 2007), Bangkok, Thailand, 9–12 April 2007. Ling, W.; Dong-Mei, F. Estimation of Missing Values Using a Weighted K-Nearest Neighbors Algorithm. In Proceedings of the International Conference on Environmental Science and Information Application Technology (ESIAT 2009), Wuhan, China, 4–5 July 2009. Hruschka, E.R.; Hruschka, E.R.; Ebecken, N.F.F., Towards Efficient Imputation by Nearest-Neighbors: A Clustering-Based Approach. In AI 2004: Advances in Artificial Intelligence; Springer: Berlin, Germany; Heidelberg, Germany, 2004; pp. 513–525. Luo, J.; Yang, T.; Wang. Y. Missing value estimation for microarray data based on fuzzy C-means clustering. In Proceedings of the Eighth International Conference on High-Performance Computing in Asia-Pacific Region, Beijing, China, 1 July 2005. Smaragdis, P.; Raj, B.; Shashanka, M. Missing Data Imputation for Time-Frequency Representations of Audio Signals. J. Signal Process. Syst. 2010, 65, 361–370. Liu, Y.; Lv, Z.; Wang, W. An Improved Generalized-Trend-Diffusion-Based Data Imputation for Steel Industry. Math. Probl. Eng. 2013, 2013, 1–10. Iacus, S.M.; Porro, G. Missing data imputation, matching and other applications of random recursive partitioning. Comput. Stat. Data Anal. 2007, 52, 773–789. Bruni, R. Discrete models for data imputation. Discret. Appl. Math. 2004, 144, 59–69. Pombo, N.G.C.C. Information Technologies for Pain Management. Ph.D. Thesis, University of Beira Interior, Covilhã, Portugal, 2014. Krichmar, J.L.; Snook, J.A. A neural approach to adaptive behavior and multi-sensor action selection in a mobile device. Rob. Autom. 2002, 4, 3864–3869. Kim, D.J.; Prabhakaran, B. Faulty and Missing Body Sensor Data Analysis. In Proceedings of the 2013 IEEE International Conference on Healthcare Informatics (ICHI), Philadelphia, PA, USA, 9–11 September 2013. Banos, O.; Damas, M.; Pomares, H.; Rojas, I. On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 2012, 12, 8039–8054. Ma, Z.; Qiao, Y.; Lee, B.; Fallon, E. Experimental evaluation of mobile phone sensors. In Proceedings of the Signals and Systems Conference (ISSC 2013), 24th IET Irish, Letterkenny, Ireland, 20–21 June 2013. Durrant-Whyte, H.; Stevens, M.; Nettleton, E. Data fusion in decentralised sensing networks. In Proceedings of the 4th International Conference on Information Fusion, Montreal, PQ, Canada, 7–10 August 2001. Aziz, A.M. A new adaptive decentralized soft decision combining rule for distributed sensor systems with data fusion. Inf. Sci. 2014, 256, 197–210. Akhoundi, M.A.A.; Valavi, E. Multi-Sensor Fuzzy Data Fusion Using Sensors with Different Characteristics. ArXiv E-Prints 2010, arXiv:1010.6096. Khaleghi, B.; Khamis, A.; Karray, F.O.; Razavi, S.N. Multisensor data fusion: A review of the state-of-the-art. Inf. Fusion 2013, 14, 28-44. Pombo, N.; Bousson, K.; Araújo, P.; Viana, J. Medical decision-making inspired from aerospace multisensor data fusion concepts. Inf. Health Soc. Care 2015, 40, 185–197. 35 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 88. 89. 90. 91. 92. 93. 94. 95. 96. 97. 98. 99. 100. 101. 102. 103. 104. 105. 106. 107. 108. 36 24 of 27 Ko, M.H.; West, G.; Venkatesh, S.; Kumar, M. Using dynamic time warping for online temporal fusion in multisensor systems. Inf. Fusion 2008, 9, 370–388. Tanveer, F.; Waheed, O.T.; ur Rehman, A. Design and Development of a Sensor Fusion based Low Cost Attitude Estimator. J. Space Technol. 2011, 1, 45–50. Zhao, L.; Wu, P.; Cao, H. RBUKF Sensor Data Fusion for Localization of Unmanned Mobile Platform. Res. J. Appl. Sci. Eng. Technol. 2013, 6, 3462–3468. Walter, O.; Schmalenstroeer, J.; Engler, A.; Haeb-Umbach, R. Smartphone-based sensor fusion for improved vehicular navigation. In Proceedings of the 10th Workshop on Positioning Navigation and Communication (WPNC), Dresden, Germay, 20–21 March 2013. Iglesias, J.; Cano, J.; Bernardos, A.M.; Casar, J.R. A ubiquitous activity-monitor to prevent sedentariness. In Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), Seattle, WA, USA, 21–25 March 2011. Yongkai, Z.; Shuangquan, W.; Zhuang, Z.; Canfeng, C.; Jian, M. A mobile device oriented framework for context information management. In Proceedings of the IEEE Youth Conference on Information, Computing and Telecommunication, Beijing, China, 20–21 September 2009. Blum, J.R.; Greencorn, D.G.; Cooperstock, J.R. Smartphone Sensor Reliability for Augmented Reality Applications. In Mobile and Ubiquitous Systems: Computing, Networking, and Services; Springer: Berlin, Germany, 2013. Neidhardt, A.; Luss, H.; Krishnan, K.R. Data fusion and optimal placement of fixed and mobile sensors. In Proceedings of the IEEE Sensors Applications Symposium, Atlanta, GA, USA, 12–14 February 2008. Haala, N.; Böhm, J. A multi-sensor system for positioning in urban environments. ISPRS J. Photogramm. Remote Sens. 2003, 58, 31–42. Theodoridis, S. Machine Learning: A Bayesian and Optimization Perspective; Academic Press: Cambridge, MA, USA, 2015. Miller, G. The Smartphone Psychology Manifesto. Perspect. Psychol. Sci. 2012, 7, 221–237. Yi, W.J.; Sarkar, O.; Mathavan, S.; Saniie, J. Wearable sensor data fusion for remote health assessment and fall detection. In Proceedings of the IEEE International Conference on Electro/Information Technology (EIT), Milwaukee, WI, USA, 5–7 June 2014. Glenn, T.; Monteith, S. New measures of mental state and behavior based on data collected from sensors, smartphones, and the Internet. Curr. Psychiatry Rep. 2014, 16, 1–10. Ponmozhi, J.; Frias, C.; Marques, T.; Frazão, O. Smart sensors/actuators for biomedical applications: Review. Measurement 2012, 45, 1675–1688. Beach, A.; Gartrell, M.; Xing, X.; Han, R.; Lv, Q.; Mishra, S.; Seada, K. Fusing mobile, sensor, and social data to fully enable context-aware computing. In Proceedings of the Eleventh Workshop on Mobile Computing Systems Applications, Annapolis, MD, USA, 22–23 February 2010. Andò, B.; Baglio, S.; Pistorio, A. A Smart Multi-Sensor Approach to Monitoring Weak People in Indoor Environments. J. Sens. Technol. 2014, 4, 24–35. Phan, T.; Kalasapur, S.; Kunjithapatham, A. Sensor fusion of physical and social data using Web SocialSense on smartphone mobile browsers. In Proceedings of the IEEE 11th Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA, 10–13 January 2014. Steed, A.; Julier, S. Behaviour-aware sensor fusion: Continuously inferring the alignment of coordinate systems from user behaviour. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Adelaide, Australia, 1–4 October 2013. Pucihar, K.C.; Coulton, P.; Hutchinson, D. Utilizing sensor fusion in markerless mobile augmented reality. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, Stockholm, Sweden, 30 August–2 September 2011. Rahman, A.S.M.M.; Hossain, M.A.; Saddik, A.E. Spatial-geometric approach to physical mobile interaction based on accelerometer and IR sensory data fusion. ACM Trans. Multimed. Comput. Commun. Appl. 2010, 6, 1–23. Grunerbl, A.; Muaremi, A.; Osmani, V.; Bahle, G.; Ohler, S.; Troester, G.; Mayora, O.; Haring, C.; Lukowicz, P. Smart-Phone Based Recognition of States and State Changes in Bipolar Disorder Patients. IEEE J Biomed. Health Inf. 2014, doi:10.1109/JBHI.2014.2343154. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 25 of 27 109. Gil, G.B.; Berlanga de Jesus, A.; Molina Lopez, J.M. inContexto: A fusion architecture to obtain mobile context. In Proceedings of the Proceedings of the 14th International Conference on Information Fusion (FUSION), Chicago, IL, USA, 5–8 July 2011. 110. Kim, J.; Gracanin, D.; Quek, F. Sensor-fusion walking-in-place interaction technique using mobile devices. In Proceedings of the 2012 IEEE Virtual Reality Short Papers and Posters (VRW), Costa Mesa, CA, USA, 4–8 March 2012. 111. Abadi, M.J.; Gu, Y.; Guan, X.; Wang, Y.; Hassan, M.; Chou, C.T. Improving Heading Accuracy in Smartphone-based PDR Systems using Multi-Pedestrian Sensor Fusion. Electr. Eng. 2013 188, 9–35. 112. Altini, M.; Vullers, R.; Van Hoof, C.; van Dort, M.; Amft, O. Self-calibration of walking speed estimations using smartphone sensors. In Proceedings of the 2014 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), Budapest, Hungary, 24–28 March 2014. 113. Tsai, P.H.; Lin, Y.J.; Ou, Y.Z.; Chu, E.T.H.; Liu, J.W.S. A Framework for Fusion of Human Sensor and Physical Sensor Data. IEEE Trans. Syst. Man Cybern. Syst. 2014, 44, 1248–1261. 114. Lee, B.G.; Chung, W.Y. A smartphone-based driver safety monitoring system using data fusion. Sensors 2012, 12, 17536–17552. 115. Chen, D.; Schmidt, A.; Gellersen, H.W. An Architecture for Multi-Sensor Fusion in Mobile Environments. Available online: http://www.cs.cmu.edu/ datong/Fusion99.pdf (accessed on 1 February 2016). 116. Sashima, A.; Ikeda, T.; Kurumatani, K. Toward Mobile Sensor Fusion Platform for Context Aware Services. Available online: http://cdn.intechopen.com/pdfs-wm/6799.pdf (accessed on 1 February 2016). 117. Ayub, S.; Bahraminisaab, A.; Honary, B. A sensor fusion method for smart phone orientation estimation. In Proceedings of the 13th Annual Post Graduate Symposium on the Convergence of Telecommunications, Networking and Broadcasting, Liverpool, UK, 25–26 June 2012 118. Zhu, W.; Liu, L.; Yin, S.; Hu, S.; Tang, E.Y.; Wei, S. Motion-sensor fusion-based gesture recognition and its VLSI architecture design for mobile devices. Int. J. Electron. 2013, 101, 621–635. 119. van de Ven, P.; Bourke, A.; Tavares, C.; Feld, R.; Nelson, J.; Rocha, A.; O Laighin, G. Integration of a suite of sensors in a wireless health sensor platform. In Proceedings of the 2009 IEEE Sensors, Christchurch, New Zealand, 25–28 October 2009 120. Chen, J.; Low, K.H.; Tan, C.K.Y.; Oran, A.; Jaillet, P.; Dolan, J.M.; Sukhatme, G.S. Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena arXiv preprint arXiv:1206.6230 2012, arXiv:1206.6230. 121. Zhao, D.; Ma, H.; Tang, S. COUPON: Cooperatively Building Sensing Maps in Mobile Opportunistic Networks. In Proceedings of the 2013 IEEE 10th International Conference on Mobile Ad-Hoc and Sensor Systems (MASS), Hangzhou, China, 14–16 October 2013. 122. Zhao, D.; Ma, H.; Tang, S.; Li, X.Y. COUPON: A Cooperative Framework for Building Sensing Maps in Mobile Opportunistic Networks. IEEE Trans. Parallel Distrib. Syst. 2015, 26, 392–402. 123. Deyun, G.; Tao, Z.; Dong, P.; Sidong, Z. A general multi-sensor node in wireless sensor networks. In Proceedings of the 2009. ICCTA ’09. IEEE International Conference on Communications Technology and Applications, Beijing, China, 16–18 October 2009. 124. Fortino, G.; Galzarano, S.; Gravina, R.; Li, W. A framework for collaborative computing and multi-sensor data fusion in body sensor networks. Inf. Fusion 2015, 22, 50–70. 125. Zheng, E.; Chen, B.; Wang, X.; Huang, Y.; Wang, Q. On the Design of a Wearable Multi sensor System for Recognizing Motion Modes and Sit to stand Transition. Int. J. Adv. Robot. Syst. 2014, doi:10.5772/57788. 126. Chen, J.I.Z. An Algorithm of Mobile Sensors Data Fusion Orientation tracking for Wireless Sensor Networks. Wirel. Pers. Commun. 2009, 58, 197–214. 127. Saeedi, S.; Moussa, A.; El-Sheimy, N. Context-aware personal navigation using embedded sensor fusion in smartphones. Sensors 2014, 14, 5742–5767. 128. Bhuiyan, M.Z.H.; Kuusniemi, H.; Chen, L.; Pei, L.; Ruotsalainen, L.; Guinness, R.; Chen, R. Performance Evaluation of Multi-Sensor Fusion Models in Indoor Navigation. Eur. J. Navig. 2013, 11, 21–28. 129. Martin, H.; Bernardos, A.M.; Tarrio, P.; Casar, J.R. Enhancing activity recognition by fusing inertial and biometric information. In Proceedings of the 14th International Conference on Information Fusion, Chicago, IL, USA, 5–8 July 2011. 37 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 26 of 27 130. Bellos, C.; Papadopoulos, A.; Rosso, R.; Fotiadis, D.I. Heterogeneous data fusion and intelligent techniques embedded in a mobile application for real-time chronic disease management. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 2011, 8303–8306. 131. Martín, H.; Bernardos, A.M.; Iglesias, J.; Casar, J.R. Activity logging using lightweight classification techniques in mobile devices. Pers. Ubiquitous Comput. 2012, 17, 675–695. 132. Thatte, G.; Li, M.; Lee, S.; Emken, B.A.; Annavaram, M.; Narayanan, S.; Spruijt-Metz, D.; Mitra, U. Optimal Time-Resource Allocation for Energy-Efficient Physical Activity Detection. IEEE Trans. Signal Process. 2011, 59, 1843–1857. 133. Scheuermann, B.; Ehlers, A.; Riazy, H.; Baumann, F.; Rosenhahn, B. Ego-motion compensated face detection on a mobile device. In Proceedings of the 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Colorado Springs, CO, USA, 20–25 June 2011. 134. Klingeberg, T.; Schilling, M. Mobile wearable device for long term monitoring of vital signs. Comput. Methods Progr. Biomed. 2012, 106, 89–96. 135. Jin, Y.Y.; Toh, H.S.; Soh, W.S.; Wong, W.C. A Robust Dead-Reckoning Pedestrian Tracking System with Low Cost Sensors. In Proceedings of the 2011 IEEE International Conference on Pervasive Computing and Communications (PerCom), Seattle, WA, USA, 21–25 March 2011. 136. Grunerbl, A.; Muaremi, A.; Osmani, V.; Bahle, G.; Ohler, S.; Troester, G.; Mayora, O.; Haring, C.; Lukowicz, P. Smart-Phone Based Recognition of States and State Changes in Bipolar Disorder Patients. IEEE J. Biomed. Health Inf. 2014, doi:10.1109/JBHI.2014.2343154. 137. García, F.; Jiménez, F.; Anaya, J.J.; Armingol, J.M.; Naranjo, J.E.; de la Escalera, A. Distributed Pedestrian Detection Alerts Based on Data Fusion with Accurate Localization. Sensors 2013, 13, 11687–11708. 138. Ou, S.; Fagg, A.H.; Shenoy, P.; Chen, L. Application Of Reinforcement Learning In Multisensor Fusion Problems With Conflicting Control Objectives. Intell. Autom. Soft Comput. 2009, 15, 223–235. 139. Wang, J.; Chen, G.; Kotz, D. A sensor-fusion approach for meeting detection. In Proceedings of the Workshop on Context Awareness at the Second International Conference on Mobile Systems, Applications, and Services, Boston, MA, USA, 6–9 June 2004. 140. Seoane, F.; Mohino-Herranz, I.; Ferreira, J.; Alvarez, L.; Buendia, R.; Ayllón, D.; Llerena, C.; Gil-Pita, R. Wearable Biomedical Measurement Systems for Assessment of Mental Stress of Combatants in Real Time. Sensors 2014, 14, 7120–7141. 141. Luque, R.; Casilari, E.; Moron, M.J.; Redondo, G. Comparison and characterization of Android-based fall detection systems. Sensors 2014, 14, 18543–18574. 142. Salah, O.; Ramadan, A.A.; Sessa, S.; Ismail, A.A.; Fujie, M.; Takanishi, A. ANFIS-based Sensor Fusion System of Sit- to- stand for Elderly People Assistive Device Protocols. Int. J. Autom. Comput. 2014, 10, 405–413. 143. Kleinberger, T.; Becker, M.; Ras, E.; Holzinger, A.; Müller, P. Ambient Intelligence in Assisted Living: Enable Elderly People to Handle Future Interfaces. In Proceedings of the 4th International Conference on Universal Access in Human-computer Interaction: Ambient Interaction, Beijing, China, 22–27 July 2007. 144. Holzinger, A.; Searle, G.; Pruckner, S.; Steinbach-Nordmann, S.; Kleinberger, T.; Hirt, E.; Temnitzer, J. Perceived usefulness among elderly people: Experiences and lessons learned during the evaluation of a wrist device. In Proceedings of the 2010 4th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth), Munich, Germany, 22–25 March 2010. 145. Ziefle, M.; Rocker, C.; Holzinger, A. Medical Technology in Smart Homes: Exploring the User’s Perspective on Privacy, Intimacy and Trust. In Proceedings of the 2011 IEEE 35th Annual Computer Software and Applications Conference Workshops (COMPSACW), Munich, Germany, 18–22 July 2011. 146. Macias, E.; Suarez, A.; Lloret, J. Mobile sensing systems. Sensors 2013, 13, 17292–17321. 147. Volkov, A.S. Accuracy bounds of non-Gaussian Bayesian tracking in a NLOS environment. Signal Process. 2015, 108, 498–508. 148. Gulrez, T.; Kavakli, M. Precision Position Tracking in Virtual Reality Environments using Sensor Networks. In Proceedings of the IEEE International Symposium on Industrial Electronics, 2007. ISIE 2007, Vigo, Spain, 4–7 June, 2007. 149. Chen, C.A.; Chen, S.L.; Huang, H.Y.; Luo, C.H. An asynchronous multi-sensor micro control unit for wireless body sensor networks (WBSNs). Sensors 2011, 11, 7022–7036. 38 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2016, 16, 184 27 of 27 150. Castro Garrido, P.; Luque Ruiz, I.; Gomez-Nieto, M.A. AGATHA: Multiagent system for user monitoring. In Proceedings of the 2012 IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin), Berlin, Germany, 3–5 September 2012. 151. Broggi, A.; Debattisti, S.; Panciroli, M.; Grisleri, P.; Cardarelli, E.; Buzzoni, M.; Versari, P. High performance multi-track recording system for automotive applications. Int. J. Autom. Technol. 2011, 13, 123–132. 152. Dong, J.; Zhuang, D.; Huang, Y.; Fu, J. Advances in multi-sensor data fusion: algorithms and applications. Sensors 2009, 9, 7771–7784. 153. Garcia, N.M. A Roadmap to the Design of a Personal Digital Life Coach. ICT Innovations 2016, 399, 21–27. © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/). 39 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 40 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 2. Validation Techniques for Sensor Data in Mobile Health Applications The following article is the second part of the chapter 2. Validation Techniques for Sensor Data in Mobile Health Applications Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta and Natalia Díaz Rodríguez Journal of Sensors (Hindawi Publishing Corporation), published, 2016. According to 2017 Journal Citation Reports published by Thomson Reuters in 2018, this journal has the following performance metrics: ISI Impact Factor (2017): 2.057 ISI Article Influence Score (2017): 0.3 Journal Ranking (2017): 259/644 (Electrical and Electronic Engineering) 41 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 42 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Hindawi Publishing Corporation Journal of Sensors Volume 2016, Article ID 2839372, 9 pages http://dx.doi.org/10.1155/2016/2839372 Review Article Validation Techniques for Sensor Data in Mobile Health Applications Ivan Miguel Pires,1,2,3 Nuno M. Garcia,1,3,4 Nuno Pombo,1,3 Francisco Flórez-Revuelta,5 and Natalia Díaz Rodríguez6 1 Instituto de Telecomunicações, Universidade of Beira Interior, Covilhã, Portugal Altranportugal, Lisbon, Portugal 3 Assisted Living Computing and Telecommunications Laboratory (ALLab), Computer Science Department, Universidade of Beira Interior, Covilhã, Portugal 4 ECATI, Universidade Lusófona de Humanidades e Tecnologias, Lisbon, Portugal 5 Department of Computer Technology, Universidad de Alicante, Alicante, Spain 6 Department of Computer Science and Artificial Intelligence, CITIC-UGR, University of Granada, Granada, Spain 2 Correspondence should be addressed to Ivan Miguel Pires; impires@it.ubi.pt Received 27 February 2016; Revised 25 August 2016; Accepted 4 September 2016 Academic Editor: Francesco Dell’Olio Copyright © 2016 Ivan Miguel Pires et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Mobile applications have become a must in every user’s smart device, and many of these applications make use of the device sensors’ to achieve its goal. Nevertheless, it remains fairly unknown to the user to which extent the data the applications use can be relied upon and, therefore, to which extent the output of a given application is trustworthy or not. To help developers and researchers and to provide a common ground of data validation algorithms and techniques, this paper presents a review of the most commonly used data validation algorithms, along with its usage scenarios, and proposes a classification for these algorithms. This paper also discusses the process of achieving statistical significance and trust for the desired output. 1. Introduction There has been an increase of the number of mobile applications that make use of sensors to achieve a plethora of goals. Many of these applications are designed and developed by amateur programmers, and that in itself is good as it confirms an increase in the overall set of skills of the developer community. Nevertheless, and even when the applications are developed by professionals or by companies, there are not many applications that publicize or disclose how the sensors’ data is processed. This is a problem, in particular when these applications are meant to be used in a scenario where they can influence their users’ lives, as for example, when the data is expected to be used to identify Activities of Daily Living (ADLs) or, to an extreme, when the applications are used in medical scenarios. Due to the nature of the mobile device itself, multiprocessing, with limited computational power and limited battery life, the data that is collected from the sensors is often unusable in its primary form, requiring further processing to allow it to be representative of the event or object that it is supposed to measure. The recording of sensor data and the sequent processing of this data need to include validation subtasks that guarantee that the data are suitable to be fed into the higher-level algorithms. Moreover, the use of the sensors’ data to feed higherlevel algorithms needs to guarantee a minimum degree of error, with this error being the difference between the output of these applications, built on limited computational mobile platforms, and the output of a golden standard. To achieve a minimum degree of error, statistical methods need to be applied to ensure that the output of the mobile application is to maximum extent similar to the output given by the relevant golden standard, if and when this is possible. To mitigate this problem, this paper presents and discusses the most used data validation algorithms and 43 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 2 techniques and their usage in a mobile application that relies on the sensors’ data to give meaningful output to its user. The algorithms are listed and their use is discussed. The discussion of the statistical process to ensure maximum reliability of the results is also presented. The remainder of this paper is organized as follows: this paragraph concludes Section 1, where a short introduction to the problem and a proposal to achieve its mitigation are disclosed; Section 2 presents the most commonly found data validation methods, along with a critical comparison of its usage scenarios; Section 3 deepens the analysis presenting a classification of the data validation methods; Section 4 discusses the applicability of these methods, including the discussion of the degree of trust the data can be expected to provide; finally, Section 5 presents relevant conclusions. 2. Data Validation Methods Sensor data validation is an important process executed during the data acquisition and data processing modules of the multisensor mobile system. This process consists of the validation of the external conditions of the data and the validity of the data for specific purpose, in order to obtain accurate and reliable results. The sequence of this validation may be applied not only in data acquisition but also in data processing since increase, as these increase the degree of confidence of the systems, with the confidence in the output being of great importance, especially for systems involved in medical diagnosis, but also for the identification of ADLs or sports monitoring. In addition, data validation methods must be used during the different phases of the conception of a new system, such as design, development, tests, and validation. Therefore, the data validation methods with verified reliability during the conception should be also used to validate the data automatically during the execution time. One of the causes for the presence of incorrect values during the data acquisition process may be existence of environmental noise. Even when the data is correctly collected, the data may still be incorrect because of noise. Therefore, very often the data captured or processed has to be cleaned, treated, or imputed to obtain better and reliable results. Following the existence of missing values at random instants of time, the causes may be the mechanical problems or power failures of sensors. At this case, data correction methods should be applied, including data imputation and data cleaning. The data validation process may be simplified as presented in Figure 1. The selection of the best technique for sensor data validation also depends on the type of data collected, the purpose of its application, and the computational platform where the algorithm will be run. Data validation techniques are commonly composed by statistical methods. Due to the characteristics of mobile devices, data validation techniques can be executed locally in the mobile device or at the server-side, depending on the amount of data to validate simultaneously, the frequency of the validation tasks, and the computational, communication, and storage resources needed for the validation. The characteristics of the sensors 44 Journal of Sensors are also important for the selection of the best techniques, which may be separated in three large groups, which are sensor performance characteristics, pervasive metrics, and environmental characteristics [1]. While data validation is important for improving the reliability of a system, it also depends on other factors, such as power instability, temperature changes, out-of-range data, internal and external noises, and synchronization problems that occur when multiple sensors are integrated into a system [2]. However, the reconstruction of the data and correction for the correct measurement is also important, and several research studies have proposed systems, methods, models, and frameworks to improve the data validation and reconstruction [3, 4]. Sensor data validation methods can be separated in three large groups, such as faulty data detection methods, data correction methods, and other assisting techniques or tools [5]. Firstly, faulty data detection methods may be either simple test based methods or physical or mathematical model based methods, and they are classified in valid data and invalid or missingness data [6, 7]. For the detection of faulty data, the authors in [7] presented an order of methods that should be applied to obtain better results, which are as follows: zero value detection, flat line detection, minimum and maximum values detection, minimum and maximum thresholds based on last values, statistical tests that follow certain distributions, multivariate statistical tests, artificial neural networks (ANNs) [8], one-class support vector machine (SVM) [9], and classification and physical models. On the one hand, simple test based methods include different techniques, such as physical range check, local realistic range detection, detection of gaps in the data, constant value detection, the signals’ gradient test, the tolerance band method, and the material redundancy detection [7, 10, 11]. On the other hand, physical or mathematical model based methods include extreme value check using statistics, drift detection by exponentially weighted moving averages, the spatial consistency method, the analytical redundancy method, gross error detection, the multivariate statistical test using Principal Component Analysis (PCA), and data mining methods [7, 12, 13]. Secondly, data correction methods can be carried out by interpolation, smoothing, data mining, and data reconciliation [10, 12, 14]. For the application of the interpolation, the authors of [11] proposed the use of the value measured from the last measurement or the use of the trend from previous sets of measurements. The smoothing methods, for example, moving average and median, may be used to filter out the random noise and convert the data into a smooth curve that is relatively unbiased by outliers [10]. The application of data mining techniques allows the replacement of the faulty values by the measurements performed with several methods, for example, ANNs [14]. The data reconciliation methods, for example, PCA, are used for the calculation of a minimal correction to the measured variables, according to the several constraints of the model [13]. Thirdly, the other assisting techniques or tools are, namely, the checking of the status of the sensors, the checking Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 3 Journal of Sensors Sensors Soft sensors Origin: data acquisition/data processing Data cleaning/noise removal Data is valid? Yes No Data is complete? Yes No Data imputation Discard the data No Data is valid? Yes Next stages: data processing/data fusion Data discarded Figure 1: Sequence of activities performed during the data validation process. of the duration after sensor maintenance, data context classification, the calibration of measuring systems, and the uncertainty consideration [6, 7, 10]. Several research studies have been performed, using data validation techniques. In [15], PCA is used for the compression of linearly correlated data. The authors compared the Auto-Associative Neural Network (AANN) and the Kernel PCA (KPCA) methods for data validation, creating a new approach named as Hybrid AANN-KPCA that uses these two methods. When compared with AANN and KPCA methods, the Hybrid AANN-KPCA achieves better performance results for the prediction or correction of inconsistent data. In [16], the authors proposed that the data validation may be performed with Kalman filtering and linear predictive coding (LPC), showing that the results using Kalman filtering are better than LPC using several types of data, but the LPC reported a smaller energy consumption. Several studies proposed the use of ANNs, for example, the Multilayer Perceptron (MLP), that can be trained to perform the identification of faulty sensors using prototype data and used to determine the near optimal subset of sensor data to produce the best results [2, 17–19]. Besides, the sensor data validation may be performed with other probabilistic methods, such as Bayesian Networks, Propagation in Trees, Probabilistic Causal Methods, and Learning Algorithms [20]. The authors of [20] proposed the anytime sensor validation algorithms that combine several probabilistic methods. On the contrary, [21] proposed the validation of data using the Sparse Bayesian Learning and the Relevance Vector Machine (RVM), which are an specialization of SVM. For the estimation of the values during data validation, the authors of [22] analysed the use of the Kalman filter, which was implemented in two methods: Algorithmic Sensor Validation (ASV), and Heuristic Sensor Validation (HSV). The ASV method implements different statistical methods, for example, mean, standard deviation, and sensor confidence that represent the uncertain nature of sensors. HSV identifies faulty sensor readings as attributable to a sensor or system failure. As an example, the authors of [23] proposed the use of the Kalman filter for the validation of the GPS data. Other used methods are the grey models, which consists of differential equations describing the behaviour of an accumulated generating operation (AGO) data sequence. As an example, [4] presented a novel self-validating strategy using grey bootstrap method (GBM) for data validation and dynamic uncertainty estimation of self-validating sensor. The GBM can evaluate the measurement uncertainty due to poor information and small sample. In [2], the Autoregressive Moving Averages (ARMA) transform the process for determining the validity of the acquired data, evaluating the levels of noise and providing a timely warning from the expected signals. The model created for ARMA includes linear regression techniques to predict the invalid values with Autoregressive (AR) and Moving Average (MA) models. Sensor Data Validation in Aeroengine Vibration Tests also implements the Autoregressive (AR) Model, complemented with the Empirical Mode Decomposition (EMD) [24]. Another method presented is the sensor validation and fusion of the Nadaraya-Watson statistical estimator [25], using a Fuzzy Logic model [26]. These methods and others, including the use of Gaussian distributions and error detection methods, may be also used to improve the quality of the measurements [27, 28]. 45 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 4 Intelligent sensor systems are able to perform the capture and validation of the sensors’ data. Staroswiecki [29] argues that the data validation is important to increase the confidence level of these systems, proposing two types of validation, such as technological and functional. Technological validation consists on the analysis of the conditions of the hardware resources of the sensors, but it does not guarantee that the estimation produced by the sensor is correct, but only that the operating conditions were not against possible correctness. On the contrary, functional validation consists of Fault Detection and Isolation (FDI) procedures, which consists of the use of algorithms to complement the Technological Validation. The authors of [30] also agreed with Staroswiecki in the separation of the data validation in two types, presenting a real time algorithm based on probabilistic methods. Other studies have been researched and developed, including the data validation techniques using intelligent sensor systems [31]. Another powerful technique for data validation consists of the use of self-validating (SEVA) sensors, which provide an estimation of the error bounds during the measurements [32]. SEVA are widely researched in literature. An example, using a Back-Propagation (BP) model, is applied into a system to obtain an estimated value and then a fault detection method called SPRT (sequential probability ratio test), identifying the validity of the system [33]. For the use of SEVA technologies, the authors of [34] also proposed the validated random fuzzy variable (VRFV) based uncertainty evaluation strategy for the online validated uncertainty (VU) estimation. In [35], the authors presented a novel strategy of using polynomial predictive filters coupled with VRFV which is proposed for the online measurements validation and validated uncertainty estimation of multifunctional self-validating sensors. These authors also performed a research about the use of some fuzzy logic rules, comparing the predicted values with the actual measurements to obtain the confidence evaluation [36]. In [37], the authors proposed an approach of sensor data validation using self-reporting, including the measurement based on the data quality, that is, validating the data loss measured by periodic sensors, the timing of data collection, and the accuracy of the detection of changes. ANNs may be used for SEVA with self-organizing maps (SOM) [38], which are trained using unsupervised learning techniques to produce a low-dimensional, discretized representation of the input space of the training samples [39]. The use of valid data is important for the developments of intelligent sensor systems, which may be used for health purposes and, consequently, for the detection of the ADLs [40–45]. The use of mobile devices allows the data acquisition anywhere and at anytime, but these devices have several constraints, such as low memory, processing power, and battery resources, but data validation may help for increasing of the performance of the measurements, reducing the resources needed [46–48]. In general, these systems use probabilistic methods to detect the failures at real-time to obtain better results. Table 1 presents a summary of the data validation methods included on each category. The methods that are mainly implemented use statistical and artificial intelligence 46 Journal of Sensors techniques, such as PCA, RVM, ANNs, and others, increasing the reliability of the data acquisition and data processing algorithms. In spite of the SVM and the ANN working in a slightly different manner, their foundations are quite similar. In fact the SVM without kernel is a single neural network neuron with a different cost function. Congruently, when the SVM has a kernel it is comparable with a 2-layer ANN. Following the methods presented at Table 1, the most studied scenarios for data validation are mainly related to health sciences, laboratory experiments, and other undifferentiated tasks. However, only a minor part of studies is related to the use of mobile devices, smart sensors, and other devices used daily. Besides, the development of healthcare solutions based on the sensors available on the mobile devices increases the requirement of the validation of the data collected by the sensors available on the mobile devices. Depending on the types of the data, for some complex data acquired, such as images, videos, GPS signal, and other complex types of data, the validation of the data should be accomplished by other auxiliary systems working at the same time, validating the data at the server-side, but a constant network connection must be available. Other topologies of systems may be susceptible for the implementation of data validation techniques. The Wireless Sensor Networks (WSN) are an example of systems where the different nodes of the network may perform the validation of the data collected for the neighbourhood nodes, and these nodes may be composed of different types of sensors. However, the main topology for the implementation with mobile devices is the self-validation using only the sensors and the data available on the mobile device. The data validation may be executed automatically and transparently for the mobile devices’ user and, commonly, at least one of the methods for each stage should be implemented in a system to perform the validation of the sensors’ data. Firstly, for faulty data detection methods, the ANNs are the most used methods for the training of the data and for the detection of the inconsistent values. Secondly, for data correction methods, the most used method is the Kalman filter. Thirdly, the other assisting techniques that are commonly applied are the data context classification, the checking of the status of sensors, and the uncertainty considerations. Applying the data validation techniques correctly, the reliability and acceptability of the systems may be increased. 3. Classification of Data Validation Methods Data validation methods may be classified in three large groups [5] as follows: faulty data detection methods, data correction methods, and other assisting techniques or tools. The faulty data detection methods and the data correction methods may be executed sequentially in a multisensor system in order to obtain the results based on valid data. The other assisting techniques or tools mainly consist of the validation of the working state of the sensors, and this validation may be executed at the same time of the execution of faulty data detection and data correction methods, because these types of failures invalidated the results of the algorithms. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 5 Journal of Sensors Table 1: Classification of the data validation methods by functionality. Groups of data validation methods Methods included ANNs (i) MLP; AANN; BP algorithm; SVM; Instance based (i) SOM Gaussian distributions Statistical methods (i) ASV; HSV Faulty data detection methods Probabilistic methods (i) Bayesian Networks; Propagation in Trees; Probabilistic Causal Methods; Learning Algorithms; Sparse Bayesian Learning; RVM; SPRT Dimensionality Reduction (i) Fuzzy logic; PCA; KPCA; others (i) Hybrid AANN-KPCA Kalman filter LPC ARMA (i) AR; MA; EMD Data correction methods Nadaraya-Watson statistical estimator Interpolation Smoothing Data mining techniques Data reconciliation techniques Checking of the status of the sensors Checking of the duration after sensor maintenance Data context classification Calibration of measuring systems Other assisting techniques or tools Uncertainty consideration Grey models (i) GBM; dynamic uncertainty estimation of self-validating sensor VRFV method These different approaches are based on either mathematical methods, for example, statistical or probabilistic methods, or complex analysis, for example, artificial intelligence methods. According to [49], the data validation methods may be classified in several types of methods, which are presented in Figure 2. As depicted in Figure 2, the faulty data detection methods, used to detect failures on the sensors’ signal, may include ANNs, dimensional reduction methods, instance based methods, probabilistic and statistical methods, and Bayesian methods. On the contrary, the data correction methods include the following methods: filtering, regression, estimation, interpolation, smoothing, data mining, and data reconciliation. These methods work specifically with the sensors’ data and the selection of the methods that can be applied by a system should consider the system’s usage scenarios. Finally, the other assisting techniques or tools are mainly related to detection of problems originated by either hardware components or its working environment. In addition, on real-time systems, these problems should be verified constantly to prevent the existence of failures in the data captured. Description Consisting of the detection of faulty or incorrect values discovered during the data acquisition and processing stages Consisting of the estimation of faulty or incorrect values obtained during the data acquisition and processing stages These are different approaches created for the correct validation of the data 4. Applicability of the Sensor Data Validation Methods Mobile devices have a plethora of sensors available for the measurement of several parameters, including the identification of the ADLs. Examples of these sensors include the accelerometer, the gyroscope, the magnetometer, the GPS, and the microphone. The data acquisition using accelerometers may fail because of several problems, including problems related with the internal electronic amplifier of the Integrated Electronic Piezoelectric (IEPE) device, the exposure to temperatures beyond the accelerometer working range, failure related with electrical components, capture of environmental noise, the multitasking and multithreading capabilities of the mobile devices that may cause irregular sampling rates, the positioning of the accelerometer, the low processing and memory power, and the battery consumption [50]. The causes of failure of an accelerometer are similar to the causes of the failure of a gyroscope, a magnetometer, or a microphone [51]. In addition, the GPS has another failure cause, which consists of the low connectivity of satellites in indoor environments [52]. 47 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 6 Journal of Sensors MLP AANN Artificial neural networks BP Hybrid AANN-KPCA PCA KPCA Hybrid AANN-KPCA Dimensionality reduction Instance based SOM SPRT Faulty data detection methods Probabilistic Causal Methods Learning Algorithms Probabilistic Propagation in Trees ASV Statistic HSV Artificial intelligence Fuzzy logic SVM RVM Bayesian Networks Sparse Bayesian Learning Bayesian Gaussian distributions Data validation techniques Filtering Kalman filter LPC ARMA Regression Data correction methods AR MA EMD Nadaraya-Watson statistical estimator Interpolation Smoothing Data mining techniques Data reconciliation techniques Checking the status of the sensors Checking the duration after sensors maintenance Data context classification Other assisting techniques or tools Calibration of measuring systems Grey models Uncertainty consideration Dynamic uncertainty estimation of selfvalidating sensors VRFV method Grey models Ensemble GBM Figure 2: Different categories of the data validation methods. The validation of the data is important, but, for critical systems, for example, clinical systems, not only should the input data be validated, but also the results should be validated to guarantee the reliability, accuracy and, consequently, acceptance of the system. The validation of the system may consist of the detection of failures and the methods that may 48 be applied are the faulty detection methods. As presented in Section 3, the methods that may be included in this category are probabilistic and statistical methods, among others, which may be used to validate the results of the system. This validation can be performed by comparing the results obtained by an equivalent system which is considered Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7 Journal of Sensors to be a gold standard [53] with the results obtained by the developed methods implemented by different sensors or devices, for example, a mobile device. Once estimated the initial error of the system, that is, how different the obtained results are from the results obtained by the gold standard system, the validation of the results of the system consist of three steps, such as the definition of the confidence level needed for the acceptance of the system, the determination of the minimum number of experiments needed to validate the application with confidence level defined, and the validation of the results when compared to a golden-standard [54]. The definition of the degree of confidence of the system is a choice of the development team. The system design leader may define what system needs to have a maximum 5% error 95% of the times. Using these parameters, a minimum number of calibration experiments need to be performed to allow the fine tuning of the algorithm. The minimum number of experiments may be measured by several statistical tests, for example, Student’s 𝑡-test [55]. After the calibration of the algorithms in the system, further tests and comparison with golden-standard systems can be done to insure that the results reported by the system have a 5% maximum error when compared to the golden standard results, for 95% of the time. Note that the 5% and 95% values are merely indicative. Moreover, the data collection stage must hold into consideration the limits for the optimal functioning of the sensors. As these limits are extremely dependent on the task the sensors must perform, we do not discuss them in this paper, for example, if the application is supposed to track the movements of a sportsperson in an open environment, it is possible that a thermal sensor reports an environment temperature of −5∘ C, yet, for an application that tracks the indoor activity of an elder, such value should raise an alarm. In this extreme case, it is even possible that more robust systems need to contain different types of sensors. 5. Conclusion The validation of the data collected by sensors in a mobile device is an important issue for two main reasons: the first one is the increasing number of devices and the applications that make use of the devices’ sensors; the other is that also increasingly users rely on these devices and applications to collect information and make decisions that may be critical for the user’s life and well-being. Despite the fact that there is a wide array and types of data validation algorithms, there is also a lack of published information on the validity of many mobile applications. Also, it is impossible to present a critical comparison of the discussed methods, even within their respective categories, as their efficiency is extremely dependent on their particular usage; for example, the efficiency of a specific method may be very dependent on the number and type of features the algorithm selects on the signal to be processed, and of course these features are chosen in view of the intended purpose of the application. Additionally, it is possible that even with the same chosen method and the same chosen set of features, different authors report different efficiency ratios; for example, their base population sample varies in size and/or type using different population sizes or using populations that are homogenous in age (elders or youngsters). This paper has presented a discussion on the different types of data validation methods such as faulty data detection, data correction, and assisting techniques or tools. Furthermore, a classification of these methods in accordance with its functionalities was provided. Finally, the relevance of the data validation methods for critical systems in terms of its reliability, accuracy and acceptance was highlighted. Complementary studies should be addressed aiming at providing an overview on the use of valid data for the identification of the ADLs. Competing Interests The authors declare that there is no conflict of interests regarding the publication of this paper. Acknowledgments This work was supported by FCT project UID/EEA/50008/ 2013 (Este trabalho foi suportado pelo projecto FCT UID/ EEA/50008/2013). The authors would also like to acknowledge the contribution of the COST Action IC1303 Architectures, Algorithms and Protocols for Enhanced Living Environments (AAPELE). References [1] A. Braun, R. Wichert, A. Kuijper, and D. W. Fellner, “A benchmarking model for sensors in smart environments,” in Ambient Intelligence: European Conference, (AmI ’14), Eindhoven, The Netherlands, November 2014. Revised Selected Papers, E. Aarts, B. de Ruyter, P. Markopoulos et al., Eds., pp. 242–257, Springer, Cham, Switzerland, 2014. [2] J. Garza-Ulloa, H. Yu, and T. Sarkodie-Gyan, “A mathematical model for the validation of the ground reaction force sensor in human gait analysis,” Measurement, vol. 45, no. 4, pp. 755–762, 2012. [3] M. À. Cugueró, M. Christodoulou, J. Quevedo, V. Puig, D. Garcı́a, and M. P. Michaelides, “Combining contaminant event diagnosis with data validation/reconstruction: application to smart buildings,” in Proceedings of the 22nd Mediterranean Conference on Control and Automation (MED ’14), pp. 293–298, June 2014. [4] Y. Chen, J. Yang, and S. Jiang, “Data validation and dynamic uncertainty estimation of self-validating sensor,” in Proceedings of the IEEE International Instrumentation and Measurement Technology Conference (I2MTC ’15), pp. 405–410, Pisa, Italy, May 2015. [5] S. Sun, J. Bertrand-krajewski, A. Lynggaard-Jensen et al., “Literature review for data validation methods,” SciTechnol, vol. 47, no. 2, pp. 95–102, 2011. [6] J.-L. Bertrand-Krajewski, S. Winkler, E. Saracevic, A. Torres, and H. Schaar, “Comparison of and uncertainties in raw sewage COD measurements by laboratory techniques and field UVvisible spectrometry,” Water Science and Technology, vol. 56, no. 11, pp. 17–25, 2007. 49 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 8 [7] N. Branisavljević, Z. Kapelan, and D. Prodanović, “Improved real-time data anomaly detection using context classification,” Journal of Hydroinformatics, vol. 13, no. 3, pp. 307–323, 2011. [8] M. H. Hassoun, Fundamentals of Artificial Neural Networks, MIT Press, 1995. [9] B. E. Boser, I. M. Guyon, and V. N. Vapnik, “A training algorithm for optimal margin classifiers,” in Proceedings of the 5th Annual Workshop on Computational Learning Theory, pp. 144–152, ACM Press, Pittsburgh, Pa, USA, 1992. [10] M. Mourad and J.-L. Bertrand-Krajewski, “A method for automatic validation of long time series of data in urban hydrology,” Water Science and Technology, vol. 45, no. 4-5, pp. 263–270, 2002. [11] G. Olsson, M. K. Nielsen, Z. Yuan, and A. Lynggaard-Jensen, Instrumentation, Control and Automation in Wastewater Systems, IWA, London, UK, 2005. [12] F. Edthofer, J. Van den Broeke, J. Ettl, W. Lettl, and A. Weingartner, “Reliable online water quality monitoring as basis for fault tolerant control,” in Proceedings of the 1st Conference on Control and Fault-Tolerant Systems (SysTol ’10), pp. 57–62, October 2010. [13] S. J. Qin and W. Li, “Detection, identification, and reconstruction of faulty sensors with maximized sensitivity,” AIChE Journal, vol. 45, no. 9, pp. 1963–1976, 1999. [14] M. A. Kramer, “Nonlinear principal component analysis using autoassociative neural networks,” AIChE Journal, vol. 37, no. 2, pp. 233–243, 1991. [15] R. Sharifi and R. Langari, “A hybrid AANN-KPCA approach to sensor data validation,” in Proceedings of the 7th Conference on 7th WSEAS International Conference on Applied Informatics and Communications, vol. 7, pp. 85–91, 2007. [16] C. C. Castello, J. R. New, and M. K. Smith, “Autonomous correction of sensor data applied to building technologies using filtering methods,” in Proceedings of the 1st IEEE Global Conference on Signal and Information Processing (GlobalSIP ’13), pp. 121–124, December 2013. [17] M. Kasinathan, B. S. Rao, N. Murali, and P. Swaminathan, “An artificial neural network approach for the discordance sensor data validation for SCRAM parameters,” in Proceedings of the 1st International Conference on Advancements in Nuclear Instrumentation, Measurement Methods and their Applications (ANIMMA ’09), pp. 1–5, Marseille, France, June 2009. [18] E. V. Zabolotskikh, L. M. Mitnik, L. P. Bobylev, and O. M. Johannessen, “Neural networks based algorithms for sea surface wind speed retrieval using SSM/I data and their validation,” in Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS ’99), vol. 2, pp. 1010–1012, Hamburg, Germany, June-July 1999. [19] E. Gaura, R. Newman, M. Kraft, A. Flewitt, and W. de Lima Monteiro, Smart MEMS and Sensor Systems, World Scientific, Singapore, 2006. [20] P. H. Ibarguengoytia, Any Time Probabilistic Sensor Validation, University of Salford, Salford, UK, 1997. [21] M. E. Tipping, “Sparse Bayesian learning and the relevance vector machine,” Journal of Machine Learning Research, vol. 1, no. 3, pp. 211–244, 2001. [22] S. Alag, A. M. Agogino, and M. Morjaria, “A methodology for intelligent sensor measurement, validation, fusion, and fault detection for equipment monitoring and diagnostics,” Artificial Intelligence for Engineering Design, Analysis and Manufacturing, vol. 15, no. 4, pp. 307–320, 2001. 50 Journal of Sensors [23] E. Shi, “An improved real-time adaptive Kalman filter for low-cost integrated GPS/INS navigation,” in Proceedings of the International Conference on Measurement, Information and Control (MIC ’12), vol. 2, pp. 1093–1098, May 2012. [24] Z. Wu and N. E. Huang, “Ensemble empirical mode decomposition: a noise-assisted data analysis method,” Advances in Adaptive Data Analysis, vol. 1, no. 1, pp. 1–41, 2009. [25] S. J. Wellington, J. K. Atkinson, and R. P. Sion, “Sensor validation and fusion using the Nadaraya-Watson statistical estimator,” in Proceedings of the 5th International Conference on Information Fusion, vol. 1, pp. 321–326, Annapolis, Md, USA, 2002. [26] K. E. Holbert, A. S. Heger, and N. K. Alang-Rashid, “Redundant sensor validation by using fuzzy logic,” Nuclear Science and Engineering, vol. 118, no. 1, pp. 54–64, 1994. [27] Y. Ai, X. Sun, C. Zhang, and B. Wang, “Research on sensor data validation in aeroengine vibration tests,” in Proceedings of the International Conference on Measuring Technology and Mechatronics Automation (ICMTMA ’10), vol. 3, pp. 162–166, March 2010. [28] F. Pfaff, B. Noack, and U. D. Hanebeck, “Data validation in the presence of stochastic and set-membership uncertainties,” in Proceedings of the 16th International Conference on Information Fusion (FUSION ’13), pp. 2125–2132, Istanbul, Turkey, July 2013. [29] M. Staroswiecki, “Intelligent sensors: a functional view,” IEEE Transactions on Industrial Informatics, vol. 1, no. 4, pp. 238–249, 2005. [30] P. H. Ibargiengoytia, L. E. Sucar, and S. Vadera, “Real time intelligent sensor validation,” IEEE Transactions on Power Systems, vol. 16, no. 4, pp. 770–775, 2001. [31] J. Rivera-Mejýa, E. Arzabala-Contreras, and Á. G. Leýn-Rubio, “Approach to the validation function of intelligent sensors based on error’s predictors,” in Proceedings of the IEEE Instrumentation and Measurement Technology Conference (I2MTC ’10), pp. 1121–1125, IEEE, May 2010. [32] M. P. Henry and D. W. Clarke, “The self-validating sensor: rationale, definitions and examples,” Control Engineering Practice, vol. 1, no. 4, pp. 585–610, 1993. [33] B. Mounika, G. Raghu, S. Sreelekha, and R. Jeyanthi, “Neural network based data validation algorithm for pressure processes,” in Proceedings of the International Conference on Control, Instrumentation, Communication and Computational Technologies (ICCICCT ’14), pp. 1223–1227, July 2014. [34] Q. Wang, Z. Shen, and F. Zhu, “A multifunctional self-validating sensor,” in Proceedings of the IEEE International Instrumentation and Measurement Technology Conference (I2MTC ’13), pp. 1283– 1288, Minneapolis, Minn, USA, May 2013. [35] Z. Shen and Q. Wang, “Data validation and validated uncertainty estimation of multifunctional self-validating sensors,” IEEE Transactions on Instrumentation and Measurement, vol. 62, no. 7, pp. 2082–2092, 2013. [36] Z. Shen and Q. Wang, “Data validation and confidence of selfvalidating multifunctional sensor,” in Proceedings of the Sensors, pp. 1–4, Taipei, Taiwan, October 2012. [37] J. Doyle, A. Kealy, J. Loane et al., “An integrated home-based self-management system to support the wellbeing of older adults,” Journal of Ambient Intelligence and Smart Environments, vol. 6, no. 4, pp. 359–383, 2014. [38] T. Kohonen, “The self-organizing map,” Proceedings of the IEEE, vol. 78, no. 9, pp. 1464–1480, 1990. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Journal of Sensors [39] B. Lamrini, E.-K. Lakhal, M.-V. Le Lann, and L. Wehenkel, “Data validation and missing data reconstruction using selforganizing map for water treatment,” Neural Computing and Applications, vol. 20, no. 4, pp. 575–588, 2011. [40] A. Pantelopoulos and N. G. Bourbakis, “A survey on wearable sensor-based systems for health monitoring and prognosis,” IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, vol. 40, no. 1, pp. 1–12, 2010. [41] S. Helal, J. W. Lee, S. Hossain, E. Kim, H. Hagras, and D. Cook, “Persim—simulator for human activities in pervasive spaces,” in Proceedings of the 7th International Conference on Intelligent Environments (IE ’11), pp. 192–199, Nottingham, UK, July 2011. [42] H. Eren, “Assessing the health of sensors using data historians,” in Proceedings of the IEEE Sensors Applications Symposium (SAS ’12), pp. 208–211, February 2012. [43] S. Oonk, F. J. Maldonado, and T. Politopoulos, “Distributed intelligent health monitoring with the coremicro Reconfigurable Embedded Smart Sensor Node,” in Proceedings of the IEEE AUTOTESTCON, pp. 233–238, Anaheim, Calif, USA, September 2012. [44] C.-M. Chen, R. Kwasnicki, B. Lo, and G. Z. Yang, “Wearable tissue oxygenation monitoring sensor and a forearm vascular phantom design for data validation,” in Proceedings of the 11th International Conference on Wearable and Implantable Body Sensor Networks (BSN ’14), pp. 64–68, June 2014. [45] F. J. Maldonado, S. Oonk, and T. Politopoulos, “Enhancing vibration analysis by embedded sensor data validation technologies,” IEEE Instrumentation and Measurement Magazine, vol. 16, no. 4, pp. 50–60, 2013. [46] E. Miluzzo, Smartphone Sensing, Dartmouth College, Hanover, New Hampshire, 2011. [47] F. J. Maldonado, S. Oonk, and T. Politopoulos, “Optimized neuro genetic fast estimator (ONGFE) for efficient distributed intelligence instantiation within embedded systems,” in Proceedings of the International Joint Conference on Neural Networks (IJCNN ’13), pp. 1–8, August 2013. [48] B. Wallace, R. Goubran, F. Knoefel et al., “Automation of the validation, anonymization, and augmentation of big data from a multi-year driving study,” in Proceedings of the IEEE International Congress on Big Data, pp. 608–614, New York, NY, USA, June 2015. [49] J. Brownlee, A Tour of Machine Learning Algorithms, Machine Learning Mastery, 2013. [50] R. Denton, “Sensor Reliability Impact on Predictive Maintenance Program Costs,” 2010, Wilcoxon Research. [51] O. Brand, G. K. Fedder, C. Hierold, J. G. Korvink, O. Tabata, and T. Tsuchiya, Reliability of MEMS: Testing of Materials and Devices, John Wiley & Sons, New York, NY, USA, 2013. [52] G. Heredia, A. Ollero, M. Bejar, and R. Mahtani, “Sensor and actuator fault detection in small autonomous helicopters,” Mechatronics, vol. 18, no. 2, pp. 90–99, 2008. [53] G. J. Prescott and P. H. Garthwaite, “A simple Bayesian analysis of misclassified binary data with a validation substudy,” Biometrics, vol. 58, no. 2, pp. 454–458, 2002. [54] V. R. Basili, R. W. Selby Jr., and T.-Y. Phillips, “Metric analysis and data validation across fortran projects,” IEEE Transactions on Software Engineering, vol. 9, no. 6, pp. 652–663, 1983. [55] K. R. Sundaram and A. Jose, “Teaching: estimation of minimum sample size and the impact of effect size and altering the type-I & II errors on IT, in clinical research,” in Data and Context in Statistics Education: Towards an Evidence-Based 9 Society. Proceedings of the Eighth International Conference on Teaching Statistics (ICOTS8, July, 2010), Ljubljana, Slovenia, C. Reading, Ed., International Statistical Institute, Voorburg, The Netherlands, 2010. 51 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 52 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 3. Audio Fingerprinting Techniques for detection of the environment of Activities of Daily Living (ADL): A Systematic Review The following article is the third part of the chapter 2. Audio Fingerprinting Techniques for detection of the environment of Activities of Daily Living (ADL): A Systematic Review Ivan Miguel Pires, Rui Santos, Nuno Pombo, Nuno M. Garcia, Francisco Flórez-Revuelta, Susanna Spinsante, Rossitza Goleva and Eftim Zdravevski Sensors (MDPI), published, 2018. According to 2016 Journal Citation Reports published by Thomson Reuters in 2017, this journal has the following performance metrics: ISI Impact Factor (2016): 2.677 ISI Article Influence Score (2016): 0.6 Journal Ranking (2016): 104/642 (Electrical and Electronic Engineering) 53 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 54 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living sensors Review Recognition of Activities of Daily Living Based on Environmental Analyses Using Audio Fingerprinting Techniques: A Systematic Review Ivan Miguel Pires 1,2,3 ID , Rui Santos 1,3 , Nuno Pombo 1,3,4 , Nuno M. Garcia 1,3,4, * Francisco Flórez-Revuelta 5 , Susanna Spinsante 6 ID , Rossitza Goleva 7 ID and Eftim Zdravevski 8 ID 1 2 3 4 5 6 7 8 * ID , Instituto de Telecomunicações, Universidade da Beira Interior, 6201-001 Covilhã, Portugal; impires@it.ubi.pt (I.M.P.); rui_17_santos@hotmail.com (R.S.); ngpombo@ubi.pt (N.P.) Altranportugal, 1990-096 Lisbon, Portugal ALLab—Assisted Living Computing and Telecommunications Laboratory, Computing Science Department, Universidade da Beira Interior, 6201-001 Covilhã, Portugal ECATI, Universidade Lusófona de Humanidades e Tecnologias, 1749-024 Lisbon, Portugal Department of Computer Technology, Universidad de Alicante, 03690 Sant Vicent del Raspeig, Alicante, Spain; francisco.florez@ua.es Department of Information Engineering, Marche Polytechnic University, 60121 Ancona, Italy; s.spinsante@univpm.it Department of Informatics, New Bulgarian University, 1618 g.k. Ovcha kupel 2 Sofia, Bulgaria; rgoleva@gmail.com Faculty of Computer Science and Engineering, University Ss Cyril and Methodius, 1000 Skopje, Macedonia; eftim.zdravevski@finki.ukim.mk Correspondence: ngarcia@di.ubi.pt; Tel.: +351-966-637-9785 Received: 28 November 2017; Accepted: 5 January 2018; Published: 9 January 2018 Abstract: An increase in the accuracy of identification of Activities of Daily Living (ADL) is very important for different goals of Enhanced Living Environments and for Ambient Assisted Living (AAL) tasks. This increase may be achieved through identification of the surrounding environment. Although this is usually used to identify the location, ADL recognition can be improved with the identification of the sound in that particular environment. This paper reviews audio fingerprinting techniques that can be used with the acoustic data acquired from mobile devices. A comprehensive literature search was conducted in order to identify relevant English language works aimed at the identification of the environment of ADLs using data acquired with mobile devices, published between 2002 and 2017. In total, 40 studies were analyzed and selected from 115 citations. The results highlight several audio fingerprinting techniques, including Modified discrete cosine transform (MDCT), Mel-frequency cepstrum coefficients (MFCC), Principal Component Analysis (PCA), Fast Fourier Transform (FFT), Gaussian mixture models (GMM), likelihood estimation, logarithmic moduled complex lapped transform (LMCLT), support vector machine (SVM), constant Q transform (CQT), symmetric pairwise boosting (SPB), Philips robust hash (PRH), linear discriminant analysis (LDA) and discrete cosine transform (DCT). Keywords: acoustic sensors; fingerprint recognition; data processing; artificial intelligence; mobile computing; signal processing algorithms; systematic review; Activities of Daily Living (ADL) Sensors 2018, 18, 160; doi:10.3390/s18010160 www.mdpi.com/journal/sensors 55 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 2 of 23 1. Introduction The identification of Activities of Daily Living (ADL) [1] is of utmost importance to build Enhanced Living Environment and Ambient Assisted Living solutions [2,3], or to allow the development of Personal Digital Life Coaching systems [4]. To achieve this, several authors have proposed the development of solutions based on mobile devices (e.g., smartphones) [5–8] for several reasons, the most prominent being the adoption ratios of these devices, its increasing computing power and memory, and the fact that these devices already come equipped with a plethora of sensors that can be used to sense and feed data to ADL identification systems. Despite the increasing complexity of ADL identification systems, the recognition of the surrounding environment is limited because of the restrictions of some location sensors. For instance, Global Positioning System (GPS) sensors, can only be reliably and accurately used in outdoor scenarios. Likewise, proximity sensors, radar sensors, Passive Infra-Red (PIR) sensors and alike require significant installation effort, thus are not widely used in real scenarios which require ADL identification. As proposed in previous works [9–11], an ADL identification framework should also be able to integrate data from the sound of the environment into the ADL identification module in order to allow the system to sense the environmental sounds, to determine the type of environment, and to increase the accuracy of the overall ADL identification solution. Most mobile devices are equipped with a microphone that can capture an acoustic signal. This signal can be processed using audio fingerprinting techniques, allowing the system to find a match between the collected signal and a database of well-known audio fingerprints. This might facilitate an increase in the accuracy of recognition of the environment where ADLs are performed. Several methods may be used to carry out audio fingerprinting, performing the pre-processing of the acoustic data (e.g., Fast Fourier Transform (FFT)), extracting relevant features, and after that, obtaining a classification or recognition (e.g., Support Vector Machine (SVM)). This review summarizes the existing methods in the literature related to audio fingerprinting techniques for the application in a system that uses mobile technology for the recognition of the environment. While acknowledging that the methods here presented are very diverse and have been tested with different data sets and different feature extraction techniques, in order to estimate which method may provide better results in a mobile computational device, this paper also presents a comparison between the different methods and features. The remainder of this paper is organized as follows: Section 2 presents the methodology for this review; the methods discovered in the literature are presented in Section 3; Section 4 discusses different methods, and finally, Section 5 present conclusions of this review. 2. Methodology 2.1. Research Questions The primary questions of this review were as follows: (RQ1) What is audio fingerprinting? (RQ2) Which audio fingerprinting techniques are useful to identify the environment of daily activities? (RQ3) Which are the audio fingerprinting techniques feasible for their use in mobile devices? 2.2. Inclusion Criteria Studies assessing ADLs using audio fingerprinting techniques were included in this review if they met the following criteria: (1) audio fingerprinting techniques adapted to mobile devices; (2) audio fingerprinting techniques used for the detection of the environment of ADL; (3) using mobile devices; (4) the accuracies of the audio fingerprinting techniques presented are reported; (5) were published between 2002 and 2017; and (6) were written in English. 56 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 3 of 23 2.3. Search Strategy The team searched for studies meeting the inclusion criteria in the following electronic databases: IEEE Xplore, and ACM Digital Library. Every study was independently evaluated by eight reviewers (IP, RS, NP, NG FR, SP, RG and EZ), and its suitability was determined with the agreement of all parties. The studies were examined to identify the characteristics of audio fingerprint and its suitability for application with mobile devices for the identification of ADL. 2.4. Extraction of Study Characteristics The following data was extracted from the studies and tabulated (see Tables 1 and 2): year of publication, population for the application of the algorithm, purpose of the study, devices used, and study outcomes of the algorithm for audio fingerprinting. For all cited studies in Tables 1 and 2, the experiments were conducted in laboratory settings. We additionally verified whether the raw data and source code are available, either publically or per request, by emailing the corresponding author of each study. Table 1. Study Analysis. Devices Raw Data Available Source Code Available To search for audio in the database by the content rather than by name Mobile Phone (Android) No No 100,000 MP3 fragments To create an MP3 sniffer system that includes audio fingerprinting Not mentioned Yes Only for feature extraction 2011 10,000 MP3 fragments Proposes an MP3 fingerprint system for the recognition of several clips Not mentioned The same data as [13] The same source code as [13] Tsai et al. [15] 2016 Multi-channel audio recordings of 75 real research group meetings, approximately 72 h of meetings in total Proposes an adaptive audio fingerprint based on spectrotemporal eigenfilters Mobile phones, tablets or laptop computers Yes No Casagranda et al. [16] 2015 1024 samples Proposes an audio fingerprinting method that uses GPS and acoustic fingerprints Smartphone No No Approximately 1,518,177 min (25,303 h) of songs Proposes a method to accelerate audio fingerprinting techniques by skipping the search for irrelevant signal sections Not mentioned Yes No 1062 10 s clips Proposes a method to analyze and classify daily activities in personal audio recordings Not mentioned Yes No Year of Publication Population Sui et al. [12] 2014 2500 pieces of 8 s advertisement audios, and randomly select 200 pieces of audio in the existing database and 50 pieces of other irrelevant audio as test audio Liu [13] 2012 Liu et al. [14] Paper Purpose of the Study ACM IEEE Nagano et al. [17] Ziaei et al. [18] 2015 2015 57 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 4 of 23 Table 1. Cont. Paper Year of Publication Population Purpose of the Study Devices Raw Data Available Source Code Available George et al. [19] 2015 1500 audio files Proposes an audio fingerprinting method based on landmarks in the audio spectrogram Computer No No Kim et al. [20] 2015 6000 television advertisements with a total time of 1110 h Proposes a television advertisement search based on audio fingerprinting in real environments Television No No Seo [21] 2014 1000 songs with classic, jazz, pop, rock, and hip-hop Proposes a binary audio fingerprint matching, using auxiliary information Not mentioned No No Rafii et al. [22] 2014 Several songs with a duration between 6 and 9 s Proposes an audio fingerprinting method for recognition of some clips Computer and Smartphone No No Not mentioned No No Naini et al. [23] 2014 1000 songs Proposes an audio fingerprinting method based on maximization of the mutual information across the distortion channel Yang et al. [24] 2014 200,000 songs Proposes a music identification system based on space-saving audio fingerprints Not mentioned No No Yin et al. [25] 2014 958 randomly chosen query excerpts Proposes an audio fingerprinting algorithm that uses compressed-domain spectral entropy Not mentioned No No Wang et al. [26] 2014 100,000 songs Proposes an audio fingerprinting method that uses GPUs Not mentioned No No 3000 TV advertisements Proposes a high-performance audio fingerprint extraction method for identifying Television commercial advertisement Television No No Smartphone, tablet, notebook, desktop, or another mobile device No No Lee et al. [27] Shibuya et al. [28] 2013 1374 television programs (792 h in total) Proposes a method of identifying media content from an audio signal recorded in reverberant and noisy environments using a mobile device Bisio et al. [29] 2013 20 sounds Proposes the Improved Real-Time TV-channel Recognition (IRTR) method Smartphone No No 2013 1000 songs as positive samples and 999 songs as negatives Proposes a method that speeds up the search process, reducing the number of database accesses Not mentioned No No Lee et al. [30] 58 2014 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 5 of 23 Table 1. Cont. Paper Year of Publication Population Purpose of the Study Devices Raw Data Available Source Code Available Bisio et al. [31] 2012 100,000 songs Proposes an audio fingerprint algorithm adapted to mobile devices Smartphone No No Several datasets Proposes an audio fingerprinting algorithm that encodes the local spectral energies around salient points selected among the main spectral peaks in a given signal Not mentioned No No 300 real-world recordings in a living room Proposes an audio fingerprinting method that combines the Fingerprinting technique with Generalized cross correlation iPad No No 20 music clips with 5 s Proposes an audio fingerprinting algorithm for recognition of some clips Not mentioned No No 835 popular songs Proposes an audio fingerprinting algorithm based on dynamic subband locating and normalized spectral subband centroid (SSC) Not mentioned No No Not mentioned No No Anguera et al. [32] Duong et al. [33] Wang et al. [34] Xiong et al. [35] 2012 2012 2012 2012 Deng et al. [36] 2011 100 audio files Proposes an audio fingerprinting algorithm based on harmonic enhancement and SSC of audio signal Pan et al. [37] 2011 62-h audio database of 1000 tracks Proposes an audio feature in spectrum, local energy centroid, for audio fingerprinting Not mentioned No No 2011 3600 s of several real-time tests Presents an audio fingerprinting method with a low-cost embedded reconfigurable platform Computer No No 1000 songs Proposes an indexing scheme and a search algorithm based on the index Computer No Only pseudo-code for fingerprint matching 7500 experiments Proposes an audio fingerprinting method for the recognition of some clips Computer No No 500 popular songs Proposes an audio fingerprinting method using sub-fingerprint masking based on the predominant pitch extraction Mobile devices Yes No Martinez et al. [38] Cha [39] Schurmann et al. [40] Son et al. [41] 2011 2011 2010 59 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 6 of 23 Table 1. Cont. Paper Chang et al. [42] Umapathy et al. [43] Kim et al. [44] Sert et al. [45] Ramalingam et al. [46] Ghouti et al. [47] 60 Population Purpose of the Study Devices Raw Data Available Source Code Available 17,208 audio clips Presents a sub-Nyquist audio fingerprinting system for music recognition, which utilizes Compressive Sampling (CS) theory Not mentioned No No 2007 213 audio signals Proposes an audio feature extraction and a multi-group classification using the local discriminant bases (LDB) technique Not mentioned No No 2007 100 Korean broadcast TV programs Proposes an audio fingerprinting method for identification of bookmarked audio segments Computer No No 2006 approximately 45 min of pop, rock, and country songs Proposes an audio fingerprinting method from the most representative section of an audio clip Not mentioned No No 2006 250 audio files Proposes and audio fingerprinting method using several features Not mentioned No No 2006 Two audio contents perceptually similar Proposes an audio fingerprinting algorithm that uses balanced multiwavelets (BMW) Not mentioned No No PDA or computer Yes No Year of Publication 2010 Cook et al. [48] 2006 7,106,069 fingerprints Proposes an audio fingerprinting algorithm for the fast indexing and searching of a metadata database Seo et al. [49] 2005 8000 classic, jazz, pop, rock, and hip-hop songs Proposes an audio fingerprinting method based on normalized SSC Not mentioned No No Not mentioned No No Not mentioned No No Haitsma et al. [50] 2003 256 sub-fingerprints Proposes to solve larger speed changes by storing the fingerprint at multiple speeds in the database or extracting the fingerprint query at multiple speeds and then to perform multiple queries on the database Haitsma et al. [51] 2002 256 sub-fingerprints Proposes an audio fingerprinting system for recognition of some clips Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 7 of 23 Table 2. Study summaries. Paper Outcomes ACM Sui et al. [12] The authors propose a two-level audio fingerprint retrieval algorithm to satisfy the demand of accurate and efficient search for advertisement audio. Based on clips with 8 s of advertisements, the authors build a database with 2500 audio fingerprints. The results show that the algorithm implemented with parallel processing yields a precision of 100%. Liu [13] The authors create an MP3 sniffer system and test it with multi-resolution local descriptions. The system has a database of 100,000 MP3 tones and authors report that the system has high performance, because 100 queries for identifying unknown MP3 tones took less than 2 s to be processed Liu et al. [14] The authors describe an MP3 fingerprinting system that compares the normalized distance between two MP3 fingerprints to detect a false identification. The authors identify the possible features of the song and build a large database. For the identification, the authors test the near neighbor searching schemes and compare with the indexing scheme, which utilizes the PCA technique, the QUery Context (QUC)-tree, and the MP3 signatures. The conclusions show that the system has a maximum average error equals to 4.26%. IEEE Tsai et al. [15] The authors propose a method for aligning a set of overlapping meeting recordings, which uses an audio fingerprint representation based on spectrotemporal eigenfilters that are learned on-the-fly in an unsupervised manner. The proposed method is able to achieve more than 99% alignment accuracy at a reasonable error tolerance of 0.1 s. The authors propose an audio fingerprinting algorithm based on the spectral features of Casagranda et al. the audio samples. The authors reported that the algorithm is noise tolerant, which is a key [16] feature for audio based group detection. Nagano et al. [17] The authors propose an approach to accelerate fingerprinting techniques and apply it to the divide-and-locate (DAL) method. The reported results show that DAL3 can reduce the computational cost of DAL to approximately 25%. Ziaei et al. [18] The authors propose a method to analyze and classify daily activities in personal audio recordings (PARs), which uses speech activity detection (SAD), speaker diarization, and a number of audio, speech and lexical features to characterize events in daily audio streams. The reported overall accuracy of the method is approximately 82%. George et al. [19] The authors propose an audio fingerprinting method that is tolerant to time-stretching and is scalable. The proposed method uses three peaks in the time slice, unlike Shazam, which uses only one. The additive noise deteriorates the lowest frequency bin, decreasing the performance of the algorithm at higher additive noise, compared to other algorithms. Kim et al. [20] The authors propose a Television advertisement search based on audio peak-pair hashing method. The reported results show that the proposed method has respectable results compared to other methods. Seo [21] The authors propose an asymmetric fingerprint matching method which utilizes an auxiliary information obtained while extracting fingerprints from the input unknown audio. The experiments carried out with one thousand songs against various distortions compare the performance of the asymmetric matching with the conventional Hamming distance. Reported results suggest that the proposed method has better performance than the conventional Hamming distance. Rafii et al. [22] The authors propose an audio fingerprinting system with two stages: fingerprinting and matching. The system uses CQT and a threshold method for fingerprinting stage, and the Hamming similarity and the Hough Transform for the matching stage, reporting an accuracy between 61% and 81%. Naini et al. [23] The authors present a method for designing fingerprints that maximizes a mutual information metric, using a greedy optimization method that relies on the information bottleneck (IB) method. The results report a maximum accuracy around 65% in the recognition. Yang et al. [24] The authors propose an efficient music identification system that utilizes a kind of space-saving audio fingerprints. The experiments were conducted on a database of 200,000 songs and a query set of 20,000 clips compressed in MP3 format with different bit rates. The author’s report that compared to other methods, this method reduces the memory consumption and keeps the recall rate at approximately 98%. 61 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 8 of 23 Table 2. Cont. Paper Outcomes IEEE 62 Yin et al. [25] The authors propose a compressed-domain audio fingerprinting algorithm for MP3 music identification in the Internet of Things. The algorithm achieves promising results on robustness and retrieval precision rates under various time-frequency audio signal distortions including the challenging pitch shifting and time-scale modification. Wang et al. [26] The authors propose parallelized schemes for audio fingerprinting over GPU. In the experiments, the speedup factors of the landmark lookup and landmark analysis are verified and the reported overall response time has been reduced. Lee et al. [27] The authors propose a salient audio peak pair fingerprint extraction based on CQT. The reported results show that the proposed method has better results compared to other methods, and is suitable for many practical portable consumer devices. Shibuya et al. [28] The authors develop a method that uses the quadratically interpolated FFT (QIFFT) for the audio fingerprint generation in order to identify media content from an audio signal recorded in a reverberant or noisy environment with an accuracy around 96%. Bisio et al. [29] The authors present an improvement of the parameter configuration used by the Philips audio fingerprint computation algorithm in order to reduce the computational load and consequent energy consumption in the smartphone client. The results show a significant reduction of computational time and power consumption of more than 90% with a limited decrease in recognition performance. Lee et al. [30] The authors propose an audio fingerprint search algorithm for music retrieval from large audio databases. The results of the proposed method achieve 80–99% search accuracy for input audio samples of 2–3 s with signal-to-noise ratio (SNR) of 10 dB or above. Bisio et al. [31] The authors present an optimization of the Philips Robust Hash audio fingerprint computation algorithm, in order to adapt it to run on a smartphone device. In the experiments, the authors report that the proposed algorithm has an accuracy of 95%. Anguera et al. [32] The authors present a novel local audio fingerprint called Masked Audio Spectral Keypoints (MASK) that is able to encode, with few bits, the audio information of any kind in an audio document. MASK fingerprints encode the local energy distribution around salient spectral points by using a compact binary vector. The authors report an accuracy around 58%. Duong et al. [33] The authors presented a new approach based on audio fingerprinting techniques. The results of this study indicate that a high level of synchronization accuracy can be achieved for a recording period as short as one second. Wang et al. [34] The authors present an audio fingerprinting algorithm, where the audio fingerprints are produced based on 2-Dimagel, reporting an accuracy between 88% and 99%. Xiong et al. [35] The authors propose an improved audio fingerprinting algorithm based on dynamic subband locating and normalized Spectral Subband Centroid (SSC). The authors claim that the algorithm can recognize unknown audio clips correctly, even in the presence of severe noise and distortion. Deng et al. [36] The authors propose an audio fingerprinting algorithm based on harmonic enhancement and Spectral Subband Centroid (SSC). The authors build a database with 100 audio files, and also implement several techniques to reduce the noise and other degradations, proving the reliability of the method when severe channel distortion is present. The results report an accuracy between 86% and 93%. Pan et al. [37] The authors propose a method for fingerprinting generation using the local energy centroid (LEC) as a feature. They report that the method is robust to different noise conditions and, when the linear speed is not changed, the audio fingerprint method based on LEC obtains an accuracy of 100%, reporting better results than Shazam’s fingerprinting. Martinez et al. [38] The authors present a music information retrieval algorithm based on audio fingerprinting techniques. The size of frame windows influences the performance of the algorithm, e.g., the best size of the frame window for shorts audio tracks is between 32 ms to 64 ms, and the best size of the frame window for audio tracks is 128 ms. Cha [39] The author proposes an indexing scheme for large audio fingerprint databases. The method shows a higher performance than the Haitsma-Kalker method with respect to accuracy and speed. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 9 of 23 Table 2. Cont. Paper Outcomes IEEE Schurmann et al. [40] The authors propose a fuzzy-cryptography scheme that is adaptable in its noise tolerance through the parameters of the error correcting code used and the audio sample length. In a laboratory environment, the authors utilized sets of recordings for five situations at three loudness levels and four relative positions of microphones and audio source. The authors derive the expected Hamming distance among audio fingerprints through 7500 experiments. The fraction of identical bits is above 0.75 for fingerprints from the same audio context, and below 0.55 otherwise. Son et al. [41] The authors present an audio fingerprinting algorithm to recognize songs in real noisy environments, which outperforms the original Philips algorithm in recognizing polyphonic music in real similar environments. Chang et al. [42] The authors introduce the Compressive Sampling (CS) theory to the audio fingerprinting system for music recognition, by proposing a CS-based sub-Nyquist audio fingerprinting system. Authors claim that this system achieves an accuracy of 93.43% in reducing the sampling rate and in the extraction of musical features. Umapathy et al. [43] The authors present a novel local discriminant bases (LDB)-based audio classification scheme covering a wide range of audio signals. After the experiments, the obtained results suggest significant potential for LDB-based audio classification in auditory scene analysis or environment detection. Kim et al. [44] The authors develop a system that retrieves desired bookmarked video segments using audio fingerprint techniques based on the logarithmic modified Discrete Cosine Transform (DCT) modulation coefficients (LMDCT-MC) feature and two-stage bit vector searching method. The author’s state that the search accuracy obtained is 99.67%. Sert et al. [45] The authors propose an audio fingerprinting model based on the Audio Spectrum Flatness (ASF) and Mel Frequency Cepstral Coefficients (MFCC) features, reporting and accuracy of 93% and 91%, respectively. The authors propose a method to create audio fingerprints by Gaussian Mixtures using Ramalingam et al. features extracted from the short-time Fourier transform (STFT) of the signal. The [46] experiments were performed on a database of 250 audio files, obtaining the highest identification rate of 99.2% with spectral centroid. Ghouti et al. [47] The authors propose a framework for robust identification of audio content by using short robust hashing codes, which applies the forward balanced multiwavelet (BMW) to transform each audio frame using 5 decomposition levels, and after the distribution of the subbands’ coefficients into 32 different blocks, the estimation quantization (EQ) scheme and the hashes are computed. Cook et al. [48] The authors propose a system that allows audio content identification and association of metadata in very restricted embedded environments. The authors report that the system has better performance than the method based on a more traditional n-dimensional hashing scheme, but it achieves results with 2% less accuracy. Seo et al. [49] The authors propose an audio fingerprinting method based on the normalized Spectral Subband Centroid (SSC), where the match is performed using the square of the Euclidean distance. The normalized SSC obtains better results than the widely-used features, such as tonality and Mel Frequency Cepstral Coefficients (MFCC). Haitsma et al. [50] The authors present an approach to audio fingerprinting, but it has negligible effects on other aspects, such as robustness and reliability. They proved that the developed method is robust in case of linear speed changes. Haitsma et al. [51] The authors present an approach to audio fingerprinting, in which the fingerprint extraction is based on the extraction of a 32-bit sub-fingerprint every 11.8 millis. They also develop a fingerprint database and implement a two-phase search algorithm, achieving an excellent performance, and allowing the analytical modeling of false acceptance rates. 3. Results As illustrated in Figure 1, our review identified 115 papers that included three duplicates, which were removed. The remaining 112 works were evaluated in terms of title, abstract, and keywords, resulting in the exclusion of 50 citations. Full text evaluation of the remaining 62 papers resulted in the exclusion of 22 papers that did not match the defined criteria. The remaining 40 papers were 63 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 10 of 23 included in the qualitative synthesis and the quantitative synthesis. In summary, our review examined 40 papers. Figure 1. Flow diagram of identification and inclusion of papers. We suggest that interested readers refer to the original cited works to find relevant information about the details of the methods analyzed in this review. Table 1 shows the year of publication, population, purpose of the study, devices, and settings of the selected papers. Table 2 shows study aims and results. As shown in Table 1, all studies have been performed in controlled environments (laboratory). The major part of the studies was performed between 2011 and 2016 with a total of 29 studies (73%), where five studies were in 2011 (13%), five studies in 2012 (13%), four studies in 2013 (10%), eight studies in 2014 (20%), six studies in 2015 (15%), and one study in 2016 (3%). Some studies indicate the devices used: eight studies used computer microphones (23%), 10 studies used mobile devices (25%), and two studies used a television (5%). Methods for Audio Fingerprinting In [12], the authors created a system that implements the framing, Fast Fourier Transform (FFT), calculation of the spectrum modules, extraction of two kinds of audio fingerprinting, and two level search of two kinds of audio fingerprinting. The two kinds were extracted calculating the sum of the spectrum modulus of every frame, getting the sum of global spectrum modulus in two stages. The authors reported that, when the signal noise rate (SNR) is 10 dB, the two level algorithm, with parallel processing, reports a precision of 100% [12]. In [14], several MP3 features were extracted, such as the Single local description, the Multiple local description, the Modified discrete cosine transform (MDCT), the Mel-frequency cepstrum coefficients (MFCC), the MPEG-7 descriptors, and the chroma vectors, using the Principal Component Analysis (PCA) technique to reduce the dimensionality and QUery Context (QUC)-tree to search for songs. 64 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 11 of 23 The tests of the methods described in [14] were performed with 10,000 MP3 fragments, reporting a maximum average error equals to 4.26%, which represents an accuracy around 96%. In [13], the same authors extracted the same features and used the same techniques, but they also used the MP3 signatures for the implementation of the audio fingerprinting method, performing tests with 100,000 MP3 fragments, which reported the errors and accuracies obtained are equals to the reported in [14]. Tsai et al. [15] presented a method to calculate audio fingerprints with 6 steps, namely compute spectrogram, collect context frames, apply eigenfilters, compute deltas, apply threshold, and bit packing. The authors reported that the developed method is more robust than the three other fingerprints (e.g., Shazam, Masked Audio Spectral Keypoints (MASK), and Panako), achieving an accuracy of 99.2% [15]. Another audio feature named local energy centroid (LEC) is used in [37] to obtain a representation of audio signals in noisy condition. The method for audio fingerprinting has several steps. First, the audio is downsampled to 8 kHz and segmented into frames, and then FFT is employed to obtain the spectrum. Later, the spectrum is optimized by applying weighted window functions with different size. Then, the LEC is saved and the amplitude components are removed, obtaining an audio spectrum that can be represented by sparse LEC set of coordinates [37]. The authors reported that the method is robust to different noise conditions, and when the linear speed is not changed, the audio fingerprint method based on LEC reports an accuracy of 100% [37]. In [36], the authors proposed an audio fingerprinting algorithm that starts with the application of low-pass filter to the audio signal and resampling to eliminate the high-frequency noise and other audio components that are perceptually insignificant for human auditory system. Afterwards, the audio is framed and weighted by Window function, and the FFT is applied [36]. Next, the Spectral Subband Centroid (SSC) is calculated and the approach of harmonic enhancement is adopted to estimate the predominant pitch of audio signal [36]. Finally, the normalized SSC is masked by the predominant pitch, and the proposed algorithm is resistant to some kinds of signal degradations in varying degrees, reporting an accuracy between 86% and 93% [36]. The authors of [35] also used the normalized SSC for the creation of an audio fingerprinting algorithm. The algorithm is structured in several phases, such as: pre-processing, framing, implementing the FFT to transform audio signals from time to frequency domain, implementing the dynamic subband locating, and applying the normalized SSC, obtaining, at the end, the audio fingerprint [35]. With the fingerprints created, the authors reported an accuracy up to 80% in normal conditions [35]. The authors of [49] also proposed an audio fingerprinting algorithm using SSC, starting with the conversion to mono and downsampling the audio to 11,025 Hz. After the downsampling, the audio signal is windowed by Hamming window (typically 371.5 ms) with 50% overlap and transformed into the frequency domain using FFT [49]. Afterwards, the audio spectrum is divided into 16 critical bands, and the frequency centroids of the 16 critical bands are used as a fingerprint of the audio frame [49], reporting an accuracy around 60% with MP3 and Random start, and an accuracy around 100% with Equalization. Another algorithm is presented in [50] that consists of the modification of an existing algorithm named Streaming Audio Fingerprinting (SAF), which apply the framing and the FFT, create energy 33 bands, and then, apply a filter and a threshold. The modification of the algorithm consists of increasing the number of the energy bands, and three new steps between the creation of energy bands and the application of a filter and threshold: auto-correction, filter and the creation of a subsample [50]. The authors reported that the algorithm is robust in case of linear speed changes [50]. In [28], the audio fingerprinting methods proposed has several steps, these are framing, application of FFT or quadratically interpolated FFT (QIFFT), time averaging, peak detection, quadratic interpolation, sinusoidal quantification, frequency-axial discretization, and time-axial warping. A fingerprint that represents the distribution of pseudosinusoidal components in the time-frequency domain is generated, showing results with an accuracy around 96% and precision of 100% [28]. 65 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 12 of 23 In [51], the authors proposed a new fingerprint algorithm based on the streaming approach, where the audio signal is segmented into overlapping frames, the FFT should be applied, and after that, the Human Auditory System (HAS) is used, reporting an accuracy of 100% for the recognition of pop-music. In [45] is proposed a system for audio fingerprinting that starts with preprocessing and framing of the audio signal. Afterwards, a general feature extraction paradigm, extended with a descriptor based on structural similarity analysis with MPEG-7 Audio Spectrum Flatness (ASF), is applied to the signal [45]. The last step, before the fingerprint construction, consists of the structural analysis that results only the feature vector of the expressive audio piece [45]. At the end, the authors reduce the dimension of the ASF feature vector in the fingerprint construction stage based on the MPEG-7 Audio Signature (AS) description scheme [45], reporting an accuracy around 93%. The authors of [46] proposed an audio fingerprinting scheme with several stages, such as preprocessing, framing, feature extraction, Gaussian mixture models (GMM) modelling, likelihood estimation, and comparison with a fingerprinting database. In the preprocessing stage, the audio signal is converted to a standard format (16-bit, pulse code modulation (PCM)) [46]. In the framing stage, the audio signals are divided into frames of length equals to 23 ms [46]. During feature extraction, the authors used the STFT, extracting several features, such as Shannon entropy, Rényi entropy, Spectral centroid, Spectral bandwidth, Spectral band energy, Spectral flatness measure, Spectral crest factor, and Mel-frequency cepstral coefficients (MFCC) [46]. Afterwards, the GMM models are applied, using the probability density function (PDF), and the Expectation-Maximization (EM) [46]. Among the features used, spectral centroid gives the highest identification rate of 99.2% [46]. The authors of [47] presented an audio fingerprint extraction algorithm, consisting of: downsampling of the input audio content of 3 s to obtain a sampling rate of 5512 Hz; applying the framing division on the downsampled content using Hamming window with an overlap factor of 31/32; computing the forward balanced multiwavelet (BMW) to transform for each audio frame using five decomposition levels; dividing the subbands’ coefficients into 32 different blocks; applying the estimation quantization (EQ) scheme using a neighbouring window of five audio samples; computing the log variances of the magnitudes of the subbands’ coefficients; computing the mean value of all the log variances for each audio frame; and at the end, extracting the sub-hash bit. Authors report that the performance of the algorithm degrades as the compression rate increases. In [48], the authors proposed an algorithm with two stages named indexing and search. The indexing is based in the construction of zone tables using the Search by Range Reduction (SRR) threshold values [48]. The search is based on the SRR test, calculating the Itakura distance between two fingerprints, and comparing it with values in the zone tables [48]. An accuracy of around 98% is reported. The authors of [43] proposed an algorithm with training and testing phases. For the training phase, the authors started with the wavelet packet decomposition, and developed a local discriminant bases (LDBs)-based automated multigroup audio classification system, which focuses on identifying discriminatory time-frequency subspaces [43]. The testing phase consists of the construction of a new wavelet tree, feature extraction, and implementation of a linear discriminant analysis (LDA) [43]. The extracted features include MFCC, spectral similarity, timbral texture, band periodicity, linear prediction coefficient derived cepstral coefficients (LPCCs), zero crossing rate, MPEG-7 descriptors, entropy, and octaves [43]. The authors of [43] reported that the average classification accuracy was between 91% and 99% [43]. The authors of [44] presents a video retrieval system (VRS) for Interactive-Television as like internet protocol television (IPTV), which implements an audio fingerprint feature of long-term logarithmic modified discrete cosine transform (DCT) modulation coefficients (LMDCT-MC) for audio indexing and retrieval, and implements two-stage search (TSS) algorithm for fast searching. In the first stage of TSS, candidate video segments are roughly determined with audio index bit vectors (IBV) 66 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 13 of 23 and then the optimal video clip is obtained by fingerprint bit vectors (FBV). An accuracy of 99.67% is reported in [44]. In [41] an audio fingerprint method using sub-fingerprint masking based on the predominant pitch extraction is proposed. It increases the accuracy of the audio fingerprinting system in a noisy environment dramatically, while requiring much less computing power compared to the expanded hash table lookup method. When applied to an audio signal without noise, the reported accuracy is 97.4%. The authors of [42] presented a sub-Nyquist audio fingerprinting system for music recognition, which utilizes Compressive Sampling (CS) theory to generate a compact audio fingerprint, and to achieve significant reduction of the dimensionality of the input signal, compared to Nyquist sampling methods [42]. The average accuracy of this method is 93.43% under various distorted environments. In [38], the authors presented an algorithm based on fingerprinting techniques implemented in a low-cost embedded reconfigurable platform. It utilizes the FFT implementation from the CUFFT library, based on the Fastest Fourier Transform in the West (FFTW) algorithm. This approach normalizes and frames the audio signal, computes the correlation and cross correlation, and applies a derivative of the audio signal. An accuracy of 94% is reported. The authors of [39] created a fingerprint database of songs and focused on the problem of effective and efficient database search. The authors proposed a new indexing scheme that overcomes the limitations of Haitsma-Kalker’s method and Miller’s k-ary tree method, adopting the inverted file as the underlying index structure and developing the techniques to apply it to the effective and efficient audio fingerprinting problem. An accuracy higher than 97% is reported in [39], which is the performance of the Haitsma-Kalker’s method. The authors of [40] explored a common audio-fingerprinting approach with the implementation of FFT, and taken into account the noise in the derived fingerprints by employing error correcting codes and applying statistical tests. Testing with several sample windows of Network Time Protocol (NTP)-based synchronization recordings, authors of [40] reported an accuracy between 60% and 70%. The authors of [31] created a system based on a client-server architecture able to recognize a live television show using audio fingerprinting. To create audio fingerprints, FFT is computed to obtain the power spectrum, which is integrated over a pre-defined set of non-overlapping, logarithmically spaced frequency bins and eventually squared to obtain an energy measure [31]. The likelihood estimation based on the cross-correlation function was used for comparison of the audio fingerprints. An accuracy of around 95% is reported in [31]. The authors of [32] presented an audio fingerprint method named Masked Audio Spectral Keypoints (MASK), which encodes the acoustic information existent in audio documents and discriminates between transformed versions of the same acoustic documents and other unrelated documents. The MASK fingerprint extraction method is composed of several tasks: time-to-frequency transformation, where the input signal is transformed from the time domain to the spectral domain, and transformed into Mel-scale; salient spectral points search; local mask application around each of the salient points; grouping of the different spectrogram values into regions; and the MASK fingerprint encoding and storage. The averaged energy values of each one of these spectrogram regions are compared to construct a fixed length binary descriptor. Authors of [32] report an accuracy around 58%. In [33], the authors implemented an audio fingerprinting algorithm based on fingerprint extraction and matching search, adapting the well-known Philips’ algorithm. The fingerprint extraction derives and encodes a set of relevant audio features, which need to be invariant to various kinds of signal distortion, including background noise, audio compression, and A/D conversion [33]. The matching search finds the best match between these fingerprints and those stored in the database [33]. The implemented audio fingerprint extraction method uses FFT, and extracts several features, such as: mel-frequency cepstral coefficients (MFCC), spectral centroid or spectral flatness [33]. The audio fingerprinting method reports an accuracy of 95% and a precision of 100% [33]. 67 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 14 of 23 The authors of [34] proposed an audio fingerprinting system with several characteristics, including robustness, granularity, and retrieval speed, reporting an accuracy between 88% and 99%. The structure of the audio fingerprinting implemented is the same as all other algorithms presented in [34], applying the FFT and an High-pass filter. The authors used the local maximum chroma energy (LMCE) to extract the perception features of Tempo-Frequency domain [34]. The work presented in [29] reviews the state-of-the-art methods for improving the power consumption and computation speed to make the smartphone implementation. It also proposed the Improved Real-Time TV-channel Recognition (IRTR), which is a fingerprint extraction method aimed at recognizing in real time what people are watching on TV without any active user interaction. The identification using the audio fingerprint is performed using a likelihood estimation [29]. The audio fingerprinting method implements linear transform and feature extraction, with several steps: the audio is recorded and divided into frames with overlap factor; each frame is filtered by means of a Hamming window function; the application of the FFT and the squared modulus; the spectrum is divided into logarithmically spaced frequency bins and the energy is computed for each bin; and the nervy of band of each frame is denoted. An accuracy about 95% is reported in [29]. In [30], an audio fingerprinting algorithm is proposed for efficient retrieval of corresponding or similar items from large audio databases, which improves the of the database search compared to the algorithm used in Haitsma’s method, without impairing the accuracy of the search results. The approach implements the FFT, the extraction of candidate songs via lookup table, the assignment of weights to candidate songs, and the database search [30], while reporting an average accuracy around 81%. The authors of [21] proposed an audio fingerprinting algorithm, which improves binary audio fingerprint matching performance by utilizing auxiliary information. The proposed matching method is based on Philips robust hash (PRH) for audio signal; Asymmetric Fingerprint Matching for PRH using the Magnitude Information, which consists of Normalization of the Subband-Energy Difference; and Fingerprint Matching Based on the Likelihood Ratio Test [21]. The proposed method yields better performance than the conventional Hamming distance [21]. The authors of [22] proposed an audio fingerprinting constituted by two stages: fingerprinting and matching. The fingerprinting module uses a log-frequency spectrogram based on the Constant Q Transform (CQT), and an adaptive thresholding method based on two-dimensional median filtering [22]. The matching uses the Hamming similarity and the Hough Transform [22]. The reported accuracy is between 61% and 81%. The authors of [23] presented a method for the construction of audio fingerprints based on: maximization of the mutual information across the distortion channel; using the information bottleneck method to optimize the filters; and quantizers that generate these fingerprints. The method starts with the application of the short time Fourier transform (STFT), and capturing the Spectral Sub-band Centroids (SSC) using 16 bins on the Bark scale. The generated features with [23] result in a maximum accuracy of around 65%. The authors of [24] implemented an audio fingerprinting algorithm composed by several steps: downsampling to 5 kHz, segmenting frames every 11.6 ms, applying the FFT, calculating the frequency bands energies, and finally, calculating the fingerprints. A recall around 98% is reported. In [25], the authors presented an audio fingerprinting algorithm based on the compressed-domain spectral entropy as audio features, showing strong robustness against various audio signal distortions such as recompression, noise interference, echo addition, equalization, band-pass filtering, pitch shifting, moderate time-scale modification, among others. The algorithm includes four steps: granule grouping, frequency alignment between long and short windows, coefficients selection and subband division, and MDCT spectral entropy calculation and fingerprint modelling [25]. It reports an accuracy above 90%. In [26], the authors presented the implementation of an audio fingerprinting system, using graphic processing units (GPUs). The system starts with the extraction of landmarks using FFT, and continues 68 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 15 of 23 with the landmark extraction, lookup, and analysis. The authors explored the use of one thread for one hash key, and one block for one hash key, reporting an accuracy around 80.96%, when there are 100,000 songs in the database [26]. The authors of [27] proposed a high-performance audio fingerprint extraction method for identifying TV commercial advertisement. The audio fingerprint extraction consists of a salient audio peak pair fingerprints based on constant Q transform (CQT). The algorithm obtains the audio fingerprints through five main steps: preprocessing; application of the CQT; application of the Mean Subtraction of Logarithmic CQT Spectrum; application of the CQT Based Salient Peak Detection using forward and Backward Filtering; and finally, application of the Fingerprint Generation using CQT Peak Pair. The reported recognition accuracy of the method based on CQT, presented in [27], is around 89.8%. The authors of [16] used a smartphone and create an audio fingerprinting algorithm based on the joint usage of GPS and acoustic fingerprints. The authors created an audio fingerprinting algorithm with noise tolerance, assessing it under several conditions [16]. The algorithm starts with the calculation of the audio sample spectrogram using the STFT, and then calculates audio sample spectrogram using the Hamming window and a high overlap [16]. Next, it takes only the first 40 frequency bins, as most of the useful audio features are in that bandwidth, averaging the logarithmic amplitude in each bin [16]. Afterwards, for each frequency bin, a 16-bit fingerprint is calculated [16]. The 16-bits fingerprint is then stored with the associated frequency and time [16]. For the comparison of the audio fingerprints, the Hamming distances between each fingerprint are calculated, looking for a minimum [16]. An accuracy of around 86% is reported. In [17], the authors proposed an approach to accelerate fingerprinting techniques by skipping the search for irrelevant sections of the signal and demonstrate its application to the divide and locate (DAL) audio fingerprint method. The method in DAL starts with the extraction of the time-frequency power spectral applied for the signals, normalizing each logarithmic power [17]. Afterwards, the normalized data is decomposed into a number of small time-frequency components of uniform size, and thus, the computational cost and memory usage are reduced in the fingerprint data [17]. The authors verified that with a reduced search threshold, the accuracy of the recognition is around 100% [17]. The authors of [18] created a method to analyze and classify daily activities in personal audio recordings (PARs). The method applies: speech activity detection (SAD), speaker diarization systems, and computing the number of audio speech and lexical features [18]. It uses a TO-Combo-SAD (Threshold Optimized Combo SAD) algorithm for separating speech from noise [18]. The Principal Component Analysis (PCA) is first applied for dimensionality reduction, and then, the remaining features are supplied to a multi-class support vector machine (SVM) with radial basis function (RBF) kernel for model training and evaluation [18]. The authors performed recognition of faculty meeting, research meeting, staff meeting, alone time, and conference call, reporting accuracies between 62.78% and 84.25% [18]. In [19], the authors proposed an audio fingerprinting method, based on landmarks in the audio spectrogram. The algorithm is based on the audio hashing of frequency peaks in the spectrogram [19]. It starts with the application of the FFT, thresholding the data, applying a high pass filter, identifying the local maximums and finding the peaks of the spectrogram [19]. The performance of the algorithm decreases at higher additive noise in comparison with other algorithms [19], reporting an accuracy around 96.71%. In [20], the authors proposed a robust TV advertisement search based on audio fingerprinting in real environments. This algorithm has several steps, such as preprocessing, logarithmic moduled complex lapped transform (LMCLT), two-are segmentation using adaptive thresholding based on median filtering, detection of prominent LMCLT spectral peaks, and fingerprinting generation using LMCLT peak pair [20]. The method applies adaptive peak-picking thresholding method to extract more salient and distinct peak pairs for comparing the query fingerprint with the original fingerprints, and the authors reported an accuracy of 86.5% [20]. 69 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 16 of 23 4. Discussion This review confirms the findings of previous studies related to the use of audio fingerprinting techniques for identification of the environment related to the different ADLs. We consider that many of the reviewed works raise important issues regarding the concept of Open Science, including, but not limited to, Reproducibility and Verifiability of the research results, and Comparability of similar research. Many of them were evaluated on unpublished data and did not publish their source code, although when commercial solutions are in perspective, a necessary degree of confidentiality is understandable. Regarding validation and comparability, only six studies used raw data available online or published its research data online. Likewise, only three studies presented some parts of the code used of the experiments. In addition, the studies that used data that is now publicly available, did not publish the research source code, making the validation of the results and further comparative research an impossible task. Therefore, we suggest to the audio fingerprinting community to become better at sharing raw data and algorithms, so as to be able to recreate and evaluate the soundness of previous studies. Nevertheless, assuming the results of the presented research studies are comparable, Tables 3–5 present a summary of the Features and Methods ordered by the number of identified studies that use these features and methods. Tables 3 and 4 present the distribution of the extracted features and methods implemented in the analyzed studies, verifying that FFT is one of the most widely used feature extraction method, because it extracts the frequencies from the audio signal, and the other most used features include thresholding, normalized Spectral Subband Centroid (SSC), Mel-frequency cepstrum coefficients (MFCC), maximum, local peaks and landmarks, Shannon entropy, Rényi entropy, MPEG-7 descriptors, Spectral bandwidth, Spectral flatness measure, Modified discrete cosine transform (MDCT), Constant Q Transform (CQT), Short-time Fourier transform (STFT), average, and the maximum and minimum. These features were used in a large part of the analyzed studies [12,14,19,24,26,28–31,33–35,38,49–51], and with them, the reported accuracy is greater than 80%, as presented in Table 3. For Tables 3 and 4, the accuracies that are equal or higher than 99% are shown in a different background color (yellow). Table 3. Distribution of the features extracted in the studies. 70 Features Average Accuracy of Features Number of Studies Fast Fourier Transform (FFT) Thresholding Normalized spectral subband centroid (SSC) Mel-frequency cepstrum coefficients (MFCC) Maximum Local peaks and landmarks Shannon entropy Rényi entropy MPEG-7 descriptors Spectral bandwidth Spectral flatness measure Modified discrete cosine transform (MDCT) Constant Q transform (CQT) Short-time Fourier transform (STFT) Average Minimum 93.85% 90.49% 93.44% 97.30% 87.57% 82.32% 99.10% 99.10% 97.50% 97.10% 97.10% 93.00% 85.40% 84.50% 83.00% 83.00% 16 6 5 4 3 3 2 2 2 2 2 2 2 2 2 2 Sum of the spectrum modulus of every frame Sum of global spectrum modulus in two stages Local energy centroid (LEC) Time-frequency power spectral 100.00% 100.00% 100.00% 100.00% 1 1 1 1 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 17 of 23 Table 3. Cont. Features Average Accuracy of Features Number of Studies Long-term logarithmic modified discrete cosine transform (DCT) modulation coefficients (LMDCT-MC) 99.67% 1 Bit packing Spectral band energy Spectral crest factor Spectral similarity Timbral texture Band periodicity Linear prediction coefficient derived cepstral coefficients (lpccs) Zero crossing rate Octaves Single local description Multiple local description Chroma vectors MP3 signatures Time averaging Quadratic interpolation Sinusoidal quantification Frequency-axial discretization Time-axial warping Logarithmic moduled complex lapped transform spectral peaks Correlation coefficient Matching score 99.20% 99.20% 99.20% 99.00% 99.00% 99.00% 99.00% 99.00% 99.00% 96.00% 96.00% 96.00% 96.00% 96.00% 96.00% 96.00% 96.00% 96.00% 86.50% 70.00% 70.00% 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Table 4. Distribution of the methods implemented in the studies. Methods Average Accuracy of Methods Number of Studies Other methods Two level search algorithm Likelihood estimation Principal Component Analysis (PCA) Hamming distances between each fingerprint Streaming audio fingerprinting (SAF) Human auditory system (HAS) Divide and locate (DAL) Gaussian mixture models (GMM) modelling 90.78% 99.84% 97.10% 90.13% 83.50% 100.00% 100.00% 100.00% 99.20% 15 2 3 2 2 1 1 1 1 Local discriminant bases (LDBS)-based automated multigroup audio classification system 99.00% 1 Linear discriminant analysis (LDA) Local maximum chroma energy (LMCE) Expanded hash table lookup method Query Context (QUC)-tree Improved Real-Time TV-channel Recognition (IRTR) Sub-Nyquist fudio fingerprinting system Logarithmic moduled complex lapped transform (LMCLT) peak pair TO-Combo-SAD (Threshold Optimized Combo SAD) algorithm Support vector machine (SVM) Hough Transform between each fingerprint Masked audio spectral keypoints (MASK) 99.00% 99.00% 97.40% 96.00% 95.00% 93.43% 86.50% 84.25% 84.25% 81.00% 58.00% 1 1 1 1 1 1 1 1 1 1 1 71 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 18 of 23 On the other hand, as verified in Table 4, a large part of the analyzed studies [12,14,19,24,26,28–31,33–35,38,49–51] do not mention the name of the method applied, presenting only the features used. Regarding the undifferentiated methods, the most used methods are the two level search algorithm, the likelihood estimation, the Principal Component Analysis (PCA), and the Hamming distances between each fingerprint, reporting accuracies also higher than 80%. Table 5 presents in a matrix format, the average of the averages of the accuracies in Methods vs. its Features. This table is a mere comparison exercise, as there are not enough studies to sustain a valid analysis of the use of different features with different methods. On the other hand, this table assumes that these results are comparable, and moreover, that any method or algorithm can be used with any set of features, which of course, is a very wide, and possibly not true assumption. Nevertheless, Table 5 shows, in a colored background the match between features and methods. For example, for method SAF (Streaming Audio Fingerprinting) the set of used features are Fast Fourier Transform, Thresholds and Energy bands, whose mean accuracies in the found studies are not higher than 99%. Also, for example for the method GMM (Gaussian Mixture Models Modelling), besides the 4 highlighted features that were used, this method uses additionally 5 other sets of features. Taking Table 5 into consideration, one can identify Shannon’s Entropy as the feature that is most used in the most accurate number of studies. Arguably, this table may propose new combinations of features and methods that can be used to devise audio-fingerprinting solutions. For a particular use, the methods to be implemented must be chosen according to their complexity, the computational power of the use case scenario, and to the purpose of its intended use. This review is focused on the use of mobile devices, but only three of the reviewed works argue that they use methods that need low resources (see Table 1). Only 19 studies compared the implemented methods with other methods published in the literature and present their accuracy, claiming an increased accuracy in the recognition of the environment using audio fingerprinting. According to the results of this review, the use of the mobile devices for the application of audio fingerprinting techniques is limited, because of the restrictions these devices impose, i.e., low power processing and battery capacity. Thus, only 10 of the analyzed studies utilize mobile devices with local processing or server-side processing of the data acquired from the mobile devices. In the case of the server-side processing, the use of the mobile devices implies a constant and stable network connection, which is not a trivial requirement both from technical perspective, but also because of battery life implications. To some extent, the using Fog and Mist Computing paradigms could overcome the challenges of the client-server architectures. The creation of lightweight techniques should be explored, as they could be executed on mobile devices (i.e., edge-nodes). The models could be recalibrated offline on the server occasionally, and then, as pre-trained models to be seamlessly redeployed on mobile devices. In conclusion, only one of the reviewed studies [38] can achieve reliable performance with reduced computational cost and memory usage. It utilizes the FFT implementation in the CUFFT library, divide and locate (DAL) audio fingerprint method, and sub-fingerprint masking based on the predominant pitch extraction methods. However, other methods could be implemented on mobile devices with some restrictions. Nonetheless, they could be amended to utilize more lightweight implementations of the underlying libraries, or by sacrificing floating point precision, for instance. 72 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 19 of 23 Table 5. Potential accuracies for the top most accurate methods vs. top most mean accurate features (mean accuracies equal or higher than 99%, according to its authors). SAF HAS DAL TLS GMM LDBS LDA LMCE Local energy centroid (LEC) Sum of global spectrum modulus in two stages Sum of the spectrum modulus of every frame Time-frequency power spectral 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 100.00% 99.84% 99.92% 99.92% 99.92% 99.92% 99.20% 99.60% 99.60% 99.60% 99.60% 99.00% 99.50% 99.50% 99.50% 99.50% 99.00% 99.50% 99.50% 99.50% 99.50% 99.00% 99.50% 99.50% 99.50% 99.50% Long-term logarithmic modified discrete cosine transform (DCT) modulation coefficients (LMDCT-MC) 99.67% 99.84% 99.84% 99.84% 99.76% 99.44% 99.34% 99.34% 99.34% Bit packing Spectral band energy Spectral crest factor Rényi entropy Shannon entropy Band periodicity Linear prediction coefficient derived cepstral coefficients (lpccs) Octaves Spectral similarity Timbral texture Zero crossing rate 99.20% 99.20% 99.20% 99.10% 99.10% 99.00% 99.00% 99.00% 99.00% 99.00% 99.00% 99.60% 99.60% 99.60% 99.55% 99.55% 99.50% 99.50% 99.50% 99.50% 99.50% 99.50% 99.60% 99.60% 99.60% 99.55% 99.55% 99.50% 99.50% 99.50% 99.50% 99.50% 99.50% 99.60% 99.60% 99.60% 99.55% 99.55% 99.50% 99.50% 99.50% 99.50% 99.50% 99.50% 99.52% 99.52% 99.52% 99.47% 99.47% 99.42% 99.42% 99.42% 99.42% 99.42% 99.42% 99.20% 99.20% 99.20% 99.15% 99.15% 99.10% 99.10% 99.10% 99.10% 99.10% 99.10% 99.10% 99.10% 99.10% 99.05% 99.05% 99.00% 99.00% 99.00% 99.00% 99.00% 99.00% 99.10% 99.10% 99.10% 99.05% 99.05% 99.00% 99.00% 99.00% 99.00% 99.00% 99.00% 99.10% 99.10% 99.10% 99.05% 99.05% 99.00% 99.00% 99.00% 99.00% 99.00% 99.00% 73 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 20 of 23 5. Conclusions This review identified and described the methodologies used for audio fingerprinting that can be applied to mobile technologies. Forty-seven studies were examined and the main findings are summarized as follows: • • • (RQ1) the audio fingerprinting is defined as the ability to recognize the scenario in which a given audio was collected and involved in, based on various methods. (RQ2) Several techniques have been applied to implement audio fingerprinting methods, including Fast Fourier Transform (FFT). Support Vector Machine (SVM), QUery Context (QUC)-tree, spectral subband centroid (SSC), Streaming Audio Fingerprinting (SAF), Human Auditory System (HAS), Gaussian mixture models (GMM) modelling, likelihood estimation, linear discriminant analysis (LDA), Compressive Sampling (CS) theory, Philips robust hash (PRH), Asymmetric Fingerprint Matching, and TO-Combo-SAD (Threshold Optimized Combo SAD). These techniques yield high accuracy, and the use of mobile devices does not influence the predictive performance, allowing the use of these techniques anywhere, anytime. (RQ3) All of the methods presented in RQ2 can be implemented on mobile devices, but the methods that require lower computational resources are FFT with the CUFFT library, divide and locate (DAL) audio fingerprint method, and sub-fingerprint masking based on the predominant pitch extraction. In addition, this review highlights the application of audio fingerprinting techniques on mobile or other devices with limited computational and battery resources. Some limitations of this review should be mentioned. First, the authors chose to exclude studies that are not focused on audio fingerprinting techniques. Second, the studies that do not utilize mobile devices have been excluded. These exclusions were performed with the analysis of the abstract and then, the full text of the papers. Finally, only English-language publications were included. Based on the analysis, we conclude that the most used methods are undifferentiated methods, two level search algorithms, likelihood estimation, Principal Component Analysis (PCA), and Hamming distances between each fingerprint. The conclusion is that the use of statistical methods reports results with an accuracy higher than 80%. Furthermore, the most used features are Fast Fourier Transform (FFT), Thresholding, normalized spectral subband centroid (SSC), Mel-frequency cepstrum coefficients (MFCC), maximum, local peaks and landmarks, Shannon entropy, Rényi entropy, MPEG-7 descriptors, Spectral bandwidth, Spectral flatness measure. Modified discrete cosine transform (MDCT), Constant Q Transform (CQT), Short-time Fourier transform (STFT), average, and minimum, which also result in accuracies greater than 80%. As future work, the extraction of features based on audio fingerprinting will be implemented in order to develop a system for the recognition of ADLs and their environments, presented in [9–11]. As presented in Table 3, the accuracy is always higher than 80%. Then, we should consider the most used features, including FFT, MFCC, average, maximum, and minimum, in order to better handle the recognition of the environment. The implementation of this framework is part of the development of a personal digital life coach [4]. Acknowledgments: This work was supported by FCT project UID/EEA/50008/2013 (Este trabalho foi suportado pelo projecto FCT UID/EEA/50008/2013). The authors would also like to acknowledge the contribution of the COST Action IC1303—AAPELE—Architectures, Algorithms and Protocols for Enhanced Living Environments. Author Contributions: All the authors have contributed with the structure, content, and writing of the paper. Conflicts of Interest: The authors declare no conflict of interest. 74 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 21 of 23 References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. Foti, D.; Koketsu, J.S. Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction, 7th ed.; Activities of daily living; Mosby: St. Louis, MI, USA, 2013; pp. 157–232. Garcia, N.M.; Rodrigues, J.J.P. Ambient Assisted Living; CRC Press: Boca Raton, FL, USA, 2015. Dobre, C.; Mavromoustakis, C.X.; Goleva, R.L. Ambient Assisted Living and Enhanced Living Environments: Principles, Technologies and Control; Butterworth-Heinemann: Oxford, UK, 2016; p. 552. Garcia, N.M. A Roadmap to the Design of a Personal Digital Life Coach. In ICT Innovations 2015; Springer: Cham, Switzerland, 2016. Da Silva, J.R.C. Smartphone Based Human Activity Prediction; Faculdade de Engenharia: Porto, Portugal, 2013. Bieber, G.; Luthardt, A.; Peter, C.; Urban, B. The Hearing Trousers Pocket—Activity Recognition by Alternative Sensors. In Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments, Crete, Greece, 25–27 May 2011. Kazushige, O.; Miwako, D. Indoor-outdoor activity recognition by a smartphone. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA, 5–8 September 2012; p. 537. Ganti, R.K.; Srinivasan, S.; Gacic, A. Multisensor Fusion in Smartphones for Lifestyle Monitoring. In Proceedings of the 2010 International Conference on Body Sensor Networks, Singapore, 7–9 June 2010. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices. Sensors 2016, 16, 184. [CrossRef] [PubMed] Pires, I.M.; Garcia, N.M.; Flórez-Revuelta, F. Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices. In Proceedings of the ECMLPKDD 2015 Doctoral Consortium, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, 7–11 September 2015. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F. Identification of Activities of Daily Living Using Sensors Available in off-the-shelf Mobile Devices: Research and Hypothesis. In Ambient Intelligence-Software and Applications, Proceedings of the 7th International Symposium on Ambient Intelligence (ISAmI 2016), Seville, Spain, 1–3 June 2016; Springer International Publishing: Cham, Switzerland, 2016. Sui, D.; Ruan, L.; Xiao, L. A Two-level Audio Fingerprint Retrieval Algorithm for Advertisement Audio. In Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia, Kaohsiung, Taiwan, 8–10 December 2014; pp. 235–239. Liu, C.-C.; Chang, P.-F. An efficient audio fingerprint design for MP3 music. In Proceedings of the 9th International Conference on Advances in Mobile Computing and Multimedia, Ho Chi Minh City, Vietnam, 5–7 December 2011; pp. 190–193. Liu, C.-C. MP3 sniffer: A system for online detecting MP3 music transmissions. In Proceedings of the 10th International Conference on Advances in Mobile Computing, Bali, Indonesia, 3–5 December 2012; pp. 93–96. Tsai, T.J.; Stolcke, A. Robust and efficient multiple alignment of unsynchronized meeting recordings. IEEE/ACM Trans. Audio Speech Lang. Proc. 2016, 24, 833–845. [CrossRef] Casagranda, P.; Sapino, M.L.; Candan, K.S. Audio assisted group detection using smartphones. In Proceedings of the 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Torino, Italy, 29 June–3 July 2015. Nagano, H.; Mukai, R.; Kurozumi, T.; Kashino, K. A fast audio search method based on skipping irrelevant signals by similarity upper-bound calculation. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, 19–24 April 2015. Ziaei, A.; Sangwan, A.; Kaushik, L.; Hansen, J.H.L. Prof-Life-Log: Analysis and classification of activities in daily audio streams. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, 19–24 April 2015. George, J.; Jhunjhunwala, A. Scalable and robust audio fingerprinting method tolerable to time-stretching. In Proceedings of the 2015 IEEE International Conference on Digital Signal Processing (DSP), Singapore, 21–24 July 2015. Kim, H.G.; Cho, H.S.; Kim, J.Y. TV Advertisement Search Based on Audio Peak-Pair Hashing in Real Environments. In Proceedings of the 2015 5th International Conference on IT Convergence and Security (ICITCS), Kuala Lumpur, Malaysia, 24–27 August 2015. 75 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39. 76 22 of 23 Seo, J.S. An Asymmetric Matching Method for a Robust Binary Audio Fingerprinting. IEEE Signal Process. Lett. 2014, 21, 844–847. Rafii, Z.; Coover, B.; Han, J. An audio fingerprinting system for live version identification using image processing techniques. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, 4–9 May 2014. Naini, R.; Moulin, P. Fingerprint information maximization for content identification. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy, 4–9 May 2014. Yang, G.; Chen, X.; Yang, D. Efficient music identification by utilizing space-saving audio fingerprinting system. In Proceedings of the 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, China, 14–18 July 2014. Yin, C.; Li, W.; Luo, Y.; Tseng, L.-C. Robust online music identification using spectral entropy in the compressed domain. In Proceedings of the Wireless Communications and Networking Conference Workshops (WCNCW), Istanbul, Turkey, 6–9 April 2014. Wang, C.C.; Jang, J.S.R.; Liou, W. Speeding up audio fingerprinting over GPUs. In Proceedings of the 2014 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, China, 7–9 July 2014. Lee, J.Y.; Kim, H.G. Audio fingerprinting to identify TV commercial advertisement in real-noisy environment. In Proceedings of the 2014 14th International Symposium on Communications and Information Technologies (ISCIT), Incheon, South Korea, 24–26 September 2014. Shibuya, T.; Abe, M.; Nishiguchi, M. Audio fingerprinting robust against reverberation and noise based on quantification of sinusoidality. In Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA, 15–19 July 2013. Bisio, I.; Delfino, A.; Lavagetto, F.; Marchese, M. A Television Channel Real-Time Detector using Smartphones. IEEE Trans. Mob. Comput. 2015, 14, 14–27. [CrossRef] Lee, S.; Yook, D.; Chang, S. An efficient audio fingerprint search algorithm for music retrieval. IEEE Trans. Consum. Electron. 2013, 59, 652–656. [CrossRef] Bisio, I.; Delfino, A.; Luzzati, G.; Lavagetto, F.; Marchese, M.; Fra, C.; Valla, M. Opportunistic estimation of television audience through smartphones. In Proceedings of the 2012 International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS), Genoa, Italy, 8–11 July 2012. Anguera, X.; Garzon, A.; Adamek, T. MASK: Robust Local Features for Audio Fingerprinting. In Proceedings of the 2012 IEEE International Conference on Multimedia and Expo, Melbourne, Australia, 9–13 July 2012. Duong, N.Q.K.; Howson, C.; Legallais, Y. Fast second screen TV synchronization combining audio fingerprint technique and generalized cross correlation. In Proceedings of the 2012 IEEE International Conference on Consumer Electronics (ICCE-Berlin), Berlin, Germany, 3–5 September 2012. Wang, H.; Yu, X.; Wan, W.; Swaminathan, R. Robust audio fingerprint extraction algorithm based on 2-D chroma. In Proceedings of the 2012 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, China, 16–18 July 2012. Xiong, W.; Yu, X.; Wan, W.; Swaminathan, R. Audio fingerprinting based on dynamic subband locating and normalized SSC. In Proceedings of the 2012 International Conference on Audio, Language and Image Processing (ICALIP), Shanghai, China, 16–18 July 2012. Jijun, D.; Wan, W.; Yu, X.; Pan, X.; Yang, W. Audio fingerprinting based on harmonic enhancement and spectral subband centroid. In Proceedings of the IET International Communication Conference on Wireless Mobile and Computing (CCWMC 2011), Shanghai, China, 14–16 November 2011. Pan, X.; Yu, X.; Deng, J.; Yang, W.; Wang, H. Audio fingerprinting based on local energy centroid. In Proceedings of the IET International Communication Conference on Wireless Mobile and Computing (CCWMC 2011), Shanghai, China, 14–16 November 2011. Martinez, J.I.; Vitola, J.; Sanabria, A.; Pedraza, C. Fast parallel audio fingerprinting implementation in reconfigurable hardware and GPUs. In Proceedings of the 2011 VII Southern Conference on Programmable Logic (SPL), Cordoba, Argentina, 13–15 April 2011. Cha, G.H. An Effective and Efficient Indexing Scheme for Audio Fingerprinting. In Proceedings of the 2011 5th FTRA International Conference on Multimedia and Ubiquitous Engineering (MUE), Crete, Greece, 28–30 June 2011. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 160 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 23 of 23 Schurmann, D.; Sigg, S. Secure Communication Based on Ambient Audio. IEEE Trans. Mob. Comput. 2013, 12, 358–370. [CrossRef] Son, W.; Cho, H.-T.; Yoon, K.; Lee, S.-P. Sub-fingerprint masking for a robust audio fingerprinting system in a real-noise environment for portable consumer devices. IEEE Trans. Consum. Electron. 2010, 56, 156–160. [CrossRef] Chang, K.K.; Pissis, S.P.; Jang, J.-S.R.; Iliopoulos, C.S. Sub-nyquist audio fingerprinting for music recognition. In Proceedings of the Computer Science and Electronic Engineering Conference (CEEC), Colchester, UK, 8–9 September 2010. Umapathy, K.; Krishnan, S.; Rao, R.K. Audio Signal Feature Extraction and Classification Using Local Discriminant Bases. IEEE Trans. Audio Speech Lang. Proc. 2007, 15, 1236–1246. [CrossRef] Kim, H.G.; Kim, J.Y.; Park, T. Video bookmark based on soundtrack identification and two-stage search for interactive-television. IEEE Trans. Consum. Electron. 2007, 53, 1712–1717. Sert, M.; Baykal, B.; Yazici, A. A Robust and Time-Efficient Fingerprinting Model for Musical Audio. In Proceedings of the 2006 IEEE International Symposium on Consumer Electronics, St Petersburg, Russia, 28 June–1 July 2006. Ramalingam, A.; Krishnan, S. Gaussian Mixture Modeling of Short-Time Fourier Transform Features for Audio Fingerprinting. IEEE Trans. Inf. Forensics Secur. 2006, 1, 457–463. [CrossRef] Ghouti, L.; Bouridane, A. A fingerprinting system for musical content. In Proceedings of the 2006 14th European Signal Processing Conference, Florence, Italy, 4–8 September 2006. Cook, R.; Cremer, M. A Tunable, Efficient, Specialized Multidimensional Range Query Algorithm. In Proceedings of the 2006 IEEE International Symposium on Signal Processing and Information Technology, Vancouver, BC, Canada, 28–30 August 2006. Seo, J.S.; Jin, M.; Lee, S.; Jang, D.; Lee, S.; Yoo, C.D. Audio fingerprinting based on normalized spectral subband centroids. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 05), Philadelphia, PA, USA, 23 March 2005. Haitsma, J.; Kalker, T. Speed-change resistant audio fingerprinting using auto-correlation. In Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 03), Hong Kong, China, 6–10 April 2003. Haitsma, J.; Kalker, T.; Oostveen, J. An efficient database search strategy for audio fingerprinting. In Proceedings of the 2002 IEEE Workshop on Multimedia Signal Processing, St.Thomas, VI, USA, 9–11 December 2002. © 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). 77 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 78 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 3 Framework for the Identification of Activities of Daily Living This chapter is related to the definition of the framework for the recognition of ADL and environments, and it is composed by one article. 1. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices The following article is the second part of the chapter 3. Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta and Susanna Spinsante Sensors (MDPI), published, 2018. According to 2016 Journal Citation Reports published by Thomson Reuters in 2017, this journal has the following performance metrics: ISI Impact Factor (2016): 2.677 ISI Article Influence Score (2016): 0.6 Journal Ranking (2016): 104/642 (Electrical and Electronic Engineering) 79 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 80 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Article Approach for the Development of a Framework for the Identification of Activities of Daily Living Using Sensors in Mobile Devices Ivan Miguel Pires 1,2,3,*, Nuno M. Garcia 1,3,4, Nuno Pombo 1,3,4, Francisco Flórez-Revuelta 5 and Susanna Spinsante 6 Instituto de Telecomunicações, Universidade da Beira Interior, 6201-001 Covilhã, Portugal; ngarcia@di.ubi.pt (N.M.G.); ngpombo@ubi.pt (N.P.) 2 Altranportugal, 1990-096 Lisbon, Portugal 3 ALLab - Assisted Living Computing and Telecommunications Laboratory, Computing Science Department, Universidade da Beira Interior, 6201-001 Covilhã, Portugal 4 ECATI, Universidade Lusófona de Humanidades e Tecnologias, 1749-024 Lisbon, Portugal 5 Department of Computer Technology, Universidad de Alicante, 03690 Sant Vicent del Raspeig, Alicante, Spain; francisco.florez@ua.es 6 Department of Information Engineering, Marche Polytechnic University, 60121 Ancona, Italy; s.spinsante@univpm.it * Correspondence: impires@it.ubi.pt; Tel.: +351-966-379-785 1 Received: 7 January 2018; Accepted: 19 February 2018; Published: 21 February 2018 Abstract: Sensors available on mobile devices allow the automatic identification of Activities of Daily Living (ADL). This paper describes an approach for the creation of a framework for the identification of ADL, taking into account several concepts, including data acquisition, data processing, data fusion, and pattern recognition. These concepts can be mapped onto different modules of the framework. The proposed framework should perform the identification of ADL without Internet connection, performing these tasks locally on the mobile device, taking in account the hardware and software limitations of these devices. The main purpose of this paper is to present a new approach for the creation of a framework for the recognition of ADL, analyzing the allowed sensors available in the mobile devices, and the existing methods available in the literature. Keywords: Activities of Daily Living (ADL); environment; sensors; mobile devices; framework; data acquisition; data processing; data fusion; pattern recognition; machine learning 1. Introduction Sensors embedded in off-the-shelf mobile devices, e.g., accelerometers, gyroscopes, magnetometers, microphones, and Global Positioning System (GPS) receivers [1], may be used in the development of algorithms for the recognition of Activities of Daily Living (ADL) [2] and the environments in which they are carried out. These algorithms are part of the development of a Personal Digital Life Coach (PDLC) [3]. According to [3], a PDLC “(…) will monitor our actions and activities, be able to recognize its user state of mind, and propose measures that not only will allow the user to achieve his/her stated goals, but also to act as an intermediate health and well-being agent between the user and his/her immediate care givers (…)”. This work is related to the development of ambient assisted living (AAL) systems, and, due to the increasing demands in our society, it is a field with high importance [4]. Due to recent advances in technology, there is an increasing number of research studies in this field for the monitoring of people with impairments and older people in a Sensors 2018, 18, 640; doi:10.3390/s18020640 www.mdpi.com/journal/sensors 81 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 2 of 22 plethora of situations by using AAL technologies, including mobile devices and smart environments [5]. Multi-sensor data fusion technologies may be implemented with mobile devices, because they incorporate several sensors, such as motion sensors, magnetic/mechanical sensors, acoustic sensors, and location sensors [6], improving the accuracy of the recognition of several types of activities, e.g., walking, running, going downstairs, going upstairs, watching TV, and standing, and environments, e.g., bar, classroom, gym, library, kitchen, street, hall, living room, and bedroom. The selection of the activities and environments that will be included in the framework was based in the activities previously recognized with best accuracies, and, in the case of the environments, there are a lack of studies related to the environment recognition, taking into account some of the environments previously recognized and the most common environments [7]. The recognition of ADL may be performed with motion, magnetic/mechanical and location sensors, and the environments may be recognized with acoustic sensors. In order to improve the recognition of the ADL, the environment recognized may be fused with the other features extracted from the other sensors. In accordance with previous works [6,8,9], the main motivation of this paper is to present the architecture of a framework for the recognition of ADL and their environments, which takes advantage of the use of a wide set of sensors available in a mobile device, also aiming at reducing the current complexity and constraints in the development of these systems. The test and validation of this framework is currently the subject of another step of this research plan [9], which includes the acquisition of a dataset that contains approximately 2.7 hours of data collected from the accelerometer, gyroscope, magnetometer, microphone and GPS receiver, related to each activity and environment. During the collection phase, the data were acquired with the mobile device located in the front pocket of the trousers by 25 subjects aged between 16 and 60 years old and different lifestyles (10 mainly active and 15 mainly sedentary) and gender (10 female and 15 male). The activities performed and the environments frequented were labelled by the user. The subjects used their personal mobile phones with their applications running, where the mainly used device was a BQ Aquarius device [10]. The identification of ADL and environments using sensors has been studied during the last years, and several methods and frameworks [11–16] have been implemented using smartphones. However, this is a complex problem that should be separated into different stages, such as data acquisition, processing, and fusion; and artificial intelligence systems. The frameworks developed in previous studies are commonly only focused on some specific parts of the problem. For example, the Acquisition Cost-Aware QUery Adaptation (ACQUA) framework [17] has been designed for data acquisition and data processing, but it does not include all the steps needed for data processing. There are no predefined standards for the creation of a framework for the recognition of the ADL [18–20], and the most implemented methods for the recognition of ADL are related to the use of motion sensors. However, there are methods and sensors that can be fused for the creation of a structured framework as a holistic approach to the identification of the ADL and environments presented in this paper. Around the concept of sensors’ data fusion, the selection of the sensors to use is the first step for the creation of the framework, defining a method for the acquisition of the data, and, consequently, their processing. The processing of the data includes data cleaning, data imputation, and extraction of the features. Data segmentation techniques are not considered, as this study was designed for local execution on mobile devices and, due to the low memory and power processing restrictions of these devices, only a short sample of the sensors’ data can be used (initial research points to 5 s samples). This strategy makes it unsuitable to apply data segmentation techniques while still making it possible to deploy the framework in scarce resource devices. The final step in the proposed framework is the selection of the best features, and then the application of artificial intelligence techniques, i.e., the implementation of three types of Artificial Neural Networks (ANN), such as Multilayer Perceptron (MLP) with Backpropagation, Feedforward Neural Networks (FNN) with Backpropagation and Deep Neural Networks (DNN), in order to choose the best method for the accurate recognition of the ADL and the environments. 82 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 3 of 22 The remaining sections of this paper are organized as follows: Section 2 presents the state of the art in this topic, presenting a set of methods for each module/stage. Section 3 presents the framework for the identification of ADL using the sensors available in off-the-shelf mobile devices, the sensors and the methods that may be used. Section 4 presents a discussion and conclusions about the new approach proposed. 2. Related Work Following previous research works related to the identification of ADL and the environment in which they are carried out, this Section reviews the state of the art on the sensors available on mobile devices (Section 2.1), data acquisition (Section 2.2), processing (Section 2.3), fusion (Section 2.4), artificial intelligence techniques (Section 2.5), and, finally, in Section 2.6, the methods to merge sensors’ data with users’ agenda. 2.1. Sensors Sensors are small components that allow the acquisition of data when they are excited responding to stimuli, often external to the device. Available in many mobile devices, namely, in smartphones, sensors can be used to infer an ADL, and the combination of the data from multiple sensors can increase the efficiency of ADL identification, and environment recognition [9]. The number and types of sensors available on mobile devices is different for each mobile platform. In general, the sensors available in mobile devices are magnetic/mechanical sensors, environmental sensors, location sensors, motion sensors, imaging/video sensors, proximity sensors, acoustic sensors, optical sensors, and force sensors, being able to capture different types of signals, such as electrical, mechanical, acoustic and others [1,21]. Based on the classification presented in [6], sensors available on Android devices include microphones, accelerometers, gyroscopes, magnetometers, altimeters, humidity sensors, ambient light sensors, temperature sensors, GPS receivers, touch screens, microphones, and cameras [22,23]. In addition to platform-dependent restrictions in the use of sensors, the hardware differences between devices can influence the availability of specific sensors. Thus, the sensors available in most of the mobile devices, presented in Table 1, are the accelerometer, the gyroscope, the magnetometer, the GPS, the microphone, the touch screen, and the camera. Table 1. List of sensors available in mobile devices. Categories: Motion sensors Magnetic/mechanical sensors Location sensors Acoustic sensors Force sensors Imaging/video sensors Sensors: Accelerometer Gyroscope Magnetometer GPS Microphone Touch screen Camera Availability Always present Present in most models Present in most models Always present Always present Always present Always present 2.2. Data Acquisition Data acquisition consists in the process of receiving the different types of data from the sensors available in the mobile devices. There are some possible problems that occur during the data acquisition process, including the influence of the unpredictable and uncontrolled external environment, the variability of the sampling rate of sensors, the number of tasks performed by the mobile device during the data acquisition, and the variability of the sensors chosen as input for a given developed framework [24]. Related to the variability of the position of the smartphone when carried by a user, to the best of the authors’ knowledge, there are no studies that solve this issue. As a standard method was not previously defined for the correct data acquisition and processing, and the sensors and capabilities of the mobile devices are different between manufacturers, the authors assumed that the results are nonetheless comparable. 83 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 4 of 22 In order to improve the data acquisition process, several frameworks have been developed, including Acquisition Cost-Aware QUery Adaptation (ACQUA) framework [17], Orchestrator framework [25], ErdOS framework [26], LittleRock prototype [27], Jigsaw continuous sensing engine [28], SociableSense framework [29], Continuous Hand Gestures (CHG) technique [30], and Barbie-Q (BBQ) approach [31]. The ACQUA framework allows to control the order of data acquisition, the correct segments of the data requested, the calibration of the data acquisition rates, the packet sizes and radio characteristics, the adaptation of the dynamic changes in query selective properties, and the support of multiple queries and heterogeneous time window semantics from all the sensors available in mobile devices, reducing the energy consumption of the real-time data acquisition [17]. The Orchestrator framework promotes the distributed execution of data acquisition using several mobile devices, and all devices execute a part of the data processing, avoiding to reduce the requirements related to the processing power and energy consumption [25]. The same purpose of Orchestrator framework is achieved from ErdOS framework and LittleRock prototype, distributing the data acquisition and processing processes by all resources available in the devices used, and reducing the energy needed to process the data collected from all sensors [26,27]. The Jigsaw continuous sensing engine implements a method to control the different sample rates, adapting the data acquisition and processing for the different capabilities of the sensors [28]. The SociableSense framework has a mechanism to adapt the different sample rates of all sensors used and it is a cloud-based framework, reducing the local data processing, but restricting the use of the framework to the availability of the Internet connection [29]. The authors of [30] implemented a CHG technique for the data acquisition with Windows Phone-based smartphones and low processing capabilities, capturing accelerometer and gyroscope data, storing the sensory data in the smartphone memory. The BBQ framework applies a multi-dimensional Gaussian probability density function from all the sensors, inferring the order of the data acquisition with conditional probabilities [31]. The data acquisition process implemented in mobile devices may be performed without the use of frameworks, improving only the data processing according to the different resource capabilities. The authors of [32–35] implement the data acquisition process from accelerometer data in Apple iPhone and Android-based smartphones for the identification of several activities, including driving, walking, sitting, standing, running, and jumping activities. The authors of [36] implemented a Cursor Movement Algorithm to detect several activities, capturing the real-time data from the accelerometer and storing them into a local database in the mobile device. Table 2 presents a summary of the data acquisition methods and their main characteristics for further implementation in the proposed new approach. Table 2. Summary of the data acquisition methods. Methods: ACQUA framework [17] Orchestrator framework [25] ErdOS framework [26] 84 Advantages: Controls of the order of the data acquisition; Controls the correct segments of the data requested; Controls the calibration of the data acquisition rates; Controls the packet sizes and radio characteristics; Controls the adaptation of the dynamic changes in query selective properties; Controls the support of multiple queries and heterogeneous time window semantics; Adapted for low processing, memory, and energy capabilities. Distributed execution of the data acquisition using several mobile devices; Adapted for low processing, memory, and energy capabilities. Distributed execution of the data acquisition using several mobile devices; Adapted for low processing, memory, and energy capabilities. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 LittleRock prototype [27] Jigsaw continuous sensing engine [28] SociableSense framework [29] CHG technique [30] BBQ framework [31] Cursor movement algorithm [36] No framework 5 of 22 Adapted for low processing, memory, and energy capabilities. Controls the different sample rates; Adapted for low processing, memory, and energy capabilities. Cloud-based framework; Needs a constant Internet connection; Adapted for low processing, memory, and energy capabilities. Stores the sensory data in the smartphone memory; Adapted for low processing, and energy capabilities. Uses a multi-dimensional Gaussian probability density function from all sensors; Adapted for low processing, memory, and energy capabilities. Stores the sensory data in the smartphone memory; Adapted for low processing, and energy capabilities. Adapted for low processing, memory, and energy capabilities. 2.3. Data Processing After the data acquisition process, the sensors’ data should be processed in order to prepare the data for the fusion from the chosen set of sensors, and, consequently, the application of the methods for ADL recognition. First, data processing should validate the integrity and quality of the data, and, then, applying data cleaning and/or data imputation techniques [37], in order to make this data available for the next stage in the processing pipeline of the framework. However, data processing depends on the environmental conditions, the types of sensors and data, the events of sensor failures, and the capabilities of the mobile devices [38]. Several techniques have been developed to reduce the memory and energy consumption of the data processing techniques. Other issues related to sensor drifting and generic noise are not specifically addressed in this paper, despite recognizing that sensors’ calibration and drift compensation may improve the outcomes of automatic recognition algorithms. Nevertheless, the application of data cleaning techniques mentioned in Section 2.3.1, and data imputation techniques mentioned in Section 2.3.2 may reduce the impact of drift and noise. Additionally, both the limited acquisition time used in the proposed framework and the fusion of data from different sensors, as discussed in [39], help in reducing the aforementioned effects. For each sensor data capture, we show that the use of only 5 s of sensors’ data is sufficient for the recognition of ADL and the environment. As a consequence the risk of failure in data acquisition or data corruption over such a short time may be assumed negligible. The ACQUA framework is also used to optimize the data processing, by automated storage and retrieval system (ASRS) algorithms [17]. Other studies have presented approaches to adapt the data processing methods to the low capabilities of the mobile devices, processing the data after splitting or using methods with limited resources needed [24,40–42]. The use of data cleaning methods, presented in Section 2.3.1, is important to decrease the influence of the environmental conditions noise or systems failures. In order to improve the results, when the data acquisition fails, Section 2.3.2 presents the possible data imputation methods to correct the data acquired. However, these methods are not addressed by the proposed framework for the identification of ADL and their environments, assuming that the data acquired is sufficient for the extraction of several features from the, presenting the feature extraction methods and possible features to extract, in Section 2.3.3. 2.3.1. Data Cleaning Data cleaning consists in the identification of the incorrect values, removing outlier values and smoothing and filtering the invalid values obtained during the data acquisition process, commonly considered as noisy values [43–45]. Using data cleaning methods, the influence of the environmental conditions, the mobile device position, and system failures occurred during the data acquisition process is reduced. The efficiency of these methods depends on the type of data acquired and spatiotemporal characteristics of the data acquired. 85 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 6 of 22 The authors of [46] proposed a weighted moving average (WMA) algorithm that collects the sensors’ data and computes the weighted moving average, applying the WMA filter for the normalization and cleaning of the sensors’ data. Three types of filters are used for the motion and magnetic/mechanical sensors: the low-pass filter (LPF), the high pass filter (HPF), and the KALMAN filter [47,48]. The WMA filter and the different types of Fourier transforms, such as Discrete Fourier Transform (DFT), Inverse Discrete Fourier Transform (IDFT), and Fast Fourier Transform (FFT) are also used to filter the acoustic data [49,50]. Table 3 presents a summary of the data cleaning methods related to the different types of sensors, discussed in section 2.1. Concerning the implementation in the development of a framework for the identification of ADL and their environments, it can be seen that the LPF is commonly used in motion and magnetic sensors, the most used technique for acoustic sensors is the FFT and that the filtering techniques are not important for location, force and imaging sensors because of the nature of the values these sensors return. Table 3. Relation between the types of sensors and the data cleaning techniques allowed. Types of Sensors: Motion sensors; Magnetic/mechanical sensors. Location sensors Acoustic sensors Force sensors Imaging/video sensors Data Cleaning Techniques: Low-Pass Filter; High-Pass Filter; KALMAN Filter; Weighted moving average (WMA) algorithm; Moving average filter. The data cleaning technique is not important for this type of data acquired. Moving average filter; Discrete Fourier Transform (DFT); Inverse Discrete Fourier Transform (IDFT); Fast Fourier Transform (FFT). The data cleaning technique is not important for this type of data acquired. 2.3.2. Data Imputation During the data processing, the verification of the existence of faulty data is performed to flag that some values are missing in some instants of the acquired data time series. The data imputation methods are mainly used for motion sensors and magnetic/mechanical sensors. However, for the development of the new approach of the framework for the identification of ADL and their environments, the data imputation techniques were not considered, assuming that data acquired by the sensors is complete. Thus, in this section, the best methods for data imputation will be presented based on a literature review. Faulty data may have different types that can be classified as Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR) [51]. When the faulty data is randomly distributed during the time interval for the data acquisition, the classification of this data is MCAR. The other types of faulty data are MAR, verified when the faulty data is randomly distributed by different subsets of the data acquired, and MNAR, and verified when the faulty data is distributed by defined instants of the data acquisition. The K-Nearest Neighbor (k-NN) method is one of the most used methods for data imputation of data acquired from motion, and magnetic/mechanical sensors [52–55]. The k-NN method has several variants that can be used for data imputation, such as MKNNimpute (K-nearest neighbor imputation method based on Mahalanobis distance), SKNNimpute (sequential K-nearest neighbor method-based imputation), and KNNimpute (K-nearest neighbor imputation) [52,53]. The clustering techniques are also used for the data imputation, including K-means clustering, K-means-based imputation, and fuzzy C-means clustering imputation [51,56,57], which are implement in the Imputation Tree (ITree) method presented in [51]. There are other methods related to data imputation, including multiple imputation [58], hot/cold imputation [59], maximum likelihood [60], Bayesian estimation [60], expectation maximization [54,61,62], discarding instances [18], pairwise deletion [18], unconditional mean imputation [18], conditional mean imputation [18], hot deck imputation [18], cold deck imputation [18], substitution method [18], linear regression [18], logistic regression [18], expectation-maximization (EM) algorithm 86 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 7 of 22 [18], probabilistic neural networks [18], fuzzy min–max neural networks [18], general regression auto associative neural network [18], tree-based methods [18], multi-matrices factorization model (MMF) [63], mean imputation (MEI) [54,62], Multivariate Imputation by Chained Equations (MICE) [54,62], Fourier method [62], and Fourier and lagged k-NN combined system (FLk-NN) [54,62,64]. In general, these methods can be applied to data collection from motion and magnetic/mechanical sensors. Data imputation methods can also be applied to the acoustic data, being the more common the k-NN methods and singular value decomposition (SVD) algorithms [65]. As the data imputation methods should be able to impute the empty instances of the data acquired by motion and magnetic/mechanical sensors, the methods that are able to be used with this purpose are MEI, EM, MICE, and FLk-NN [54]. However, k-NN can be applied with the comparison between the history of the data acquisition that is similar to the data acquired in the stream with faulty values [54]. It emerges from the reviewed literature that data imputation may be avoided for acoustic and location sensors, because of the slow variability of their signals. 2.3.3. Feature Extraction The correct definition of the features extracted from the sensors’ data increases the accuracy of the identification of ADL and their environments. This definition depends on the types of sensors and the data acquired, but also on the purpose of their final use. For the correct extraction of the features for the motion and magnetic/mechanical sensors’ data, the Euclidean norm for each instant of outputs from the sensors defined as magnitude of vector (MV). Thus, the features that should be extracted from the motion and magnetic/mechanical sensors are the mean for each axis [66–69], variance of MV [70,71], mean of MV [67,70–75], median of MV [70,74], maximum of MV [66,70,71,73], minimum of MV [66,70,71,73], standard deviation of MV [66,67, 70–75], Root Mean Square (RMS) of MV [66,70], average of peak frequency (APF) of each axis [66], maximum of each axis [66,69,74], minimum of each axis [66,69,74], standard deviation of each axis [66,68,69], RMS of each axis [66], cross-axis signals correlation [66,67,69,73,76], Fast Fourier Transform (FFT) spectral energy [70,76], frequency domain entropy [76], FFT coefficients [70,73], logarithm of FFT [76], skewness of each axis [67], kurtosis of each axis [67], average absolute deviation of each axis [67], time between peaks [72], Interquartile range of MV [71,73], skewness of MV [71], kurtosis of MV [71], wavelet energy of MV [73], average of peak values [77], average of peak rising time [77], average of peak fall time [77], average time per sample [77], average time between peaks [77], slope for each axis [74], binned distribution for each axis [68], percentiles of MV [75], and zero crossing rate for each axis [69]. Related to the motion and magnetic/mechanical sensors’ data, the most used features are mean, standard deviation, maximum, minimum, median, correlation, variance, and FFT spectral energy of MV. For the correct extraction of the features for the acoustic sensors’ data, the features that should be extracted are average [78], thresholding [78], minimum [78], maximum [78], distance [78], and MFCC (Mel-frequency cepstrum coefficients) [79,80]. For the location sensors, the feature that should be extracted is the distance travelled between a time interval, in order to identify ADL with high distance travelled. The distance between two points captured by a GPS receiver is the ellipsoidal distance, because the two points are acquired in the geodetic coordinate system, where the calculation of this distance is measured with the Vincenty formula [81–83]. Table 4 presents a summary of the features extracted for each type of sensors presented in the Section 2.1, for further implementation the in new approach for the development of a framework for the identification of ADL and their environments. 87 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 8 of 22 Table 4. Relation between sensors and extracted features. Types of Sensors: Motion sensors; Magnetic/mechanical sensors. Location sensors Acoustic sensors Force sensors; Imaging/video sensors. Features: Mean [67,70–75], average of peak frequency (APF) [66], maximum [66,70,71,73], minimum [66,70,71,73], standard deviation [66,67, 70–75], Root Mean Square (RMS) [66,70], cross-axis signals correlation [66,67,69,73,76], skewness [67], kurtosis [67], average absolute deviation [67], slope [74], binned distribution [68], and zero crossing rate for each axis [69]; Mean [67,70–75], median [70,74], variance [70,71], maximum [66,70,71,73], minimum [66,70,71,73], standard deviation [66,67,70–75], Root Mean Square (RMS) [66,70], Fast Fourier Transform (FFT) spectral energy [70,76], frequency domain entropy [76], FFT coefficients [70,73], logarithm of FFT [76], Interquartile range [71,73], skewness [67], kurtosis [67], wavelet energy [73], and percentiles of MV [75]; Time between peaks [72], average of peak values [77], average of peak rising time [77], average of peak fall time [77], average time between peaks [77]. Distance between two points. Average [78], Thresholding [78], Minimum [78], Maximum [78], Distance [78], MFCC (Mel-frequency cepstrum coefficients) [79,80]. These sensors are not useful for the development of the framework for the Identification of ADL and their environments. 2.4. Data Fusion After the extraction of the features, the data acquired from all sensors should be fused to improve the accuracy of the ADL identification and their environments in the new approach for the framework proposed in this study [11]. The data fusion methods implemented should be related with the final purpose of the framework presented in Section 2.6. Based on the literature studies presented by several authors [12,20,84,85], the data fusion methods are grouped in four categories [12,84,85]. These are: probabilistic methods, statistical methods, knowledge base theory methods and evidence reasoning methods. The probabilistic methods [12,20,84,85] include Bayesian analysis methods, maximum likelihood methods, state-space models, evidential reasoning, possibility theory, Kalman Filter [86,87], Particle filtering, k-Nearest Neighbor (k-NN), k-Means, optimal theory, uncertainty ellipsoids, Gaussian mixture model (GMM), weighted averages, and regularization. The statistical methods [12,84,85] for data fusion include covariance intersection, crosscovariance, and other robust statistics. However, other statistical methods used for data fusion are dynamic time warping (DTW) [88], which measures the similarity between two temporal sequences, based on the raw data or the features extracted. The knowledge base theory methods [12,20,84,85,89] for data fusion include Artificial Neural Networks (ANN), Support Vector Machines (SVM), Decision Trees, Deep Learning, Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN), Fuzzy Logic, Topic models, and Genetics Algorithms. The evidence reasoning methods [12,84,85] for data fusion include evidence theory, Bayesian network, Dempster-Shafer, and recursive operators. Based on these categories of data fusion methods, several implementations have been performed and presented in several studies for the identification of a plethora of a real-life activities and environments. The Rao-Blackwellization unscented Kalman filter (RBUKF) [90] was implemented to fuse the data acquired from a compass, a gyroscope, and a GPS receiver. The Kalman filter was used to fuse the data acquired from the GPS receiver and the gyroscope in order to support a navigation system [91]. The Naïve Bayes classifier is used to fuse the data acquired from acoustic, accelerometer and GPS sensors to recognize different situations during daily life [92]. The AutoregressiveCorrelated Gaussian Model was implemented in the KNOWME system [93]. Bayesian analysis and Kalman filter where used to data acquired from the several sensors available in mobile devices for the identification of the ADL [94]. The CHRONIOUS system implements several methods to 88 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 9 of 22 recognize several ADL, such as Support Vector Machine (SVM), random forests, Artificial Neural Networks (ANN), decision trees, decision tables, and Naïve Bayes classifier, in order to fuse the data collection from several sensors available in mobile devices [95]. In [96], the authors used the empirical mode decomposition (EMD) applied to the inertial sensors available in a mobile device, including accelerometer, gyroscope, and magnetometer, for the identification of several ADL. The authors of [97] implements several methods for data fusion, including SVM, random forest, hidden Markov models (HMMs), conditional random fields (CRFs), Fisher kernel learning (FKL), and ANN for several sensors, such as Accelerometer, RFID, and Vital monitoring sensors for the correct identification of ADL. Table 5 presents a summary of the data fusion methods that can be applied for each type of sensors presented in Section 2.1, for further implementation in a new approach for the development of a framework for the identification of ADL and their environments. Table 5. Relation between the different types of sensors and some data fusion methods. Types of sensors: Motion sensors; Magnetic/mechanical sensors; Location sensors; Acoustic sensors. Force sensors; Imaging/video sensors. Data fusion methods: Autoregressive-Correlated Gaussian Model; Fuzzy Logic; Dempster-Shafer; Evidence Theory; Recursive Operators; Support Vector Machine (SVM); Random Forests; Artificial Neural Networks (ANN); Decision Trees; Naïve Bayes classifier; Bayesian analysis; Kalman Filter; k-Nearest Neighbor (k-NN); Least squares-based estimation methods; Optimal Theory; Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); Uncertainty Ellipsoids. These sensors are not useful for the development of the framework for the Identification of ADL and their environments. 2.5. Identification of Activities of Daily Living The definition of the methods for ADL identification represents the final module of the new proposed framework, presented in Figure 1. The identification of the ADL and their environments depends on the sensors’ data used, therefore, if a method uses the data acquired from motion and/or magnetic/mechanical sensors, it will probably be used to identify the ADL. If a method uses the data acquired from acoustic sensors, it will probably be used to identify the external environments. Finally, if the implemented method uses the location sensors, it is probably identifying activities with fast movement, e.g., driving, or it is probably trying to identify the place where the ADL is performed. In general, the identification of ADL is performed at the same time of the data fusion, because the methods use the same techniques. The machine learning is a set of several techniques for artificial intelligence, including the techniques for the identification of ADL and their environments. The concept of machine learning will be presented in the Section 2.5.1. In Section 2.5.2, the pattern recognition methods are presented, which consists in a subset of the machine learning techniques. 89 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 10 of 22 Figure 1. Schema for the framework for the recognition of Activities of Daily Living (ADL). 2.5.1. Machine Learning Artificial Intelligence (AI) is one of the main areas for the development of computer science systems, and machine learning is composed by a subset of AI methods, where the computers have the ability to learn and perform some tasks, taking into account the external conditions of the system in order to change the execution of some methods for obtaining of better results [98]. Machine learning methods are based on the creation and implementation of algorithms for the recognition and prediction of several situations based on the data acquired, and these methods are commonly classified in four categories [99,100], such as Supervised learning, Unsupervised learning, Reinforcement learning, and Semi-supervised Learning and Active Learning. Supervised learning methods are based on the automatic adjustment of the network parameters, comparing the actual network output with the desired output previously defined in the data acquired, where the error obtained is the mean squared error (MSE) [100]. The input data involved in the supervised leaning should be labeled, in order to perform the comparisons. Unsupervised learning methods consist on the correction of the results obtained based on the input data, attempting to obtain the signification patterns or features in the unlabeled input data, automatically learning with intuitive primitives like neural competition and cooperation [100]. Reinforcement learning methods are similar to supervised learning methods, but the exact desired output is a priori unknown [100]. Thus, these methods are learning based on the feedback provided during the execution of the algorithm by an artificial agent in order to maximize the total expected reward [100]. Semi-supervised Learning and Active Learning methods are methods that should be applied to datasets with a large collection of unlabeled input data and a few labeled examples to generalize the results and performance of the method, based on assumptions related to the probability of occurrence of some output. For the development of a new approach for the development of a framework for the identification of ADL and their environments, the machine learning may be used, as it can be adapted to bioinformatics and human-related systems [101–104]. Pattern recognition methods, described in Section 2.5.2, consist on a subset of machine learning methods for the recognition of patterns [105], which are very useful in the development of the framework for the identification of ADL and their environments. 2.5.2. Pattern Recognition The use of pattern recognition methods is the final part of research for the creation of a new approach for a framework for the identification of ADL and their environments. Several sensors, presented in Section 2.1, may be used with pattern recognition methods, which should be applied to the features extracted from the input data. 90 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 11 of 22 The methods implemented during the pattern recognition step are similar to the methods implemented for the data fusion, presented in Section 2.4. As reported early in this paper, the data fusion and pattern recognition may be confused, and the pattern recognition is performed at the same time of the data fusion. The categorization of the methods is similar to the methods applied for data fusion, and they are separated in four categories [12,84,85], these are the probabilistic methods, the statistical methods, the knowledge base theory methods and the evidence reasoning methods. Several ADL may be recognized with pattern recognition methods, as example for the recognition of standing, and walking activities may be used ANN [106]. Several authors [13–16, 66–69,71–76,89,107–120] proposed the use of the ANN, probabilistic neural networks (PNN), deep neural networks (DNN), Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN), SVM, Random Forest, Bayesian Network, Sequential Minimal Optimization (SMO), Logistic Regression, Naïve Bayes, C4.5 Decision Tree, Logistic Model Trees (LMT), J48 Decision tree, K-Nearest Neighbor (KNN), and Simple Logistic Logit Boost methods for the recognition of walking, running, jogging, jumping, dancing, driving, cycling, sitting, standing, lying, walking on stairs, going up on an escalator, laying down, walking on a ramp activities, cleaning, cooking, medication, sweeping, washing hands, and watering plants. The Hidden Markov Model (HMM) and their variants are also a pattern recognition implemented in several studies related with the identification of ADL and their environments, such as the Hidden Markov Model (HMM) [71], the Hidden Markov Model Ensemble (HMME) [121], the Sliding-Window-based Hidden Markov Model (SW-HMM) [113]. The ADLs commonly identified by the HMM method are walking, walking on stairs, standing, running, sitting, and laying. Table 6 presents a summary of the pattern recognition methods that can be applied for each type of sensors presented in Section 2.1, for further implementation in the proposed approach for the identification of ADL and their environments. As shown in the Table, the HMM method is commonly used for the recognition of walking, walking on stairs, standing, running, sitting and laying activities, whereas the SVM, ANN and their variants, HMM and Random Forest methods, are useful for the recognition of complex activities (e.g., cleaning, cooking, medication, sweeping, washing hands and watering plants). However, all of the described methods in this study may be used for the recognition of simple activities (e.g., walking, running, jogging, jumping, dancing, driving, cycling, sitting, standing, lying, walking on stairs, going up on an escalator, laying down and walking on a ramp) with reliable accuracy. Table 6. Relation between the different types of sensors and some pattern recognition methods. Types of sensors: Motion sensors; Magnetic/mechanical sensors; Location sensors; Acoustic sensors. Pattern recognition methods: Support Vector Machines (SVM); Decision trees (J48, C4.5); Artificial Neural Networks (ANN); Probabilistic Neural Networks (PNN); Deep Neural Networks (DNN); Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); k-Nearest Neighbour (KNN); Naïve Bayes; Random Forest; Logistic Regression; Bayesian network; Sequential minimal optimization (SMO); ADL recognized: Walking; running; jogging; jumping; dancing; driving, cycling; sitting; standing; lying; walking on stairs; going up on an escalator; laying down; walking on a ramp. 91 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 Force sensors; Imaging/video sensors. 12 of 22 Logistic Model Trees (LMT); Simple Logistic Logit Boost. Support Vector Machines (SVM); Artificial Neural Networks (ANN); Probabilistic Neural Networks (PNN); Deep Neural Networks Cleaning; cooking; medication; sweeping; washing (DNN); hands; watering plants. Long Short Term Memory (LSTM) Recurrent Neural Networks (RNN); Hidden Markov model (HMM); Random Forest. Hidden Markov model Walking; walking on stairs; standing; running; sitting; (HMM). laying. These sensors are not useful for the development of the framework for the Identification of ADL and their environments. 2.6. Relation between the Identification of Activities of Daily Living and User Agenda After the identification of the ADL and their environments with machine learning methods, the results obtained should be compared with the users’ agenda for the validation of the scheduled activities performed during the daily life. By comparing the identified ADL with the user’s agenda, it will be possible to monitor the lifestyle [122] and provide feedback regarding planned activities and executed activities. However, the inputs from agenda can also be used to validate the accuracy of the framework developed [123]. 3. Methods and Expected Results The new approach proposed for the creation of the framework for the identification of ADL (Figure 1) is based on [6,8,9], and it is composed by several stages. They are: the selection of the sensors, the data and processing, including data cleaning, imputation, and feature extraction, data fusion, the identification of ADL with artificial intelligence, including pattern recognition, and other machine learning techniques, and, at the end, the combination of the results obtained with the data available in the users’ agenda. In order to create a new approach for a framework for the identification of ADL and their environments, the architecture, presented in Figure 1, and set of methods presented in Section 2 are proposed for obtaining results with reliable accuracy. Following the list of sensors available in off-the-shelf mobile devices, presented in Section 2.1, the sensors that will be used in the framework should be dynamically selected, according to the sensors available in the mobile device. Thus, the types of sensors selected to use in the framework will be motion sensors, magnetic/mechanical sensors, acoustic sensors, and location sensors. The accelerometer is available in all mobile devices, but the gyroscope is only available on some devices, therefore, to cover the execution of the framework in all devices, two different methods should be implemented, one considering the data from the accelerometer and the gyroscope, and another considering only the data from the accelerometer. The magnetometer is only available on some devices, therefore this sensor should be managed similarly. Related to the acoustic sensors, the microphone is available in all mobile devices. As to the location sensors, the GPS is available in most of the mobile devices and its data should be used in the framework whenever possible. The data acquisition methods are not directly related to the development of the framework, because the different manufacturers of the mobile operating systems have different methodologies to acquire the different types of sensors’ data. Thus, the data acquisition methods, presented in Section 2.2, should take in account the limitations of the mobile devices. Based on previous research 92 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 13 of 22 studies and preliminary experiments, acquiring only 5 seconds of data from the selected sensors every 5 min is sufficient for the identification of the ADL and environments. Following the creation of the new approach for a framework for the identification of ADL and their environments, the selection of data processing methods, presented in Section 2.3, should contain the data cleaning, data imputation, and feature extraction methods. The data cleaning methods adapted for the framework depends on the types of sensors. On the one hand, for the accelerometer, gyroscope, and magnetometer sensors, the data cleaning method that should be applied is a low pass filter to remove the noise and the value of the gravity acquired during the data acquisition process. On the other hand, for the acoustic sensors, the data cleaning method that should be applied is the FFT in order to extract the frequencies of the audio. As the location sensors return values that are in nature already a result (e.g., GPS coordinates), data cleaning methods are not significant. Nevertheless, and as future work, it may be necessary to implement algorithms that increase the accuracy of these sensors as to better contribute to a quality data fusion process. The data imputation methods is not important to implement in the development of a new approach for a framework for the identification of ADL and their environments, assuming that the data acquired from all sensors is always filled. Related to the feature extraction, the features needed to recognize the ADL and their environments should be selected based on the type of sensors and on the selected features already reported in the literature and presented in Section 2.3.3. Firstly, the features selected for the accelerometer, gyroscope, and magnetometer sensors are the five greater distances between the maximum peaks, the average of the maximum peaks, the standard deviation of the maximum peaks, the variance of the maximum peaks, the median of the maximum peaks, the standard deviation of the raw signal, the average of the raw signal, the maximum value of the raw signal, the minimum value of the raw signal, the variance of the of the raw signal, and the median of the raw signal. Secondly, the features selected for the microphone are the standard deviation of the raw signal, the average of the raw signal, the maximum value of the raw signal, the minimum value of the raw signal, the variance of the of the raw signal, the median of the raw signal, and 26 MFCC coefficients. Finally, the features selected for the GPS receiver are the distance travelled during the acquisition time. Before the presentation of the data fusion and pattern recognition methods that should be used for in the framework, the ADL and environments to recognize should be defined. This process should be executed with several sensors, that will be combined as presented in the Figure 2 and Table 7, being these the necessary stages: 1. 2. 3. Firstly, the ADL are recognized with motion and magnetic/mechanical sensors; Secondly, the identification of the environments is performed with acoustic sensors; Finally, there are two options, being these: o o The identification of standing activities with the fusion of the data acquired from motion and magnetic/mechanical sensors, and the environment recognized, where the number of ADL recognized depends on the number of sensors available; The identification of standing activities with the fusion of the data acquired from motion, magnetic/mechanical and location sensors, and the environment recognized, where the number of ADL recognized depends on the number of sensors available. In identifying the environments, what is intended is to identify the associated activity, i.e., the sound generated in a classroom is not only the sound of the room itself, but rather the sound of a class who is having a lesson in a classroom. This is to say that an environment is to be considered as a place where some activity occurs in a given time of the day or the week, so there will be the need to consider different types of “Street” environments as they will have different audio signatures at different times of the day or week and of course, in different streets. All the proposed environments shown in Figure 2 are expected to be plural. 93 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 14 of 22 Figure 2. Sensors used for the recognition of Activities of Daily Living (ADL) and environments for each phase of development. Environments Activities Table 7. Sensors, Activities of Daily Living (ADL), and environments for recognition with the framework proposed. Going Downstairs Going Upstairs Running Walking Standing Sleeping Driving Bar Classroom Gym Library Kitchen Street Hall Watching tv Bedroom Accelerometer ✓ ✓ ✓ ✓ ✓ ✓ ✓ Gyroscope ✓ ✓ ✓ ✓ ✓ ✓ ✓ Magnetometer ✓ ✓ ✓ ✓ ✓ ✓ ✓ Microphone GPS ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Firstly, the ADL to be identified with the framework will be going downstairs, going upstairs, running, walking, and standing, because they are part of the most recognized ADL in previous studies with reliable accuracy [7]. Secondly, the proposed environments to identify with the framework will be bar, classroom, gym, kitchen, library, street, hall, watching TV, and bedroom, because the existence of previous studies related to the recognition of environments is very limited, the proposed framework will take in account the most common environments and some of the environments previously recognized [7]. Thirdly, the proposed ADL to distinct with the framework 94 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 15 of 22 will be sleeping, and standing, because the ADL may be confused as standing ADL and the inclusion of the environment recognized as an input for the classification method will help in the accurate recognition of these ADL. Finally, the proposed ADL to distinct with the framework are sleeping, standing, and driving, because the driving may also confused as standing ADL and, in order to accurately distinct these ADL, the environment recognized and the features extracted from the GPS receiver should be included. As the data for the creation of the methods for the recognition of ADL and environments was acquired will several conditions and different people, the generated method with ANN will be generic and the calibration of sensor is not needed. Based on the list of data fusion methods and pattern recognition methods, defined in Sections 2.4 and 2.5, the method selected for the implementation in the new approach for a framework for the identification of ADL and their environments will be based in ANN methods, because, based on the literature, it is one of the methods that reports the best accuracies. However, the selection of the best type of ANN will be done with the comparison of the results obtained with three types of ANN selected. The types of ANN that will be tested to the acquired data are: • • • MLP with Backpropagation; FNN with Backpropagation; DNN. Regarding the data acquired from GPS receiver, it can be useful to increase the accuracy of the identification of the ADL and their environments, but it can also be used for the identification of the location where the ADL are executed, in order to improve the comparison with the users’ agenda presented in Section 2.6. 4. Discussion and Conclusions This paper presents the architecture of a new approach for a framework for the identification of ADL and their environments, using methods with a reported good accuracy. The development of the new approach for the development of a framework for the identification of ADL and their environments, based on the system presented in [6,8,9], is one of the steps for the creation of a personal digital life coach [3] using mobile devices. The framework will be composed by several modules several, such as data acquisition, data processing, data fusion, and a module to implement artificial intelligence techniques for the identification of the ADL and their environments. The sensors used in the framework will be accelerometer, gyroscope, magnetometer, microphone, and GPS receiver, in order to recognize several ADL, including going downstairs, going upstairs, running, walking, standing, sleeping, and driving, and their environments, including bar, classroom, gym, kitchen, library, street, hall, watching TV, and bedroom. The sensors’ data should be acquired and, before the extraction of the features of the sensors’ data, filters such as low pass filter and FFT, should be applied. Afterwards, the data fusion and pattern recognition methods should be applied for the recognition of ADL and environments. This paper consists on a conceptual definition of the framework for the recognition of the ADL and their environments, proposing three possible methods for this purpose, based on the use of the ANN methods. In order to define the best method, the future implementation of the proposed methods will compare the differences between them, including the accuracy, performance, and adaptability for the development of a local processing framework for mobile devices. It will include the acquisition of a large set of sensors’ data related to the ADL and environments proposed for the creation of training and testing sets and further validation of the developed methods. Additionally, and also as future work, the framework will allow each user to validate the ADL identified by the framework when this is not the real performed activity. Due to the inexistence of previous studies that review the use of all sensors available in current off-the-shelf mobile devices, our proposed framework is a function of the number of sensors available in the mobile device used, proving a reliable feedback in almost real-time. 95 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 16 of 22 Acknowledgments: This work was supported by FCT project UID/EEA/50008/2013. The authors would also like to acknowledge the contribution of the COST Action IC1303–AAPELE–Architectures, Algorithms and Protocols for Enhanced Living Environments. Author Contributions: All the authors have contributed with the structure, content, and writing of the paper. Conflicts of Interest: The authors declare no conflict of interest. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 96 Salazar, L.H.A.; Lacerda, T.; Nunes, J.V.; von Wangenheim, C.G. A Systematic Literature Review on Usability Heuristics for Mobile Phones. Int. J. Mob. Hum. Comput. Interact. 2013, 5, 50–61. Foti, D.; Koketsu, J.S. Activities of daily living. In Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction; Elsevier Health Sciences: Amsterdam, Netherlands, 2013; Vol. 7, pp. 157–232. Garcia, N.M. A Roadmap to the Design of a Personal Digital Life Coach. In Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2016; Vol. 399. Kleinberger, T.; Becker, M.; Ras, E.; Holzinger, A.; Müller, P. Ambient intelligence in assisted living: Enable elderly people to handle future interfaces. In Lecture Notes in Computer Science; Springer: Berlin, Germany, 2007; Vol. 4555. Singh, D.; Kropf, J.; Hanke, S.; Holzinger, A. Ambient Assisted Living Technologies from the Perspectives of Older People and Professionals. In Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2017; Vol.10410. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F. From Data Acquisition to Data Fusion: A Comprehensive Review and a Roadmap for the Identification of Activities of Daily Living Using Mobile Devices. Sensors 2016, 16, 184. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F.; Zdravevski, E.; Spinsante, S. Machine Learning Algorithms for the Identification of Activities of Daily Living Using Mobile Devices: A Comprehensive Review. engrXiv, engrxiv.org/k6rxa, 2018. (In Review) Pires, I.M.; Garcia, N.M.; Flórez-Revuelta, F. Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices. In Proceedings of the ECMLPKDD 2015 ECML PKDD— European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, 7–11 September 2015. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F. Identification of Activities of Daily Living Using Sensors Available in off-the-shelf Mobile Devices: Research and Hypothesis. In Advances in Intelligent Systems and Computing; Springer: Cham, Switzerland, 2016; Vol. 476. Smartphones: BQ Aquaris and BQ Portugal. Available online: https://www.bq.com/pt/smartphones (accessed on 2 September 2017). Banos, O.; Damas, M.; Pomares, H.; Rojas, I. On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 2012, 12, 8039–8054. Akhoundi, M.A.A.; Valavi, E.; Multi-Sensor Fuzzy Data Fusion Using Sensors with Different Characteristics. arXiv preprint, arXiv:1010.6096, 2010. Paul, P.; George, T. An Effective Approach for Human Activity Recognition on Smartphone. In Proceedings of the 2015 IEEE International Conference on Engineering and Technology (ICETECH), Coimbatore, India, 20–20 March 2015; pp. 45–47. Hsu, Y.W.; Chen, K.H.; Yang, J.J.; Jaw, F.S. Smartphone-based fall detection algorithm using feature extraction. In Proceedings of the 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Datong, China, 15–17 October 2016. Dernbach, S.; Das, B.; Krishnan, N.C.; Thomas, B.L.; Cook, D.J. Simple and Complex Activity Recognition through Smart Phones. In Proceedings of the 8th International Conference on Intelligent Environments (IE) Guanajuato, Mexico, 26–29 June 2012. Shen, C.; Chen, Y.F.; Yang, G.S. On Motion-Sensor Behavior Analysis for Human-Activity Recognition via Smartphones. In Proceedings of the 2016 Ieee International Conference on Identity, Security and Behavior Analysis (ISBA), Sendai, Japan, 29 February–2 March 2016. Misra, A.; Lim, L. Optimizing Sensor Data Acquisition for Energy-Efficient Smartphone-Based Continuous Event Processing. In Proceedings of the 12th IEEE International Conference on Mobile Data Management (MDM), Lulea, Sweden, 6–9 June 2011; pp. 88–97. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 17 of 22 D’Ambrosio, A.; Aria, M.; Siciliano, R. Accurate Tree-based Missing Data Imputation and Data Fusion within the Statistical Learning Paradigm. J. Classif. 2012, 29, 227–258. Dong, J.; Zhuang, D.; Huang, Y.; Fu, J. Advances in multi-sensor data fusion: algorithms and applications. Sensors 2009, 9, 7771–7784. King, R.C.; Villeneuve, E.; White, R.J.; Sherratt, R.S.; Holderbaum, W.; Harwin, W.S. Application of data fusion techniques and technologies for wearable health monitoring. Med. Eng. Phys. 2017, 42, 1–12. White, R.M. A Sensor Classification Scheme. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 1987, 34, 124– 126. Bojinov, H.; Michalevsky, Y.; Nakibly, G.; Boneh, D. Mobile device identification via sensor fingerprinting. arXiv preprint, arXiv:1408.1416, 2014. Katevas, K.; Haddadi, H.; Tokarchuk, L. Sensingkit: Evaluating the sensor power consumption in ios devices. In Proceedings of the 12th International Conference on Intelligent Environments (IE), London, UK, 14–16 September 2016. Bersch, S.D.; Azzi, D.; Khusainov, R.; Achumba, I.E.; Ries, J. Sensor data acquisition and processing parameters for human activity classification. Sensors 2014, 14, 4239–70. Kang, S.; Lee, Y.; Min, C.; Ju, Y.; Park, T.; Lee, J.; Rhee, Y.; Song, J. Orchestrator: An active resource orchestration framework for mobile context monitoring in sensor-rich mobile environments. In Proceedings of the 2010 IEEE International Conference on Pervasive Computing and Communications (PerCom), Mannheim, Germany, 29 March–2 April 2010. Vallina-Rodriguez, N.; Crowcroft, J. ErdOS: Achieving energy savings in mobile OS. In Proceedings of the sixth international workshop on MobiArch. Bethesda, MD, USA, 28 June 2011; pp. 37–42. Priyantha, B.; Lymberopoulos, D.; Jie, L. LittleRock: Enabling Energy-Efficient Continuous Sensing on Mobile Phones. IEEE Pervasive Comput. 2011, 10, 12–15. Lu, H.; Yang, J.; Liu, Z.; Lane, N.D.; Choudhury, T.; Campbell, A.T. The Jigsaw continuous sensing engine for mobile phone applications. In Proceedings of the 8th ACM Conference on Embedded Networked Sensor Systems, Zürich, Switzerland, 3–5 November 2010; pp. 71–84. Rachuri, K.K.; Mascolo, C.; Musolesi, M.; Rentfrow, P.J. SociableSense: exploring the trade-offs of adaptive sampling and computation offloading for social sensing. In Proceedings of the 17th annual international conference on Mobile computing and networking, Las Vegas, NV, USA, 19–23 September 2011; pp. 73–84. Gupta, H.P.; Chudgar, H.S.; Mukherjee, S.; Dutta, T.; Sharma, K. A continuous hand gestures recognition technique for human-machine interaction using accelerometer and gyroscope sensors. IEEE Sens. J. 2016, 16, 6425–6432. Deshpande, A.; Guestrin, C.; Madden, S.R.; Hellerstein, J.M.; Hong, W. Model-driven data acquisition in sensor networks. In Proceedings of the Thirtieth international conference on Very large data bases—2004, VLDB Endowment: Toronto, Canada, 31 August–3 September 2004; Vol. 30, pp. 588–599. Kubota, H.; Kyokane, M.; Imai, Y.; Ando, K.; Masuda, S.I. A Study of Data Acquisition and Analysis for Driver’s Behavior and Characteristics through Application of Smart Devices and Data Mining. In Proceedings of the Third International Conference on Computer Science, Computer Engineering, and Education Technologies, Lodz, Poland, 19–21 September 2016. Ayu, M.A.; Mantoro, T.; Matin, A.F. A.; Basamh, S.S. Recognizing user activity based on accelerometer data from a mobile phone. In Proceedings of the 2011 IEEE Symposium on Computers & Informatics (ISCI), Kuala Lumpur, Malaysia, 20–23 March 2011. Banos, O.; Garcia, R.; Holgado-Terriza, J.A.; Damas, M.; Pomares, H.; Rojas, I.; Saez, A.; Villalonga, C. mHealthDroid: A novel framework for agile development of mobile health applications. In Lecture Notes in Comput. Sci. 2014. Springer: Cham, Switzerland, 2014; Vol. 8868. Chavan, V.B.; Mhala, N. Development of Hand Gesture Recognition Framework Using Surface EMG and Accelerometer Sensor for Mobile Devices. 2015. Available online: https://www.irjet.net/archives/V2/i5/IRJET-V2I542.pdf (accessed on 23rd December 2017). Sarkar, M.; Haider, M.Z.; Chowdhury, D.; Rabbi, G. An Android based human computer interactive system with motion recognition and voice command activation. In Proceedings of the 5th International Conference on Informatics, Electronics and Vision (ICIEV), Dhaka, Bangladesh, 13–14 May 2016. Pires, I.M.; Garcia, N.M.; Pombo, N.; Flórez-Revuelta, F.; Rodríguez, N.D. Validation Techniques for Sensor Data in Mobile Health Applications. J. Sens. 2016, 2016, 1687–725. 97 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 38. 39. 40. 41. 42. 43. 44. 45. 46. 47. 48. 49. 50. 51. 52. 53. 54. 55. 56. 57. 58. 98 18 of 22 Lane, N.D.; Miluzzo, E.; Lu, H.; Peebles, D.; Choudhury, T.; Campbell, A.T. A survey of mobile phone sensing. IEEE Commun. Mag. 2010, 48, doi: 10.1109/MCOM.2010.5560598. Banos, O.; Toth, M.A.; Damas, M.; Pomares, H.; Rojas, I Dealing with the effects of sensor displacement in wearable activity recognition. Sensors 2014, 14, 9995–10023. Pejovic, V.; Musolesi, M. Anticipatory Mobile Computing. ACM Comput. Surv. 2015, 47, 1–29. Lin, F.X.; Rahmati, A.; Zhong, L. Dandelion: A framework for transparently programming phone-centered wireless body sensor applications for health. In Proceedings of the 10th Wireless Health, San Diego, CA, USA, 5–7 October 2010. Postolache, O.; Girão, P.S.; Ribeiro, M.; Guerra, M.; Pincho, J.; Santiago, F.; Pena, A. Enabling telecare assessment with pervasive sensing and Android OS smartphone. In Proceedings of the 2011 IEEE International Workshop on Medical Measurements and Applications Proceedings (MeMeA), Bari, Italy, 30–31 May 2011. Jeffery, S.R.; Alonso, G.; Franklin, M.J.; Hong, W.; Widom, J. Declarative Support for Sensor Data Cleaning. In Lecture Notes in Computer Science; Springer: Berlin, Germany, 2006; Vol. 2006. Tomar, D.; Agarwal, S. A survey on pre-processing and post-processing techniques in data mining. Int. J.Database Theory Appl. 2014, 7, 99–128. Park, K.; Becker, E.; Vinjumur, J.K.; Le, Z.; Makedon, F. Human behavioral detection and data cleaning in assisted living environment using wireless sensor networks. In Proceedings of the 2nd International Conference on PErvasive Technologies Related to Assistive Environments, Corfu, Greece, 9–13 June 2009. Zhuang, Y.; Chen, L.; Wang, X.S.; Lian, J. A weighted moving average-based approach for cleaning sensor data. in Distributed Computing Systems, In Proceedings of the 27th International Conference on Distributed Computing Systems (ICDCS'07), Toronto, ON, Canada, 25–27 June 2007. Li, Z.; Wang, J.; Gao, J.; Li, B.; Zhou, F. A vondrak low pass filter for IMU sensor initial alignment on a disturbed base. Sensors 2014, 14, 23803–23821. Graizer, V. Effect of low‐pass filtering and re‐sampling on spectral and peak ground acceleration in strong‐ motion records. In Proceedings of the 15th World Conference of Earthquake Engineering, Lisbon, Portugal, 24–28 September 2012. UiO: Fourier analysis and applications to sound processing. Available online: http://www.uio.no/studier/emner/matnat/math/MAT.../v12/part1.pdf (accessed on 27 August 2017) Ninness, B. Spectral Analysis Using the FFT. Available online: https://pdfs.semanticscholar.org/dd74/4c224d569bd9ae907b7527e7f2a92fafa19c.pdf (accessed on 27 August 2017) Vateekul, P.; Sarinnapakorn, K. Tree-Based Approach to Missing Data Imputation. In Proceedings of IEEE International Conference on 2009 Data Mining Workshops (ICDMW '09), Miami, FL, USA, 6 December 2009; pp. 70–75. Ling, W.; Dong, M. Estimation of Missing Values Using a Weighted K-Nearest Neighbors Algorithm. In Proceedings of the International Conference on 2009 Environmental Science and Information Application Technology (ESIAT 2009), Wuhan, China; pp. 660–663. García-Laencina, P.J.; Sancho-Gómez, J. L.; Figueiras-Vidal, A. R.; Verleysen, M. K nearest neighbours with mutual information for simultaneous classification and missing data imputation. Neurocomputing 2009, 72, 1483–1493. Rahman, S.A.; Rahman, S.A.; Huang, Y.; Claassen, J.; Kleinberg, S. Imputation of Missing Values in Time Series with Lagged Correlations. In Proceedings of the 2014 IEEE International Conference on Data Mining Workshop (ICDMW), Shenzhen, China, 14 December 2014. Batista, G.E.; Monard, M.C. A Study of K-Nearest Neighbour as an Imputation Method. HIS 2002, 87, 251– 260. Hruschka, E.R.; Hruschka, E.R.; Ebecken, N.F.F. Towards Efficient Imputation by Nearest-Neighbors: A Clustering-Based Approach. In AI 2004: Advances in Artificial Intelligence; Springer: Berlin/Heidelberg, Germany, 2004; pp. 513–525. Luo, J.W.; Yang, T.; Wang, Y. Missing value estimation for microarray data based on fuzzy C-means clustering. In Proceedings of Eighth International Conference on High-Performance Computing in AsiaPacific Region, Beijing, China, 30 November–3 December 2005. Ni, D.; Leonard, J.D.; Guin, A.; Feng, C. Multiple Imputation Scheme for Overcoming the Missing Values and Variability Issues in ITS Data. J. Trans. Eng. 2005, 131, 931–938. Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 59. 60. 61. 62. 63. 64. 65. 66. 67. 68. 69. 70. 71. 72. 73. 74. 75. 76. 77. 19 of 22 Smith, B.; Scherer, W.; Conklin, J. Exploring Imputation Techniques for Missing Data in Transportation Management Systems. Trans. Res. Rec. 2003, 1836(1), 132–142. Qu, L.; Zhang, Y.; Hu, J.; Jia, L.; Li, L. A BPCA based missing value imputing method for traffic flow volume data. In Proceedings of 2008 IEEE Intelligent Vehicles Symposium, Eindhoven, Netherlands, 4–6 June 2008; pp. 985–990. Jiang, N.; Gruenwald, L. Estimating Missing Data in Data Streams. In Proceedings of the 12th International Conference on Database Systems for Advanced Applications (DASFAA'07), Bangkok, Thailand, 9–12 April 2007; pp. 981–987. Rahman, S.A.; Huang, Y.; Claassen, J.; Heintzman, N.; Kleinberg, S. Combining Fourier and lagged knearest neighbor imputation for biomedical time series data. J. Biomed. Inform. 2015, 58, 198–207. Huang, X.-Y.; Li, W.; Chen, K.; Xiang, X.-H.; Pan, R.; Li, L.; Cai, W.-X. Multi-matrices factorization with application to missing sensor data imputation. Sensors 2013, 13, 15172–15186. Rahman, S.A.; Huang, Y.; Claassen, J.; Kleinberg, S. Imputation of missing values in time series with lagged correlations. In Proceedings of the 2014 IEEE International Conference on Data Mining Workshop (ICDMW), Shenzhen, China, 14 December 2014; pp. 753–762. Smaragdis, P.; Raj, B.; Shashanka, M. Missing Data Imputation for Time-Frequency Representations of Audio Signals. J. Signal Process. Syst. 2010, 65, 361–370. Bayat, A.; Pomplun, M.; Tran, D.A. A Study on Human Activity Recognition Using Accelerometer Data from Smartphones. In Proceedings of 9th International Conference on Future Networks and Communications (Fnc'14) / the 11th International Conference on Mobile Systems and Pervasive Computing (Mobispc'14)/Affiliated Workshops, Ontario, Canada, 17–20 August 2014; pp. 450–457. Khalifa, S.; Hassan, M.; Seneviratne, A. Feature selection for floor-changing activity recognition in multifloor pedestrian navigation. In Proceedings of 2014 Seventh International Conference on Mobile Computing and Ubiquitous Networking (ICMU), Singapore, 6–8 January 2014. Zhao, K.L.; Du, J.; Li, C.; Zhang, C.; Liu, H.; Xu, C. Healthy: A Diary System Based on Activity Recognition Using Smartphone. In Proceedings of 2013 IEEE 10th International Conference on Mobile Ad-Hoc and Sensor Systems (Mass 2013), Hangzhou, China, 14–16 October 2013; pp. 290–294. Zainudin, M.N.S.; Sulaiman, M.N.; Mustapha, N.; Perumal, T. Activity Recognition based on Accelerometer Sensor using Combinational Classifiers. In Proceedings of 2015 IEEE Conference on Open Systems (ICOS), Bandar Melaka, Malaysia, 24–26 August 2015; pp. 68–73. Fan, L.; Wang, Z.M.; Wang, H. Human activity recognition model based on decision tree. In Proceedings of 2013 International Conference on Advanced Cloud and Big Data (CBD), Nanjing, China, 13–15 December 2013; pp. 64–68. Liu, Y.Y.; Fang, Z.; Wenhua, S.; Haiyong, Z. An Hidden Markov Model based Complex Walking Pattern Recognition Algorithm. In Proceedings of 2016 Fourth International Conference on Ubiquitous Positioning, Indoor Navigation and Location Based Services (IEEE UPINLBS 2016), Shanghai, China, 2–4 November 2016; pp. 223–229. Piyare, R.; Lee, S.R. Mobile Sensing Platform for Personal Health Management. In Proceedings of 18th IEEE International Symposium on Consumer Electronics (ISCE 2014), JeJu Island, South Korea, 22–25 June 2014; pp. 1–2. Chen, Y.F.; Shen, C. Performance Analysis of Smartphone-Sensor Behavior for Human Activity Recognition. IEEE Access 2017, 5, 3095–3110. Vavoulas, G.; Chatzaki, C.; Malliotakis, T.; Pediaditis, M.; Tsiknakis, M. The MobiAct Dataset: Recognition of Activities of Daily Living using Smartphones. In Proceedings of the International Conference on Information and Communication Technologies for Ageing Well and E-Health (ICT4AWE), Rome, Italy, 21– 22 April 2016; pp. 143–151. Torres-Huitzil, C.; Nuno-Maganda, M. Robust smartphone-based human activity recognition using a triaxial accelerometer. In Proceedings of 2015 IEEE 6th Latin American Symposium on Circuits & Systems (Lascas), Montevideo, Uruguay, 24–27 February 2015; pp. 1–4. Anjum, A.; Ilyas, M.U. Activity Recognition Using Smartphone Sensors. In Proceedings of 2013 IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, USA, 11–14 January 2013; pp. 914–919. Kumar, A.; Gupta, S. Human Activity Recognition through Smartphone’s Tri-Axial Accelerometer using Time Domain Wave Analysis and Machine Learning. Int. Comput. Appl. 2015, 127, 22–26. 99 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 78. 20 of 22 Hon, T.K.; Wang, L.; Reiss, J.D.; Cavallaro, A. Audio Fingerprinting for Multi-Device Self-Localization. IEEE/ACM Trans. Audio, Speech Lang. Process. 2015, 23, 1623–1636. 79. Sert, M.; Baykal, B.; Yazici, A. A Robust and Time-Efficient Fingerprinting Model for Musical Audio. In Proceedings of 2006 IEEE International Symposium on Consumer Electronics, St. Petersburg, Russia, 28 June–1 July 2007. 80. Ramalingam, A.; Krishnan, S. Gaussian Mixture Modeling of Short-Time Fourier Transform Features for Audio Fingerprinting. IEEE Trans. Inform. Forens. Secu. 2006, 1, 457–463. 81. Vincenty, T. Direct and Inverse Solutions of Geodesics on the Ellipsoid with Application of Nested equations. Surv. Rev. 1975, 22, 88–93. 82. Karney, C.F.F. Algorithms for Geodesics. J. Geodesy 2013, 87, 43–55. 83. Karney, C.F.F.; Deakin, R.E. The calculation of longitude and latitude from geodesic measurements. Astron. Nachr. 2010, 331, 852–861. 84. Khaleghi, B.; Khamisa, A.; Karraya, F.O.; Razavib, S.N. Multisensor data fusion: A review of the state-ofthe-art. Inf. Fusion 2013, 14, 28–44. 85. Pombo, N.; Bousson, K.; Araújo, P.; Viana J. Medical decision-making inspired from aerospace multisensor data fusion concepts. Inform Health Soc. Care 2015, 40, 185–97. 86. Durrant-Whyte, H.; Stevens, M.; Nettleton, E. Data fusion in decentralised sensing networks. In Proceedings of the 4th International Conference on Information Fusion, Montreal, Canada, 7–10 August 2001. 87. Tanveer, F.; Waheed, O.T.; Atiq-ur-Rehman. Design and Development of a Sensor Fusion based Low Cost Attitude Estimator. J.Space Technol. 2011, 1, 45–50. 88. Ko, M.H.; Westa, G.; Venkatesha, S.; Kumarb, M. Using dynamic time warping for online temporal fusion in multisensor systems. Inf. Fusion 2008, 9, 370–388. 89. Singh, D.; Merdivan, E., Psychoula, I.; Kropf, J.; Hanke, S.; Geist, M.; Holzinger, A. Human activity recognition using recurrent neural networks. In Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Reggio, Italy, 29 August–1 September, 2017. 90. Zhao, L.; Wu, P.; Cao, H. RBUKF Sensor Data Fusion for Localization of Unmanned Mobile Platform. Res. J. Appl. Sci. Eng. Technol. 2013, 6, 3462–3468. 91. Walter, O.; Schmalenstroeer, J.; Engler, A.; Haeb-Umbach, R. Smartphone-based sensor fusion for improved vehicular navigation. In Proceedings of 2013 10th Workshop on Positioning Navigation and Communication (WPNC), Dresden, Germany, 20–21 March 2013. 92. Grunerbl, A.; Muaremi A.; Osmani V.; Bahle G.; Ohler S.; Tröster G.; Mayora O.; Haring C.; Lukowicz P. Smart-Phone Based Recognition of States and State Changes in Bipolar Disorder Patients. IEEE J Biomed. Health Inform 2015, 15, 140–148. 93. Thatte, G.; Li, M.; Lee, S.; Emken, B.A.; Annavaram, M.; Narayanan, S.; Narayanan, D.; Mitra, U. Optimal Time-Resource Allocation for Energy-Efficient Physical Activity Detection. IEEE Trans. Signal Process 2011, 59, 1843–1857. 94. Bhuiyan, M.Z.H.; Kuusniemi, H.; Chen, L.; Pei, L.; Ruotsalainen, L.; Guinness, R.; Chen, R. Performance Evaluation of Multi-Sensor Fusion Models in Indoor Navigation. Eur. J. Navig. 2013, 11, 21–28. 95. Bellos, C.; Papadopoulos, A.; Rosso, R.; Fotiadis, D.I. Heterogeneous data fusion and intelligent techniques embedded in a mobile application for real-time chronic disease management. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2011, 2011, 8303–8306. 96. Ayachi, F.S.; Nguyen, H.; Goubault, E.; Boissy, P.; Duval, C. The use of empirical mode decompositionbased algorithm and inertial measurement units to auto-detect daily living activities of healthy adults. IEEE Trans. Neural Syst. Rehabilit. Eng. 2016, 24, 1060–1070. 97. Debes, C.; Merentitis, A.; Sukhanov, S.; Niessen, M.; Frangiadakis, N.; Bauer, A. Monitoring activities of daily living in smart homes: Understanding human behavior. IEEE Signal Process. Mag. 2016, 33, 81–94. 98. Koza, J.R.; Bennett, F.H.; Andre, D.; Keane, M.A. Automated design of both the topology and sizing of analog electrical circuits using genetic programming. In Artificial Intelligence in Design’96; Springer: Berlin, Germany, 1996, pp. 151–170. 99. Russell, S.; P. Norvig, P.; Canny, C.F.; Malik, J.M. A. Artificial Intelligence: A Modern Approach; Upper Saddle River: Bergen, NJ, USA, 1995. 100. Du, K.-L.; Swamy, M.N.S. Fundamentals of Machine Learning. In Neural Networks and Statistical Learning. Springer: Berlin, Germany, 2014; pp. 15–65. 100 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 21 of 22 101. Zhang, Y.; Rajapakse, J.C. Machine Learning in Bioinformatics. John Wiley & Sons: Hoboken, NJ, USA, 2009. 102. Witten, I.H.; Frank, E.; Hall, M.A.; Data Mining: Practical Machine Learning Tools and Techniques; Morgan Kaufmann: Burlington, MA, USA, 2016. 103. Schapire, R.E. The boosting approach to machine learning: An overview. In Nonlinear Estimation and Classification, Springer: Berlin, Germany, 2003; pp. 149–171. 104. Michalski, R.S.; Carbonell, J.G.; Mitchell, T.M.X. Machine Learning: An Artificial Intelligence Approach. Springer Science & Business Media: Berlin, Germany, 2013 105. Bishop, C.M. Pattern Recognition and Machine Learning. Springer: Berlin, Germany, 2006. 106. Lorenzi, P.; Rao, R.; Romano, G.; Kita, A.; Irrera, F.; Mobile Devices for the Real-Time Detection of Specific Human Motion Disorders. IEEE Sens. J. 2016, 16, 8220–8227. 107. Lau, S.L.; König, I.; David, K.; Parandian, B.; Carius-Düssel, C.; Schultz, M. Supporting patient monitoring using activity recognition with a smartphone. In Proceedings of 2010 7th International Symposium on Wireless Communication Systems (ISWCS), York, UK, 19–22 September 2010. 108. Lau, S.L. Comparison of orientation-independent-based-independent-based movement recognition system using classification algorithms. In Proceedings of 2013 IEEE Symposium on Wireless Technology and Applications (ISWTA), Kuching, Malaysia, 22–25 September 2013. 109. Duarte, F.; Lourenco, A.; Abrantes, A. Activity classification using a smartphone. In Proceedings of 2013 IEEE 15th International Conference on e-Health Networking, Applications & Services (Healthcom), Lisbon, Portugal, 9–12 October 2013 110. Fahim, M.; Lee, S.; Yoon, Y. SUPAR: Smartphone as a ubiquitous physical activity recognizer for uhealthcare services. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2014, 2014, 3666–3669. 111. Bajpai, A.; Jilla, V.; Tiwari, V.N.; Venkatesan, S.M.; Narayanan, R. Quantifiable fitness tracking using wearable devices. In Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015. 112. Nguyen, P.; Akiyama, T.; Ohashi, H.; Nakahara, G.; Yamasaki, K.; Hikaru, S. User-friendly Activity Recognition Using SVM Classifier and Informative Features. In Proceedings of the 2015 International Conference on Indoor Positioning and Indoor Navigation (Ipin), Banff, AB, Canada, 13–16 October 2015; pp. 1–8. 113. Wang, C.; Xu, Y.; Zhang, J.; Yu, W. SW-HMM: A Method for Evaluating Confidence of Smartphone-Based Activity Recognition. In Proceedings of the 2016 IEEE Trustcom/BigDataSE/ISPA, Tianjin, China, 23–26 August 2016. 114. Lau, S.L.; David, K. Movement recognition using the accelerometer in smartphones. In Proceedings of the Future Network and Mobile Summit, Florence, Italy, 16–18 June 2010. 115. Zhang, L.; Wu, X.; Luo, D. Real-Time Activity Recognition on Smartphones Using Deep Neural Networks. In Proceedings of the Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), Beijing, China, 10–14 August 2015. 116. Cardoso, N.; Madureira, J.; Pereira, N. Smartphone-based Transport Mode Detection for Elderly Care. 2016 In Proceedings of the IEEE 18th International Conference on E-Health Networking, Applications and Services (Healthcom), Munich, Germany, 14–16 September 2016; pp. 261–266. 117. Vallabh, P.; Malekian, R.; Ye, N.; Bogatinoska, D.C. Fall Detection Using Machine Learning Algorithms. In Proceedings of the 24th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, 22–24 September 2016; pp. 51–59. 118. Filios, G.; Nikoletseas, S.; Pavlopoulou, C.; Rapti, M.; Ziegler, S. Hierarchical Algorithm for Daily Activity Recognition via Smartphone Sensors. In Proceedings of the 2015 IEEE 2nd World Forum on Internet of Things (WF-IOT), Milan, Italy, 14–16 December 2015; pp. 381–386. 119. Tang, C.X.; Phoha, V.V. An Empirical Evaluation of Activities and Classifiers for User Identification on Smartphones. In Proceedings of the 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems (BTAS), Niagara Falls, NY, USA, 6–9 September 2016; pp. 1–8. 120. Li, P.; Wang, Y.; Tian, Y.; Zhou, T.S.; Li, J.S. An Automatic User-Adapted Physical Activity Classification Method Using Smartphones. IEEE Trans. Biomed. Eng. 2017, 64, 706–714. 121. Kim, Y.J.; Kang, B.N.; Kim, D. Hidden Markov Model Ensemble for Activity Recognition using Tri-axis Accelerometer. In Proceedings of the 2015 IEEE International Conference on Systems, Man, and 101 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Sensors 2018, 18, 640 22 of 22 Cybernetics (Smc 2015): Big Data Analytics for Human-Centric Systems, Kowloon, China, 9–12 October 2015; pp. 3036–3041. 122. Brdiczka, O.; Bellotti, V. Identifying routine and telltale activity patterns in knowledge work. In Proceedings of the Fifth IEEE International Conference on Semantic Computing (ICSC), Palo Alto, CA, USA, 18–21 September 2011. 123. Costa, Â.; Castillo, J.C.; Novais, P.; Fernández-Caballero, A.; Simoes, R. Sensor-driven agenda for intelligent home care of the elderly. Exp. Syst. Appl. 2012, 39, 12192–12204. © 2018 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/). 102 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 4 Data fusion for the Identification of Activities of Daily Living This chapter is related to the research and development of the framework for the recognition of ADL and environments, and it is composed by two articles, each presented in its section. 1. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices The following article is the first part of the chapter 4. Identification of Activities of Daily Living through Data Fusion on Motion and Magnetic Sensors embedded on Mobile Devices Ivan Miguel Pires, Nuno M. Garcia, Nuno Pombo, Francisco Flórez-Revuelta, Susanna Spinsante and Maria Canavarro Teixeira Pervasive and Mobile Computing (Elsevier), published, 2018. According to 2016 Journal Citation Reports published by Thomson Reuters in 2017, this journal has the following performance metrics: ISI Impact Factor (2016): 2.349 ISI Article Influence Score (2016): 0.7 Journal Ranking (2016): 6/32 (Computer Science (miscellaneous)) 103 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 104 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Pervasive and Mobile Computing 47 (2018) 78–93 Contents lists available at ScienceDirect Pervasive and Mobile Computing journal homepage: www.elsevier.com/locate/pmc Identification of activities of daily living through data fusion on motion and magnetic sensors embedded on mobile devices Ivan Miguel Pires a,b,c , , Nuno M. Garcia a , Nuno Pombo a , Francisco Flórez-Revuelta d , Susanna Spinsante e , Maria Canavarro Teixeira f,g ∗ a Instituto de Telecomunicações, Universidade da Beira Interior, Covilhã, Portugal Altranportugal, Lisbon, Portugal ALLab — Assisted Living Computing and Telecommunications Laboratory, Computer Science Department, Universidade da Beira Interior, Covilhã, Portugal d Department of Computer Technology, Universidad de Alicante, Spain e Università Politecnica delle Marche, Ancona, Italy f UTC de Recursos Naturais e Desenvolvimento Sustentável, Polytechnique Institute of Castelo Branco, Castelo Branco, Portugal g CERNAS — Research Centre for Natural Resources, Environment and Society, Polytechnique Institute of Castelo Branco, Castelo Branco, Portugal b c article info Article history: Received 23 January 2018 Received in revised form 6 May 2018 Accepted 18 May 2018 Available online xxxx Keywords: Mobile devices sensors Sensor data fusion Artificial neural networks Identification of activities of daily living a b s t r a c t Several types of sensors have been available in off-the-shelf mobile devices, including motion, magnetic, vision, acoustic, and location sensors. This paper focuses on the fusion of the data acquired from motion and magnetic sensors, i.e., accelerometer, gyroscope and magnetometer sensors, for the recognition of Activities of Daily Living (ADL). Based on pattern recognition techniques, the system developed in this study includes data acquisition, data processing, data fusion, and classification methods like Artificial Neural Networks (ANN). Multiple settings of the ANN were implemented and evaluated in which the best accuracy obtained, with Deep Neural Networks (DNN), was 89.51%. This novel approach applies L2 regularization and normalization techniques on the sensors’ data proved it suitability and reliability for the ADL recognition. © 2018 Elsevier B.V. All rights reserved. 1. Introduction Off-the-shelf mobile devices have several sensors available, which are capable for the acquisition of several physical and physiological parameters [1], including the accelerometer, magnetometer, and gyroscope sensors, allowing the recognition of Activities of Daily Living (ADL) [2]. The correct identification of ADL is one of the stages for the development of a personal digital life coach [3], which can be used in several areas, including sports and geriatrics, among others. For the use of several sensors in the development of a method for the recognition of ADL, data fusion techniques should be used before the application of the classification methods. This paper focuses on the use of motion and magnetic sensors available on mobile devices, where the most commonly available are the accelerometer, the magnetometer, and the gyroscope, proposing the recognition of ADL with movement, including running, walking, walking on stairs, and standing. The architecture for the method for the recognition of ADL was proposed in [4–6], which is composed by data acquisition, data processing, data fusion, and classification methods. Taking in account that the data acquired from the sensors is fulfilled, ∗ Correspondence to: Computer Science Department, Universidade da Beira Interior, Rua Marques d’Avila e Bolama, 6200-001 Covilhã, Portugal. E-mail addresses: impires@it.ubi.pt (I.M. Pires), ngarcia@di.ubi.pt (N.M. Garcia), ngpombo@di.ubi.pt (N. Pombo), francisco.florez@ua.es (F. Flórez-Revuelta), s.spinsante@univpm.it (S. Spinsante), ccanavarro@ipcb.pt (M.C. Teixeira). https://doi.org/10.1016/j.pmcj.2018.05.005 1574-1192/© 2018 Elsevier B.V. All rights reserved. 105 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 79 the data processing methods are forked in two types of methods, namely data cleaning, and features extraction methods. After the feature extraction, the data will be fused and the classification methods will be applied. Following the use of the data fusion techniques with the accelerometer, the gyroscope and the magnetometer sensors, this study proposes a creation of a new method that combine multiple sensors suitable for the off-the-shelf mobile devices such as smartphones. Recently, several studies have been done regarding the recognition of ADL using several sensors [7–12]. Artificial Neural Networks (ANN) are widely used, proving their reliability for the automatic learning and recognition of several patterns of sensors’ data [13,14]. Due to the limitation of number of sensors available in the off-the-shelf mobile devices, and based on the previous study [15] that uses only the accelerometer sensor, this study proposes the creation of two different methods for the recognition of ADL using different number of sensors in order to adapt the method according to the number of sensors available. Firstly, it proposes the fusion of the data acquired from the accelerometer, and the magnetometer sensors. Secondly, it proposes the fusion of the data acquired from the accelerometer, gyroscope, and magnetometer sensors. The ADL proposed for recognition are running, walking, going upstairs, going downstairs, and standing, consisting this research on the analysis of the performance of three implementations of ANN, namely, Multilayer Perception (MLP) with Backpropagation, Feedforward Neural Network (FNN) with Backpropagation, and Deep Neural Networks (DNN). A dataset used for this research is composed by the sensors’ data acquired in several experiments with users aged between 16 and 60 years old, having distinct lifestyles, and a mobile device in the front pocket of their trousers, performing the proposed ADL. This research was conducted with the use of three Java libraries, Neuroph [16], Encog [17], and DeepLearning4j [18], and different datasets of features, in order to identify the best dataset of features and implementation of ANN for the recognition of ADL, verifying that the best accuracy for the recognition of ADL with the two different methods proposed was achieved with Deep Learning methods. This paragraph concludes Section 1, and this paper is organized as follows: Section 2 summarizes the literature review for the use of data fusion techniques with accelerometer, gyroscope, and magnetometer sensors; Section 3 presents the methods used on each stage of the architecture proposed. The results obtained are presented in Section 4, presenting the discussion about these results in Section 5. The conclusions of this study are presented in Section 6. 2. Related work Data fusion techniques may be used with the data acquired from motion and magnetic sensors available in the off-theshelf mobile devices, i.e., accelerometer, gyroscope, and magnetometer, in order to improve the reliability of the methods for the recognition of Activities of Daily Living (ADL) [2]. Following the main focus of this paper, the accelerometer, the gyroscope, and the magnetometer are used by the authors of [19] with the Random Forest classifier for the recognition of standing, going downstairs, going upstairs, sitting, walking, and running activities. Based on a myriad of features such as the variance, the mean, the frequency of the point with maximum amplitude, the energy of the extremum value, the value of the point with maximum amplitude, the mean and the period of the extremum value, the sum of the difference between extremum values, the maximum, the minimum and the mean value around the midpoint, reporting an average accuracy of 99.7%. In addition, Shoaib et al. [20] presented a method that also uses the Global Positioning System (GPS) receiver, implementing ANN, i.e., Multi-Layer Perceptron (MLP), Support Vector Machine (SVM), Naïve Bayes, Logistic regression, decision tree, K-Nearest Neighbor (KNN), and rule based classifiers. The features extracted included the mean and the standard deviation of the raw signal from the accelerometer, gyroscope, and magnetometer, and the distance from the GPS data, in order to recognize running, walking, standing, sitting, going downstairs, and going upstairs with a reported accuracy between 69% and 99%. For the recognition of going upstairs and downstairs, standing, walking on an escalator and taking an elevator, the authors of [21] extracted several features from the accelerometer, magnetometer and gyroscope sensors, including mean, median, variance, standard deviation, 75th percentile, inter-quartile range, average absolute difference, binned distribution, energy, Signal Magnitude Area (SMA), Zero-Crossing Rate, Number of Peaks, Absolute Value of short-time Fourier Transform, Power of short-time Fourier Transform and Power Spectral Centroid. They used these features with a decision tree method, reporting an accuracy between 80% and 90% [21]. The majority of the studies in the literature only fuses the accelerometer and gyroscope data. For example the authors of [22] do not present the features extracted, but they implemented the Random Forests (RF) variable importance in order to help in the selection of the best features for the recognition of walking, going upstairs and downstairs, sitting, standing and laying activities. After the feature extraction, the implementation of the Two-stage continuous Hidden Markov Model (HMM) reported an accuracy of 91.76%. The Hidden Markov Model (HMM) was also implemented in [23], which also implemented the decision tree and Random Forest methods, with accelerometer and gyroscope data for the recognition of going downstairs and upstairs, and walking. The features included: variance, mean, standard deviation, maximum, minimum, median, interquartile range, skewness, kurtosis, and spectrum peak position of the accelerometer and gyroscope data, reporting an accuracy of 93.8%. In [24], the authors recognized walking, standing, running, laying activities, going downstairs and upstairs, and with accelerometer and gyroscope data, extracting the mean, the energy, the standard deviation, the correlation, and the entropy 106 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 80 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 of the sensors’ data. Several models were implemented such as the J48 decision tree, the logistic regression, the MLP, and the SVM reporting an accuracy between 89.3% and 100%. The authors of [25] implemented the Signal Magnitude Vector (SMV) algorithm with a Threshold based algorithm for feature extraction in order to recognize some ADL, such as walking, standing, sitting, and running with a reported accuracy around 90%. According to [26], the Gaussian mixture model (GMM) and the time series shapelets, applied to the accelerometer and gyroscope data, allow the recognition of sitting, standing, walking, and running activities with mean and standard deviation as features, reporting an accuracy of 88.64%. The authors of [27] also used the mean and standard deviation as features for the application of KNN and SVM methods, in order to recognize walking, resting, running, going downstairs, and going upstairs with a reported accuracy higher than 90%. The standard deviation, maximum, minimum, correlation coefficients, interquartile range, mean, Dynamic time warping distance (DTW), Fast Fourier Transform (FFT) coefficients, and wavelet energy were extracted as features from accelerometer and gyroscope sensors. In order to recognize walking, jumping, running, going downstairs and upstairs several methods were implemented, such as SVM, KNN, MLP, and Random Forest, reporting an accuracy between 84.97% and 90.65% [12]. The authors of [28] extracted the same features for the recognition of walking, going upstairs and downstairs, jumping, and jogging activities, The KNN, the Random Forests and the SVM methods were implemented, reporting an accuracy of 95%. The authors of [29] extracted the variance, mean, minimum and maximum along the Y axis of the accelerometer, and the variance and mean along the X axis of the gyroscope, and implemented the SVM method for the recognition of running, walking, going downstairs and upstairs, standing, cycling and sitting, which reports an accuracy of 96%. In [30], the authors extracted the skewness, mean, minimum, maximum, standard deviation, kurtosis, median, and interquartile range from the accelerometer and gyroscope data, implementing the MLP, the SVM, the Least Squares Method (LSM), and the Naïve Bayes classifiers for the recognition of falling activities with a reported accuracy of 87.5%. The SVM, Random Forest, J48 decision tree, Naïve Bayes, MLP, Rpart, JRip, Bagging, and KNN were implemented in [31] for the recognition of going downstairs and upstairs, lying, standing, and walking with the mean and standard deviation along the X , Y and Z axis of the accelerometer and the gyroscope signal as features, reporting an accuracy higher than 90%. The Root Mean Square (RMS), minimum, maximum, and zero crossing rate for X , Y , and Z axis were extracted from the accelerometer and gyroscope data, and the ANOVA method was applied for the correct recognition of sitting, resting, turning, and walking with a maximum reported accuracy of 100% [32]. The driving, walking, running, cycling, resting, and jogging were recognized by ANN with mean, minimum, maximum, standard deviation, difference between maximum and minimum, Parseval’s Energy either in the frequency range 0–2.5 Hz, or in the frequencies greater than 2.5 Hz. In addition, RMS, kurtosis, correlation between axis, ratio of the maximum and minimum values in the FFT, skewness, difference between the maximum and minimum values in the FFT, median of troughs, median of peaks, number of troughs, number of peaks, average distance between two consecutive troughs, average distance between two consecutive peaks, indices of the 8 highest peaks after the application of the FFT, and ratio of the average values of peaks were also considered. The observed accuracy varies between 57.53% and 97.58% [33]. The Threshold Based Algorithm (TBA) was applied to the values of the acceleration, and the difference between adjacent elements of the heading, extracted from the accelerometer and gyroscope sensors, in order to recognize going downstairs and upstairs, running, walking, and jumping with a reported accuracy of 83% [34]. The median absolute deviation, minimum, maximum, absolute mean, interquartile range, Signal Magnitude Range, skewness, and kurtosis were extracted from accelerometer and gyroscope signal for the application of KNN, SVM, Sparse Representation Classifier, and Kernel-Extreme Learning Machine, in order to recognize standing, running, going upstairs, walking, and going downstairs, reporting an average accuracy of 94.5% [35]. The jogging and walking activities are recognized with mean, variance, minimum, and maximum of the X , Y and Z axis of the accelerometer and gyroscope sensors as features applied to the SVM method, reporting an accuracy of 95.5% [36]. The authors of [37] implemented sparse approximation, KNN, SVM, Spearman correlation, Fuzzy c-means, MLP, and linear regression classifiers for the recognition of running, cycling, sitting, walking, and standing, using the standard deviation, mean, median, power ratio of the frequency bands, peak acceleration, and energy extracted from the accelerometer and gyroscope signal, reporting an accuracy of 98%. In [38], the implementation of SVM and Random Forest methods was used for the recognition of standing, sitting, laying, walking, going downstairs and upstairs, with the extraction of the angle, the minimum, the maximum, and the mean values of the accelerometer and gyroscope signal, reporting an accuracy around 100%. The authors of [39] used the accelerometer and gyroscope sensors for the recognition of the movements related to up and down buses, implementing the C4.5 decision tree, Naïve Bayes, KNN, logistic regression, SVM, and MLP with mean, standard deviation, energy, correlation between axis, and magnitude of FFT components as features, reporting an accuracy of 95.3%. The accelerometer, gyroscope, barometer, and GPS were used for the recognition of standing, sitting, washing dishes, going downstairs and upstairs, walking, running, and cycling with standard deviation, mean, interquartile range, mean squared, altitude difference in meters, and speed as features applied to the SVM method, whose the authors reported an accuracy around 90% [40]. For the recognition of walking, lying, running, cycling, jogging, washing dishes, vacuum cleaning, playing piano, playing cello, playing tennis, brushing teeth, wiping cupboard, driving, taking an elevator, doing laundry, working on a computer, eating, reading a book, going downstairs, going upstairs, and folding laundry, the authors of [41] used the features extracted 107 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 81 Table 1 Distribution of the ADL extracted in the studies analyzed. from the accelerometer, the gyroscope, and the camera, including the variance and mean for each axis, the movement intensity, the energy, the energy consumption, and the periodicity, applying them to the HMM, the SVM, the Naïve Bayes methods, and obtaining a reported accuracy of 81.5%. The J48 decision tree, IBk, MLP, and Logistic regression methods were implemented with the median, the mean, the standard deviation, the kurtosis, the skewness, the maximum, the minimum, the slope, difference between maximum and minimum, the spectral centroid, the entropy of the energy in 10 equal sized blocks, the short time energy, the spectral roll off, the zero crossing rate, the spectral flux, and the spectral centroid for each axis and the absolute value of the accelerometer, gyroscope, and orientation sensors [42], in order to recognize walking, standing, jogging, going downstairs, going upstairs, jumping, and sitting activities, reporting an accuracy of 94%. According to the analysis previously presented, Table 1 shows the ADL recognized, in more than one study, with the use of the accelerometer, gyroscope and/or magnetometer sensors, verifying that the walking, standing/resting, going downstairs and upstairs, running, and sitting are the most recognized ADL. The lines in Table 1 are sorted in decreasing manner regarding the number of studies found for each activity highlighting the activities reported in at least 10 papers. Based on the literature review, the features used for the recognition of the ADL, in more than 1 studies, are presented in Table 2, showing that the mean, standard deviation, maximum, minimum, energy, inter-quartile range, correlation coefficients, median, and variance are the most used features, with more relevance for mean, standard deviation, maximum, and minimum. The Table 2 is sorted in decreasing order of the number of studies that reportedly used a specific feature highlighting that ones used in 6 or more papers. Finally, the methods implemented in the literature that report the achieved accuracy, are presented in Table 3, concluding that the methods with an accuracy higher than 90% are MLP, logistic regression, random forest and decision tree methods, verifying that the method that reports the best average accuracy in the recognition of ADL is the MLP, with an average accuracy equal to 93.86%. As the ANN are the methods that report the best accuracies in the literature for the recognition of ADL, this paper focuses on the research on what implementation of ANN reports the best results in the recognition of a set of 5 ADL (i.e., walking, going downstairs, going upstairs, standing and running) with a dataset previously acquired. Moreover, the comparison of our results with the results available in the literature is not possible, because the authors did not make their datasets publicly available and the efficiency of the methods depends on the number of ADL each tries to recognize. The following sections of this paper focus on the comparison between three implementations of ANN, such as the MLP method with backpropagation, the FNN method with backpropagation and the DNN method. Their implementation parameters will be presented in the next section. 3. Methods Based on the related work presented in the previous section and the proposed architecture of a framework for the recognition of ADL previously presented in [4–6], there are several modules for the creation of the final method, such as data acquisition, data processing, data fusion, and classification methods. Assuming that the data acquired from all sensors is fulfilled, the data processing module is composed by data cleaning and feature extraction methods. After that, the data fusion method will be applied for further classification of the data. In Fig. 1, a simplified schema for the development of a framework for the identification of ADL is presented. Section 3.1 presents the methodology for the data acquisition. Data processing methods are presented in Section 3.2 and, finally, in Section 3.3, the data fusion and classification methods are presented. 108 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 82 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 Table 2 Distribution of the features extracted in the studies analyzed. Table 3 Distribution of the classification methods used in the studies analyzed. 3.1. Data acquisition A mobile application developed for Android devices [43,44] was installed in the BQ Aquaris 5.7 smartphone, which it has a Quad Core CPU and 16 GB of internal memory [45] for the acquisition of the sensors’ data, saving the data captured from the accelerometer, magnetometer, and gyroscope sensors into text files. The mobile application captures the data in 5 s slots every 5 min, where the frequency of the data acquisition is around 10 ms per sample. For the definition of the experiments, 25 individuals aged between 16 and 60 years old were selected. The individuals selected had distinct lifestyles, being 10 individuals mainly active and the remaining 15 individuals, mainly sedentary. During the data acquisition, the mobile device was in the front pocket of the user’s trousers, and using the data collection application, the user manually defined a label of the ADL performed in each of the 5 s of data captured. Based on the ADL that are the most identified in the previous research studies [4–6,15], the selected ADL for this study in the mobile application are: running, walking, going upstairs, going downstairs, and standing. After the selection of the individuals for the experimental study and the definition of the main rules previously discussed, the environment of the data acquisition needs to be characterized. The environment is strictly defined for the different types of ADL performed, where the walking and running activities are performed in outdoor environments (e.g. street) at different 109 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 83 times of the day, the standing activity is performed in the living room at different times of the day, and the going upstairs and going downstairs activities are performed in indoor environments at different times of the day. In total, there are around 36 h of captures for each activity, resulting in 2000 captures with 5 s of raw sensors’ data. By default, without user selection, the mobile application will label the acquired data as unknown activity, therefore, before starting the data acquisition, the user must select in the mobile application the ADL that will be performed during a certain time for completion of the classification method. After this selection the user must put the mobile device is front pocket of the trousers. The unlabeled data is not stored in the final dataset for the creation of the classification method. Nevertheless, all the data used in this research is publicly available at the ALLab MediaWiki [46]. One of the problems related to the data acquisition is the battery lifetime and power processing capabilities of the mobile device, already described in [47], verifying that the acquisition of 5 s of raw data every 5 min using the oldest tested devices has possible for a minimum of 16 h with a normal use of the mobile device. As the mobile devices usually require a daily recharge, it is possible to consider that the acquisition of the sensors’ data with this method can be implemented in a real-life scenario. The performance of the mobile device is also not significantly affected during the data acquisition. In conclusion, the acquisition of small datasets sampled in a defined time interval allows the use of this method, but it only characterizes the data acquired on each interval, discarding the activities performed during the time where the data acquisition is in standby (the 5 min every 5 s). It will be probably sufficient for the characterization of lifestyles, but it may miss some important events that may not be identified, including other activities e.g. users’ falls, that are out of the scope of this research. The development of a method that implements a more significant sampling strategy without decreasing the performance and the availability of resources at the mobile device requires additional research. 3.2. Data processing The data processing is the second step of the method for the recognition of ADL, which is executed after the data acquisition. Data cleaning methods are executed for noise reduction, as presented in Section 3.2.1. After cleaning the data, the features were extracted for further analysis, as discussed in Section 3.2.2. 3.2.1. Data cleaning Data cleaning is a process to filter the data acquired from the accelerometer, magnetometer, and gyroscope sensors, in order to remove the noise. The data cleaning method should be selected according to the types of sensors used, nevertheless, the low-pass filter is the best method for the data acquired from the sensors used in this study [48], removing the noise and allowing the correct extraction of the selected features. 3.2.2. Feature extraction After the data cleaning, and based on the features most commonly extracted in previous research studies [4–6,15] (see Table 2), several features were extracted from the accelerometer, magnetometer, and gyroscope sensors. These are the five greatest distances between the maximum peaks, the average, standard deviation, variance and median of the maximum peaks, and the standard deviation, average, maximum value, minimum value, variance and median of the raw signal. The feature extraction process starts with the calculation of the maximum peaks of the 5 s of data acquired related to each sensor used. After the calculation of the maximum peaks, the distance between the different peaks (in milliseconds) is calculated, using the five highest distances between the highest peaks. After that, the remaining features related to the maximum peaks are calculated, including the average, standard deviation, variance and median. Finally, the standard deviation, average, maximum value, minimum value, variance and median are calculated from the original acquired data. These features will be used in the different datasets presented in Section 3.3. 3.3. Identification of activities of daily living with data fusion Extending a previous study [15] that used only the accelerometer sensor, this study fuses the features extracted from the accelerometer and magnetometer sensors (Section 3.3.1), and the features extracted from the accelerometer, gyroscope and magnetometer sensors (Section 3.3.2). Finally, the classification methods for the identification of ADL are presented in Section 3.3.3. 3.3.1. Data fusion with accelerometer and magnetometer sensors Regarding the features extracted from each ADL, five datasets have been constructed with features extracted from the data acquired from the accelerometer and magnetometer during the performance of the five ADL, resulting in 2000 records from each ADL. The datasets defined are: • Dataset 5: Composed by the average, and the standard deviation of the raw signal extracted from both accelerometer and magnetometer sensors; • Dataset 4: Composed by the features of the Dataset 5 plus the variance and the median of the raw signal extracted from both accelerometer and magnetometer sensors; 110 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 84 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 Fig. 1. Simplified diagram for the framework for the identification of ADL. • Dataset 3: Composed by the features of the Dataset 4 plus the maximum and the minimum values of the raw signal extracted from both accelerometer and magnetometer sensors; • Dataset 2: Composed by the features of the Dataset 3 plus the average, the standard deviation, the variance and the median of the maximum peaks obtained from the raw signal extracted from both accelerometer and magnetometer sensors; • Dataset 1: Composed by the features of the Dataset 2 plus the five greatest distances between the maximum peaks obtained from the raw signal extracted from both accelerometer and magnetometer sensor. 3.3.2. Data fusion with accelerometer, magnetometer and gyroscope sensors Regarding the features extracted from each ADL, five datasets have been similarly constructed with features extracted from the data captured from the accelerometer, magnetometer and gyroscope, again resulting in 2000 records from each ADL. The datasets defined are: • Dataset 5: Composed by the average, and the standard deviation of the raw signal extracted from both accelerometer, magnetometer and gyroscope sensors; • Dataset 4: Composed by the features of the Dataset 5 plus the variance and the median of the raw signal extracted from both accelerometer, magnetometer and gyroscope sensors; • Dataset 3: Composed by the features of the Dataset 4 plus the maximum and the minimum values of the raw signal extracted from both accelerometer, magnetometer and gyroscope sensors; • Dataset 2: Composed by the features of the Dataset 3 plus the average, the standard deviation, the variance and the median of the maximum peaks obtained from the raw signal extracted from both accelerometer, magnetometer and gyroscope sensors; • Dataset 1: Composed by the features of the Dataset 2 plus the five greatest distances between the maximum peaks obtained from the raw signal extracted from both accelerometer, magnetometer and gyroscope sensors. 3.3.3. Classification Based on the results reported by the literature review presented in Section 2, one of the most used methods for the recognition of ADL based on the use of the mobiles’ sensors is the ANN, and this method reports a better accuracy than SVM, KNN, Random Forest, and Naïve Bayes. Following the datasets defined in Sections 3.3.1 and 3.3.2, this study implements three implementations of ANN, such as MLP, FNN, and DNN, in order to identify the best ANN for the recognition of ADL, these are: • MLP method with Backpropagation, applied with the Neuroph framework [16]; • FNN method with Backpropagation, applied with the Encog framework [17]; • DNN method, applied with the DeepLearning4j framework [18]. 111 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 85 Table 4 Configurations of the ANN methods implemented. Parameters MLP FNN DNN Activation function Learning rate Momentum Maximum number of training iterations Number of hidden layers Weight function Seed value Backpropagation Regularization Sigmoid 0.6 0.4 4 × 106 0 N/A N/A Yes N/A Sigmoid 0.6 0.4 4 × 106 0 N/A N/A Yes N/A Sigmoid 0.1 N/A 4 × 106 3 Xavier 6 Yes L2 The configurations between the three implementations of ANN methods used are presented in Table 4, where it is visible that the Sigmoid as activation function and backpropagation parameters are implemented in all methods. The used of more efficient activation functions, resulting in a better energy efficiency of the proposed method, will be subject to additional research. Based on the parameters of the three implementations of ANN presented in Table 4, the architecture of the MLP and the FNN methods is presented in Fig. 2a and the architecture of the DNN method is presented in Fig. 2b. In order to improve the results obtained by the ANN, the MIN/MAX normalizer [49] was applied to the defined datasets, implementing the MLP method with Backpropagation, and the FNN method with Backpropagation with normalized and non-normalized at different stages. Before the application of the DNN method, the L2 regularization [50] was applied to the defined datasets. After the application of the L2 regularization, the normalization with mean and standard deviation [51] was applied to the datasets, implementing the DNN method with normalized and non-normalized data at different stages. The number of training iterations may influence the results of the ANN, defining the maximum number of 106 , 2 × 106 and 4 × 106 iterations, in order to identify the best number of training iterations with best results. After this research, the methods that should be implemented in the framework for the recognition of ADL defined in [4–6] are a function of the number of sensors available in the off-the-shelf mobile device. According to the results available in [15], if the mobile device has only the accelerometer sensor, the method that should be implemented is the DNN method, verifying with this research the best methods for the use of the datasets defined in Sections 3.3.1 and 3.3.2. 4. Results This research consists in the creation of two different methods for the recognition of ADL with different number of sensors. Firstly, the results of the creation of a method with accelerometer and magnetometer sensors are presented in Section 4.1. Finally, the results of the creation of a method with accelerometer, magnetometer, and gyroscope sensors are presented in Section 4.2. 4.1. Identification of activities of daily living with accelerometer and magnetometer sensors Based on the datasets defined in Section 3.3.1, the three implementations of ANN proposed in Section 3.3.3 are implemented with the frameworks proposed, these are the MLP method with Backpropagation, the FNN method with Backpropagation, and the DNN method. The defined training dataset has 10 000 records, where each ADL has 2000 records. Firstly, the results of the implementation of the MLP method with Backpropagation using the Neuroph framework are presented in Fig. 3, verifying that the results have very low accuracy with all datasets, achieving values between 20% and 40% with non-normalized data (Fig. 3a), and values between 20% and 30% with normalized data (Fig. 3b). Secondly, the results of the implementation of the FNN method with Backpropagation using the Encog framework are presented in Fig. 4. In general, this implementation of ANN achieves low accuracy results with both non-normalized and normalized data, reporting the maximum results around 40%. With non-normalized data (Fig. 4a), the ANN reports results above 30% with the dataset 1 trained over 106 and 4 × 106 iterations, the dataset 2 trained over 106 iterations, the dataset 3 trained over 2 × 106 iterations, the dataset 4 trained over 106 , 2 × 106 and 4 × 106 iterations, and the dataset 5 trained over 106 and 4 × 106 iterations. With normalized data (Fig. 4b), the results reported are lower than 40%, with an exception for the ANN trained over 2 × 106 iterations with the dataset 5 that reports results higher than 60%. Finally, the results of the implementation of the DNN method with the DeepLearning4j framework are presented in Fig. 5. With non-normalized data (Fig. 5a), the results obtained are below the expectations (around 20%) for the datasets 2, 3 and 4, and the results obtained with dataset 5 are around 70%. On the other hand, with normalized data (Fig. 5b), the results reported are always higher than 70%, achieving better results with the dataset 1, decreasing with the reduction of the number of features in the dataset. 112 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 86 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 Fig. 2. Schema of the architecture of the different implementations of ANN used in this study. The Figure (a) shows the architecture related to the MLP and FNN methods. The Figure (b) shows the architecture of the DNN method. In Table 5, the maximum accuracies achieved with the different implementations of ANN are presented with the relation of the different datasets used for accelerometer and magnetometer data, and the computational complexity [52] and the maximum number of iterations, verifying that the use of the DNN method with normalized data reports better results than others. The computational complexity taken in account in this study consists on the absolute value of the lower bounds of time complexity calculated with Big-Oh [52]. Regarding the results obtained, in the case of the use of accelerometer and magnetometer sensors in the framework for the identification of ADL, the implementation of ANN that should be used is the DNN method with normalized data, because the results obtained are always higher than 80%. 4.2. Identification of activities of daily living with accelerometer, magnetometer and gyroscope sensors Based on the datasets defined in Section 3.3.2, the three implementations of ANN proposed in Section 3.3.3 are implemented with the frameworks proposed, these are the MLP method with Backpropagation, the FNN method with Backpropagation, and the DNN method. The defined training dataset has 10 000 records, where each ADL has 2000 records. Firstly, the results of the implementation of the MLP method with Backpropagation using the Neuroph framework are presented in Fig. 6, verifying that the results have very low accuracy with all datasets. With non-normalized data (Fig. 6a), the 113 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 87 Fig. 3. Results obtained with the Neuroph framework for the different datasets of accelerometer and magnetometer sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. Fig. 4. Results obtained with the Encog framework for the different datasets of accelerometer and magnetometer sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. Fig. 5. Results obtained with DeepLearning4j framework for the different datasets of accelerometer and magnetometer sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. results achieved are between 20% and 40%, where the better accuracy was achieved with the dataset 2. And, with normalized data (Fig. 6b), the results obtained are between 30% and 40%, with lower results for the dataset 5. Secondly, the results of the implementation of the FNN method with Backpropagation using the Encog framework are presented in Fig. 7. In general, this implementation of ANN achieves low accuracy results with non-normalized and normalized data, reporting the maximum results around 40%. With non-normalized data (Fig. 7a), the ANN reports results above 30% with the dataset 2 trained over 2 × 106 iterations, the dataset 3 trained over 4 × 106 iterations, and the dataset 4 trained over 4 × 106 iterations, reporting an accuracy higher than 70% with the dataset 2 trained over 2 × 106 iterations. With normalized data (Fig. 7b), the results reported are lower than 20%, with an exception for the ANN with the dataset 3 trained over 4 × 106 iterations, the dataset 4 trained over 2 × 106 iterations, and the dataset 5 trained over 106 and 2 × 106 iterations. Finally, the results of the implementation of the DNN method with the DeepLearning4j framework are presented in Fig. 8. With non-normalized data (Fig. 8a), the results obtained are below the expectations (around 40%) for the datasets 2, 3 and 4, and the results obtained with datasets 1 and 5 are around 70%. On the other hand, with normalized data (Fig. 8b), the results reported are always higher than 80%, achieving better results with the dataset 1, decreasing with the reduction of the number of features in the dataset. 114 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 88 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 Fig. 6. Results obtained with the Neuroph framework for the different datasets of accelerometer, magnetometer and gyroscope sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. Fig. 7. Results obtained with the Encog framework for the different datasets of accelerometer, magnetometer and gyroscope sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. Fig. 8. Results obtained with the DeepLearning4j framework for the different datasets of accelerometer, magnetometer and gyroscope sensors (horizontal axis) and different maximum number of iterations (series), obtaining the accuracy in percentage (vertical axis). The Figure (a) shows the results with data without normalization. The Figure (b) shows the results with normalized data. In Table 6, the maximum accuracies achieved with the different implementations of ANN are presented with the relation of the different datasets used for the accelerometer, magnetometer and gyroscope data, and the computational complexity [52] and the maximum number of iterations, verifying that the use of the DNN method with normalized data reports better results than others. The computational complexity taken in account in this study consists on the absolute value of the lower bounds of time complexity calculated with Big-Oh [52]. Regarding the results obtained, in the case of the use of accelerometer, magnetometer and gyroscope sensors in the framework for the identification of ADL, the implementation of ANN that should be used is the DNN method with normalized data, because the results obtained are always higher than 80%, and the best result was achieved with dataset 1 that was equals to 89.51%. 5. Discussion Based on the previous studies available in the literature, there are only 3 studies in the literature that focus on the use of the accelerometer, the gyroscope and the magnetometer sensors, but datasets and detailed specifications of the 115 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 89 Table 5 Best accuracies obtained with the different frameworks, datasets and number of iterations. Framework Not normalized data Normalized data Absolute value of the lower bound of the time complexity Dataset Iterations needed for training 6 Best accuracy achieved (%) Neuroph Encog Deep learning 9.53674 × 10 0.00390625 4 −27 2 5 5 10 106 4 × 106 35.15 42.75 70.43 Neuroph Encog Deep learning 4.85694 × 10−45 0.00390625 4 1 5 1 4 × 106 2 × 106 4 × 106 24.93 64.94 86.49 Iterations needed for training Best accuracy achieved (%) Table 6 Best accuracies obtained with the different frameworks, datasets and number of iterations. Framework Absolute value of the lower bound of the time complexity −45 Datasets 6 Not normalized data Neuroph Encog Deep learning 4.85694 × 10 4.85694 × 10−45 4 2 2 1 10 2 × 106 2 × 106 38.32 76.13 74.47 Normalized data Neuroph Encog Deep learning 4.03122 × 10−75 1.12157 × 10−13 4 1 4 1 2 × 106 2 × 106 4 × 106 37.13 29.54 89.51 Table 7 Best accuracies achieved by the method using only the accelerometer sensor. Framework Best accuracy achieved (%) Not normalized data Neuroph Encog Deep learning 34.76 74.45 80.35 Normalized data Neuroph Encog Deep learning 24.03 37.07 85.89 methods implemented are not available. In line with this, and due to the absence of previous studies focused on the proposed topology, we compared the use of three Java frameworks with different implementations of ANN, i.e., MLP method with Backpropagation, FNN method with Backpropagation and DNN method. The main goal is to research on the most efficient implementation for the recognition of the ADL, in order to may provide both the sequence of the steps used for the recognition of ADL and the comparison based on the accuracy of the recognition of ADL. Following the research for the development of a framework for the identification of the ADL using motion and magnetic sensors, presented in [4–6], there are several modules, such as data acquisition, data cleaning, feature extraction, data fusion, and classification methods. The choice of the methods for data fusion, and classification modules, depends on the number of sensors available on the mobile device, aiming to use the maximum number of sensors available on the mobile device, in order to increase the reliability of the method. According to the previous study based only in the use of the accelerometer sensor for the recognition of ADL, presented in [15], the best results achieved for each implementation of ANN are presented in Table 7, verifying that the best method is the DNN method with normalized data, reporting an accuracy of 85.89%. In the case of the mobile device only has the accelerometer sensor available, DNN method with normalized data should be implemented in the framework for the recognition of ADL, removing the data fusion, as presented in Fig. 1. Based on results obtained with the use of accelerometer and magnetometer sensors, presented in Section 4.1, the comparison of the results between the use of the accelerometer sensor, and the use of accelerometer and magnetometer sensors is presented in Table 8. In general, the accuracy increases with the use of normalized data, and decreases with the use of non-normalized data, where the highest difference was verified with the use of the accelerometer and magnetometer sensors with the implementation of FNN method with Backpropagation using the Encog framework, reporting a difference of 27.87%. However, the DNN method continues to report the best results with an accuracy of 86.49%. In the case of the mobile device only having the accelerometer and magnetometer sensors available, the DNN method with normalized data should be implemented in the framework for the recognition of ADL. Based on results obtained with the use of accelerometer, magnetometer and gyroscope sensors, presented in Section 4.2, the comparison of the results between the use of the accelerometer sensor, and the use of accelerometer, magnetometer and gyroscope sensors is presented in Table 9. In general, the accuracy increases, except in the cases of the use of the DNN method with non-normalized data and the FNN method with Backpropagation using the Encog framework with normalized data. The highest difference in the accuracy is verified with the use of MLP method with Backpropagation using the Neuroph framework, where the accuracy results increased 13.1% with normalized data, but the DNN method achieves better results with an accuracy of 89.51%. However, the computational complexity of the DNN method is higher 116 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 90 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 Table 8 Comparison between the best results achieved only using the accelerometer sensor, and using the accelerometer and magnetometer sensors. Framework Best accuracy achieved (%) Accelerometer Accelerometer Magnetometer Difference (%) Not normalized data Neuroph Encog Deep learning 34.76 74.45 80.35 35.15 42.75 70.43 +0.39 −31.70 −9.92 Normalized data Neuroph Encog Deep learning 24.03 37.07 85.89 24.93 64.94 86.49 +0.90 +27.87 +0.60 Table 9 Comparison between the best results achieved only using the accelerometer sensor, and using the accelerometer, magnetometer and gyroscope sensors. Framework Best accuracy achieved (%) Accelerometer Accelerometer Magnetometer Gyroscope Difference (%) Not normalized data Neuroph Encog Deep learning 34.76 74.45 80.35 38.32 76.13 74.47 +3.56 +1.46 −5.88 Normalized data Neuroph Encog Deep learning 24.03 37.07 85.89 37.13 29.54 89.51 +13.10 −7.53 +3.62 Table 10 Comparison between the best results achieved only using the accelerometer and magnetometer sensors, and using the accelerometer, magnetometer and gyroscope sensors. Framework Best accuracy achieved (%) Accelerometer Accelerometer Magnetometer Magnetometer Gyroscope Difference (%) Not normalized data Neuroph Encog Deep learning 35.15 42.75 70.43 38.32 76.13 74.47 +3.17 +33.38 +4.04 Normalized data Neuroph Encog Deep learning 24.93 64.94 86.49 37.13 29.54 89.51 +12.20 −35.40 +3.02 than the MLP and FNN methods, but the results obtained are significantly higher than the others in the recognition of the proposed ADL. Based on results obtained with the use of accelerometer, magnetometer and gyroscope sensors, presented in Section 4.2, and the results obtained with the use of accelerometer and magnetometer sensors, presented in Section 4.1, the comparison between these results is presented in Table 10. In general, the accuracy increases, except in the case of the use of the FNN method with Backpropagation using the Encog framework with normalized data. The highest difference in the accuracy is verified with the use of the FNN method with Backpropagation using the Encog framework with non-normalized data, where the accuracy results increased 33.38% with non-normalized data, but the DNN method continues achieving the better results with an accuracy of 89.51%. Thus, in the case of the mobile device has the accelerometer, magnetometer and gyroscope sensors available, the DNN method with normalized data should be implemented in the framework for the recognition of ADL. In conclusion, when compared with MLP method with Backpropagation using the Neuroph framework and FNN method with Backpropagation using the Encog framework, the DNN method with normalized data achieves better results for the recognition of the ADL with accuracies between 85% and 90% with the different datasets. The fusion of the data acquired from the sensors available in the mobile device requires the acquisition, processing, fusion and classification of the different sensors’ data as defined in the schema of the framework proposed (see Fig. 1). The energy consumptions for the different sensors is very different, being the accelerometer consumption range of one order of magnitude, the magnetometer in the range of two orders of magnitude, and the gyroscope in the range of three orders of magnitude, unbalancing and increasing the resources (e.g., battery and power processing capabilities) needed for the execution of the framework. However, as presented in [47], with the method proposed in this study the effects are minimized and only a daily recharge is needed for the execution of the framework with the three sensors. The devices used in [47] were used to study different combinations of sensors both in number and type, including a configuration where the oldest devices had only the accelerometer sensor, and the remaining devices were used for other configurations. Regarding 117 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 91 the use of only the accelerometer and the use of the fusion of one accelerometer and one magnetometer, the accuracy has a marginal improvement (+0.60%), but it increases the reliability of the framework and reduces the problems with the possible failures of one of the sensors during data acquisition. Finally, the use of the three sensors improves the accuracy of the framework in 3.02%, the main problem consisting in the fact that some commercially available devices do not integrate the set comprising the gyroscope, magnetometer and accelerometer sensors. In conclusion, data fusion increases the reliability of the framework, proving the viability of the use of the proposed framework. 6. Conclusions The sensors that are available in the mobile devices, including accelerometer, gyroscope, and magnetometer sensors, allow the capture of data that can be used to the recognition of ADL [2]. This study focused on the architecture defined in [4–6], composed by data acquisition, data cleaning, feature extraction, data fusion and classification methods. Based on the literature review, the proposed ADL for the recognition with motion and magnetic sensors are running, walking, going upstairs, going downstairs, and standing. Based on the data acquired, several features must be extracted from the sensors’ signal, such as the five greatest distances between the maximum peaks, the average, standard deviation, variance, and median of the maximum peaks, plus the average, the standard deviation, the variance, the median, and the minimum, the maximum of the raw signal have been extracted from the accelerometer, magnetometer and/or gyroscope sensors available on the off-the-shelf mobile device. The fusion of the features implemented in the framework for the recognition of ADL should be a function of the number of sensors available. The method implemented in the framework for the recognition of ADL should also adapted with the software and hardware limitations of the mobile devices, including the number of sensors available and the low memory and power processing capabilities. For the development of the framework for the recognition of the ADL, three implementations of ANN were created in order to identify the best framework and implementation of ANN for the development of each step of the framework for the recognition of ADL, such as the MLP method with Backpropagation using the Neuroph framework [16], the FNN method with Backpropagation using the Encog framework [17], and the DNN method using DeepLearning4j framework [18], verifying that the DNN method achieves better results than others. Due to the limitations of mobile devices and regarding the results obtained with the method for the recognition of ADL with the accelerometer previously performed, presented in [15], it was verified that the best results were obtained with the DNN method with L2 regularization and normalized data with an accuracy of 85.89%. Related to the development of a method for the recognition of ADL, this study proves that the best accuracy are always achieved with the DNN method with L2 regularization and normalized data, and the data fusion increases the accuracy of the method, reporting an accuracy of 86.49% with the fusion of the data acquired from two sensors (i.e., accelerometer and magnetometer), and 89.51% with the fusion of the data acquired from three sensors (i.e., accelerometer, magnetometer and gyroscope). On the other hand, the MLP method with Backpropagation and the FNN method with Backpropagation achieve low accuracies, because the networks are overfitting, and this problem may be solved with several strategies, these being the stopping of the training when the network error increases for several iterations, the application of dropout regularization, the application of L2 regularization, the application of the batch normalization, or the reduction of the number of features in the ANN implemented. As future work, the methods for the recognition of ADL presented in this study should be implemented during the development of the framework for the identification of ADL, adapting the method to the number of sensors available on the mobile device, but the method that should be implemented is the DNN method. The data related to this research is available in a free repository [46]. Acknowledgments This work was supported by FCT project UID/EEA/50008/2013 (Este trabalho foi suportado pelo projecto FCT UID/EEA/50008/2013). The authors would also like to acknowledge the contribution of the COST Action IC1303 – AAPELE – Architectures, Algorithms and Protocols for Enhanced Living Environments. References [1] L.H.A. Salazar, T. Lacerda, J.V. Nunes, C. Gresse von Wangenheim, A systematic literature review on usability heuristics for mobile phones, Int. J. Mob. Hum. Comput. Interact. 5 (2013) 50–61. http://dx.doi.org/10.4018/jmhci.2013040103. [2] D. Foti, J.S. Koketsu, Activities of daily living, in: Pedretti’s Occupational Therapy: Practical Skills for Physical Dysfunction, vol. 7 2013, pp. 157–232. [3] N.M. Garcia, A roadmap to the design of a personal digital life coach, in: ICT Innovations 2015, Springer, 2016. [4] I. Pires, N. Garcia, N. Pombo, F. Flórez-Revuelta, From data acquisition to data fusion: A comprehensive review and a roadmap for the identification of activities of daily living using mobile devices, Sensors 16 (2016) 184. [5] I.M. Pires, N.M. Garcia, F. Flórez-Revuelta, Multi-sensor data fusion techniques for the identification of activities of daily living using mobile devices, in: Proceedings of the ECMLPKDD 2015 Doctoral Consortium, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Porto, Portugal, 2015. 118 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 92 I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 [6] I.M. Pires, N.M. Garcia, N. Pombo, F. Flórez-Revuelta, Identification of activities of daily living using sensors available in off-the-shelf mobile devices: Research and hypothesis, in: Ambient Intelligence-Software and Applications–7th International Symposium on Ambient Intelligence, ISAmI 2016, 2016, pp. 121–130. [7] O. Banos, M. Damas, H. Pomares, I. Rojas, On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition, Sensors (Basel) 12 (2012) 8039–8054. http://dx.doi.org/10.3390/s120608039. [8] M.A.A. Akhoundi, E. Valavi, Multi-sensor fuzzy data fusion using sensors with different characteristics, 2010, arXiv preprint arXiv:1010.6096. [9] P. Paul, T. George, An effective approach for human activity recognition on smartphone, in: 2015 IEEE International Conference on Engineering and Technology, ICETech, 2015, pp. 45–47. http://dx.doi.org/10.1109/icetech.2015.7275024. [10] Y.-W. Hsu, K.-H. Chen, J.-J. Yang, F.-S. Jaw, Smartphone-based fall detection algorithm using feature extraction, in: 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, CISP-BMEI, Datong, China, 2016, pp. 1535–1540. [11] S. Dernbach, B. Das, N.C. Krishnan, B.L. Thomas, D.J. Cook, Simple and complex activity recognition through smart phones, in: 2012 8th International Conference on Intelligent Environments, IE, Guanajuato, Mexico, 2012, pp. 214–221. [12] C. Shen, Y.F. Chen, G.S. Yang, On motion-sensor behavior analysis for human-activity recognition via smartphones, in: 2016 IEEE International Conference on Identity, Security and Behavior Analysis, Isba, Sendai, Japan, 2016, pp. 1–6. [13] D. Wang, Pattern recognition: neural networks in perspective, IEEE Expert 8 (1993) 52–60. http://dx.doi.org/10.1109/64.223991. [14] K. Doya, D. Wang, Exciting time for neural networks, Neural Netw. 61 (2015) xv–xvi. http://dx.doi.org/10.1016/s0893-6080(14)00260-3. [15] I.M. Pires, N.M. Garcia, N. Pombo, F. Flórez-Revuelta, S. Spinsante, Pattern recognition techniques for the identification of activities of daily living using mobile device accelerometer, 2017, submitted for publication, ed. arXiv:1711.00096. [16] Neuroph, Java Neural Network Framework Neuroph, 2 Sep. 2017. Available: http://neuroph.sourceforge.net/. [17] H. Research, Encog Machine Learning Framework, 2 Sep. 2017. Available: http://www.heatonresearch.com/encog/. [18] A. Chris Nicholson, Deeplearning4j: Open-source, Distributed Deep Learning for the JVM, 2 Sep. 2017. Available: https://deeplearning4j.org/. [19] Q. Guo, B. Liu, C.W. Chen, A two-layer and multi-strategy framework for human activity recognition using smartphone, in: 2016 IEEE International Conference on Communications, ICC, 2016, pp. 1–6, http://dx.oi.org/d10.1109/icc.2016.7511487. [20] M. Shoaib, H. Scholten, P.J.M. Havinga, Towards physical activity recognition using smartphone sensors, in: 2013 IEEE 10th International Conference on and 10th International Conference on Autonomic and Trusted Computing (UIC/ATC) Ubiquitous Intelligence and Computing, 2013, pp. 80–87. http://dx.doi.org/10.1109/Uic-Atc.2013.43. [21] M. Elhoushi, J. Georgy, A. Wahdan, M. Korenberg, A. Noureldin, Using portable device sensors to recognize height changing modes of motion, in: 2014 IEEE International Instrumentation and Measurement Technology Conference, I2MTC, Proceedings, 2014, pp. 477–481. http://dx.doi.org/10.1109/ i2mtc.2014.6860791. [22] C.A. Ronao, S.B. Cho, Human activity recognition using smartphone sensors with two-stage continuous hidden Markov models, in: 2014 10th International Conference on Natural Computation, ICNC, 2014, pp. 681–686. http://dx.doi.org/10.1109/icnc.2014.6975918. [23] Y.Y. Liu, F. Zhao, W.H. Shao, H.Y. Luo, An hidden Markov model based complex walking pattern recognition algorithm, in: Proceedings of 2016 Fourth International Conference on Ubiquitous Positioning, Indoor Navigation and Location Based Services, IEEE UPINLBS 2016, 2016, pp. 223–229. http://dx.doi.org/10.1109/upinlbs.2016.7809976. [24] W.C. Hung, F. Shen, Y.L. Wu, M.K. Hor, C.Y. Tang, Activity recognition with sensors on mobile devices, in: Proceedings of 2014 International Conference on Machine Learning and Cybernetics, ICMLC, Vol. 2, 2014, pp. 449–454. http://dx.doi.org/10.1109/icmlc.2014.7009650. [25] M.B. Rasheed, N. Javaid, T.A. Alghamdi, S. Mukhtar, U. Qasim, Z.A. Khan, et al., Evaluation of human activity recognition and fall detection using android phone, in: 2015 IEEE 29th International Conference on Advanced Information Networking and Applications, IEEE AINA 2015, 2015, pp. 163–170. http://dx.doi.org/10.1109/Aina.2015.181. [26] F. Kawsar, M.K. Hasan, R. Love, S.I. Ahamed, A novel activity detection system using plantar pressure sensors and smartphone, in: 39th Annual IEEE Computers, Software and Applications Conference, COMPSAC 2015, Vol. 1, 2015, pp. 44–49. http://dx.doi.org/10.1109/Compsac.2015.201. [27] V. Simonetti, W. Baccinelli, M. Bulgheroni, E. d’Amico, Free context smartphone based application for motor activity levels recognition, in: 2016 IEEE 2nd International Forum on Research and Technologies for Society and Industry Leveraging a Better Tomorrow, RTSI, 2016, pp. 292–295. http://dx.doi.org/10.1109/rtsi.2016.7740601. [28] Y.F. Chen, C. Shen, Performance analysis of smartphone-sensor behavior for human activity recognition, IEEE Access 5 (2017) 3095–3110. http: //dx.doi.org/10.1109/access.2017.2676168. [29] Y. Tian, W.J. Chen, MEMS-based human activity recognition using smartphone, in: Proceedings of the 35th Chinese Control Conference 2016, 2016. pp. 3984–3989. http://dx.doi.org/10.1109/ChiCC.2016.7553975. [30] P. Vallabh, R. Malekian, N. Ye, D.C. Bogatinoska, Fall detection using machine learning algorithms, in: 2016 24th International Conference on Software, Telecommunications and Computer Networks, Softcom, 2016, pp. 51–59. http://dx.doi.org/10.1109/softcom.2016.7772142. [31] C.X. Tang, V.V. Phoha, An empirical evaluation of activities and classifiers for user identification on smartphones, in: 2016 IEEE 8th International Conference on Biometrics Theory, Applications and Systems, BTAS, 2016, pp. 1–8. http://dx.doi.org/10.1109/btas.2016.7791159. [32] T.H. Sanders, M.A. Clements, Multimodal monitoring for neurological disorders, in: 2014 40th Annual Northeast Bioengineering Conference, NEBEC, 2014, pp. 1-2. http://dx.doi.org/10.1109/nebec.2014.6972928. [33] A. Bajpai, V. Jilla, V.N. Tiwari, S.M. Venkatesan, R. Narayanan, Quantifiable fitness tracking using wearable devices, in: Conf Proc IEEE Eng Med Biol Soc, Vol. 2015, Aug. 2015, pp. 1633–1637. http://dx.doi.org/10.1109/EMBC.2015.7318688. [34] F. De Cillisy, F. De Simioy, F. Guidoy, R.A. Incalzi, R. Setolay, Fall-detection solution for mobile platforms using accelerometer and gyroscope data, in: Conf Proc IEEE Eng Med Biol Soc, Vol. 2015, Aug. 2015, pp. 3727–3730. http://dx.doi.org/10.1109/EMBC.2015.7319203. [35] Q.C. Zhu, Z.H. Chen, Y.C. Soh, Smartphone-based human activity recognition in buildings using locality-constrained linear coding, in: Proceedings of the 2015 10th IEEE Conference on Industrial Electronics and Applications, 2015, pp. 214–219. http://dx.doi.org/10.1109/iciea.2015.7334113. [36] Y.Y. Fan, L. Xie, Y.F. Yin, S.L. Lu, A context aware energy-saving scheme for smart camera phones based on activity sensing, in: 2015 IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems, MASS, 2015, pp. 64–72. http://dx.doi.org/10.1109/Mass.2015.17. [37] P. Li, Y. Wang, Y. Tian, T.S. Zhou, J.S. Li, An automatic user-adapted physical activity classification method using smartphones, IEEE Trans. Biomed. Eng. 64 (2017) 706–714. http://dx.doi.org/10.1109/TBME.2016.2573045. [38] M.T. Uddin, M.M. Billah, M.F. Hossain, Random forests based recognition of human activities and postural transitions on smartphone, in: 2016 5th International Conference on Informatics, Electronics and Vision, ICIEV, 2016, pp. 250–255. http://dx.doi.org/10.1109/iciev.2016.7760005. [39] L. Fang, S. Yishui, C. Wei, Up and down buses activity recognition using smartphone accelerometer, in: 2016 IEEE Information Technology, Networking, Electronic and Automation Control Conference, 2016, pp. 761–765. [40] M. Altini, R. Vullers, C. Van Hoof, M. van Dort, O. Amft, Self-calibration of walking speed estimations using smartphone sensors, in: 2014 IEEE International Conference on Pervasive Computing and Communications Workshops, Percom Workshops, 2014, pp. 10–18. http://dx.doi.org/10.1109/ PerComW.2014.6815158. [41] J. Windau, L. Itti, Situation awareness via sensor-equipped eyeglasses, in: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, 2013, pp. 5674–5679. http://dx.doi.org/10.1109/iros.2013.6697178. 119 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living I.M. Pires et al. / Pervasive and Mobile Computing 47 (2018) 78–93 93 [42] G. Vavoulas, C. Chatzaki, T. Malliotakis, M. Pediaditis, M. Tsiknakis, The MobiAct dataset: Recognition of activities of daily living using smartphones, in: Proceedings of the International Conference on Information and Communication Technologies for Ageing Well and E-Health, ICT4AWE, 2016, pp. 143–151. http://dx.doi.org/10.5220/0005792401430151. [43] H. Bojinov, Y. Michalevsky, G. Nakibly, D. Boneh, Mobile device identification via sensor fingerprinting, 2014, arXiv preprint arXiv:1408.1416. [44] K. Katevas, H. Haddadi, L. Tokarchuk, Sensingkit: Evaluating the sensor power consumption in IOS devices, in: Intelligent Environments, IE, 2016 12th International Conference on, 2016, pp. 222–225. [45] Bq.com., Smartphones BQ Aquaris / BQ Portugal, 2 Sep. 2017. Available: https://www.bq.com/pt/smartphones. [46] ALLab, August 2017- Multi-sensor data fusion in mobile devices for the identification of activities of daily living - ALLab Signals, September 2nd. Available: https://allab.di.ubi.pt/mediawiki/index.php/August_2017-_Multi-sensor_data_fusion_in_mobile_devices_for_the_identification_of_ activities_of_daily_living. [47] I.M. Pires, N.M. Garcia, N. Pombo, F. Flórez-Revuelta, Limitations of the use of mobile devices and smart environments for the monitoring of ageing people, in: ICT4AWE 2018 4th International Conference on Information and Communication Technologies for Ageing Well and e-Health, Madeira, Portugal, 2018. [48] V. Graizer, Effect of low-pass filtering and re-sampling on spectral and peak ground acceleration in strong-motion records, in: Proc. 15th World Conference of Earthquake Engineering, Lisbon, Portugal, 2012, pp. 24–28. [49] A. Jain, K. Nandakumar, A. Ross, Score normalization in multimodal biometric systems, Pattern Recognit. 38 (2005) 2270–2285. http://dx.doi.org/10. 1016/j.patcog.2005.01.012. [50] A.Y. Ng, Feature selection, L 1 vs. L 2 regularization, and rotational invariance, in: Proceedings Of the Twenty-First International Conference on Machine Learning, 2004, p. 78. [51] L. Brocca, F. Melone, T. Moramarco, W. Wagner, V. Naeimi, Z. Bartalis, et al., Improving runoff prediction through the assimilation of the ASCAT soil moisture product, Hydrol. Earth Syst. Sci. 14 (2010) 1881–1893. http://dx.doi.org/10.5194/hess-14-1881-2010. [52] M. Bianchini, F. Scarselli, On the complexity of shallow and deep neural network classifiers, in: ESANN, 2014. 120 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 2. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions The following article is the second part of the chapter 4. Android library for recognition of activities of daily living: implementation considerations, challenges, and solutions Ivan Miguel Pires, Maria Canavarro Teixeira, Nuno Pombo, Nuno M. Garcia, Francisco FlórezRevuelta, Susanna Spinsante, Rossitza Goleva and Eftim Zdravevski The Open Bioinformatics Journal (Bentham Science Publishers B.V.), published, 2018. According to 2016 Journal Citation Reports published by SCImago in 2017, this journal has the following performance metrics: CiteScore (2016): 4.86 Journal Ranking (2016): 2/32 (Computer Science (miscellaneous)) 121 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 122 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 6HQG2UGHUVIRU5HSULQWVWRUHSULQWV#EHQWKDPVFLHQFHDH  7KH2SHQ%LRLQIRUPDWLFV-RXUQDO 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO &RQWHQWOLVWDYDLODEOHDWZZZEHQWKDPRSHQFRP72%,2,- '2, 5(6($5&+$57,&/( $QGURLG /LEUDU\ IRU 5HFRJQLWLRQ RI $FWLYLWLHV RI 'DLO\ /LYLQJ ,PSOHPHQWDWLRQ&RQVLGHUDWLRQV&KDOOHQJHVDQG6ROXWLRQV      0DULD &DQDYDUUR 7HL[HLUD  1XQR 3RPER  1XQR 0 *DUFLD  )UDQFLVFR ,YDQ 0LJXHO 3LUHV )OhUH]5HYXHOWD6XVDQQD6SLQVDQWH5RVVLW]D*ROHYDDQG(IWLP=GUDYHYVNL  ,QVWLWXWRGH7HOHFRPXQLFDdeHV8QLYHUVLGDGHGD%HLUD,QWHULRU&RYLOKc3RUWXJDO $OWUDQSRUWXJDO/LVERQ3RUWXJDO  $//DE  $VVLVWHG /LYLQJ &RPSXWLQJ DQG 7HOHFRPPXQLFDWLRQV /DERUDWRU\ &RPSXWLQJ 6FLHQFH 'HSDUWPHQW 8QLYHUVLGDGH GD %HLUD ,QWHULRU &RYLOKc 3RUWXJDO  87&GH5HFXUVRV1DWXUDLVH'HVHQYROYLPHQWR6XVWHQWbYHO3RO\WHFKQLTXH,QVWLWXWHRI&DVWHOR%UDQFR&DVWHOR%UDQFR 3RUWXJDO  &(51$6  5HVHDUFK &HQWUH IRU 1DWXUDO 5HVRXUFHV (QYLURQPHQW DQG 6RFLHW\ 3RO\WHFKQLTXH ,QVWLWXWH RI &DVWHOR %UDQFR&DVWHOR%UDQFR3RUWXJDO  'HSDUWPHQWRI&RPSXWHU7HFKQRORJ\8QLYHUVLGDGGH$OLFDQWH$OLFDQWH6SDLQ  'HSDUWPHQWRI,QIRUPDWLRQ(QJLQHHULQJ0DUFKH3RO\WHFKQLF8QLYHUVLW\$QFRQD,WDO\  'HSDUWPHQWRI,QIRUPDWLFV1HZ%XOJDULDQ8QLYHUVLW\6RILD%XOJDULD  )DFXOW\RI&RPSXWHU6FLHQFHDQG(QJLQHHULQJ8QLYHUVLW\6V&\ULODQG0HWKRGLXV6NRSMH0DFHGRQLD  5HFHLYHG)HEUXDU\ 5HYLVHG$SULO $FFHSWHG$SULO $EVWUDFW %DFNJURXQG 2IIWKHVKHOIPRELOHGHYLFHVKDYHVHYHUDOVHQVRUVDYDLODEOHRQERDUGWKDWPD\EHXVHGIRUWKHUHFRJQLWLRQRI$FWLYLWLHVRI'DLO\ /LYLQJ $'/ DQGWKHHQYLURQPHQWVZKHUHWKH\DUHSHUIRUPHG7KLVUHVHDUFKLVIRFXVHGRQWKHGHYHORSPHQWRI$PELHQW$VVLVWHG /LYLQJ $$/ V\VWHPVXVLQJPRELOHGHYLFHVIRUWKHDFTXLVLWLRQRIWKHGLIIHUHQWW\SHVRIGDWDUHODWHGWRWKHSK\VLFDODQGSK\VLRORJLFDO FRQGLWLRQVRIWKHVXEMHFWVDQGWKHHQYLURQPHQWV0RELOHGHYLFHVZLWKWKH$QGURLG2SHUDWLQJ6\VWHPVDUHWKHOHDVWH[SHQVLYHDQG H[KLELWWKHELJJHVWPDUNHWZKLOHSURYLGLQJDYDULHW\RIPRGHOVDQGRQERDUGVHQVRUV 2EMHFWLYH 7KLVSDSHUGHVFULEHVWKHLPSOHPHQWDWLRQFRQVLGHUDWLRQVFKDOOHQJHVDQGVROXWLRQVDERXWDIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQG WKHHQYLURQPHQWVSURYLGHGDVDQ$QGURLGOLEUDU\7KHIUDPHZRUNLVDIXQFWLRQRIWKHQXPEHURIVHQVRUVDYDLODEOHLQGLIIHUHQWPRELOH GHYLFHVDQGXWLOL]HVDYDULHW\RIDFWLYLW\UHFRJQLWLRQDOJRULWKPVWRSURYLGHDUDSLGIHHGEDFNWRWKHXVHU 0HWKRGV 7KH$QGURLGOLEUDU\LQFOXGHVGDWDIXVLRQGDWDSURFHVVLQJIHDWXUHVHQJLQHHULQJDQGFODVVLILFDWLRQPHWKRGV7KHVHQVRUVWKDWPD\EH XVHGDUHWKHDFFHOHURPHWHUWKHJ\URVFRSHWKHPDJQHWRPHWHUWKH*OREDO3RVLWLRQLQJ6\VWHP *36 UHFHLYHUDQGWKHPLFURSKRQH 7KHGDWDSURFHVVLQJLQFOXGHVWKHDSSOLFDWLRQRIGDWDFOHDQLQJPHWKRGVDQGWKHH[WUDFWLRQRIIHDWXUHVZKLFKDUHXVHGZLWK'HHS 1HXUDO1HWZRUNV '11 IRUWKHFODVVLILFDWLRQRI$'/DQGHQYLURQPHQW7KURXJKRXWWKLVZRUNWKHOLPLWDWLRQVRIWKHPRELOHGHYLFHV ZHUHH[SORUHGDQGWKHLUHIIHFWVKDYHEHHQPLQLPL]HG $GGUHVVFRUUHVSRQGHQFHWRWKLVDXWKRUDWWKH&RPSXWHU6FLHQFH'HSDUWPHQW8QLYHUVLGDGHGD%HLUD,QWHULRU5XD0DUTXeVGlmYLODH%RODPD &RYLOKb3RUWXJDO7HO(PDLOLPSLUHV#LWXELSW  %HQWKDP2SHQ 123 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO 5HVXOWV 7KHLPSOHPHQWDWLRQRIWKH$QGURLGOLEUDU\UHSRUWHGDQRYHUDOODFFXUDF\EHWZHHQDQGGHSHQGLQJRQWKHQXPEHURI VHQVRUV XVHG DQG WKH QXPEHU RI $'/ DQG HQYLURQPHQWV UHFRJQL]HG &RPSDUHG ZLWK WKH UHVXOWV DYDLODEOH LQ WKH OLWHUDWXUH WKH SHUIRUPDQFHRIWKHOLEUDU\UHSRUWHGDPHDQLPSURYHPHQWRIDQGWKH\GRQRWGLIIHUDWWKHPD[LPXPIRXQGLQSULRUZRUNWKDW EDVHGRQWKH6WXGHQWlVWWHVW &RQFOXVLRQ 7KLVVWXG\SURYHVWKDW$'/OLNHZDONLQJJRLQJXSVWDLUVDQGGRZQVWDLUVUXQQLQJZDWFKLQJ79GULYLQJVOHHSLQJDQGVWDQGLQJ DFWLYLWLHV DQG WKH EHGURRP FRRNLQJNLWFKHQ J\P FODVVURRP KDOO OLYLQJ URRP EDU OLEUDU\ DQG VWUHHW HQYLURQPHQWV PD\ EH UHFRJQL]HGZLWKWKHVHQVRUVDYDLODEOHLQRIIWKHVKHOIPRELOHGHYLFHV)LQDOO\WKHVHUHVXOWVPD\DFWDVDSUHOLPLQDU\UHVHDUFKIRUWKH GHYHORSPHQWRIDSHUVRQDOGLJLWDOOLIHFRDFKZLWKDPXOWLVHQVRUPRELOHGHYLFHFRPPRQO\XVHGGDLO\ .H\ZRUGV$FWLYLWLHVRIGDLO\OLYLQJ6HQVRUV0RELOHGHYLFHV3DWWHUQUHFRJQLWLRQ'DWDIXVLRQ$QGURLGOLEUDU\$UWLILFLDOQHXUDO QHWZRUNV5HFRJQLWLRQ ,1752'8&7,21 %DFNJURXQG 0RELOHGHYLFHVOLNHVPDUWSKRQHVFRPPRQO\XVHGLQGDLO\OLIHFDQEHFODVVLILHGDVPXOWLVHQVRUGHYLFHV>@EHFDXVH WKH\KRVWVHYHUDOVHQVRUVHJDFFHOHURPHWHUJ\URVFRSHPDJQHWRPHWHUPLFURSKRQH*OREDO3RVLWLRQLQJ6\VWHP *36 UHFHLYHU DQG FDPHUD WKDW DOORZ WKH DFTXLVLWLRQ RI GLIIHUHQW W\SHV RI GDWD LQFOXGLQJ VRPH RI WKH SK\VLFDO DQG SK\VLRORJLFDOGDWDSHUWDLQLQJWRWKHXVHUWKXVDOORZLQJIRUWKHUHFRJQLWLRQRIWKH$FWLYLWLHVRI'DLO\/LYLQJ $'/ > @ 7KH GHYHORSPHQW RI $PELHQW $VVLVWHG /LYLQJ  $$/ V\VWHPV KDV PDQ\ FKDOOHQJHV >  @ VXFK DV WLPHO\ DQG DFFXUDWHUHFRJQLWLRQRI$'/>@GHVLJQLQJDUFKLWHFWXUHVWKDWDUHDSSOLFDEOHIRU(QKDQFHG/LYLQJ(QYLURQPHQWV> @SURYLGLQJUHOLDEOHV\VWHPVDQGLQWHJUDWLRQZLWKRWKHUV\VWHPV>@$'/UHFRJQLWLRQLVDOVRLPSRUWDQWWRGHVLJQD SHUVRQDOGLJLWDOOLIHFRDFK>@7KHGHYHORSPHQWRIWKHVHV\VWHPVLVVSHFLDOO\LPSRUWDQWWRVXSSRUWWKHDXWRQRP\RI ROGHUXVHUVSDWLHQWVZLWKFKURQLFGLVHDVHVDQGXVHUVWKDWPD\KDYHVRPHW\SHRIGLVDELOLW\>@+RZHYHUWKHVH V\VWHPVPD\EHXVHIXOIRUHYHU\RQHLQFOXGLQJDWKOHWHVDQG\RXQJXVHUVDVWKHSURSRVHGIUDPHZRUNFDQEHLQWHJUDWHG LQWRDWRROIRUWKHPRQLWRULQJDQGWUDLQLQJRIOLIHVW\OHV>@ 0RWLYDWLRQ 7KHUHFRJQLWLRQRI$'/DQGWKHHQYLURQPHQWZKHUHWKHVXEMHFWLVDWDJLYHQPRPHQWLVZLGHO\LPSRUWDQWIRUWKH FUHDWLRQ RI V\VWHPV DLPHG DW VHYHUDO SXUSRVHV VXFK DV WKH SHUVRQDO FRDFKLQJ WKH KHDOWK PRQLWRULQJ WKH OLIHVW\OH PRQLWRULQJDQGRWKHUV1RZDGD\VVXFKUHFRJQLWLRQLVFRPPRQO\SHUIRUPHGE\FRPSOH[DQGH[SHQVLYHV\VWHPVZLWK KLJK SRZHU SURFHVVLQJ DQG PHPRU\ FDSDELOLWLHV >     @ EXW LW FRXOG EH SHUIRUPHG ZLWK FRPPRQO\ XVHG HTXLSPHQWEDVHGRQORFDOSURFHVVLQJWHFKQLTXHVLQRUGHUWRSURYLGHDIHHGEDFNUHODWHGWRWKH$'/SHUIRUPHGDQG HQYLURQPHQWIUHTXHQWHGLQDOPRVWUHDOWLPH 2XUPRWLYDWLRQLVWRVROYHWKLVSUREOHPZLWKWKHGHYHORSPHQWRIDQ$QGURLGOLEUDU\WKDWSHUIRUPVWKHUHFRJQLWLRQRI $'/DQGWKHDVVRFLDWHGHQYLURQPHQWVXVLQJRQO\WKHGDWDDFTXLUHGIURPWKHVHQVRUVDYDLODEOHRQERDUGRIIWKHVKHOI PRELOHGHYLFHV LHVPDUWSKRQHV DQGORFDOSURFHVVLQJ:HVHOHFWHGWKH$QGURLGSODWIRUPIRUWKHGHYHORSPHQWRIWKH OLEUDU\EHFDXVHWKLVLVWKHPRVWXVHGRSHQSODWIRUPDYDLODEOHLQWKHPDUNHW>@DQGWKHDUFKLWHFWXUHRIDQ$QGURLG OLEUDU\IRUWKHUHFRJQLWLRQRI$'/DQGHQYLURQPHQWVKDVEHHQDOUHDG\SURSRVHGLQ>@ 7KHXVHRIWKHRIIWKHVKHOIPRELOHGHYLFHVIRUWKLVSXUSRVHUHTXLUHVWKHGHYHORSPHQWRIOLJKWZHLJKWSURFHVVLQJ WHFKQLTXHV ,Q IDFW GHVSLWH UHFHQW GHYHORSPHQWV DQG LPSURYHPHQWV WKHVH GHYLFHV XVXDOO\ KDYH VHYHUDO OLPLWDWLRQV LQFOXGLQJUHGXFHGSURFHVVLQJFDSDELOLWLHVORZFDSDFLW\EDWWHULHVDQGVPDOOLQWHUQDOVWRUDJHVSDFH +RZHYHU EDVHG RQ SUHYLRXV VWXGLHV >  @ ZH FRQILUP WKDW WKH XVH RI WKH VHQVRUV DYDLODEOH LQ RIIWKHVKHOI PRELOHGHYLFHVPD\IXHOWKHGHYHORSPHQWRI$'/UHFRJQLWLRQV\VWHPVKDYLQJWKHSRWHQWLDOWRLPSURYHWKHTXDOLW\RI OLIH RI LWV XVHUV :LWK WKLV VWXG\ ZH SUHVHQW D UHDOWLPH PRELOH DSSOLFDWLRQ WKDW UHOLDEO\ LGHQWLILHV $'/V WKH HQYLURQPHQW DQG ORFDWLRQ SURYLGLQJ IHHGEDFN WR WKH XVHU 7KH UHFRJQLWLRQ RI WKH $'/ IHDWXULQJ WKH VXEMHFWlV PRYHPHQWLVEDVHGRQWKHGDWDDFTXLUHGIURPWKHPRWLRQDQGRUPDJQHWLFVHQVRUV LHDFFHOHURPHWHUJ\URVFRSHDQG PDJQHWRPHWHU EXWWKHUHFRJQLWLRQRIWKHHQYLURQPHQWLVEDVHGRQDFRXVWLFGDWDDFTXLUHGIURPWKHPLFURSKRQH,QRUGHU 124 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH WRVXSSRUWWKHEHWWHUUHFRJQLWLRQRIDQ$'/ZKHQWKHXVHULVVWDQGLQJVWLOOWKHIXVLRQRIWKHIHDWXUHVH[WUDFWHGIURPWKH GDWDDFTXLUHGIURPPRWLRQDQGPDJQHWLFVHQVRUVZLWKWKHHQYLURQPHQWSUHYLRXVO\UHFRJQL]HGDQGWKHIHDWXUHVH[WUDFWHG IURPWKHGDWDDFTXLUHGIURPWKH*36UHFHLYHULVSHUIRUPHG 3ULRU:RUN 7R GDWH DQG EDVHG RQ WKH OLWHUDWXUH WKHUH DUH VHYHUDO VWXGLHV XVLQJ GLIIHUHQW VXEVHWV RI VHQVRUV DYDLODEOH RQ VPDUWSKRQHV<HWVWXGLHVUHSRUWLQJPHWKRGVWKDWUHO\RQDODUJHUVHWRIVHQVRUVDYDLODEOHLQRIIWKHVKHOIPRELOHGHYLFHV IRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWVZHUHQRWIRXQG)RUWKHSXUSRVHRIFRPSDULVRQZLWKWKLVUHVHDUFK ZHDQDO\]HGVWXGLHVDYDLODEOHLQWKHOLWHUDWXUHWKDWXVH$UWLILFLDO1HXUDO1HWZRUNV $11 IRUWKHUHFRJQLWLRQRI$'/ DQGHQYLURQPHQWV7KHLPSOHPHQWDWLRQRI$11ZLWKPRELOHGHYLFHVLVZLGHO\VWXGLHGDQGVHYHUDOIUDPHZRUNVKDYH EHHQ GHYHORSHG ,Q WKLV FDVH RWKHU W\SHV RI PHWKRGV KDYH EHHQ DOUHDG\ XVHG LQ WKH OLWHUDWXUH LQFOXGLQJ WKH QHXUDO QHWZRUNVDQGWKHHQVHPEOHOHDUQLQJPHWKRGV>@EXWWKHHQVHPEOHOHDUQLQJPHWKRGVRIWHQLQFOXGHDFRPELQDWLRQRI PHWKRGV WKXVUHTXLULQJKLJKHUFRPSXWDWLRQDOUHVRXUFHV DQGFDQEHXVHGZLWKODUJHUGDWDVHWVWKDQWKHQHXUDOQHWZRUNV PHWKRGVZKLFKZLOOEHWKHFDVHIRUWKHGDWDVHWVDYDLODEOHLQWKHVWRUDJHRIPRELOHGHYLFHV>@ 7KHXVHRIWKH$11ZLWKYDULDQWVWRUHFRJQL]HGLIIHUHQWW\SHVRI$'/LQYROYLQJWKHVXEMHFWlVPRYHPHQWVLVTXLWH ZHOOGRFXPHQWHGLQWKHOLWHUDWXUH6HYHUDOZRUNVDVWKHIROORZLQJGLVFXVVLRQVKRZVDGGUHVVHGWKLVPHWKRGSURYLGLQJ GLIIHUHQWDFFXUDF\SHUIRUPDQFHRQGLIIHUHQWGDWDVHWV7KHUHDUHVHYHUDOVWXGLHVXVLQJRQO\WKHDFFHOHURPHWHUVHQVRU ZKHUHWKHDXWKRUVRI>@UHFRJQL]HGWKHZDONLQJDQGVWDQGLQJDFWLYLWLHVXVLQJ0XOWLOD\HU3HUFHSWURQ 0/3 ZLWKD UHSRUWHG DFFXUDF\ DURXQG  %D\DW HW DO >@ LPSOHPHQWHG WKH 0/3 PHWKRG XVLQJ IHDWXUHV H[WUDFWHG IURP WKH DFFHOHURPHWHUGDWDIRUWKHUHFRJQLWLRQRIUXQQLQJZDONLQJGDQFLQJJRLQJXSVWDLUVDQGJRLQJGRZQVWDLUVUHSRUWLQJDQ DFFXUDF\ RI  ,Q >@ WKH DXWKRUV XVHG WKH DFFHOHURPHWHU GDWD DQG LPSOHPHQWHG WKH 0/3 PHWKRG IRU WKH UHFRJQLWLRQRIJRLQJXSVWDLUVJRLQJXSRQDQHVFDODWRUDQGZDONLQJUHSRUWLQJDQDYHUDJHDFFXUDF\RI7KHVWXG\ RI>@LPSOHPHQWHGWKH0/3PHWKRGIRUWKHUHFRJQLWLRQRIVHYHUDODFWLYLWLHVVXFKDVZDONLQJMRJJLQJJRLQJXSVWDLUV JRLQJGRZQVWDLUVVLWWLQJVWDQGLQJDQGO\LQJUHSRUWLQJDPD[LPXPDFFXUDF\RI7KHVWDQGLQJZDONLQJDQG UXQQLQJDFWLYLWLHVUHSRUWHGDPLQLPXPDFFXUDF\RIXVLQJWKH0/3PHWKRG>@7KHDXWKRUVRI>@LPSOHPHQWHG WKH0/3PHWKRGIRUWKHFRPSDULVRQRIWKHUHVXOWVREWDLQHGZLWKWKH6OLGLQJ:LQGRZEDVHG+LGGHQ0DUNRY0RGHO 6:+00 UHSRUWLQJDQDFFXUDF\DOZD\VKLJKHUWKDQLQWKHUHFRJQLWLRQRIZDONLQJVWDQGLQJUXQQLQJJRLQJ GRZQVWDLUVDQGJRLQJXSVWDLUV0/3PHWKRGZDVLPSOHPHQWHGE\WKHDXWKRUVRI>@IRUWKHUHFRJQLWLRQRIVWDQGLQJ VLWWLQJZDONLQJO\LQJDQGIDOOLQJDFWLYLWLHVUHSRUWLQJDQDFFXUDF\KLJKHUWKDQ8VLQJWKHVDPHPHWKRGLQ>@WKH DXWKRUVUHSRUWHGWKDWWKHUHFRJQLWLRQRIWKHVWDQGLQJZDONLQJMRJJLQJMXPSLQJJRLQJXSVWDLUVJRLQJGRZQVWDLUVDQG VLWWLQJDFWLYLWLHVKDVDQDFFXUDF\KLJKHUWKDQ,Q>@WKHZDONLQJDQGVWDQGLQJDFWLYLWLHVUHSRUWHGDUHFRJQLWLRQ DFFXUDF\EHWZHHQDQGXVLQJWKH0/3PHWKRG7KH0/3PHWKRGZDVXVHGE\WKHDXWKRUVRI>@IRUWKH UHFRJQLWLRQRIVWDQGLQJZDONLQJJRLQJXSVWDLUVJRLQJGRZQVWDLUVDQGUXQQLQJDFWLYLWLHVUHSRUWLQJDQDFFXUDF\RI &\FOLQJVWDQGLQJZDONLQJMRJJLQJGULYLQJDQGUXQQLQJDUHRWKHUDFWLYLWLHVUHFRJQL]HGE\WKHDXWKRUVRI>@ZLWKWKH 0/3 PHWKRG UHSRUWLQJ DQ DFFXUDF\ EHWZHHQ  DQG  ,Q >@ WKH 0/3 PHWKRG ZDV XVHG IRU WKH UHFRJQLWLRQRIZDONLQJJRLQJXSVWDLUVJRLQJGRZQVWDLUVVWDQGLQJVLWWLQJDQGMRJJLQJDFWLYLWLHVUHSRUWLQJDQDFFXUDF\ DURXQG,QRUGHUWRLGHQWLI\WKHDFWLYLWLHVSHUIRUPHGGXULQJDIRRWEDOOJDPHWKHDXWKRUVRI>@XVHGWKH0/3 PHWKRGZLWKDUHSRUWHGDFFXUDF\KLJKHUWKDQ,Q>@WKHZDONLQJJRLQJXSVWDLUVMRJJLQJJRLQJGRZQVWDLUV VWDQGLQJDQGVLWWLQJDFWLYLWLHVZHUHUHFRJQL]HGZLWKWKH0/3PHWKRGZKLFKUHSRUWHGDQDFFXUDF\RI7KHVWXG\ >@SUHVHQWHGWKHLPSOHPHQWDWLRQRIWKH0/3PHWKRGIRUWKHUHFRJQLWLRQRIZDONLQJMRJJLQJJRLQJXSVWDLUVJRLQJ GRZQVWDLUVVLWWLQJDQGVWDQGLQJDFWLYLWLHVUHSRUWHGDQDYHUDJHDFFXUDF\RI,Q>@WKH0/3PHWKRGZDVXVHG IRUWKHUHFRJQLWLRQRIF\FOLQJUXQQLQJVLWWLQJVWDQGLQJDQGZDONLQJZLWKDPLQLPXPUHSRUWHGDFFXUDF\RI,Q DGGLWLRQWRWKHXVHRIWKH0/3PHWKRGZLWKWKHDFFHOHURPHWHUGDWDDVLQSXWRWKHUPHWKRGVDUHJURZLQJVXFKDV'11 DQG3UREDELOLVWLF1HXUDO1HWZRUNV 311 PHWKRGV'11PHWKRGZDVXVHGE\WKHDXWKRUVRI>@IRUWKHUHFRJQLWLRQRI ZDONLQJ UXQQLQJ VWDQGLQJ VLWWLQJ O\LQJ JRLQJ XSVWDLUV DQG JRLQJ GRZQVWDLUV DFWLYLWLHV ZLWK D UHSRUWHG DFFXUDF\ EHWZHHQDQG311PHWKRGZDVXVHGE\WKHDXWKRUVRI>@IRUWKHUHFRJQLWLRQRIZDONLQJMRJJLQJF\FOLQJ JRLQJXSVWDLUVDQGJRLQJGRZQVWDLUVDFWLYLWLHVUHSRUWLQJUHVXOWVZLWKDQDYHUDJHDFFXUDF\RI )ROORZLQJWKHVWXGLHVWKDWIXVHWKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUDQGWKHJ\URVFRSHVHQVRUVWKHDXWKRUVRI >@XVHGWKH0/3PHWKRGIRUWKHUHFRJQLWLRQRIVWDQGLQJZDONLQJUXQQLQJVLWWLQJO\LQJF\FOLQJGULYLQJFOHDQLQJ FRRNLQJWDNLQJPHGLFDWLRQVZHHSLQJZDVKLQJKDQGVDQGZDWHULQJSODQWVDFWLYLWLHVUHSRUWLQJDQDYHUDJHDFFXUDF\RI 7KH0/3PHWKRGZDVXVHGLQ>@IRUWKHUHFRJQLWLRQRIZDONLQJMXPSLQJUXQQLQJJRLQJGRZQVWDLUVDQG JRLQJXSVWDLUVUHSRUWLQJDQDYHUDJHDFFXUDF\RI)RUWKHUHFRJQLWLRQRIIDOOLQJDFWLYLWLHVWKHDXWKRUVRI>@ LPSOHPHQWHGWKH0/3PHWKRGDQGUHSRUWHGDQDFFXUDF\RI7KHZDONLQJMRJJLQJVLWWLQJJRLQJXSVWDLUVDQG 125 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO JRLQJGRZQVWDLUVZHUHUHFRJQL]HGZLWKWKHXVHRIWKH0/3PHWKRGUHSRUWLQJDQDYHUDJHDFFXUDF\RI>@,Q >@WKHDXWKRUVLPSOHPHQWHGWKH0/3PHWKRGIRUWKHUHFRJQLWLRQRIPRYHPHQWVUHODWHGWRJHWWLQJXSDQGGRZQIURP EXVHVUHSRUWLQJDQDYHUDJHDFFXUDF\RI $ V\VWHP WKDW XVHV 6XSSRUW 9HFWRU 0DFKLQHV  690V ZLWK WXQHG SDUDPHWHUV DQG IHDWXUHV EDVHG RQ KLVWRJUDPV H[WUDFWHG IURP PXOWLSOH DFFHOHURPHWHUV DQG J\URVFRSHV LV SURSRVHG LQ >@ 7KLV LV D YDOLG DSSURDFK EHFDXVH VXFK IHDWXUHVDUHHDV\WRFRPSXWH$EDODQFHGDFFXUDF\RYHULVUHSRUWHG $QDSSURDFKEDVHGRQDXWRPDWLFIHDWXUHHQJLQHHULQJLVXVHGLQ>@IRUWKHUHFRJQLWLRQRIDYDULHW\RI$'/VXVLQJ VHQVRUVSODFHGRQPXOWLSOHORFDWLRQV'HSHQGLQJRQWKHGDWDVHWDQDFFXUDF\IURPWRLVUHSRUWHG,QWHUHVWLQJO\ WKHEHVWDFFXUDF\DFKLHYHGZLWK690V5DQGRP)RUHVWDQG([WUD7UHHVLVFRPSDUDEOHWRWKHDFFXUDF\RIPXFKPRUH OLJKWZHLJKWDSSURDFKHVOLNHORJLVWLFUHJUHVVLRQDQG1DgYH%D\HV$VLPLODUDSSURDFKZDVSURYHQWRZRUNIRUMRJJLQJ GHWHFWLRQLQ>@HYHQRQILHOGFRQGLWLRQVZKHQWKHVXEMHFWZHDUVDFFHOHURPHWHUVDOOWKHWLPH ,Q>@WKHDFFHOHURPHWHUDQGJUDYLW\VHQVRUVlGDWDZHUHIXVHGDQGWKH0/3PHWKRGXVHGIRUWKHUHFRJQLWLRQRI ZDONLQJJRLQJXSVWDLUVJRLQJGRZQVWDLUVVWDQGLQJDQGOD\LQJDFWLYLWLHVUHSRUWLQJDQDFFXUDF\KLJKHUWKDQ &RPELQLQJWKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUDQGWKH*36UHFHLYHUWKH0/3PHWKRGKDQGOHVWKHUHFRJQLWLRQ RIVWDQGLQJZDONLQJDQGWUDYHOOLQJE\FDUDQGWUDLQUHSRUWLQJDPLQLPXPDFFXUDF\RI>@ ,Q>@WKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUWKHJ\URVFRSHDQGWKH*36UHFHLYHUZHUH IXVHG DQG WKH 0/3 PHWKRG ZDV DSSOLHG IRU WKH UHFRJQLWLRQ RI UXQQLQJ ZDONLQJ VLWWLQJ JRLQJ XSVWDLUV JRLQJ GRZQVWDLUV DQG VLWWLQJ DFWLYLWLHV UHSRUWLQJ D UHFRJQLWLRQ DFFXUDF\ KLJKHU WKDQ  7KHDFRXVWLFGDWDPD\EHH[SORLWHGIRUWKHUHFRJQLWLRQRIWKHGLIIHUHQWHQYLURQPHQWVXVLQJ$11$XWKRUVRI>@ LPSOHPHQWHG WKH 0/3 PHWKRG IRU WKH UHFRJQLWLRQ RI VRXQGV RI HPHUJHQF\ YHKLFOHV UHSRUWLQJ DQ DFFXUDF\ DURXQG ,Q>@VHYHUDOVRXQGVKDYHEHHQUHFRJQL]HGZLWKWKH0/3PHWKRGVXFKDVEROOLPSDFWPHWDOLPSDFWZRRG LPSDFWSODVWLFLPSDFWRSHQFORVHGRRUW\SLQJNQRFNLQJWHOHSKRQHULQJLQJJUDLQVIDOOLQJVSUD\DQGZKLVWOHUHSRUWLQJ DQDYHUDJHDFFXUDF\LQHQYLURQPHQWUHFRJQLWLRQRI7KHVRXQGVRIVQHH]LQJGRJEDUNLQJFORFNWLFNLQJEDE\ FU\LQJFURZLQJURRVWHUUDLQLQJVHDZDYHVILUHFUDFNOLQJKHOLFRSWHUDQGFKDLQVDZZHUHUHFRJQL]HGLQ>@XVLQJWKH 0/3 PHWKRG DQG UHSRUWLQJ DQ DFFXUDF\ DURXQG  7KH DXWKRUV RI >@ UHFRJQL]HG VHYHUDO VRXQGV LQFOXGLQJ DODUPVELUGVFODSSLQJGRJVPRWRUF\FOHVUDLQLQJVHDZDYHVDQGZLQGEDVHGRQWKHDSSOLFDWLRQRIWKH0/3PHWKRG DQGUHSRUWHGDQDFFXUDF\DURXQG2WKHU$11PHWKRGVKDYHEHHQXVHGVXFKDV)HHGIRUZDUGQHXUDOQHWZRUNV )11 5HFXUUHQW1HXUDO1HWZRUNV 511 DQG'11PHWKRGV7KH)11ZHUHLPSOHPHQWHGE\WKHDXWKRUVRI>@IRU WKHUHFRJQLWLRQRIHPHUJHQF\YHKLFOHVFDUKRUQVDQGQRUPDOVWUHHWVRXQGVUHSRUWLQJDQDFFXUDF\EHWZHHQDQG 7KHDXWKRUVRI>@LPSOHPHQWHG'11DQG511PHWKRGVIRUWKHUHFRJQLWLRQRIWKHVRXQGVUHODWHGWREDE\ FU\LQJDQGVPRNLQJDODUPUHSRUWLQJDQDFFXUDF\DURXQG7KH'11PHWKRGVZDVDOVRXVHGIRUWKHUHFRJQLWLRQRI GLIIHUHQWVRXQGVLQFOXGLQJODXJKLQJVLQJLQJFU\LQJDUJXLQJVLJKLQJYRLFHDQGPXVLFUHSRUWLQJDFFXUDFLHVDURXQG >@ 7KHDFRXVWLFGDWDPD\DOVREHXVHGWRKHOSLQWKHUHFRJQLWLRQRIVRPH$'/WKHDXWKRUVRI>@UHFRJQL]HGWKH IDOOLQJHYHQWVZLWKWKH0/3PHWKRGUHSRUWLQJDQDFFXUDF\DURXQG $VXPPDU\RIWKHOLWHUDWXUHRYHUYLHZGLVFXVVHGDERYHLVSURYLGHGLQ7DEOHVWRZKHUHGHWDLOVUHJDUGLQJWKH QXPEHU RI $'/V DQDO\]HG E\ HDFK VWXG\ DQG WKH DFFXUDF\ REWDLQHG RQ DYHUDJH DQG RQ WKH VLQJOH W\SH RI $'/ FRQVLGHUHGDUHSUHVHQWHG+RZHYHUWKHDFFXUDF\YDOXHVUHSRUWHGE\WKHVWXGLHVDQDO\]HGDUHQRWGLUHFWO\FRPSDUDEOHDV WKH\ZHUHREWDLQHGIURPGLIIHUHQWGDWDVHWV 7DEOH  6XPPDU\ RI WKH OLWHUDWXUH VWXGLHV UHODWHG WR $'/ UHFRJQLWLRQ EDVHG RQ WKH $11 PHWKRG 7KH QXPEHU RI $'/ DQDO\]HGWKHDYHUDJHDFFXUDF\DQGWKHDFFXUDF\UHODWHGWRHDFK$'/DUHUHSRUWHG  $YHUDJH $'/ $FFXUDF\ $FFXUDF\ $FFXUDF\ SHU *RLQJ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ 6WXG\ SHU $FFXUDF\ $FFXUDF\ $FFXUDF\ *RLQJ 5HIV VWXG\ VWXG\ 6WDQGLQJ :DONLQJ 5XQQLQJ 8SVWDLUV 'RZQVWDLUV -RJJLQJ 6LWWLQJ /\LQJ )DOOLQJ &\FOLQJ 'ULYLQJ >@     k k k k k k k k k >@   k     k k k k k k >@   k  k  k k k k k k k >@     k      k k k >@      k k k k k k k k 126 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH (Table 1) contd.....  $YHUDJH $'/ $FFXUDF\ $FFXUDF\ $FFXUDF\ SHU *RLQJ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ 6WXG\ SHU $FFXUDF\ $FFXUDF\ $FFXUDF\ *RLQJ 5HIV VWXG\ VWXG\ 6WDQGLQJ :DONLQJ 5XQQLQJ 8SVWDLUV 'RZQVWDLUV -RJJLQJ 6LWWLQJ /\LQJ )DOOLQJ &\FOLQJ 'ULYLQJ >@       k k k k k k >@      k k k k    k k >@     k     k k k k >@     k k k k k k k k k >@        k k k k k k >@      k k  k k k   >@     k     k k k k >@        k   k k k >@     k     k k k k >@     k     k k k k >@      k k k   k   >@   k     k k k k k k >@   k k k k k k k k  k k >@     k   k k  k k k >@   k  k     k k k k >@      k k k  k k  k >@     k k k k k k k k k >@        k  k k k k >@   k  k    k k k  k >@   k k k k k k k k  k k >@        k k k k k k 7DEOH  6XPPDU\ RI WKH OLWHUDWXUH VWXGLHV UHODWHG WR $'/ UHFRJQLWLRQ EDVHG RQ WKH $11 PHWKRG 7KH QXPEHU RI $'/ DQDO\]HGWKHDYHUDJHDFFXUDF\DQGWKHDFFXUDF\UHODWHGWRHDFK$'/DUHUHSRUWHG FRQW   $YHUDJH $FFXUDF\ $FFXUDF\ $'/ $FFXUDF\ *RLQJRQ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ *RLQJ SHU $FFXUDF\ DQ 6WXG\ SHU $FFXUDF\ $FFXUDF\ $FFXUDF\ 3OD\ 7DNLQJ $FFXUDF\ :DVKLQJ :DWHULQJ 7UDYHOOLQJ 7UDYHOOLQJ *RLQJXS 'RZQWR %XV 5HIV VWXG\ VWXG\ 'DQFLQJ (VFDODWRU -XPSLQJ &\FOLQJ 'ULYLQJ )RRVEDOO 0HGLFDWLRQ 6ZHHSLQJ +DQGV 3ODQWV E\&DU E\7UDLQ WR%XV >@    k k k k k k k k k k k k k >@   k  k k k k k k k k k k k k >@   k k k  k k k k k k k k k k >@   k k k k       k k k k >@   k k  k k k k k k k k k k k >@   k k k k k k k k k k   k k >@   k k k k k k k k k k k k   7DEOH6XPPDU\RIWKHOLWHUDWXUHVWXGLHVUHODWHGWRHQYLURQPHQWUHFRJQLWLRQEDVHGRQWKH$11PHWKRG7KHQXPEHURI HQYLURQPHQWVDQDO\]HGWKHDYHUDJHDFFXUDF\DQGWKHDFFXUDF\UHODWHGWRHDFKHQYLURQPHQWDUHUHSRUWHG $YHUDJH  $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ %DE\ &URZLQJ $FFXUDF\ 6HD )LUH 6WXG\ (QYLURQPHQWV $FFXUDF\ $FFXUDF\ $FFXUDF\ SHU (PHUJHQF\ $FFXUDF\ $FFXUDF\ &ORFN SHUVWXG\ 9HKLFOHV 6QHH]LQJ 'RJV 7LFNLQJ &U\LQJ 5RRVWHU 5DLQLQJ :DYHV &UDFNOLQJ +HOLFRSWHU &KDLQVDZ 6WUHHW 5HIV VWXG\ >@    k k k k k k k k k k >@   k           k k >@    k k k k k k k k k k  >@   k k k k  k k k k k k k >@   k k k k  k k k k k k k >@    k  k k k   k k k 127 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO 7DEOH6XPPDU\RIWKHOLWHUDWXUHVWXGLHVUHODWHGWRHQYLURQPHQWUHFRJQLWLRQEDVHGRQWKH$11PHWKRG7KHQXPEHURI HQYLURQPHQWVDQDO\]HGWKHDYHUDJHDFFXUDF\DQGWKHDFFXUDF\UHODWHGWRHDFKHQYLURQPHQWDUHUHSRUWHG FRQW  $YHUDJH $FFXUDF\  $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ 2SHQRU %ROO 0HWDO :RRG 3ODVWLF *UDLQV $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ SHU &ORVH $FFXUDF\ $FFXUDF\ 3KRQH 6WXG\ (QYLURQPHQWV SHUVWXG\ ,PSDFW ,PSDFW ,PSDFW ,PSDFW VWXG\ 'RRU 5HIV 7\SLQJ .QRFNLQJ 5LQJLQJ )DOOLQJ 6SUD\ :KLVWOH 9RLFH 0XVLF >@              k k >@   k k k k k k k k k k k   7DEOH6XPPDU\RIWKHOLWHUDWXUHVWXGLHVUHODWHGWRHQYLURQPHQWUHFRJQLWLRQEDVHGRQWKH$11PHWKRG7KHQXPEHURI HQYLURQPHQWVDQDO\]HGWKHDYHUDJHDFFXUDF\DQGWKHDFFXUDF\UHODWHGWRHDFKHQYLURQPHQWDUHUHSRUWHG FRQW  $YHUDJH $FFXUDF\  $FFXUDF\ 6WXG\ (QYLURQPHQWV $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ 6PRNLQJ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ $FFXUDF\ SHU 5HIV 7UDIILF %LUGV &ODSSLQJ )RRWVWHSV 0RWRUF\FOHV 5LYHUV :LQG VWXG\ /DXJKLQJ 6LQJLQJ $UJXLQJ 6LJKLQJ :DWHU SHUVWXG\ $ODUP >@       k k k k k k k k k >@   k k k k   k k k k k k k >@   k k k k k k  k k k k k k >@   k k k k k k        7DNLQJLQWRDFFRXQWWKHVHWRIVWXGLHVZHVHOHFWHGDQGDQDO\]HGIURPWKHOLWHUDWXUHDVOLVWHGLQ7DEOHVWRDQGLQ WKH ELEOLRJUDSK\ VHFWLRQ RI WKLV ZRUN ZH FDQ FRQFOXGH WKDW WKH PDMRULW\ RI WKH VWXGLHV   XVHV RQO\ WKH DFFHOHURPHWHUIRUWKHUHFRJQLWLRQRI$'/DQGRQO\VWXGLHV  XVHGDFRXVWLFGDWDIRUWKHUHFRJQLWLRQRIWKH HQYLURQPHQWV1HYHUWKHOHVVVWXGLHV  SUHVHQWPHWKRGVWKDWIXVHWKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUDQG J\URVFRSHVHQVRUV2WKHUFRPELQDWLRQVDUHLQYHVWLJDWHGLQWKHOLWHUDWXUHVXFKDVWKHIXVLRQRIWKHGDWDDFTXLUHGIURPWKH DFFHOHURPHWHUDQG*36UHFHLYHU  WKHIXVLRQRIWKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUDQGWKHJUDYLW\VHQVRUV  DQGWKHIXVLRQRIWKHGDWDDFTXLUHGIURPWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUWKHJ\URVFRSHDQGWKH*36 UHFHLYHU   7KHQXPEHURI$'/UHFRJQL]HGE\HDFKVWXG\YDULHVEHWZHHQDQGEXWLQRIWKHVWXGLHVWKDWQXPEHUYDULHV EHWZHHQDQG,QKDOIRIWKDWVWXGLHVWKHQXPEHURI$'/UHFRJQL]HGLVXSWRLQFOXGHG,QFDVHRIWKHQXPEHURI HQYLURQPHQWVUHFRJQL]HGE\HDFKVWXG\WKH\YDU\EHWZHHQDQGEXWLWVGLVSHUVLRQLVHQRUPRXV,QRIWKDW VWXGLHVWKHQXPEHURIHQYLURQPHQWVUHFRJQL]HGLVXSWRDQGWKHUHVW  DUHDQG 7DEOHVWRSUHVHQWWKHPDSSLQJEHWZHHQWKHVWXG\DQDO\VHVDQGWKH$'/DQGHQYLURQPHQWVUHFRJQL]HGLQFOXGLQJ WKHDFFXUDFLHVUHSRUWHGIRUIXUWKHUFRPSDULVRQZLWKRXUVWXG\7KHDFFXUDFLHVSUHVHQWHGDUHUHODWHGWRHDFK$'/DQG HQYLURQPHQWUHFRJQLWLRQ )LQDOO\7DEOHVWRVXPPDUL]HWKHDYHUDJHVWDQGDUGGHYLDWLRQDQGWKHFRHIILFLHQWRIYDULDWLRQRIWKHVWDQGDUG GHYLDWLRQRIWKHDFFXUDF\IRUHDFK$'/DQGHQYLURQPHQWUHFRJQL]HGLQWKHSUHYLRXVZRUNDQDO\]HG7KHORZFRHIILFLHQW RIYDULDWLRQRIVWDQGDUGGHYLDWLRQYDOXHV  SHUPLWFRQFOXGHWKDWERWKDYHUDJHDFFXUDF\RI$'/DQGHQYLURQPHQWV UHFRJQL]HGDUHUHSUHVHQWDWLYH 7DEOH$YHUDJHDFFXUDF\RIWKH$'/UHFRJQL]HG 128 $'/ $YHUDJH$FFXUDF\ 6WDQGDUG'HYLDWLRQ &OHDQLQJ   &RHIILFLHQWRI9DULDWLRQ  &\FOLQJ    'DQFLQJ    'ULYLQJ    )DOOLQJ    *RLQJGRZQWREXV    *RLQJGRZQVWDLUV    *RLQJRQDQHVFDODWRU    *RLQJXSWRDEXV    *RLQJXSVWDLUV    -RJJLQJ    -XPSLQJ    /\LQJ    Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH (Table 6) contd..... $'/ $YHUDJH$FFXUDF\ 6WDQGDUG'HYLDWLRQ 3OD\IRRVEDOO   &RHIILFLHQWRI9DULDWLRQ  5XQQLQJ    6LWWLQJ    6WDQGLQJ    6ZHHSLQJ    7DNLQJPHGLFDWLRQ    7UDYHOOLQJE\FDU    7UDYHOOLQJE\WUDLQ    :DONLQJ    :DVKLQJKDQGV    :DWHULQJSODQWV    &RHIILFLHQWRI9DULDWLRQ 7DEOH$YHUDJHDFFXUDF\RIWKHHQYLURQPHQWVUHFRJQL]HG (QYLURQPHQW $YHUDJH$FFXUDF\ 6WDQGDUG'HYLDWLRQ $UJXLQJ    %DE\FU\LQJ    %LUGV    %ROOLPSDFW    &KDLQVDZ    &ODSSLQJ    &ORFNWLFNLQJ    &RRNLQJ.LWFKHQ    &URZLQJURRVWHU    'RJV    'RRURSHQLQJFORVLQJ    (PHUJHQF\YHKLFOHV    )LUHFUDFNOLQJ    )RRWVWHSV    *UDLQVIDOOLQJ    +HOLFRSWHU    .QRFNLQJ    /DXJKLQJ    0HWDOLPSDFW    0RWRUF\FOHV    0XVLF    3ODVWLFLPSDFW    5DLQLQJ    5LYHUV    6HDZDYHV    6LJKLQJ    6LQJLQJ    6PRNLQJDODUP    6QHH]LQJ    6SUD\    6WUHHW    7HOHSKRQHULQJLQJ    7UDIILF    7\SLQJ    9RLFH    :DWHU    :KLVWOH    :LQG    :RRGLPSDFW    129 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO 3XUSRVHRIWKLV6WXG\ %DVHGRQRXUSUHYLRXVZRUN>@WKHSXUSRVHRIWKLVVWXG\FRQVLVWVLQWKHGHYHORSPHQWRIDQ$QGURLGOLEUDU\ WKDWGHSHQGLQJRQWKHQXPEHUDQGW\SHRIVHQVRUVDYDLODEOHRQERDUGWKHPRELOHGHYLFHLPSOHPHQWVWKHIUDPHZRUN SUHYLRXVO\GHVLJQHGIRUWKHUHFRJQLWLRQRIWKH$'/DQGWKHLUHQYLURQPHQWVH[SORLWLQJWKHPDMRUQXPEHURIVHQVRUV DYDLODEOHLQWKHPRELOHGHYLFHXVHG7KHLPSOHPHQWDWLRQVKRXOGXVHOLJKWZHLJKWPHWKRGVLQDPRELOHDSSOLFDWLRQLQ RUGHUWRJLYHDUDSLGIHHGEDFNWRWKHXVHUZLWKWKHGDWDSURFHVVLQJPHWKRGVLPSOHPHQWHGORFDOO\RQWKHPRELOHGHYLFH 7KHSUHYLRXVZRUNH[SORUHGWKHXVHRIGLIIHUHQWLPSOHPHQWDWLRQVRI$11VXFKDV0/3ZLWK%DFNSURSDJDWLRQ )11 PHWKRG ZLWK %DFNSURSDJDWLRQ DQG '11 PHWKRG ZLWK QRUPDOL]HG DQG QRQQRUPDOL]HG GDWD 7KH IUDPHZRUN LPSOHPHQWHGLQWKLVUHVHDUFKLVFRPSRVHGRIVHYHUDOVWDJHVWKHVHEHLQJGDWDDFTXLVLWLRQGDWDSURFHVVLQJGDWDIXVLRQ DQG FODVVLILFDWLRQ PHWKRGV 7KH GDWD DFTXLVLWLRQ DQG SURFHVVLQJ GHSHQGV RQ WKH W\SHV RI VHQVRUV ZKHUH IRU WKH DFFHOHURPHWHUJ\URVFRSHDQGPDJQHWRPHWHUGDWDDORZSDVVILOWHULVDSSOLHGIRUWKHDFRXVWLFGDWDWKH)DVW)RXULHU 7UDQVIRUP ))7 IRUWKHH[WUDFWLRQRIWKHUHOHYDQWIUHTXHQFLHVRIWKHDXGLRVLJQDOLVDSSOLHGDQGIRUWKHORFDWLRQGDWDQR ILOWHUVDUHDSSOLHGDVWKHUDZGDWDDFTXLUHGLVXVHGIRUWKHPHDVXUHPHQWRIWKHGLVWDQFHWUDYHOOHG>@)ROORZLQJWKH DSSOLFDWLRQRIWKHORZSDVVILOWHUDQGWKH))7WRFOHDQWKHGDWDDFTXLUHGVRPHIHDWXUHVUHODWHGWRWKHVHQVRUVlVLJQDOV ZHUH H[WUDFWHG DQG IXVHG )LQDOO\ WKH FODVVLILFDWLRQ PHWKRGV ZHUH DSSOLHG IRU WKH UHFRJQLWLRQ RI $'/ DQG HQYLURQPHQWV 7KH UHFRJQLWLRQ RI $'/ DQG WKHLU HQYLURQPHQWV DUH VHSDUDWHG LQWR WKUHH VWDJHV 7KHVH DUH WKH UHFRJQLWLRQ RI FRPPRQ$'/ LHZDONLQJJRLQJXSVWDLUVJRLQJGRZQVWDLUVUXQQLQJVWDQGLQJDQGZDWFKLQJ79 WKHUHFRJQLWLRQRI WKHHQYLURQPHQWV LHEDUFODVVURRPEHGURRPJ\PKDOONLWFKHQOLEUDU\VWUHHWDQGOLYLQJURRP DQGWKHUHFRJQLWLRQ RI$'/ZLWKRXWERG\PRWLRQ LHVOHHSLQJZDWFKLQJ79DQGGULYLQJ %DVHGRQWKHRXWFRPHVRISUHYLRXVZRUNV ILUVWO\ IRU WKH UHFRJQLWLRQ RI WKH FRPPRQ $'/ WKH PHWKRG WKDW UHSRUWHG EHWWHU UHVXOWV ZDV '11 PHWKRG ZLWK QRUPDOL]HG GDWD 6HFRQGO\ IRU WKH UHFRJQLWLRQ RI HQYLURQPHQWV WKH PHWKRG WKDW UHSRUWHG EHWWHU UHVXOWV ZDV )11 PHWKRGZLWKQRQQRUPDOL]HGGDWDEXWZKHQFRPSDUHGZLWKWKHSUHYLRXVUHVXOWVWKLVPHWKRGUHSRUWVDORZSHUIRUPDQFH DQGDFFXUDF\GXULQJWKHLPSOHPHQWDWLRQRIWKH$QGURLGOLEUDU\ZKHUHWKHUHVXOWVREWDLQHGDUHEHWWHUZLWK'11PHWKRG ZLWK QRUPDOL]HG GDWD )LQDOO\ IRU WKH UHFRJQLWLRQ RI $'/ ZLWKRXW ERG\ PRWLRQ WKH PHWKRG WKDW UHSRUWHG EHWWHU DFFXUDF\ZDVDOVR'11PHWKRGZLWKQRUPDOL]HGGDWD7KHGDWDXVHGLQWKLVVWXG\LVDYDLODEOHLQWKH$//DE0HGLD:LNL >@ 6WUXFWXUHRIWKLV6WXG\ 7KLVVHFWLRQSUHVHQWVRXUPRWLYDWLRQDQGUHODWHGZRUNIRUWKHGHYHORSPHQWRIDQ$QGURLGOLEUDU\IRUWKHUHFRJQLWLRQ RI $'/ DQG WKHLU HQYLURQPHQWV 7KH IROORZLQJ VHFWLRQV SUHVHQW WKH PHWKRGV XVHG DQG UHVXOWV REWDLQHG ZLWK WKH GHYHORSHG$QGURLGOLEUDU\LQFOXGLQJWKHLUGLVFXVVLRQDQGFRPSDULVRQZLWKUHVHDUFKVWXGLHVDYDLODEOHLQWKHOLWHUDWXUH ZLWKDVLPLODUSXUSRVH 0(7+2'6 6WXG\'HVLJQDQG3DUWLFLSDQWV 2IIWKHVKHOI PRELOH GHYLFHV DUH HTXLSSHG ZLWK VHYHUDO VHQVRUV HJ DFFHOHURPHWHU J\URVFRSH PDJQHWRPHWHU PLFURSKRQHDQG*36UHFHLYHUZKLFKDOORZWKHDFTXLVLWLRQRIVHYHUDOSK\VLFDODQGSK\VLRORJLFDOSDUDPHWHUV 7KLVVWXG\LVIRFXVHGRQWKHGHYHORSPHQWRIDQ$QGURLGOLEUDU\>@IRUWKHSURFHVVLQJRIWKHVHQVRUVGDWDlDQG UHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWV7KHPRELOHGHYLFHXVHGIRUWKHWHVWVRIWKH$QGURLGOLEUDU\ZDVD%4 $TXDULXVGHYLFH>@EXWWKHVDPSOLQJIUHTXHQF\RIWKHVHQVRUVlGDWDDFTXLVLWLRQLVGLIILFXOWWRPHDVXUHEHFDXVHWKHVH GHYLFHVKDYHVHYHUDOFRQVWUDLQWVPRVWO\EHFDXVHDVWKHV\VWHPRSHUDWHVLQDWLPHVKDULQJPDQQHULWLVLPSRVVLEOHWR DVVXUHDVWHDG\PLFURSURFHVVRUDWWHQWLRQWRWKHVHQVRUUHODWHGSURFHVVHV 6XEMHFWV PDOHVDQGIHPDOHV DJHGEHWZHHQDQG\HDUVROGZHUHVHOHFWHGIRUWKHGDWDDFTXLVLWLRQZLWKD PRELOHDSSOLFDWLRQWKDWXVHVWKH$QGURLGOLEUDU\GHYHORSHG,QGHSHQGHQWO\RIWKHJHQGHUZHKDYHVHOHFWHGSHRSOHZLWK GLVWLQFWOLIHVW\OHVZKHUHVXEMHFWVVHOIDVVHVVHGWKHLUOLYLQJVW\OHDVPDLQO\DFWLYHDQGWKHRWKHUVXEMHFWVVHOI DVVHVVHG WKHLU OLYLQJ VW\OH DV PDLQO\ VHGHQWDU\ 'XULQJ WKH XVH RI WKH PRELOH DSSOLFDWLRQ WKH PRELOH GHYLFH LV SHUIRUPLQJ RWKHU WDVNV VXFK DV UHFHLYLQJ FDOOV XVLQJ WKH ,QWHUQHW FRQQHFWLRQ DQG XVLQJ RWKHU PRELOH DSSOLFDWLRQV +RZHYHUWKHPDMRUSDUWRIWKHWLPHWKHPRELOHGHYLFHLVLQWKHIURQWSRFNHWRIWKHXVHUlVSDQWVIRUWKHFRUUHFWGDWD DFTXLVLWLRQ 130 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 7KHGHYHORSHGPRELOHDSSOLFDWLRQDFTXLUHGWKHVHQVRUVlGDWDDVDEDFNJURXQGVHUYLFHDFTXLULQJDQGSURFHVVLQJVORWV RIVHFRQGVRIGDWD:LWKRQO\VHFRQGVRIGDWDWKH$'/DQGHQYLURQPHQWVZLOOEHUHFRJQL]HG7KHGDWDDFTXLVLWLRQ LV SHUIRUPHG HYHU\  PLQXWHV DQG  FDSWXUHV IRU HDFK $'/  LH ZDONLQJ UXQQLQJ JRLQJ XSVWDLUV VWDQGLQJZDWFKLQJ79JRLQJGRZQVWDLUVGULYLQJDQGVOHHSLQJ DQGHQYLURQPHQW LHEHGURRPEDUFODVVURRPJ\P NLWFKHQOLYLQJURRPKDOOVWUHHWDQGOLEUDU\ ZKLFKFRUUHVSRQGVWRVHFRQGV DSSUR[LPDWHO\KRXUV IRUHDFK $'/DQGHQYLURQPHQW'XULQJWKHSHUIRUPDQFHRIWKHH[SHULPHQWVWKHXVHUVKRXOGODEHOWKH$'/DQGHQYLURQPHQW DFTXLUHGEHFDXVHWKHVHGDWDZHUHXVHGWRWUDLQDQGWHVWWKHPHWKRGVIRUWKHFODVVLILFDWLRQZKLFKDUHLPSOHPHQWHGLQWKH $QGURLGOLEUDU\ 'HVFULSWLRQRIWKH)UDPHZRUN )ROORZLQJWKHSUHYLRXVZRUNWKDWH[SODLQVWKHDUFKLWHFWXUHDQGWKHPHWKRGZLWKEHVWUHVXOWVLQWKHUHFRJQLWLRQRI $'/ DQG WKHLU HQYLURQPHQWV >     @ WKH IUDPHZRUN LPSOHPHQWHG LV D FROOHFWLRQ RI WKH PHWKRGV WKDW DFFRUGLQJWRWKHOLWHUDWXUHSURYLGHWKHEHVWSHUIRUPDQFHRQUHFRJQLWLRQRI$'/DQGHQYLURQPHQWVVXEMHFWHGWRWKH DYDLODEOHVHQVRUVLQWKHPRELOHGHYLFHZKHUHWKHIUDPHZRUNLVH[HFXWHG7KXVLWVGHYHORSPHQWLVVHSDUDWHGLQWRVHYHUDO PRGXOHVVXFKDVGDWDDFTXLVLWLRQGDWDSURFHVVLQJGDWDIXVLRQDQGFODVVLILFDWLRQPHWKRGV 'DWDDFTXLVLWLRQPHWKRGVDUHQRWSURYLGHGE\WKH$QGURLGOLEUDU\SHUIRUPLQJRQO\WKHGDWDSURFHVVLQJGDWDIXVLRQ DQGFODVVLILFDWLRQPHWKRGV'DWDDFTXLVLWLRQRIWKHDFFHOHURPHWHUJ\URVFRSHPDJQHWRPHWHUPLFURSKRQHDQG*36GDWD VKRXOGEHSHUIRUPHGZLWKWKHVWDQGDUGPHWKRGVRI$QGURLGSODWIRUPXVLQJDPRELOHDSSOLFDWLRQ)RUWKHUHPDLQLQJ PRGXOHVWKH$QGURLGOLEUDU\VKRXOGEHDIXQFWLRQRIWKHQXPEHURIVHQVRUVDYDLODEOHLQWKHPRELOHGHYLFH 'DWDSURFHVVLQJPHWKRGVLPSOHPHQWHGLQWKH$QGURLGOLEUDU\FRQVLVWRIWKHGDWDFOHDQLQJDQGIHDWXUHH[WUDFWLRQ PHWKRGV 7KH GDWD FOHDQLQJ PHWKRG XVHG GHSHQGV RQ WKH W\SH RI VHQVRUV LH IRU DFFHOHURPHWHU J\URVFRSH DQG PDJQHWRPHWHUGDWDWKHORZSDVVILOWHULVDSSOLHGDQGIRUWKHDFRXVWLFGDWDWKH))7LVDSSOLHGIRUWKHH[WUDFWLRQRIWKH UHOHYDQWIUHTXHQFLHV7KHIHDWXUHH[WUDFWLRQDOVRGHSHQGVRQWKHW\SHRIVHQVRUVLHIRUDFFHOHURPHWHUJ\URVFRSHDQG PDJQHWRPHWHUGDWDWKHIHDWXUHVH[WUDFWHGDUHWKHILYHJUHDWHVWGLVWDQFHVEHWZHHQWKHPD[LPXPSHDNVFRPELQHGZLWK WKH$YHUDJH6WDQGDUG'HYLDWLRQ9DULDQFHDQG0HGLDQRIWKHPD[LPXPSHDNVDQGWKH6WDQGDUG'HYLDWLRQ$YHUDJH 0D[LPXP0LQLPXP9DULDQFHDQG0HGLDQRIWKHUDZVLJQDO)RUWKHDFRXVWLFGDWDWKHIHDWXUHVH[WUDFWHGDUHWKH 0HO)UHTXHQF\&HSVWUDO&RHIILFLHQWV 0)&& FRPELQHGZLWKWKH6WDQGDUG'HYLDWLRQ$YHUDJH0D[LPXP0LQLPXP 9DULDQFHDQG0HGLDQRIWKHUDZVLJQDO)LQDOO\IRUWKHORFDWLRQGDWDWKHIHDWXUHH[WUDFWHGLVWKHGLVWDQFHWUDYHOOHG $IWHUWKHGDWDIXVLRQRIWKHIHDWXUHVH[WUDFWHGIURPWKHVHQVRUVlGDWDWKHQRUPDOL]DWLRQDQGFODVVLILFDWLRQPHWKRGV DUHDSSOLHGLQWKUHHVWDJHV 6WDJH 6HQVRUVDFFHOHURPHWHUJ\URVFRSHDQGRUPDJQHWRPHWHU 3XUSRVH5HFRJQLWLRQRIZDONLQJVWDQGLQJUXQQLQJJRLQJXSVWDLUVDQGJRLQJGRZQVWDLUVDFWLYLWLHVXVLQJ '11PHWKRGZLWKQRUPDOL]HGGDWD 6WDJH 6HQVRUV0LFURSKRQH 3XUSRVH 5HFRJQLWLRQ RI EDU EHGURRP FODVVURRP J\P KDOO NLWFKHQ VWUHHW OLEUDU\ DQG ZDWFKLQJ 79OLYLQJURRPHQYLURQPHQWVXVLQJ)11PHWKRGZLWKQRQQRUPDOL]HGGDWDDQGRU'11PHWKRGZLWK QRUPDOL]HGGDWD 6WDJH 6HQVRUVDFFHOHURPHWHUJ\URVFRSHPDJQHWRPHWHUDQGRU*36UHFHLYHU 3XUSRVH 5HFRJQLWLRQ RI ZDWFKLQJ 79 VOHHSLQJ DQG GULYLQJ DFWLYLWLHV XVLQJ '11 PHWKRG ZLWK QRUPDOL]HGGDWD 5HODWHG WR WKH VWDJH  WKH SUHYLRXV ZRUN >@ UHFRPPHQGV WKH XVH RI )11 PHWKRG ZLWK QRQQRUPDOL]HG GDWD +RZHYHU DV LW ZLOO EH SUHVHQWHG LQ WKH IROORZLQJ VHFWLRQV ZH YHULILHG WKDW WKH DFFXUDFLHV FRXOG EH LPSURYHG E\ FRPELQLQJGLIIHUHQWPHWKRGRORJLHV  8VHRI)11PHWKRGZLWKQRQQRUPDOL]HGGDWDDQGZKHQWKHUHVXOWVDUHQRWFOHDUO\LGHQWLILHGWKHXVH'11 PHWKRGZLWKQRUPDOL]HGGDWDIRUWKHVHFDVHVPD\SHUIRUPDEHWWHUUHFRJQLWLRQRIWKHHQYLURQPHQWV )LJ$   8VHRQO\WKH'11PHWKRGZLWKQRUPDOL]HGGDWD )LJ%  131 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH !"#$%&' 3LUHVHWDO !"#$%&' !""#$%&'()#*+&' ,(-$./+0%)#).&. !"#$%&( !""#$%&'()#*+&' ,(-$./+0%)#).&. !"#$%&( 2""#$%&'()#*+&' ,(,1,(-$./+0%)#).&. !"#$%&) !""#$%&'()#*+&' ,(-$./+0%)#).&. !"#$%&) !""#$%&'()#*+&' ,(-$./+0%)#).&. *+&&,%"-./.0.$1&' !""#$%&'()#*+&' ,(-$./+0%)#).&. 2+&&,%"-./.0.$1&( )LJ  6FKHPDRIWKHFODVVLILFDWLRQVWDJHRIWKHIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWV 7KH)11PHWKRGZLWKQRQQRUPDOL]HGGDWDZDVLPSOHPHQWHGWUDLQHGDQGWHVWHGZLWKWKH(QFRJIUDPHZRUN>@LQ D -DYD GHVNWRS DSSOLFDWLRQ FUHDWLQJ D QHXUDO QHWZRUN IRU IXUWKHU DSSOLFDWLRQ LQ WKH PRELOH DSSOLFDWLRQ 7KH '11 PHWKRGVZLWKQRUPDOL]HGGDWDZHUHLPSOHPHQWHGWUDLQHGDQGWHVWHGZLWK'HHSOHDUQLQJMIUDPHZRUN>@LQD-DYD GHVNWRSDSSOLFDWLRQFUHDWLQJDQHXUDOQHWZRUNIRUIXUWKHUDSSOLFDWLRQLQWKHPRELOHDSSOLFDWLRQ7KHWUDLQRIWKHQHXUDO QHWZRUNVQHHGVDVLJQLILFDQWSURFHVVLQJHIIRUWWKDWLVSHUIRUPHGSUHYLRXVO\RQDGHVNWRSFRPSXWHUXVLQJRQO\WKH PHWKRGGHYHORSHGLQWKHPRELOHDSSOLFDWLRQ7KHUHDUHVHYHUDOSDUDPHWHUVXVHGIRUWKHFUHDWLRQDQGWUDLQLQJRIWKH QHXUDOQHWZRUNLQFOXGLQJWKHDFWLYDWLRQIXQFWLRQDQGWKHOHDUQLQJUDWHDQGPRPHQWXPYDOXHVXVHGZKLFKDUHSUHVHQWHG LQ7DEOH7KHVHFRQILJXUDWLRQVPD\LQIOXHQFHWKHVSHHGRIWKHWUDLQLQJVWDJHDVZHOODVWKHUHVXOWVREWDLQHGZLWKWKH WUDLQHGQHWZRUNILOH 7DEOH&RQILJXUDWLRQVRIWKHFODVVLILFDWLRQPHWKRGV 3DUDPHWHUV )11PHWKRGZLWK%DFNSURSDJDWLRQ '110HWKRG $FWLYDWLRQIXQFWLRQ 6LJPRLG 6LJPRLG /HDUQLQJUDWH   0RPHQWXP  1$ 0D[LPXPQXPEHURIWUDLQLQJLWHUDWLRQV [ [ 1XPEHURIKLGGHQOD\HUV   :HLJKWIXQFWLRQ 1$ ;DYLHU 6HHGYDOXH 1$  %DFNSURSDJDWLRQ <HV <HV 5HJXODUL]DWLRQ 1$ / 7HVWRIWKH0HWKRG :HFUHDWHGDVHWRIPHWKRGVWRWHVWWKHFRUUHFWLGHQWLILFDWLRQRIWKH$'/DQGHQYLURQPHQWVXVLQJWKH$QGURLGOLEUDU\ ZHGHYHORSHG)RUWKLVSXUSRVHZHDFTXLUHGWKHVHQVRUVlVLJQDOZLWKDPRELOHGHYLFHUHODWHGWRDURXQGKRXUVIRU HDFK $'/ DQG HQYLURQPHQWV FRUUHVSRQGHQW WR  UHFRUGV ZLWK D OHQJWK RI  VHFRQGV ZKLFK DUH ODEHOOHG LQ WKH PRELOH DSSOLFDWLRQ LQ RUGHU WR DOORZ WKH PHDVXUHPHQW RI WKH DFFXUDF\ RI WKH $QGURLG OLEUDU\ GHYHORSHG :H 132 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH LPSOHPHQWHGXQLWWHVWVWRHYDOXDWHWKHSUHYLRXVO\DFTXLUHGUHFRUGVUHSRUWLQJWKH$'/RUHQYLURQPHQWUHFRJQL]HGZLWK HDFKUHFRUGLQRUGHUWRPHDVXUHWKHRYHUDOOUHFRJQLWLRQDFFXUDF\ SHUFHQWDJH RIWKH$QGURLGOLEUDU\DVZHOODVWKH UHFRJQLWLRQDFFXUDF\ SHUFHQWDJH RIHDFK$'/DQGHQYLURQPHQWZLWKWKH$QGURLGOLEUDU\ 5(68/76 3URWRW\SH 7KHPDLQUHVXOWRIWKLVVWXG\FRQVLVWVRIWKHGHYHORSPHQWRIDQ$QGURLGOLEUDU\WKDWLPSOHPHQWHGWKHIUDPHZRUNIRU WKH UHFRJQLWLRQ RI $'/ DQG WKHLU HQYLURQPHQWV ,Q DGGLWLRQ D PRELOH DSSOLFDWLRQ ZLWK D *UDSKLF 8VHU ,QWHUIDFH SUHVHQWHGLQ)LJ  WKDWHPEHGVWKH$QGURLGOLEUDU\GHYHORSHGZDVLPSOHPHQWHGIRULWVYDOLGDWLRQ > > = ? =@ -A = ? =@ -A 678 99:67 678 23!)4.#"5*$0&"* 99:69 23!)4.#"5*$0&"* /4BC5B/5C /D6CD3 !"#$%&'$(")*"+$, $?0.@)69)1&*A0./)B&0C"A0 /.*/"@/D)($0$)$#EA&/&0&"* .1 *+,-".(,/01 2/345603,7'89(" :"#)';(<1,:"#)';(< = ? =@ -A I%0&1")4.5&/0" !"#$%# &'(#)" > -./0.)1"1.*0")./0$ 678 99:67 K&/0"@&#" = >/0$J$ MP MQ MR 69 L M 6 !EFG"8HEG%9E 7 N O P Q R L9 LL LM L6 L7 LN LO LP LQ LR M9 ML MM M6 M7 MN MO MP MQ MR 69 6L L M 6 7 N O P !"#$%&'$(")*"+$, 2 !EFG"8HEG%9E <F.* 2//.//1.*0 H$%.*(.@ .1 I.+,9(F,J<%.E%#(F,B+ ;<=4>)-<; > <F.*)G%$**&*5 H$%.*($@ 3 > 678 99:67 > = ? =@ -A S H@.$0.)*.B G%$* = ? =@ -A = ? =@ -A 678 2(&#&"*$@)F%$*.$1.*0" G%$*.$1.*0$ <F.*)3$&%U 2//.//1.*0 > H 99:67 M9LPVLMVP 3$0$)(.%*&#&" 3$0/)(.)S&1 MP MQ MR 69 L M 6 7 N O P Q R L9 LL LM L6 L7 LN LO LP 3.$#.@)>$#$("/ LQ LR M9 ML MM M6 M7 MN MO MP MQ MR 69 6L L M 6 7 N O P TA$@0" )LJ  'HVLJQRIWKHLQWHUIDFHRIWKH$QGURLGDSSOLFDWLRQGHYHORSHG)LJXUHVD DQGE SUHVHQWWKHPDLQVFUHHQ)LJXUHF SUHVHQWV WKHFDOHQGDUZLWKWKHGDLO\SODQQLQJ)LJXUHG SUHVHQWVWKHIRUPIRUWKHFUHDWLRQRIDSODQQLQJUHFRUG)LJXUHH SUHVHQWVDFDOHQGDU WKDWKLJKOLJKWVWKHGD\VWKDWKDYH$'/DQGHQYLURQPHQWVUHFRJQL]HG)LJXUHI SUHVHQWVDOLVWRI$'/DQGHQYLURQPHQWVSHUIRUPHG HDFKGD\ 7KHPDLQIXQFWLRQDOLW\RIWKHPRELOHDSSOLFDWLRQGHYHORSHGFRQVLVWVLQWKHSUHVHQWDWLRQRIWKHFXUUHQWRUWKHODVW $'/HQYLURQPHQWRUORFDWLRQUHFRJQL]HG )LJ$DQG% %DVHGRQWKHFDOHQGDUWKHPRELOHDSSOLFDWLRQKDVDPRGXOH WRPDQDJHWKH$'/DQGHQYLURQPHQWVSODQQHG )LJ&DQG' )LQDOO\WKHPRELOHDSSOLFDWLRQLQFOXGHVDPRGXOHWR VKRZWKHOLVWRIWKH$'/DQGHQYLURQPHQWVUHFRJQL]HGE\HDFKFDOHQGDUGD\ )LJ) LQFOXGLQJWKHKLJKOLJKWRIWKH UHVSHFWLYHFDOHQGDUGD\VWKDWLQFOXGHGWKH$'/DQGHQYLURQPHQWVUHFRJQL]HG )LJ(  ,QWKHIROORZLQJVHFWLRQVWKHYDOLGDWLRQDQGUHVXOWVRIWKH$QGURLGOLEUDU\HPEHGGHGLQWKHPRELOHDSSOLFDWLRQZLOO 133 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO EHVKRZQSURYLQJWKHUHOLDELOLW\RIWKHIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWV 5HTXLUHPHQWV 7KLV VWXG\ FRQVLVWV RI WKH XVH RI WKH VHQVRUV DYDLODEOH LQ FRQVXPHU PRELOH GHYLFHV 'XH WR WKH GLIIHUHQW PDQXIDFWXUHUV LQ WKH PDUNHW WKHVH GHYLFHV KDYH D GLIIHUHQW QXPEHU RI VHQVRUV DQG WKH $QGURLG OLEUDU\ GHYHORSHG VKRXOGEHIOH[LEOHDQGDGDSWDEOHWRWKHQXPEHURIVHQVRUVHIIHFWLYHO\DYDLODEOHDOORZLQJWKHLUXVHZLWKDOOGHYLFHV DYDLODEOHRQWKHPDUNHW 7RXVHRXU$QGURLGOLEUDU\WKHPRELOHGHYLFHVKRXOGKDYHDPLQLPXPQXPEHURIVHQVRUV LHDFFHOHURPHWHUDQG PLFURSKRQH EXWGHSHQGLQJRQWKHQXPEHURIVHQVRUVLQFOXGHGLQHDFKPRELOHWKHILQDODFFXUDF\RIWKHUHFRJQLWLRQRI $'/ DQG WKHLU HQYLURQPHQWV ZLOO EH GLIIHUHQW 7KXV LI WKH PRELOH GHYLFH LQFOXGHV WKH *36 UHFHLYHU WKH PRELOH DSSOLFDWLRQZLOOLQFUHDVHWKHQXPEHURI$'/FRUUHFWO\UHFRJQL]HG$QGLIWKHPRELOHGHYLFHKDVDQHWZRUNFRQQHFWLRQ WKHXVHUlVORFDWLRQZLOOEHVKRZQLQWKHPRELOHDSSOLFDWLRQ ,QFRQFOXVLRQGHSHQGLQJRQWKHQXPEHURIVHQVRUVDYDLODEOHLQWKHPRELOHGHYLFHWKH$QGURLGOLEUDU\LVDEOHWR UHFRJQL]HEHWZHHQDQG$'/DQGHQYLURQPHQWVDVVKRZQLQWKHQH[WVHFWLRQ 9DOLGDWLRQ 2QFHWKHIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQGHQYLURQPHQWVLQWKH$QGURLGOLEUDU\ZDVLPSOHPHQWHGZH WHVWHGERWKWKHGLIIHUHQWVWDJHVRIWKHIUDPHZRUNLQGHSHQGHQWO\DQGWKHRYHUDOOH[HFXWLRQRIWKHIUDPHZRUN)RUWKHVH WHVWVZHWRRNLQWRDFFRXQWWKHGLIIHUHQWFRPELQDWLRQVRIVHQVRUVSRVVLEO\DYDLODEOHLQWKHPRELOHGHYLFHV7KHVHWHVWV ZHUHSHUIRUPHGZLWKWKHGDWDVHWSUHYLRXVO\DFTXLUHGDQGDYDLODEOHDW>@ %DVHGRQWKHLPSOHPHQWDWLRQRIVWDJHIRUWKHUHFRJQLWLRQRIFRPPRQ$'/ LHVWDQGLQJZDONLQJUXQQLQJJRLQJ XSVWDLUV DQG JRLQJ GRZQVWDLUV XVLQJ WKH '11 PHWKRG ZLWK QRUPDOL]HG GDWD ZH DFKLHYHG DQ DYHUDJH DFFXUDF\ RI ZLWKWKHDFFHOHURPHWHUVHQVRU 7DEOH ZLWKWKHDFFHOHURPHWHUDQGPDJQHWRPHWHUVHQVRUV 7DEOH  DQGZLWKWKHDFFHOHURPHWHUPDJQHWRPHWHUDQGJ\URVFRSHVHQVRUV 7DEOH  7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHZLWKWKHDFFHOHURPHWHUVHQVRU b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ :DONLQJ $FWXDO&ODVV *RLQJ'RZQVWDLUV      *RLQJ8SVWDLUV      5XQQLQJ      6WDQGLQJ      :DONLQJ      7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHZLWKWKHDFFHOHURPHWHUDQGPDJQHWRPHWHUVHQVRUV b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ :DONLQJ $FWXDO&ODVV *RLQJ'RZQVWDLUV      *RLQJ8SVWDLUV      5XQQLQJ      6WDQGLQJ      :DONLQJ      'XULQJ WKH LPSOHPHQWDWLRQ RI VWDJH  IRU WKH UHFRJQLWLRQ RI HQYLURQPHQWV  LH EHGURRP EDU FODVVURRP J\P NLWFKHQZDWFKLQJ79OLYLQJURRPKDOOVWUHHWDQGOLEUDU\ ZHYHULILHGWKDWWKH)11PHWKRGZLWK%DFNSURSDJDWLRQ XVHGLQWKHSUHYLRXVVWXG\>@UHSRUWHGFRQIXVHGUHVXOWV,QIDFWXVLQJWKH)11PHWKRGZLWK%DFNSURSDJDWLRQWKH IUDPHZRUNUHSRUWVDQDYHUDJHDFFXUDF\RILQWKHUHFRJQLWLRQRIWKHHQYLURQPHQWVUHFRJQL]LQJRQO\RXWRI HQYLURQPHQWV 7DEOH $VWKHUHVXOWVZHUHQRWJRRGZHFUHDWHGDQHZPHWKRGLQWKHIUDPHZRUNWKDWILUVWWULHVWKH UHFRJQLWLRQRIWKHHQYLURQPHQWVXVLQJDQ)11PHWKRGZLWK%DFNSURSDJDWLRQDQGZKHQWKHUHVXOWVDUHQRWFOHDUO\ LGHQWLILHGDXWRPDWLFDOO\DSSOLHVWKH'11PHWKRGZLWKQRUPDOL]HGGDWD7KLVZD\ZHREWDLQHGDQDYHUDJHDFFXUDF\RI  ZLWK D VORZLQJ GRZQ SHUIRUPDQFH RI WKH PRELOH GHYLFH UHFRJQL]LQJ  RXW RI  HQYLURQPHQWV  7DEOH   )LQDOO\ ZH LPSOHPHQWHG WKH '11 PHWKRG RQO\ ZLWK QRUPDOL]HG GDWD IRU WKH UHFRJQLWLRQ RI WKH HQYLURQPHQWV 134 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH LQFUHDVLQJWKHSHUIRUPDQFHRIWKH$QGURLGOLEUDU\DQGUHSRUWLQJDQDYHUDJHDFFXUDF\RIUHFRJQL]LQJDOOWKH HQYLURQPHQWV 7DEOH ,QWKHQH[WDQDO\VLVUHODWHGWRWKHUHFRJQLWLRQRIWKHHQYLURQPHQWVWKHUHVXOWVREWDLQHGZLWK WKH'11RQO\PHWKRGZLOOEHWDNHQLQWRDFFRXQW 7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHZLWKWKHDFFHOHURPHWHUPDJQHWRPHWHUDQGJ\URVFRSHVHQVRUV b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ :DONLQJ *RLQJ'RZQVWDLUV      *RLQJ8SVWDLUV      5XQQLQJ      6WDQGLQJ      :DONLQJ      $FWXDO&ODVV 7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHXVLQJ)11PHWKRGZLWK%DFNSURSDJDWLRQ b 3UHGLFWHG&ODVV %HGURRP %DU &ODVVURRP *\P +DOO .LWFKHQ /LEUDU\ 6WUHHW :DWFKLQJ79OLYLQJURRP %HGURRP          %DU          &ODVVURRP          *\P          +DOO          .LWFKHQ          /LEUDU\          6WUHHW          :DWFKLQJ79OLYLQJURRP          $FWXDO &ODVV 7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHXVLQJ)11PHWKRGZLWK%DFNSURSDJDWLRQDQG'11PHWKRG b 3UHGLFWHG&ODVV %HGURRP %DU &ODVVURRP *\P +DOO .LWFKHQ /LEUDU\ 6WUHHW $FWXDO &ODVV :DWFKLQJ79OLYLQJURRP %HGURRP          %DU          &ODVVURRP          *\P          +DOO          .LWFKHQ          /LEUDU\          6WUHHW          :DWFKLQJ79OLYLQJURRP          %DVHG RQ WKH LPSOHPHQWDWLRQ RI VWDJH  IRU WKH UHFRJQLWLRQ RI $'/ ZLWKRXW ERG\ PRWLRQ  LH ZDWFKLQJ 79 VOHHSLQJDQGGULYLQJ WKHUHVXOWVREWDLQHGDUHDOZD\VEXWWKHVHUHVXOWVGHSHQGRQWKHUHVXOWVREWDLQHGGXULQJWKH SUHYLRXVVWDJHVRIWKHIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWVEHFDXVHWKHRXWFRPHVGHSHQGRQ WKHFRUUHFWUHFRJQLWLRQRIWKHHQYLURQPHQW VWDJH WKHFRUUHFWUHFRJQLWLRQRIWKH$'/DVVWDQGLQJ VWDJH DQGWKH FRUUHFWH[WUDFWLRQRIWKH*36IHDWXUHV )LQDOO\ ZH WHVWHG WKH RYHUDOO UHFRJQLWLRQ RI WKH $'/  LH ZDONLQJ UXQQLQJ JRLQJ XSVWDLUV JRLQJ GRZQVWDLUV ZDWFKLQJ79VOHHSLQJDQGGULYLQJ ZLWKWKHVHTXHQWLDOH[HFXWLRQRIWKHWKUHHVWDJHVXVLQJRQO\WKH'11PHWKRGZLWK QRUPDOL]HG GDWD UHSRUWLQJ DQ DYHUDJH DFFXUDF\ RI  ZLWK WKH XVH RI WKH DFFHOHURPHWHU DQG WKH HQYLURQPHQW UHFRJQL]HG 7DEOH ZLWKWKHXVHRIWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUDQGWKHHQYLURQPHQWUHFRJQL]HG 7DEOH ZLWKWKHXVHRIWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUWKHJ\URVFRSHDQGWKHHQYLURQPHQWUHFRJQL]HG 7DEOH ZLWKWKHXVHRIWKHDFFHOHURPHWHUWKH*36UHFHLYHUDQGWKHHQYLURQPHQWUHFRJQL]HG 7DEOH  ZLWKWKHXVHRIWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUWKH*36UHFHLYHUDQGWKHHQYLURQPHQWUHFRJQL]HG 7DEOH 135 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO  DQGILQDOO\ZLWKWKHXVHRIWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUWKHJ\URVFRSHWKH*36UHFHLYHUDQGWKH HQYLURQPHQWUHFRJQL]HG 7DEOH  7DEOH$QDO\VLVRIWKHUHVXOWVRIVWDJHXVLQJWKH'11PHWKRG b 3UHGLFWHG&ODVV %HGURRP %DU &ODVVURRP *\P +DOO .LWFKHQ /LEUDU\ 6WUHHW :DWFKLQJ79OLYLQJURRP %HGURRP          %DU          &ODVVURRP          *\P          +DOO          .LWFKHQ          /LEUDU\          6WUHHW          :DWFKLQJ79OLYLQJURRP          $FWXDO &ODVV 7DEOH$QDO\VLVRIWKHRYHUDOOUHFRJQLWLRQRIWKH$'/ZLWKDFFHOHURPHWHUDQGHQYLURQPHQWUHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ $FWXDO&ODVV 6WDQGLQJ:DWFKLQJ79 :DONLQJ 6OHHSLQJ *RLQJ'RZQVWDLUV       *RLQJ8SVWDLUV       5XQQLQJ       6WDQGLQJ:DWFKLQJ79       :DONLQJ       6OHHSLQJ       7DEOH$QDO\VLVRIWKHRYHUDOOUHFRJQLWLRQRIWKH$'/ZLWKDFFHOHURPHWHUPDJQHWRPHWHUDQGHQYLURQPHQWUHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ $FWXDO&ODVV 6WDQGLQJ:DWFKLQJ79 :DONLQJ 6OHHSLQJ *RLQJ'RZQVWDLUV       *RLQJ8SVWDLUV       5XQQLQJ       6WDQGLQJ:DWFKLQJ79       :DONLQJ       6OHHSLQJ       7DEOH$QDO\VLVRIWKHRYHUDOOUHFRJQLWLRQRIWKH$'/ZLWKDFFHOHURPHWHUPDJQHWRPHWHUJ\URVFRSHDQGHQYLURQPHQW UHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV *RLQJ8SVWDLUV 5XQQLQJ $FWXDO&ODVV 136 6WDQGLQJ:DWFKLQJ79 :DONLQJ 6OHHSLQJ *RLQJ'RZQVWDLUV       *RLQJ8SVWDLUV       5XQQLQJ       6WDQGLQJ:DWFKLQJ79       :DONLQJ       6OHHSLQJ       Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 7DEOH$QDO\VLVRIWKHRYHUDOOUHFRJQLWLRQRIWKH$'/ZLWKDQDFFHOHURPHWHU*36UHFHLYHUDQGHQYLURQPHQWUHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV $FWXDO &ODVV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ:DWFKLQJ 79 :DONLQJ 6OHHSLQJ 'ULYLQJ *RLQJ'RZQVWDLUV        *RLQJ8SVWDLUV        5XQQLQJ        6WDQGLQJ:DWFKLQJ79        :DONLQJ        6OHHSLQJ        'ULYLQJ        7DEOH  $QDO\VLV RI WKH RYHUDOO UHFRJQLWLRQ RI WKH $'/ ZLWK DQ DFFHOHURPHWHU PDJQHWRPHWHU *36 UHFHLYHU DQG HQYLURQPHQW UHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV $FWXDO &ODVV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ:DWFKLQJ 79 :DONLQJ 6OHHSLQJ 'ULYLQJ *RLQJ'RZQVWDLUV        *RLQJ8SVWDLUV        5XQQLQJ        6WDQGLQJ:DWFKLQJ79        :DONLQJ        6OHHSLQJ        'ULYLQJ        7DEOH$QDO\VLVRIWKHRYHUDOOUHFRJQLWLRQRIWKH$'/ZLWKDQDFFHOHURPHWHUPDJQHWRPHWHUJ\URVFRSH*36UHFHLYHUDQG HQYLURQPHQWUHFRJQL]HG b 3UHGLFWHG&ODVV *RLQJ'RZQVWDLUV $FWXDO &ODVV *RLQJ8SVWDLUV 5XQQLQJ 6WDQGLQJ:DWFKLQJ 79 :DONLQJ 6OHHSLQJ 'ULYLQJ *RLQJ'RZQVWDLUV        *RLQJ8SVWDLUV        5XQQLQJ        6WDQGLQJ:DWFKLQJ79        :DONLQJ        6OHHSLQJ        'ULYLQJ        ',6&866,21 0DLQ)LQGLQJV %DVHG RQ WKH SDUWLDO UHVXOWV REWDLQHG LQ WKH SUHYLRXV ZRUN >  @ ZH SUHVHQW D VXPPDU\ RI WKH DFFXUDFLHV REWDLQHGLQHDFKVWDJHRIWKHIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWV 7DEOHVWR IRU IXUWKHUFRPSDULVRQZLWKWKHUHVXOWVREWDLQHGZLWKWKH$QGURLGOLEUDU\GHYHORSHGIRUWKLVVWXG\ 7DEOHVWR ZKLFK FRPELQHVWKHGLIIHUHQWVWDJHVDOOWRJHWKHU 7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGLQWKHVWXGLHV>@XVLQJWKHDFFHOHURPHWHUVHQVRU 6WDJHV $FFHOHURPHWHU $FFHOHURPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH *36 6WDJH    6WDJH    6WDJH    137 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO (Table 21) contd..... 6WDJHV $FFHOHURPHWHU $FFHOHURPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH *36 2YHUDOO    7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGLQWKHVWXGLHV>@XVLQJWKHDFFHOHURPHWHUDQGWKHPDJQHWRPHWHUVHQVRUV 6WDJHV $FFHOHURPHWHU 0DJQHWRPHWHU $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH *36 6WDJH    6WDJH    6WDJH    2YHUDOO    7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGLQWKHVWXGLHV>@XVLQJWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUDQGWKH J\URVFRSHVHQVRUV 6WDJHV $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH *36 6WDJH    6WDJH    6WDJH    2YHUDOO    7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGE\WKH$QGURLGOLEUDU\XVLQJWKHDFFHOHURPHWHUVHQVRU 6WDJHV $FFHOHURPHWHU $FFHOHURPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH *36 6WDJH    6WDJH    6WDJH    2YHUDOO    7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGE\WKH$QGURLGOLEUDU\XVLQJWKHDFFHOHURPHWHUDQGWKHPDJQHWRPHWHUVHQVRUV 6WDJHV $FFHOHURPHWHU 0DJQHWRPHWHU $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH *36 6WDJH    6WDJH    6WDJH    2YHUDOO    7DEOH$QDO\VLVRIWKHDFFXUDFLHVUHSRUWHGE\WKH$QGURLGOLEUDU\XVLQJWKHDFFHOHURPHWHUWKHPDJQHWRPHWHUDQGWKH J\URVFRSHVHQVRUV 138 6WDJHV $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH *36 6WDJH    Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH (Table 26) contd..... 6WDJHV $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH *36 6WDJH    6WDJH    2YHUDOO    )ROORZLQJ WKH UHVXOWV REWDLQHG LQ WKH VWDJH  UHODWHG WR WKH UHFRJQLWLRQ RI WKH FRPPRQ $'/ ZLWK WKH LPSOHPHQWDWLRQRIWKH'11PHWKRGZLWKQRUPDOL]HGGDWDZHYHULILHGWKDWZLWKWKH$QGURLGOLEUDU\GHYHORSHGWKH $'/DUHUHFRJQL]HGZLWKPRUHDFFXUDF\  ZLWKWKHDFFHOHURPHWHUVHQVRUZLWKWKHVDPHDFFXUDF\ZLWKWKHMRLQW DFFHOHURPHWHUDQGPDJQHWRPHWHUVHQVRUVDQGZLWKOHVVDFFXUDF\  ZLWKWKHDFFHOHURPHWHUPDJQHWRPHWHUDQG J\URVFRSH VHQVRUV WKDQ WKH DFFXUDFLHV REWDLQHG LQ WKH SUHYLRXV ZRUN >  @ 7DEOH  VKRZV WKH YDOXHV RI WKH DFFXUDF\ $ WKHVSHFLILFLW\ 6 WKHUHFDOO 5 DQGWKHSUHFLVLRQ 3 RIWKHUHFRJQLWLRQRIHDFK$'/DVZHOODVWKH DYHUDJHRIWKHVHSDUDPHWHUVLQWKH$QGURLGOLEUDU\GHYHORSHGYHULI\LQJWKDWWKH$'/ZLWKKLJKHVWUHFRJQLWLRQDFFXUDF\ LVWKHVWDQGLQJZLWKDSUHFLVLRQRIDQGDUHFDOORILQFUHDVLQJWKHUHVXOWVLQWKHIUDPHZRUNEHFDXVHLW LQIOXHQFHVWKHUHVXOWVRIVWDJH 7DEOH5HFRJQLWLRQRIHDFK$'/LQVWDJHPHDQYDOXHV   $'/ $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH $FFHOHURPHWHU 0DJQHWRPHWHU $FFHOHURPHWHU $  6  5  3  $  6  5  3  $  6  5  3  *RLQJ'RZQVWDLUV             *RLQJ8SVWDLUV             5XQQLQJ             6WDQGLQJ             :DONLQJ             )ROORZLQJWKHUHVXOWVREWDLQHGLQVWDJHUHODWHGWRWKHUHFRJQLWLRQRIWKHHQYLURQPHQWVZHWULHGWRXVHWKHVDPH PHWKRG LPSOHPHQWHG LQ WKH SUHYLRXV ZRUNV >  @ ZKLFK LV WKH )11 PHWKRG ZLWK %DFNSURSDJDWLRQ DQG QRQ QRUPDOL]HGGDWDEXWLQVWHDGRIWKHRIWKHUHFRJQLWLRQDFFXUDF\WKH$QGURLGOLEUDU\UHSRUWHGDQDFFXUDF\RI  DQG RQO\  HQYLURQPHQWV ZHUH UHFRJQL]HG +RZHYHU GXH WR WKH FDSDELOLWLHV RI WKH PRELOH GHYLFHV WKH UHFRJQLWLRQREWDLQHGZDVYHU\ORZDQGZHWULHGWRFRPELQHWKHXVHRIWKH)11PHWKRGZLWK%DFNSURSDJDWLRQDQGQRQ QRUPDOL]HG GDWD ZLWK WKH XVH RI WKH '11 PHWKRG ZLWK QRUPDOL]HG GDWD ZKLFK SURYHV WKDW WKH DFFXUDF\ REWDLQHG LQFUHDVHVWRUHFRJQL]LQJRIWKHSURSRVHGHQYLURQPHQWV$VZHYHULILHGWKDWWKHXVHRIWKH'11PHWKRG ZLWKQRUPDOL]HGGDWDLQFUHDVHVWKHUHFRJQLWLRQDFFXUDF\RIWKHHQYLURQPHQWVZHWULHGWRLPSOHPHQWRQO\WKH'11 PHWKRG ZLWK QRUPDOL]HG GDWD UHSRUWLQJ DQ DFFXUDF\ RI  UHFRJQL]LQJ DOO RI  HQYLURQPHQWV ZLWK EHWWHU SHUIRUPDQFHEHFDXVHWKHH[HFXWLRQRIWKH'11PHWKRGUHTXLUHVOHVVUHVRXUFHVWKDQWKHH[HFXWLRQRIWKH)11PHWKRG ZLWK %DFNSURSDJDWLRQ 7DEOH  VKRZV WKH YDOXHV RI WKH DFFXUDF\  $  WKH VSHFLILFLW\  6  WKH UHFDOO  5 DQG WKH SUHFLVLRQ 3 RIWKHUHFRJQLWLRQRIHDFKHQYLURQPHQWDVZHOODVWKHDYHUDJHRIWKHVHSDUDPHWHUVLQWKH$QGURLGOLEUDU\ GHYHORSHG YHULI\LQJ WKDW WKH UHVXOWV ZLWK WKH KLJKHVW SUHFLVLRQ ZHUH DFKLHYHG LQ WKH VWUHHW DQG WKH EHGURRP HQYLURQPHQWV DQG WKH KLJKHVW UHFDOO YDOXH LQ WKH UHFRJQLWLRQ RI WKH EHGURRP HQYLURQPHQW 7DEOH5HFRJQLWLRQRIHDFKHQYLURQPHQWLQVWDJHPHDQYDOXHV   (QYLURQPHQWV 0LFURSKRQH $  6  5  3  %HGURRP     %DU     &ODVVURRP     *\P     +DOO     .LWFKHQ     /LEUDU\     6WUHHW     139 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO (Table 28) contd..... 0LFURSKRQH (QYLURQPHQWV :DWFKLQJ79OLYLQJURRP $  6  5  3      )ROORZLQJWKHUHVXOWVREWDLQHGLQVWDJHUHODWHGWRWKHUHFRJQLWLRQRIWKH$'/ZLWKRXWERG\PRWLRQZLWK$QGURLG OLEUDU\GHYHORSHGWKH$'/ZLWKRXWERG\PRWLRQLVUHFRJQL]HGZLWKRIUHFRJQLWLRQDFFXUDF\ZKLFKLVWKHVDPH DFFXUDF\UHSRUWHGE\WKHSUHYLRXVZRUNV>@7DEOHVDQGVKRZWKHYDOXHVRIWKHDFFXUDF\ $ WKHVSHFLILFLW\ 6 WKHUHFDOO 5 DQGWKHSUHFLVLRQ 3 RIWKHUHFRJQLWLRQRIHDFK$'/ZLWKRXWERG\PRWLRQDVZHOODVWKHDYHUDJHRI WKHVHSDUDPHWHUVLQWKH$QGURLGOLEUDU\GHYHORSHGYHULI\LQJWKDWWKHYDOXHVRIWKHDFFXUDF\UHFDOOVSHFLILFLW\DQG SUHFLVLRQDUHDOZD\V 7DEOH6WDWLVWLFDODQDO\VLVRIWKHUHFRJQLWLRQRIHDFK$'/LQVWDJH $'/ $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH $  6  5  3  $  6  5  3  $  6  5  3  :DWFKLQJ796WDQGLQJ             6OHHSLQJ             'ULYLQJ             7DEOH6WDWLVWLFDODQDO\VLVRIWKHUHFRJQLWLRQRIHDFK$'/LQVWDJH FRQW  $'/ $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH *36 $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH *36 $FFHOHURPHWHU 0LFURSKRQH *36 $  6  5  3  $  6  5  3  $  6  5  3  :DWFKLQJ796WDQGLQJ             6OHHSLQJ             'ULYLQJ             )ROORZLQJWKHRYHUDOOUHVXOWVREWDLQHGE\WKHIUDPHZRUNLQWKHSUHYLRXVZRUNV>@IRUWKHPHDVXUHPHQWRIWKH RYHUDOODFFXUDF\ZHWRRNLQDFFRXQWWKDWZKHQWKH$'/ZDVUHFRJQL]HGDVDQDFWLYLW\ZLWKRXWPRWLRQLWZDVDOZD\V FRUUHFWO\ FODVVLILHG 7KXV XVLQJ WKH VDPH GDWDVHW IRU WKH WHVWV LQ WKH $QGURLG OLEUDU\ ZH WHVWHG WKH IORZ RI WKH IUDPHZRUN IRU WKH UHFRJQLWLRQ RI $'/ UHSRUWLQJ DFFXUDFLHV EHWZHHQ  DQG  ZLWK WKH DFFHOHURPHWHU VHQVRU EHWZHHQ  DQG  ZLWK WKH DFFHOHURPHWHU DQG PDJQHWRPHWHU VHQVRUV DQG EHWZHHQ  DQG GHSHQGLQJRQWKHQXPEHURI$'/UHFRJQL]HGWKDWYDULHVEHWZHHQDQG$'/7DEOHVDQGVKRZWKH YDOXHVRIWKHDFFXUDF\ $ WKHVSHFLILFLW\ 6 WKHUHFDOO 5 DQGWKHSUHFLVLRQ 3 RIWKHUHFRJQLWLRQRIHDFK$'/DV ZHOO DV WKH DYHUDJH RI WKHVH SDUDPHWHUV LQ WKH $QGURLG OLEUDU\ GHYHORSHG YHULI\LQJ LQ JHQHUDO WKH UHVXOWV REWDLQHG UHSRUWHG KLJKHVW SUHFLVLRQ DQG VSHFLILFLW\ ZKHUH WKH ORZHU DFFXUDFLHV ZHUH UHSRUWHG IRU WKH JRLQJ XSVWDLUV DQG GRZQVWDLUVDFWLYLWLHV 7DEOH6WDWLVWLFDODQDO\VLVRIWKHUHFRJQLWLRQRIHDFK$'/LQWKHIUDPHZRUN $'/ 140 $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH $  6  5  3  $  6  5  3  $  6  5  3  *RLQJ'RZQVWDLUV             *RLQJ8SVWDLUV             5XQQLQJ             6WDQGLQJ             :DONLQJ             6OHHSLQJ             Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH (Table 31) contd..... $'/ 'ULYLQJ $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH $FFHOHURPHWHU 0LFURSKRQH $  6  5  3  $  6  5  3  $  6  5  3              7DEOH6WDWLVWLFDODQDO\VLVRIWKHUHFRJQLWLRQRIHDFK$'/LQWKHIUDPHZRUN FRQW  $'/ $FFHOHURPHWHU 0LFURSKRQH *36 $FFHOHURPHWHU 0DJQHWRPHWHU 0LFURSKRQH *36 $FFHOHURPHWHU 0DJQHWRPHWHU *\URVFRSH 0LFURSKRQH *36 $  6  5  3  $  6  5  3  $  6  5  3  *RLQJ'RZQVWDLUV             *RLQJ8SVWDLUV             5XQQLQJ             6WDQGLQJ             :DONLQJ             6OHHSLQJ             'ULYLQJ             2QH RI WKH SRVVLEOH FDXVHV RI WKH YDULDELOLW\ RI WKH UHVXOWV EHWZHHQ WKH SUHYLRXV ZRUN DQG WKH $QGURLG OLEUDU\ GHYHORSHGPD\EHWKHUHVWULFWLRQVLQKDUGZDUHDQGVRIWZDUHYHULILHGLQWKHPRELOHGHYLFHVWKDWZLOOEHDQDO\]HGLQWKH QH[WVHFWLRQ /LPLWDWLRQV 'XH WR WKH VRIWZDUH DQG KDUGZDUH OLPLWDWLRQV RI WKH PRELOH GHYLFHV WKHLU XVH IRU WKH UHFRJQLWLRQ RI $'/ DQG HQYLURQPHQWVKDVVHYHUDOFKDOOHQJHVZKLFKDUHHYLGHQFHGLQVHYHUDOVWXGLHV>@7KHVHDUH 7KHXVHRIVHYHUDOW\SHVRIFRQQHFWLRQ HJ:L)LDQG%OXHWRRWK GHFUHDVHVWKHXVDEOHEDWWHU\WLPH 'XULQJWKHSURFHVVLQJRIWKHVHQVRUVlGDWDWKHSHUIRUPDQFHRIWKHPRELOHGHYLFHPD\GHFUHDVH 'XHWRWKHH[HFXWLRQRIVHYHUDOWDVNVDWWKHVDPHWLPHDQGWKHUHGXFHGUHVRXUFHVRIWKHVHGHYLFHVWKHIUHTXHQF\ RIWKHGDWDDFTXLVLWLRQLVQRWFRQVWDQWRYHUWKHWLPH 7KHSODFHPHQWRIWKHPRELOHGHYLFHLVRQHRIWKHPRVWLPSRUWDQWSUREOHPVEHFDXVHLWLVYHU\LPSRUWDQWIRUWKH DFTXLVLWLRQRIYDOLGGDWDZKHUHWKHUHDUHQRVWXGLHVSURYLGLQJWKHDEVROXWHRSWLPDOSRVLWLRQWKDWFDQEHJRRGIRU DQ\W\SHRIUHFRJQLWLRQRQHZDQWVWRSHUIRUPEHFDXVHLWGHSHQGVRQWKHSXUSRVHRIWKHVWXG\ ,IWKHSURFHVVLQJRIWKHVHQVRUVlGDWDLVSHUIRUPHGDWDUHPRWHVHUYHUDSHUPDQHQWGDWDFRQQHFWLRQVKRXOGEH DYDLODEOHWKLVZLOODOVRFRQWULEXWHWRIXUWKHUGHFUHDVHWKHXVDEOHEDWWHU\WLPH 7KH QXPEHU RI VHQVRUV DYDLODEOH LQ WKH FRQVXPHU PRELOH GHYLFHV LV GLIIHUHQW E\ KDUGZDUH YHUVLRQ DQG PDQXIDFWXUHUVEXWWKHGHYHORSHGPHWKRGVVKRXOGEHDIXQFWLRQRIWKHQXPEHURIVHQVRUVDYDLODEOHRQWKHVH GHYLFHVPRUHRYHUGLIIHUHQWPDQXIDFWXUHUVPD\UHSRUWGLIIHUHQWHUURUPDUJLQVDQGHYHQGLIIHUHQWRXWSXWYDOXH UDQJHV 5HJDUGLQJ WKH XVH RI PXOWLSOH GHYLFHV WKH VHFXULW\ SULYDF\ DQG UHVLOLHQFH RI WKH GDWD H[FKDQJHG RYHU WKH GLIIHUHQWW\SHVRIQHWZRUNFRQQHFWLRQVVKRXOGEHWDNHQLQWRDFFRXQWIRUWKHFRUUHFWXVHRIWKHPRELOHV\VWHPV )RUWKHPLQLPL]DWLRQRIWKHHIIHFWVRIWKHDIRUHPHQWLRQHGOLPLWDWLRQVVHYHUDOVWUDWHJLHVDUHSURSRVHGLQRXUVWXG\ WKHVHDUH 7KHVHUHVHDUFKVWXGLHVPD\RQO\XVHWKHVHQVRUVDYDLODEOHLQFRQVXPHUPRELOHGHYLFHVUHGXFLQJWKHFRQVWUDLQWV ZLWKWKHFRQQHFWLRQWRH[WHUQDOVHQVRUVDQGWKHEDWWHU\OLIHWLPH 7KHXVHRIOLJKWZHLJKWPHWKRGVPD\UHGXFHWKHSUREOHPVZLWKWKHUHGXFHGSRZHUSURFHVVLQJRIWKHVHGHYLFHV $QRWKHUSRVVLEOHVROXWLRQLVWRHQDEOHDQGGLVDEOHWKHDFTXLVLWLRQRIWKHVHQVRUVlGDWDRYHUWKHWLPHEHFDXVHLWLV QRWQHFHVVDU\WRKDYHDFRQWLQXRXVUHDOWLPHVHQVRUGDWDDFTXLVLWLRQLQRUGHUWRUHFRJQL]HWKH$'/DQGWKHLU HQYLURQPHQWV 7KH FUHDWLRQ RI PHWKRGV DGDSWHG WR WKH GLIIHUHQW VHWV RI VHQVRUV DYDLODEOH LQ WKH FRQVXPHU PRELOH GHYLFHV 141 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO LQFUHDVHVWKHDYDLODELOLW\RIWKHPHWKRGVIRUDODUJHJURXSRISHRSOH 7KHXVHRIORFDOGDWDSURFHVVLQJUHGXFHVWKHWLPHWRJLYHIHHGEDFNWRWKHXVHUEXWLWDOVRUHGXFHVWKHSUREOHPV ZLWKWKHVHFXULW\DQGSULYDF\RIWKHSHUVRQDOGDWDGHVSLWHLQFUHDVLQJWKHFRPSXWDWLRQEXUGHQRIWKHGHYLFH &RPSDULVRQZLWK3ULRU:RUN %DVHGRQWKHSUHYLRXVUHVHDUFKVWXGLHVDYDLODEOHLQWKHOLWHUDWXUH7DEOHSUHVHQWVWKHFRPSDULVRQEHWZHHQWKH PLQLPXPPD[LPXPDQGDYHUDJHRIWKHDFFXUDF\DFKLHYHGDQGWKHUHVXOWVUHSRUWHGE\WKH$QGURLGOLEUDU\GHYHORSHG 7KHUHVXOWVWKDWDUHFRPSDUDEOHDUHKLJKOLJKWHGLQ7DEOHYHULI\LQJWKDWWKHKLJKHVWLPSURYHPHQWZDVYHULILHGLQWKH UXQQLQJDFWLYLW\ZKHQZHFRPSDUHGRXUUHVXOWVZKLWWKHDYHUDJHDFFXUDF\DFKLHYHGLQSULRUZRUN)RUWKDWDFWLYLW\WKH PLQLPXPDQGWKHPD[LPXPDFFXUDF\IRXQGHGDUHDQGUHVSHFWLYHO\UHWXUQHGDSRVLWLYH  DQG QHJDWLYHLPSURYHPHQW   !""# $"# %"# &"# '"# ("# )"# *"# F0E/8-0G603> M:8<96>:A3> 73?0B/01 D8/E/01 ;-/01<B-C0>3?/8> @?A./01 ;-/01<:=>3?/8> 9:00/01 ,--./012./34560 738663 +"# HDI HE68?16<H44:8?4K J?L/G:0<H44:8?4K J/0/G:G<H44:8?4K )LJ  &RPSDULVRQEHWZHHQWKHPLQLPXPPD[LPXPDQGDYHUDJHRIWKHDFFXUDFLHVREWDLQHGLQWKHOLWHUDWXUHUHYLHZZLWKWKH FRPPRQ$'/DQGHQYLURQPHQWVUHFRJQL]HGE\WKH$QGURLGOLEUDU\GHYHORSHG 7DEOH&RPSDULVRQEHWZHHQWKHPLQLPXPPD[LPXPDQGDYHUDJHRIWKHDFFXUDFLHVREWDLQHGLQWKHOLWHUDWXUHUHYLHZDQG WKHDFFXUDF\RIRXUUHVXOWVLQWKHUHFRJQLWLRQRIWKHVHOHFWHG$'/DQGHQYLURQPHQWV b (QYLURQPHQWV $'/(QYLURQPHQWV %HGURRP &RRNLQJNLWFKHQ $YHUDJH $FFXUDF\ $YJ 1$   2XU5HVXOWV 25    ,PSURYHPHQWV 250LQ 250D[ 25$YJ       *\P 1$     &ODVVURRP 1$     +DOO 1$     :DWFKLQJ79OLYLQJURRP 1$     %DU 1$     /LEUDU\ 1$       6WUHHW 142 0LQLPXP 0D[LPXP $FFXUDF\ 0LQ $FFXUDF\ 0D[      Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH (Table 33) contd..... $'/(QYLURQPHQWV b $'/ 0LQLPXP 0D[LPXP $FFXUDF\ 0LQ $FFXUDF\ 0D[ $YHUDJH $FFXUDF\ $YJ 2XU5HVXOWV 25 ,PSURYHPHQWV 250LQ 250D[ 25$YJ 5XQQLQJ        *RLQJXSVWDLUV        :DONLQJ        *RLQJGRZQVWDLUV        'ULYLQJ        6WDQGLQJ        6OHHSLQJ 1$     2EV1$PHDQVWKDWWKH$'/DQGHQYLURQPHQWVZHUHQRWUHFRJQL]HGLQUHVHDUFKVWXGLHVDYDLODEOHLQWKHOLWHUDWXUH )ROORZLQJ WKH FRPSDUDEOH UHVXOWV SUHVHQWHG LQ )LJ    ZH YHULILHG WKDW RXU UHVXOWV DUH DOZD\V KLJKHU WKDQ WKH DYHUDJHRIWKHUHVXOWVREWDLQHGLQWKHSUHYLRXVUHVHDUFKVWXGLHVLQWKHOLWHUDWXUH +RZHYHUWKHVLJQLILFDQFHRIWKHLPSURYHPHQWV VHH25$YJLQ7DEOH ZDVHYDOXDWHGZLWKWKH6WXGHQWlVWWHVW IRUDVDPSOH$VSUHVHQWHGLQ7DEOHWKHQRUPDOLW\DVVXPSWLRQZDVDFKLHYHGDQGLWZDVDVVHVVHGE\WKH6KDSLUR :LONWHVW 76:   SBYDOXH  63666WDWLVWLFV6RIWZDUH Y,%06366&KLFDJR,/ >@ZDVXVHGWR SHUIRUP WKH VWDWLVWLFDO WHVWV 7KH LPSURYHPHQWV IRU ZKLFK WKH SYDOXH RI WKH WHVW LV ORZHU WKDQ  DUH FRQVLGHUHG VWDWLVWLFDOO\VLJQLILFDQWDWOHYHORIFRQILGHQFH 7DEOH1RUPDOLW\WHVWV b .ROPRJRURY6PLUQRY6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\ 6KDSLUR:LON 6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\       25 $YJ 7KHUHVXOWVRIWKH6WXGHQWlVWWHVWIRUWKHVDPSOHXVHGDUHSUHVHQWHGLQ7DEOHUHSRUWLQJDYDOXHRIWKHDYHUDJHRI LPSURYHPHQWV HTXDOV WR  $FFRUGLQJ WR WKH 6WXGHQWlV WWHVW WKLV YDOXH LV VWDWLVWLFDOO\ VLJQLILFDQW  W   SBYDOXH   7DEOH9DOXHVRIWKH6WXGHQW VWWHVWIRUDVDPSOHPHDQIRUFRPSDULQJRXUUHVXOWVDQGDYHUDJHDFFXUDF\ 7HVW9DOXH  b b WVFRUH 'HJUHHVRI)UHHGRP 6LJQLILFDQFH3UREDELOLW\ %LODWHUDO 25$YJ    $YHUDJHRIWKH'LIIHUHQFH   b &RQILGHQFH,QWHUYDORI'LIIHUHQFH /RZHVW +LJKHVW   ,QDGGLWLRQWKHVDPHSURFHGXUHZDVGRQHIRUDQDO\]HWKHLPSURYHPHQWVREWDLQHGZKHQFRPSDUHGRXUUHVXOWVZKLW WKHPLQLPXPDQGPD[LPXPUHVXOWIRXQGHGLQWKHOLWHUDWXUH VHH250LQDQG250D[LQ7DEOHUHVSHFWLYHO\  ,Q7DEOHWKHQRUPDOLW\DVVXPSWLRQZDVDFKLHYHG 6KDSLUR:LONWHVWIRU250LQDQG250D[UHVSHFWLYHO\ 76:   SBYDOXH  DQG 76:   SBYDOXH   7DEOH1RUPDOLW\WHVWV b .ROPRJRURY6PLUQRY6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\ 6KDSLUR:LON 6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\ 250LQ       250D[       %DVHGLQ7DEOHWKHUHVXOWVRI6WXGHQWlVWHVWVKRZLQRQHKDQGWKDWRXUUHVXOWVDQGWKHPLQLPXPDFFXUDF\IRXQG LQWKHOLWHUDWXUHDUHVWDWLVWLFDOO\GLIIHUHQW W   SBYDOXH  DQGWKH\DUHKLJKHU DYHUDJHRIWKHGLIIHUHQFH EHWZHHQ25DQG0LQLVHTXDO ,QWKHRWKHUKDQGZKHQFRPSDULQJRXUUHVXOWVDQGWKHPD[LPXPDFKLHYHGLQ OLWHUDWXUHZHFDQDFFHSWWKHQHJDWLYHDYHUDJHRIWKHGLIIHUHQFHEHWZHHQWKHP DYHUDJHRIWKHGLIIHUHQFHEHWZHHQ25 DQG0D[LVHTXDO LVHTXDO]HUR W   SBYDOXH   143 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO 7DEOH9DOXHVRIWKH6WXGHQW VWWHVWIRUDVDPSOHPHDQIRUFRPSDULQJRXUUHVXOWVZLWKPLQLPXPDQGRXUUHVXOWVZLWK PD[LPXPDFFXUDF\ 7HVW9DOXH  b b b &RQILGHQFH,QWHUYDORI'LIIHUHQFH WVFRUH 'HJUHHVRI )UHHGRP 6LJQLILFDQFH3UREDELOLW\ %LODWHUDO $YHUDJHRIWKH'LIIHUHQFH  /RZHVW +LJKHVW 250LQ       250D[       )LQDOO\EDVHGRQWKHDFFXUDFLHVSUHVHQWHGLQWKH7DEOHDQGWKHQXPEHURI$'/(QYLURQPHQWVUHFRJQL]HGLQWKH OLWHUDWXUHSUHVHQWHGLQWKH7DEOHZHFDQFRQFOXGHWKDWRXUPHWKRGLVEHWWHUWKDQRWKHUDYDLODEOHLQWKHOLWHUDWXUH UHYLHZEHFDXVHRXULPSURYHPHQWLVUHOLDEOHEHFDXVHZHUHFRJQL]HDPDMRUQXPEHURI$'/(QYLURQPHQWVWKDQWKH RWKHUSUHYLRXVZRUNVDQGZHREWDLQHGVWDWLVWLFDOO\VLJQLILFDQWUHVXOWV 7DEOH  1XPEHU RI $'/(QYLURQPHQWV UHFRJQL]HG E\ WKH VWXGLHV ZKHUH WKH PLQLPXP PD[LPXP DQG DYHUDJH RI WKH DFFXUDFLHVREWDLQHGLQWKHOLWHUDWXUHUHYLHZZHUHYHULILHG $'/(QYLURQPHQWV 1XPEHURI$'/ (QYLURQPHQWVLQWKH 6WXG\ZLWK0D[LPXP $FFXUDF\ 1XPEHURI$'/ (QYLURQPHQWVLQWKH 6WXG\ZLWK0LQLPXP $FFXUDF\ $YHUDJHRIWKHQXPEHURI $'/(QYLURQPHQWV 5HFRJQL]HGLQWKH /LWHUDWXUH &RRNLQJNLWFKHQ    6WUHHW    5XQQLQJ    *RLQJXSVWDLUV    :DONLQJ    *RLQJGRZQVWDLUV    'ULYLQJ    6WDQGLQJ    b (QYLURQPHQWV $'/ 2XU1XPEHURI $'/ (QYLURQPHQWV  ,QDGGLWLRQZHSHUIRUPHGWKHFRPSDULVRQEHWZHHQRXWQXPEHURI$'/(QYLURQPHQWVUHFRJQL]HGZLWKWKHPHDQ QXPEHU RI $'/(QYLURQPHQWV UHFRJQL]HG LQ WKH OLWHUDWXUH HYDOXDWLQJ WKH VLJQLILFDQFH RI RXU QXPEHU RI $'/(QYLURQPHQWV UHFRJQL]HG ZLWK WKH 6WXGHQWlV WWHVW IRU D VDPSOH $V SUHVHQWHG LQ 7DEOH  WKH QRUPDOLW\ DVVXPSWLRQ ZDV DFKLHYHG DQG LW ZDV DVVHVVHG E\ WKH 6KDSLUR:LON WHVW  76:    SBYDOXH   6366 6WDWLVWLFV6RIWZDUH Y,%06366&KLFDJR,/ >@ZDVXVHGWRSHUIRUPWKHVWDWLVWLFDOWHVWV7KHLPSURYHPHQWVIRU ZKLFKWKHSYDOXHRIWKHWHVWLVORZHUWKDQDUHFRQVLGHUHGVWDWLVWLFDOO\VLJQLILFDQWDWOHYHORIFRQILGHQFH 7DEOH1RUPDOLW\WHVWV b 0HDQQXPEHURI$'/ (QYLURQPHQWV .ROPRJRURY6PLUQRY 6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\ 6KDSLUR:LON 6WDWLVWLFV 'HJUHHVRI )UHHGRP 6LJQLILFDQFH 3UREDELOLW\       7KHUHVXOWVRIWKH6WXGHQWlVWWHVWIRUWKHVDPSOHXVHGDUHSUHVHQWHGLQ7DEOHUHSRUWLQJDYDOXHRIWKHQXPEHURI $'/(QYLURQPHQWV UHFRJQL]HG HTXDOV WR  $FFRUGLQJ WR WKH 6WXGHQWlV WWHVW WKLV YDOXH LV VWDWLVWLFDOO\ VLJQLILFDQW W  SBYDOXH   7DEOH9DOXHVRIWKH6WXGHQW VWWHVWIRUDVDPSOHPHDQIRUFRPSDULQJRXUQXPEHURI$'/(QYLURQPHQWVZLWKWKHDYHUDJH QXPEHURI$'/(QYLURQPHQWVUHFRJQL]HGLQWKHOLWHUDWXUH 7HVW9DOXH  b b WVFRUH 0HDQQXPEHURI$'/ (QYLURQPHQWV  'HJUHHVRI )UHHGRP 6LJQLILFDQFH3UREDELOLW\ %LODWHUDO $YHUDJHRIWKH 'LIIHUHQFH     b &RQILGHQFH,QWHUYDORI'LIIHUHQFH /RZHVW +LJKHVW   7KH GHYHORSPHQW RI WKH OLEUDU\ KDV VHYHUDO FKDOOHQJHV LQFOXGLQJ WKH OLPLWHG EDWWHU\ OLIHWLPH DQG WKH UHGXFHG 144 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH SURFHVVLQJSRZHURIWKHPRELOHGHYLFHV>@+RZHYHUWKHDFTXLVLWLRQRIVHFRQGVRIUDZGDWDHYHU\PLQXWHVXVLQJ WKHROGHVWGHYLFHVWHVWHGKDVVKRZHGDPLQLPXPRIKRXUVRIEDWWHU\OLIHZLWKDQRUPDOXVDJHDQGDVFXUUHQWO\WKH PRELOHGHYLFHVUHTXLUHGDGDLO\UHFKDUJHLWLVSRVVLEOHWRFRQVLGHUWKDWWKHDFTXLVLWLRQRIWKHVHQVRUVlGDWDZLWKWKLV PHWKRGFDQEHLPSOHPHQWHG7KHSHUIRUPDQFHRIWKHPRELOHGHYLFHLVRQO\VWULFWO\DIIHFWHGGXULQJDVPDOOWLPHHYHU\ PLQXWHVDQGWKHHIIHFWVLQWKHSHUIRUPDQFHDUHQRWVLJQLILFDQW,WZLOOEHSUREDEO\VXIILFLHQWIRUWKHFKDUDFWHUL]DWLRQRI OLIHVW\OHV EXW LW PD\ PLVV VRPH LPSRUWDQW HYHQWV WKDW PD\ QRW EH LGHQWLILHG LQFOXGLQJ IDOOLQJ DFWLYLWLHV 7KH GHYHORSPHQWRIDPHWKRGWKDWLPSOHPHQWVDPRUHVLJQLILFDQWVDPSOLQJVWUDWHJ\ZLWKRXWGHFUHDVLQJWKHSHUIRUPDQFHDQG WKH DYDLODELOLW\ RI UHVRXUFHV DW WKH PRELOH GHYLFH ZKLOH LGHQWLI\LQJ D ZLGHU QXPEHU RI $'/V UHTXLUHV DGGLWLRQDO UHVHDUFK 7KHGDWDSURFHVVLQJLVH[HFXWHGDIWHUWKHGDWDDFTXLVLWLRQIRUWKHSUHSDUDWLRQRIWKHGDWDIRUIXUWKHUFODVVLILFDWLRQLQ WKHODVWVWDJH7KHGDWDSURFHVVLQJZLOOFDOFXODWHWKHIHDWXUHVIURPWKHUDZGDWDDQGDIWHUWKDWLIWKHPLQLPXPUHTXLUHG GDWD LV DYDLODEOH  LH WKH DFFHOHURPHWHU GDWD LV FRUUHFWO\ DFTXLUHG  WKH FDOFXODWHG IHDWXUHV ZLOO EH VHQW WR WKH FODVVLILFDWLRQPRGXOHZKHUHWKHYHULILFDWLRQRIWKH$'/H[HFXWHGLVSHUIRUPHGDQGLQFDVHRIDVLPLODUSDWWHUQLVQRW DOUHDG\WUDLQHGLQ$11FUHDWHGIRUWKHGHYHORSPHQWRIWKHPHWKRGLQ>@WKHGDWDZLOOEHGLVFDUGHG1HYHUWKHOHVV WKHGDWDLVRQO\VWRUHGXQWLOWKHFODVVLILFDWLRQDIWHUZKLFKLWZLOOEHDXWRPDWLFDOO\GLVFDUGHGDYRLGLQJSUREOHPVUHODWHG WRWKHVWRUDJHFDSDFLW\RIWKHVHGHYLFHV &21&/86,21 7KH DXWRPDWLF UHFRJQLWLRQ RI $'/ DQG WKHLU HQYLURQPHQWV PD\ EH SHUIRUPHG ZLWK WKH VHQVRUV DYDLODEOH LQ FRQVXPHUPRELOHGHYLFHVLQFOXGLQJWKHDFFHOHURPHWHUWKHJ\URVFRSHWKHPDJQHWRPHWHUWKHPLFURSKRQHDQGWKH*36 UHFHLYHU7KLVSDSHUFRQVLVWVRIWKHFUHDWLRQRIDQ$QGURLGOLEUDU\EDVHGRQDIUDPHZRUNIRUWKHUHFRJQLWLRQRI$'/DQG WKHLUHQYLURQPHQWVFUHDWHGE\SURSHUO\PHUJLQJPHWKRGVDQGWRROVDYDLODEOHDW>@7KHOLEUDU\GHYHORSHG VKRXOGEHDIXQFWLRQRIWKHQXPEHURIVHQVRUVDYDLODEOHLQWKHPRELOHGHYLFHVDQGDEOHWRSURYLGHDUDSLGIHHGEDFNWR WKHXVHUWKDQNVWRWKHORFDOSURFHVVLQJRIWKHVHQVRUVlGDWD 7KH $QGURLG OLEUDU\ LPSOHPHQWHG LQFOXGHV VRPH VWDJHV RI WKH IUDPHZRUN WKDW KDYH EHHQ DOUHDG\ SURSRVHG LQFOXGLQJ GDWD SURFHVVLQJ GDWD IXVLRQ DQG FODVVLILFDWLRQ PHWKRGV 7KH GDWD DFTXLVLWLRQ VWDJH KDV WR EH SHUIRUPHG DFFRUGLQJWRWKHPHWKRGVVXSSRUWHGE\WKH$QGURLG2SHUDWLQJ6\VWHPLQHDFKGHYLFH5HODWHGWRWKHGDWDSURFHVVLQJ SURSHU GDWD FOHDQLQJ PHWKRGV VKRXOG EH DSSOLHG D ORZ SDVV ILOWHU VKRXOG EH DSSOLHG IRU WKH PRWLRQ DQG PDJQHWLF VHQVRUVlGDWD))7VKRXOGEHDSSOLHGIRUWKHDFRXVWLFVHQVRUVlGDWDDQGFRUUHFWIHDWXUHVVKRXOGEHH[WUDFWHGIRUIXUWKHU IXVLRQDQGFODVVLILFDWLRQRIWKHVHQVRUVGDWD$IWHUWKHGDWDSURFHVVLQJVWHSWKHIHDWXUHVVKRXOGEHIXVHGQRUPDOL]HG DQGWKH'11PHWKRGVKRXOGEHDSSOLHG7KHVWUXFWXUHRIWKHIUDPHZRUNIRUWKHFODVVLILFDWLRQLVVHSDUDWHGLQWRWKUHH VWDJHVWKHUHFRJQLWLRQRIFRPPRQ$'/WKHUHFRJQLWLRQRIHQYLURQPHQWVDQGWKHUHFRJQLWLRQRI$'/ZLWKRXWERG\ PRWLRQ7KLVVWUXFWXUHLVSRVVLEOHEHFDXVHHDFKVWDJHXVHVGLIIHUHQWVHQVRUVlGDWD)LUVWO\WKHUHFRJQLWLRQRIFRPPRQ $'/ PD\ XVH WKH DFFHOHURPHWHU PDJQHWRPHWHU DQG J\URVFRSH VHQVRUVl GDWD ZLWK WKH '11 PHWKRG UHSRUWLQJ DQ DYHUDJHDFFXUDF\EHWZHHQDQGIRUWKHUHFRJQLWLRQRIJRLQJXSVWDLUVDQGGRZQVWDLUVZDONLQJUXQQLQJ DQG VWDQGLQJ 6HFRQGO\ WKH UHFRJQLWLRQ RI WKH HQYLURQPHQWV PD\ XVH WKH PLFURSKRQH GDWD ZLWK WKH '11 PHWKRG UHSRUWLQJ DQ DFFXUDF\ RI  IRU WKH UHFRJQLWLRQ RI GLIIHUHQW VRXQG HQYLURQPHQWV LQFOXGLQJ EHGURRP FRRNLQJNLWFKHQJ\PFODVVURRPKDOOZDWFKLQJ79OLYLQJURRPEDUOLEUDU\DQGVWUHHW)LQDOO\WKHUHFRJQLWLRQRI $'/ZLWKRXWERG\PRWLRQPD\EHSHUIRUPHGZLWKWKHHQYLURQPHQWUHFRJQL]HGDQGWKHDFFHOHURPHWHUPDJQHWRPHWHU J\URVFRSHDQG*36UHFHLYHUVHQVRUVlGDWDUHSRUWLQJDQDFFXUDF\RIIRUWKHUHFRJQLWLRQRIZDWFKLQJ79OLYLQJ URRPVOHHSLQJGULYLQJDQGRWKHUXQGLIIHUHQWLDWHGVWDQGLQJDFWLYLWLHV,QVXPPDU\WKHRYHUDOODFFXUDF\RIWKH$QGURLG OLEUDU\ LV EHWZHHQ  DQG  GHSHQGLQJ RQ WKH QXPEHU RI VHQVRUV XVHG DQG WKH QXPEHU RI $'/ DQG HQYLURQPHQWVUHFRJQL]HG 7KHUHDUHVHYHUDOOLPLWDWLRQVUHODWHGWRWKHXVHRIPRELOHGHYLFHVWKDWFDQEHPLQLPL]HGZLWKVHYHUDOWHFKQLTXHV LQFOXGLQJWKHXVHRIOLJKWZHLJKWPHWKRGVDQGORFDOSURFHVVLQJ7KHUHDUHQRRWKHUGDWDVHWVDYDLODEOHWKDWFDQEHXVHGWR VLPXODWH WKH UHVXOWV REWDLQHG ZLWK WKH IUDPHZRUN ZH GHYHORSHG EHFDXVH WKH UHVXOWV DYDLODEOH LQ WKH OLWHUDWXUH DUH SURYLGHGVHSDUDWHO\IRU$'/DQGHQYLURQPHQWV6RZHFRPSDUHGWKHDFFXUDFLHVREWDLQHGYHULI\LQJWKDWRXUUHVXOWV H[KLELWDQRYHUDOOLPSURYHPHQWHTXDOVWRZKHQFRPSDUHGZLWKWKHDYHUDJHRIDFFXUDF\IRXQGLQWKHOLWHUDWXUH :HDOVRFRPSDUHGRXUUHVXOWVZLWKWKHPLQLPXPDQGWKHPD[LPXPDFFXUDF\DYDLODEOHLQWKHOLWHUDWXUHWKDWSHUPLW FRQFOXGHVWKDWRXUUHVXOWVGRQRWGLIIHUDWWKHPD[LPXPEXWWKH\DUHVWDWLFDOO\KLJKHUDWWKHPLQLPXPIRXQGLQWKH OLWHUDWXUH 7RGDWHWKHUHDUHGLIIHUHQWW\SHVRIPRELOHGHYLFHVDYDLODEOHLQWKHPDUNHW7KHUHIRUHWKLVVWXG\ZDVIRFXVHGRQWKH 145 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO LPSOHPHQWDWLRQRIDQ$QGURLGOLEUDU\IRUWKHUHFRJQLWLRQRI$'/DQGWKHLUHQYLURQPHQWVZLWKWKHXVHRIDQ$QGURLG VPDUWSKRQH7KHVHYHUDOW\SHVRIPRELOHGHYLFHVKDYHGLIIHUHQWDUFKLWHFWXUHVDQGVHQVRUVWKDWPD\UHWXUQGLIIHUHQWW\SHV RIVHQVRUVlGDWD2XUPHWKRGLVRQO\DSSOLFDEOHZKHQWKHPRELOHGHYLFHLVLQWKHIURQWSRFNHWRIWKHXVHUlVSDQWVWKH RWKHUSRVLWLRQVPD\UHWXUQLQFRUUHFWUHVXOWV)RUWKHXVHRIGLIIHUHQWGHYLFHV LHVPDUWZDWFK WKHPHWKRGVKRXOGEH DGDSWHGEHFDXVHWKHVPDUWZDWFKLVQRUPDOO\RQWKHDUPRIWKHXVHUDQGWKHDFTXLUHGSDWWHUQVRIWKHVHQVRUVlGDWDDUH GLIIHUHQWDFFRUGLQJWRWKHSRVLWLRQDQGWKHKDUGZDUHRIWKHVHGHYLFHV7KHFUHDWLRQRIDOLEUDU\WKDWVKRXOGEHDIXQFWLRQ RIWKHQXPEHURIVHQVRUVWKHSRVLWLRQLQJRIGHYLFHDQGGLIIHUHQWDUFKLWHFWXUHVH[LVWLQJLQWKHPDUNHWLVDUHVHDUFKWKDW VKRXOGEHGRQHLQWKHIXWXUH ,QFRQFOXVLRQWKH$QGURLGOLEUDU\GHYHORSHGSURYHVWKHUHOLDELOLW\RIWKHPRELOHGHYLFHVIRUWKHUHFRJQLWLRQRI$'/ DQGWKHLUHQYLURQPHQWVZLWKKLJKSUHFLVLRQ7KHOLEUDU\PD\EHXVHGLQIXWXUHVWXGLHVFRPELQLQJWKHUHFRJQLWLRQRIWKH $'/DQGWKHLUHQYLURQPHQWVZLWKRWKHUSXUSRVHVIRUWKHGHYHORSPHQWRIDSHUVRQDOGLJLWDOOLIHFRDFK (7+,&6$33529$/$1'&216(17723$57,&,3$7( 1RWDSSOLFDEOH +80$1$1'$1,0$/5,*+76 1RDQLPDOVKXPDQVZHUHXVHGIRUVWXGLHVWKDWDUHWKHEDVLVRIWKLVUHVHDUFK &216(17)2538%/,&$7,21 1RWDSSOLFDEOH &21)/,&762),17(5(67 7KHDXWKRUVGHFODUHQRFRQIOLFWRILQWHUHVWILQDQFLDORURWKHUZLVH $&.12:/('*(0(176 7KLVZRUNZDVVXSSRUWHGE\)&7SURMHFW8,'(($ (VWHWUDEDOKRIRLVXSRUWDGRSHORSURMHFWR)&7 8,'(($  7KH DXWKRUV ZRXOG DOVR OLNH WR DFNQRZOHGJH WKH FRQWULEXWLRQ RI WKH &267 $FWLRQ ,& k $$3(/( k $UFKLWHFWXUHV $OJRULWKPV DQG 3URWRFROV IRU (QKDQFHG /LYLQJ (QYLURQPHQWV 5()(5(1&(6 >@ 6DOD]DU/+$/DFHUGD71XQHV-9*UHVVHYRQ:DQJHQKHLP&$V\VWHPDWLFOLWHUDWXUHUHYLHZRQXVDELOLW\KHXULVWLFVIRUPRELOHSKRQHV,QW0RELOH+XP&RPSXW,QWHUDFW >KWWSG[GRLRUJMPKFL@ >@ )RWL'.RNHWVX-6$FWLYLWLHVRIGDLO\OLYLQJ3HGUHWWLlVRFFXSDWLRQDOWKHUDS\3UDFWLFDOVNLOOVIRUSK\VLFDOG\VIXQFWLRQ >@ 7UDMNRYLN99ODKX*MRUJLHYVND(.RFHVNL6.XOHY,*HQHUDODVVLVWHGOLYLQJV\VWHPDUFKLWHFWXUHPRGHO,QWHUQDWLRQDO&RQIHUHQFHRQ0RELOH 1HWZRUNVDQG0DQDJHPHQW >@ 6LHJHO&+RFKJDWWHUHU$'RUQHU7(&RQWULEXWLRQVRIDPELHQWDVVLVWHGOLYLQJIRUKHDOWKDQGTXDOLW\RIOLIHLQWKHHOGHUO\DQGFDUHVHUYLFHVD TXDOLWDWLYHDQDO\VLVIURPWKHH[SHUWVlSHUVSHFWLYHRIFDUHVHUYLFHSURIHVVLRQDOV%0&*HULDWU >KWWSG[GRLRUJ@>30,'@ >@ %RWLD-$9LOOD$3DOPD-$PELHQW$VVLVWHG/LYLQJV\VWHPIRULQKRPHPRQLWRULQJRIKHDOWK\LQGHSHQGHQWHOGHUV([SHUW6\VW$SSO  >KWWSG[GRLRUJMHVZD@ >@ *DUFLD105RGULJXHV--3&(OLDV'&'LDV06$PELHQW$VVLVWHG/LYLQJ7D\ORU )UDQFLV >@ 'LPLWULHYVNL$=GUDYHYVNL(/DPHVNL37UDMNRYLN9$VXUYH\RI$PELHQW$VVLVWHG/LYLQJV\VWHPV&KDOOHQJHVDQGRSSRUWXQLWLHV,Q ,QWHOOLJHQW&RPSXWHU&RPPXQLFDWLRQDQG3URFHVVLQJ ,&&3 ,(((WK,QWHUQDWLRQDO&RQIHUHQFHRQSS >KWWSG[GRLRUJ,&&3@ >@ 6RXVD36DEXJXHLUR')HOL]DUGR9&RXWR53LUHV,*DUFLD1P+HDOWK6HQVRUVDQG$SSOLFDWLRQVIRU3HUVRQDO$LG,Q0RELOH+HDOWK SS >KWWSG[GRLRUJB@ >@ =GUDYHYVNL(/DPHVNL37UDMNRYLN9.XODNRY$&KRUEHY,*ROHYD5HWDO,PSURYLQJ$FWLYLW\5HFRJQLWLRQ$FFXUDF\LQ$PELHQW $VVLVWHG/LYLQJ6\VWHPVE\$XWRPDWHG)HDWXUH(QJLQHHULQJ,((($FFHVV >KWWSG[GRLRUJ$&&(66@ 146 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH >@ &LSSLWHOOL(*DVSDUULQL6*DPEL(6SLQVDQWH6:cKVOdQ\-2UKDQ\,HWDO7LPHV\QFKURQL]DWLRQDQGGDWDIXVLRQIRU5*%GHSWKFDPHUDV DQGLQHUWLDOVHQVRUVLQ$$/DSSOLFDWLRQV&RPPXQLFDWLRQ:RUNVKRS ,&&:  >KWWSG[GRLRUJ,&&:@ >@ 'REUH&0DYURPRXVWDNLV&[*DUFLD1*ROHYD5,0DVWRUDNLV*$PELHQW$VVLVWHG/LYLQJDQG(QKDQFHG/LYLQJ(QYLURQPHQWV3ULQFLSOHV 7HFKQRORJLHVDQG&RQWURO%XWWHUZRUWK+HLQHPDQQ >@ *ROHYD5,*DUFLD100DYURPRXVWDNLV&;'REUH&0DVWRUDNLV*6WDLQRY5HWDO$$/DQG(/(3ODWIRUP$UFKLWHFWXUH >KWWSG[GRLRUJ%@ >@ $XJXVWR-&+XFK0.DPHDV$HWDO(GV+DQGERRNRI$PELHQW$VVLVWHG/LYLQJ7HFKQRORJ\IRU+HDOWKFDUH5HKDELOLWDWLRQDQG:HOO EHLQJ$PVWHUGDP1HWKHUODQGV,263UHVV >@ *DUFLD10$5RDGPDSWRWKH'HVLJQRID3HUVRQDO'LJLWDO/LIH&RDFK,Q >KWWSG[GRLRUJB@ >@ .DUD0/DPRXFKL25DPGDQH&KHULI$$4XDOLW\0RGHOIRUWKH(YDOXDWLRQ$$/6\VWHPV >KWWSG[GRLRUJMSURFV@ >@ 6SLQVDQWH6*DPEL(5DIIDHOL/0RQWDQLQL/3DFLHOOR/%HYLODFTXD5HWDO7HFKQRORJ\EDVHGDVVLVWDQFHRISHRSOHZLWKGHPHQWLDVWDWH RIWKHDUWRSHQFKDOOHQJHVDQGIXWXUHGHYHORSPHQWV+XPDQ0RQLWRULQJ6PDUW+HDOWKDQG$VVLVWHG/LYLQJ7HFKQLTXHVDQG7HFKQRORJLHV  >@ 0    -DQXUD\ QG  6PDUWSKRQH 26 PDUNHW VKDUH IRUHFDVW  _ 6WDWLVWLF  $YDLODEOH KWWSVZZZVWDWLVWDFRP VWDWLVWLFVPDUNHWVKDUHIRUHFDVWIRUVPDUWSKRQHRSHUDWLQJV\VWHPV >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD))URP'DWD$FTXLVLWLRQWR'DWD)XVLRQ$&RPSUHKHQVLYH5HYLHZDQGD5RDGPDSIRUWKH ,GHQWLILFDWLRQRI$FWLYLWLHVRI'DLO\/LYLQJ8VLQJ0RELOH'HYLFHV6HQVRUV %DVHO    >KWWSG[GRLRUJV@>30,'@ >@ 3LUHV,0*DUFLD10)OhUH]5HYXHOWD)0XOWLVHQVRUGDWDIXVLRQWHFKQLTXHVIRUWKHLGHQWLILFDWLRQRIDFWLYLWLHVRIGDLO\OLYLQJXVLQJPRELOH GHYLFHV3URFHHGLQJVRIWKH(&0/3.'''RFWRUDO&RQVRUWLXP(XURSHDQ&RQIHUHQFHRQ0DFKLQH/HDUQLQJDQG3ULQFLSOHVDQG3UDFWLFH RI.QRZOHGJH'LVFRYHU\LQ'DWDEDVHV3RUWR3RUWXJDO >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD),GHQWLILFDWLRQRI$FWLYLWLHVRI'DLO\/LYLQJ8VLQJ6HQVRUV$YDLODEOHLQRIIWKHVKHOI 0RELOH'HYLFHV5HVHDUFKDQG+\SRWKHVLV >KWWSG[GRLRUJB@ >@ ,YDVFX7&LQFDU.'LQLV$1HJUX9$FWLYLWLHVRIGDLO\OLYLQJDQGIDOOVUHFRJQLWLRQDQGFODVVLILFDWLRQIURPWKHZHDUDEOHVHQVRUVGDWD,Q  >KWWSG[GRLRUJ(+%@ >@ .LOLQF2'DO]HOO$8OXWXUN,8\VDO,,QHUWLD%DVHG5HFRJQLWLRQRI'DLO\$FWLYLWLHVZLWK$11VDQG6SHFWURWHPSRUDO)HDWXUHV >KWWSG[GRLRUJ,&0/$@ >@ 3UDERZR200XWLMDUVD.6XSDQJNDW6+0LVVLQJGDWDKDQGOLQJXVLQJPDFKLQHOHDUQLQJIRUKXPDQDFWLYLW\UHFRJQLWLRQRQPRELOHGHYLFH ,QWHUQDWLRQDO&RQIHUHQFHRQ,&7)RU6PDUW6RFLHW\ ,&,66  >KWWSG[GRLRUJ,&766@ >@ %XMDUL $ /LFDU % 3DOD]]L &( 0RYHPHQW SDWWHUQ UHFRJQLWLRQ WKURXJK VPDUWSKRQHlV DFFHOHURPHWHU &RQVXPHU &RPPXQLFDWLRQV DQG 1HWZRUNLQJ&RQIHUHQFH &&1&  >KWWSG[GRLRUJ&&1&@ >@ 9DOODEK30DOHNLDQ5<H1%RJDWLQRVND'&)DOOGHWHFWLRQXVLQJPDFKLQHOHDUQLQJDOJRULWKPV >KWWSG[GRLRUJ62)7&20@ >@ 3RPER1*DUFLD1%RXVVRQ.&ODVVLILFDWLRQWHFKQLTXHVRQFRPSXWHUL]HGV\VWHPVWRSUHGLFWDQGRUWRGHWHFW$SQHD$V\VWHPDWLFUHYLHZ &RPSXW0HWKRGV3URJUDPV%LRPHG >KWWSG[GRLRUJMFPSE@>30,'@ >@ 'UXFNHU+&RUWHV&-DFNHO/'/H&XQ<9DSQLN9%RRVWLQJDQG2WKHU(QVHPEOH0HWKRGV1HXUDO&RPSXW >KWWSG[GRLRUJQHFR@ >@ /RUHQ]L35DR55RPDQR*.LWD$,UUHUD)0RELOH'HYLFHVIRUWKH5HDO7LPH'HWHFWLRQRI6SHFLILF+XPDQ0RWLRQ'LVRUGHUV,(((6HQV - >KWWSG[GRLRUJ-VHQ@ >@ %D\DW$3RPSOXQ07UDQ'$$6WXG\RQ+XPDQ$FWLYLW\5HFRJQLWLRQ8VLQJ$FFHOHURPHWHU'DWDIURP6PDUWSKRQHVWK,QWHUQDWLRQDO &RQIHUHQFH RQ )XWXUH 1HWZRUNV DQG &RPPXQLFDWLRQV  )QFl  WKH WK ,QWHUQDWLRQDO &RQIHUHQFH RQ 0RELOH 6\VWHPV DQG 3HUYDVLYH &RPSXWLQJ 0RELVSFl $IILOLDWHG:RUNVKRSVYRO >KWWSG[GRLRUJMSURFV@ >@ .KDOLID6+DVVDQ06HQHYLUDWQH$)HDWXUHVHOHFWLRQIRUIORRUFKDQJLQJDFWLYLW\UHFRJQLWLRQLQPXOWLIORRUSHGHVWULDQQDYLJDWLRQ >KWWSG[GRLRUJ,&08@ >@ 3L\DUH5/HH650RELOH6HQVLQJ3ODWIRUPIRU3HUVRQDO+HDOWK0DQDJHPHQWWK ,HHH,QWHUQDWLRQDO6\PSRVLXPRQ&RQVXPHU(OHFWURQLFV ,VFH  147 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO >KWWSG[GRLRUJ,6&(@ >@ .ZRQ<.DQJ.%DH&$QDO\VLVDQG(YDOXDWLRQRI6PDUWSKRQHEDVHG+XPDQ$FWLYLW\5HFRJQLWLRQ8VLQJD1HXUDO1HWZRUN$SSURDFK ,QWHUQDWLRQDO-RLQW&RQIHUHQFHRQ1HXUDO1HWZRUNV ,-&11  >KWWSG[GRLRUJ,-&11@ >@ :DQJ&;X<=KDQJ-<X:6:+00$0HWKRGIRU(YDOXDWLQJ&RQILGHQFHRI6PDUWSKRQH%DVHG$FWLYLW\5HFRJQLWLRQ,Q >KWWSG[GRLRUJ7UXVW&RP@ >@ +VX<:&KHQ.+<DQJ---DZ)66PDUWSKRQHEDVHGIDOOGHWHFWLRQDOJRULWKPXVLQJIHDWXUHH[WUDFWLRQ >KWWSG[GRLRUJ&,63%0(,@ >@ 9DYRXODV*&KDW]DNL&0DOOLRWDNLV73HGLDGLWLV07VLNQDNLV07KH0REL$FW'DWDVHW5HFRJQLWLRQRI$FWLYLWLHVRI'DLO\/LYLQJXVLQJ 6PDUWSKRQHV3URFHHGLQJVRIWKH,QWHUQDWLRQDO&RQIHUHQFHRQ,QIRUPDWLRQDQG&RPPXQLFDWLRQ7HFKQRORJLHVIRU$JHLQJ:HOODQG(+HDOWK ,FWDZH  >KWWSG[GRLRUJ@ >@ %XMDUL$/LFDU%3DOD]]L&(0RYHPHQW3DWWHUQ5HFRJQLWLRQWKURXJK6PDUWSKRQHlV$FFHOHURPHWHU,(((&RQVXPHU&RPPXQLFDWLRQV DQG1HWZRUNLQJ&RQIHUHQFH&FQFSS >@ 7RUUHV+XLW]LO&1XQR0DJDQGD05REXVWVPDUWSKRQHEDVHGKXPDQDFWLYLW\UHFRJQLWLRQXVLQJDWULD[LDODFFHOHURPHWHU >KWWSG[GRLRUJ/$6&$6@ >@ %DMSDL$-LOOD97LZDUL919HQNDWHVDQ601DUD\DQDQ54XDQWLILDEOHILWQHVVWUDFNLQJXVLQJZHDUDEOHGHYLFHV&RQI3URF,((((QJ0HG %LRO6RF >KWWSG[GRLRUJ(0%&@>30,'@ >@ =DLQXGLQ 016 6XODLPDQ 01 0XVWDSKD 1 3HUXPDO 7 $FWLYLW\ 5HFRJQLWLRQ EDVHG RQ $FFHOHURPHWHU 6HQVRU XVLQJ &RPELQDWLRQDO &ODVVLILHUV,(((&RQIHUHQFHRQ2SHQ6\VWHPV ,FRV  >KWWSG[GRLRUJ,&26@ >@ 6HQ65DFKXUL..0XNKHUML$0LVUD$'LG\RXWDNHDEUHDNWRGD\"'HWHFWLQJSOD\LQJIRRVEDOOXVLQJ\RXUVPDUWZDWFK >KWWSG[GRLRUJ3(5&20:@ >@ .ZDSLV]-5:HLVV*00RRUH6$$FWLYLW\UHFRJQLWLRQXVLQJFHOOSKRQHDFFHOHURPHWHUV6,*.''([SORU >KWWSG[GRLRUJ@ >@ 0LWFKHOO(0RQDJKDQ'2l&RQQRU1(&ODVVLILFDWLRQRIVSRUWLQJDFWLYLWLHVXVLQJVPDUWSKRQHDFFHOHURPHWHUV6HQVRUV %DVHO     >KWWSG[GRLRUJV@>30,'@ >@ /L3:DQJ<7LDQ<=KRX76/L-6$Q$XWRPDWLF8VHU$GDSWHG3K\VLFDO$FWLYLW\&ODVVLILFDWLRQ0HWKRG8VLQJ6PDUWSKRQHV,(((7UDQV %LRPHG(QJ   >KWWSG[GRLRUJ7%0(@>30,'@ >@ =KDQJ/:X;/XR'5HDO7LPH$FWLYLW\5HFRJQLWLRQRQ6PDUWSKRQHV8VLQJ'HHS1HXUDO1HWZRUNV >KWWSG[GRLRUJ8,&$7&6FDO&RP&%'&RP,R3@ >@ =KDR./'X-=/L&4=KDQJ&//LX+;X&+HDOWK\$'LDU\6\VWHP%DVHGRQ$FWLYLW\5HFRJQLWLRQ8VLQJ6PDUWSKRQH >KWWSG[GRLRUJ0$66@ >@ 'HUQEDFK6'DV%.ULVKQDQ1&7KRPDV%/&RRN'-6LPSOHDQG&RPSOH[$FWLYLW\5HFRJQLWLRQWKURXJK6PDUW3KRQHV >KWWSG[GRLRUJ,(@ >@ 6KHQ&&KHQ<)<DQJ*62Q0RWLRQ6HQVRU%HKDYLRU$QDO\VLVIRU+XPDQ$FWLYLW\5HFRJQLWLRQYLD6PDUWSKRQHV,HHH,QWHUQDWLRQDO &RQIHUHQFHRQ,GHQWLW\6HFXULW\DQG%HKDYLRU$QDO\VLV ,VED 6HQGDL-DSDQSS >KWWSG[GRLRUJ,6%$@ >@ 9DOODEK30DOHNLDQ5<H1%RJDWLQRVND'&)DOO'HWHFWLRQ8VLQJ0DFKLQH/HDUQLQJ$OJRULWKPV >KWWSG[GRLRUJ62)7&20@ >@ :X:'DVJXSWD65DPLUH]((3HWHUVRQ&1RUPDQ*-&ODVVLILFDWLRQDFFXUDFLHVRISK\VLFDODFWLYLWLHVXVLQJVPDUWSKRQHPRWLRQVHQVRUV0HG,QWHUQHW5HV  H >KWWSG[GRLRUJMPLU@>30,'@ >@ )DQJ / <LVKXL 6 :HL & 8S DQG GRZQ EXVHV DFWLYLW\ UHFRJQLWLRQ XVLQJ VPDUWSKRQH DFFHOHURPHWHU (OHFWURQLF DQG $XWRPDWLRQ &RQWURO &RQIHUHQFH >KWWSG[GRLRUJ,71(&@ >@ =GUDYHYVNL(/DPHVNL30LQJRY5.XODNRY$*MRUJMHYLNM'5REXVWKLVWRJUDPEDVHGIHDWXUHHQJLQHHULQJRIWLPHVHULHVGDWD >KWWSG[GRLRUJ)@ >@ =GUDYHYVNL ( 5LVWHVND 6WRMNRVND % 6WDQGO 0 6FKXO] + $XWRPDWLF PDFKLQHOHDUQLQJ EDVHG LGHQWLILFDWLRQ RI MRJJLQJ SHULRGV IURP DFFHOHURPHWHUPHDVXUHPHQWVRIDGROHVFHQWVXQGHUILHOGFRQGLWLRQV3/R62QH  H >KWWSG[GRLRUJMRXUQDOSRQH@>30,'@ >@ 7DQJ&;3KRKD99$Q(PSLULFDO(YDOXDWLRQRI$FWLYLWLHVDQG&ODVVLILHUVIRU8VHU,GHQWLILFDWLRQRQ6PDUWSKRQHV >KWWSG[GRLRUJ%7$6@ 148 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living $QGURLG/LEUDU\IRU5HFRJQLWLRQRI$FWLYLWLHV 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH >@ %ORFK$(UGLQ50H\HU6.HOOHU7GH6SLQGOHU$%DWWHU\(IILFLHQW7UDQVSRUWDWLRQ0RGH'HWHFWLRQRQ0RELOH'HYLFHV >KWWSG[GRLRUJ0'0@ >@ 6KRDLE06FKROWHQ++DYLQJD3-07RZDUGV3K\VLFDO$FWLYLW\5HFRJQLWLRQ8VLQJ6PDUWSKRQH6HQVRUV >KWWSG[GRLRUJ8,&$7&@ >@ 0LHONH 0 %UiFN 5 6PDUWSKRQH DSSOLFDWLRQ IRU DXWRPDWLF FODVVLILFDWLRQ RI HQYLURQPHQWDO VRXQG 3URFHHGLQJV RI WKH WK ,QWHUQDWLRQDO &RQIHUHQFH0L[HG'HVLJQRI,QWHJUDWHG&LUFXLWVDQG6\VWHPV0,;'(6 >@ *XR;7R\RGD</L++XDQJ-'LQJ6/LX<(QYLURQPHQWDOVRXQGUHFRJQLWLRQXVLQJWLPHIUHTXHQF\LQWHUVHFWLRQSDWWHUQV >@ 3LOORV$$OJKDPLGL.$O]DPHO13DYORY90DFKDQDYDMKDOD6$UHDOWLPHHQYLURQPHQWDOVRXQGUHFRJQLWLRQV\VWHPIRUWKH$QGURLG26  >@ %RXQWRXUDNLV99U\VLV/3DSDQLNRODRX*0DFKLQH/HDUQLQJ$OJRULWKPVIRU(QYLURQPHQWDO6RXQG5HFRJQLWLRQ7RZDUGV6RXQGVFDSH 6HPDQWLFV3URFHHGLQJVRIWKH$XGLR0RVWO\RQ,QWHUDFWLRQ:LWK6RXQG7KHVVDORQLNL*UHHFH >KWWSG[GRLRUJ@ >@ 0LHONH0%UXHFN5'HVLJQDQGHYDOXDWLRQRIDVPDUWSKRQHDSSOLFDWLRQIRUQRQVSHHFKVRXQGDZDUHQHVVIRUSHRSOHZLWKKHDULQJORVV&RQI 3URF,((((QJ0HG%LRO6RF >KWWSG[GRLRUJ(0%&@ >@ 6LJWLD66WDUN$0.UVWXORYLF63OXPEOH\0'$XWRPDWLF(QYLURQPHQWDO6RXQG5HFRJQLWLRQ3HUIRUPDQFH9HUVXV&RPSXWDWLRQDO&RVW ,((($&07UDQV$XGLR6SHHFK/DQJ3URFHVV >KWWSG[GRLRUJ7$6/3@ >@ 'XEH\+0HKO050DQNRGL\D.%LJ($5,QIHUULQJWKHDPELHQWDQGHPRWLRQDOFRUUHODWHVIURPVPDUWSKRQHEDVHGDFRXVWLFELJGDWD VW ,((( ,QWHUQDWLRQDO&RQIHUHQFHRQ&RQQHFWHG+HDOWK$SSOLFDWLRQV6\VWHPVDQG(QJLQHHULQJ7HFKQRORJLHV &+$6(  >KWWSG[GRLRUJ&+$6(@ >@ /DQH1'*HRUJLHY34HQGUR/'HHS(DUUREXVWVPDUWSKRQHDXGLRVHQVLQJLQXQFRQVWUDLQHGDFRXVWLFHQYLURQPHQWVXVLQJGHHSOHDUQLQJ 3URFHHGLQJVRIWKH$&0,QWHUQDWLRQDO-RLQW&RQIHUHQFHRQ3HUYDVLYHDQG8ELTXLWRXV&RPSXWLQJ2VDND-DSDQ >KWWSG[GRLRUJ@ >@ &KHIIHQD0)DOO'HWHFWLRQ8VLQJ6PDUWSKRQH$XGLR)HDWXUHV,(((-%LRPHG+HDOWK,QIRUP   >KWWSG[GRLRUJ-%+,@>30,'@ >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD)8VHU(QYLURQPHQW'HWHFWLRQZLWK$FRXVWLF6HQVRUV(PEHGGHGRQ0RELOH'HYLFHVIRUWKH 5HFRJQLWLRQRI$FWLYLWLHVRI'DLO\/LYLQJ,Q5HYLHZ >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD)$0XOWLSOH6RXUFH)UDPHZRUNIRUWKH,GHQWLILFDWLRQRI$FWLYLWLHVRI'DLO\/LYLQJ%DVHG RQ0RELOH'HYLFH'DWD,Q5HYLHZ >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD)6SLQVDQWH63DWWHUQ5HFRJQLWLRQ7HFKQLTXHVIRUWKH,GHQWLILFDWLRQRI$FWLYLWLHVRI'DLO\ /LYLQJXVLQJ0RELOH'HYLFH$FFHOHURPHWHU,Q5HYLHZ >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD)6SLQVDQWH6'DWD)XVLRQRQ0RWLRQDQG0DJQHWLF6HQVRUVHPEHGGHGRQ0RELOH'HYLFHV IRUWKH,GHQWLILFDWLRQRI$FWLYLWLHVRI'DLO\/LYLQJ,QHQJU[LYRUJ[U],Q5HYLHZ >@ 3LUHV,0*DUFLD103RPER1)OhUH]5HYXHOWD)5RGUfJXH]1'9DOLGDWLRQ7HFKQLTXHVIRU6HQVRU'DWDLQ0RELOH+HDOWK$SSOLFDWLRQV6HQV >KWWSG[GRLRUJ@ >@ $XJXVW  0XOWLVHQVRU GDWD IXVLRQ LQ PRELOH GHYLFHV IRU WKH LGHQWLILFDWLRQ RI DFWLYLWLHV RI GDLO\ OLYLQJ $YDLODEOH IURP KWWSVDOODEGLXELSW PHGLDZLNLLQGH[SKS$XJXVWBB 0XOWL VHQVRUBGDWDBIXVLRQBLQBPRELOHBGHYLFHVBIRUBWKHBLGHQWLILFDWLRQBRIBDFWLYLWLHVBRIBGDLO\BOLYLQJ >@ %RMLQRY+0LFKDOHYVN\<1DNLEO\*%RQHK'0RELOHGHYLFHLGHQWLILFDWLRQYLDVHQVRUILQJHUSULQWLQJDU;LYSUHSULQWDU;LY >@ .DWHYDV.+DGGDGL+7RNDUFKXN/6HQVLQJNLW(YDOXDWLQJWKHVHQVRUSRZHUFRQVXPSWLRQLQL26GHYLFHVDU[LY >KWWSG[GRLRUJ,(@ >@ 6PDUWSKRQHV$YDLODEOHIURPKWWSVZZZETFRPSWVPDUWSKRQHV >@ (QFRJ0DFKLQH/HDUQLQJ)UDPHZRUN$YDLODEOHIURPKWWSZZZKHDWRQUHVHDUFKFRPHQFRJ >@ 'HHSOHDUQLQJM2SHQVRXUFH'LVWULEXWHG'HHS/HDUQLQJIRUWKH-90$YDLODEOHIURPKWWSVGHHSOHDUQLQJMRUJ >@ 6DQWRV-5RGULJXHV--3&6LOYD%0&&DVDO-6DOHHP.'HQLVRY9$Q,R7EDVHGPRELOHJDWHZD\IRULQWHOOLJHQWSHUVRQDODVVLVWDQWVRQ PRELOHKHDOWKHQYLURQPHQWV-1HWZ&RPSXW$SSO >KWWSG[GRLRUJMMQFD@ >@ %HUW)*LDFRPHWWL0*XDODQR056LOLTXLQL56PDUWSKRQHVDQGKHDOWKSURPRWLRQDUHYLHZRIWKHHYLGHQFH-0HG6\VW   >KWWSG[GRLRUJV@>30,'@ >@ 3RVODG68ELTXLWRXVFRPSXWLQJVPDUWGHYLFHVHQYLURQPHQWVDQGLQWHUDFWLRQV-RKQ:LOH\ 6RQV >@ &KRXGKXU\7%RUULHOOR*&RQVROYR6+DHKQHO'+DUULVRQ%+HPLQJZD\%HWDO7KH0RELOH6HQVLQJ3ODWIRUP$Q(PEHGGHG$FWLYLW\ 5HFRJQLWLRQ6\VWHP,(((3HUYDVLYH&RPSXW 149 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living 7KH2SHQ%LRLQIRUPDWLFV-RXUQDO9ROXPH 3LUHVHWDO >KWWSG[GRLRUJ0359@ >@ 0RQWRUR0DQULTXH*+D\D&ROO36FKQHOOH:DOND',QWHUQHWRI7KLQJV)URP5),'6\VWHPVWR6PDUW$SSOLFDWLRQV >@ $ULI0-(O(PDU\,0.RXWVRXULV''$UHYLHZRQWKHWHFKQRORJLHVDQGVHUYLFHVXVHGLQWKHVHOIPDQDJHPHQWRIKHDOWKDQGLQGHSHQGHQW OLYLQJRIHOGHUO\7HFKQRO+HDOWK&DUH   >KWWSG[GRLRUJ7+&@>30,'@ >@ )LHOG$'LVFRYHULQJVWDWLVWLFVXVLQJ,%06366VWDWLVWLFV j3LUHVHWDO 7KLVLVDQRSHQDFFHVVDUWLFOHGLVWULEXWHGXQGHUWKHWHUPVRIWKH&UHDWLYH&RPPRQV$WWULEXWLRQ,QWHUQDWLRQDO3XEOLF/LFHQVH &&%< D FRS\RIZKLFKLVDYDLODEOHDW KWWSVFUHDWLYHFRPPRQVRUJOLFHQVHVE\OHJDOFRGH 7KLVOLFHQVHSHUPLWVXQUHVWULFWHGXVHGLVWULEXWLRQDQG UHSURGXFWLRQLQDQ\PHGLXPSURYLGHGWKHRULJLQDODXWKRUDQGVRXUFHDUHFUHGLWHG 150 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living Chapter 5 Conclusions and Future Work This chapter presents the main conclusions that result from the research work described in this Thesis. Furthermore, it discusses research topics related with the work developed in the doctoral programme, which may be addressed in the future. 1. Final Conclusions The focus of this Thesis is related to the development of a novel framework for the recognition of ADL and its environments that includes the selection of the best methods for different stages, such as Data Acquisition, Data Fusion, Data Processing, Data Cleaning, Feature Extraction and Classification, based on the sensors available in a smartphone. The most important stages consist in the Data Fusion and Classification methods that mostly influence the results obtained by the framework for the purpose of this Thesis. Mobile devices have several available sensors, including the accelerometer, the gyroscope, the magnetometer, the microphone and the Global Positioning System (GPS) receiver, which allow the acquisition of different types of physical and physiological parameters. The mobile devices may be considered as multi-sensor devices, which is the first part of the title of this Thesis. There is a variety of mobile devices, but the scope of this Thesis focuses only in the use of smartphones. The acquisition of the different parameters allows the recognition of Activities of Daily Living (ADL), where the work of this Thesis is to implement data fusion and classification methods for the recognition of ADL and its environments. There are several ADL that can be recognized, but this Thesis focuses in the recognition of 7 ADL, i.e., walking, running, going upstairs, going downstairs, standing/watching TV, sleeping and driving activities and 9 environments, i.e., bedroom, bar, classroom, gym, kitchen, living room, hall, street and library, with the sensors available in the mobile devices. These devices have several limitations, such as limited power, processing and memory capabilities, which are minimized with the 151 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living implementation of different methods. Finally, the developed framework may be used for the mapping of lifestyles with the ADL and environments recognized. The main objective of the work reported in this Thesis is the creation of a new method and framework for the recognition of ADL and environments with the fusion and classification of the data acquired from a large set of sensors available in off-the-shelf mobile devices. The developed method is a function of the number of available sensors embedded in the different mobile devices. It should be implemented with lightweight algorithms in order to allow its execution locally in the mobile devices. The main objective was divided into six smaller objectives, which started with the literature review, verifying that there are no available studies that use a large set of sensors available in the mobile devices and fuse the data acquired with them, but it is a topic that is widely growing with different research works. The literature review also reveals that Artificial Neural Networks (ANN) are widely used, and reported better results than other methods. Finally, the literature review helped in the definition of the ADL and environments to be recognized, and in the definition of the architecture of the novel framework with Data Acquisition, Data Fusion, Data Processing, Data Cleaning, Feature Extraction and Classification as stages. Among them, Data Fusion and Classification are the main modules of the framework. Data Acquisition based using mobile devices may be performed with standard methods. Data Processing, Data Cleaning and Feature Extraction methods depend on the types of sensors used as presented in chapter 3, which is focused on the definition of the architecture of the framework for the recognition of ADL and its environments. Next, data acquisition was performed with a mobile application that allows the labelling of the ADL and environments for further processing. In continuation, the acquired data was analyzed and the different methods were implemented, including low pass filter, Fast Fourier Transform (FFT), and so on. After the extraction of the features, three different implementations of ANN methods had been tested, including Multilayer Perceptron (MLP) with Backpropagation, Feedforward Neural Networks (FNN) with Backpropagation and Deep Neural Networks (DNN), where these implementation were detailed in chapter 4. The different implementations reported that the method that obtains the best results is the DNN method with normalized data. The first part of this Thesis, presented in chapter 2, consists in the literature review on the different concepts related to the creation of a novel framework for the recognition of ADL and its environments. Initially, the main proposed concepts are Data Acquisition, Data Processing, Data Imputation and Data Fusion, which are explored and several frameworks were described for each concept as well as their applicability. However, these solutions are difficult to be adapted to the mobile devices, because the development of solutions for these devices should take into account the different hardware restrictions that depend on the manufacturers. Moreover, during the first part of the literature review, it was verified that Data Imputation may be a part of Data Processing, therefore it was excluded from this research, as the 152 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living classification methods use the features extracted from the sensors’ signal, the inexistence of data in some instants are not very relevant. The second part of the literature review consists on the research on the validation methods, including Data Cleaning and Data Imputation methods in the Data Processing module. Where, the Data Cleaning methods are mainly filtering methods, including low pass filter, high pass filter and others; and the Data Imputation methods reviewed are mainly the K-Nearest Neighbor (KNN) and their variants. Next, the classification methods for the identification of the ADL and environments were reviewed. It was verified that one of the most used methods with better results are ANN. As the acoustic data was also used for the recognition of the environments, audio fingerprinting methods were also reviewed, verifying that FFT should be implemented for the extraction of the frequencies. It was concluded that for all types of sensors’ data, the most used features with better results are statistical features. The second part of this Thesis, presented in chapter 3, consists in the definition of the architecture of the framework for the recognition of the ADL. In this chapter it is mentioned that the execution starts with the Data Acquisition process. After the acquisition of the data, it is sent to the Data Processing, including Data Cleaning, Data Imputation and Feature Extraction. Data Cleaning will take into account the sensors used, implemented the low pass filter method for the motion and magnetic sensors and the FFT for the acoustic sensors. Firstly, for motion and magnetic sensors, the features extracted are the five greatest distances between the maximum peaks combined with the average, standard deviation, variance and median of the maximum peaks, and the standard deviation, average, maximum, minimum, variance and median of the raw signal. Secondly, for the acoustic sensors, the features extracted are the 26 Mel-Frequency Cepstral Coefficients (MFCC) combined with the standard deviation, average, maximum, minimum, variance and median of the raw signal. Finally, for the location sensors, the feature extracted is the distance travelled. After the feature extraction and based on ANN, the classification method were implemented. The output should be combined with the users’ agenda in future developments. The third part of this Thesis, presented in chapter 4, consists in the explanation of the main achievement of this Thesis, which is the implementation of the different methods in the framework for the recognition of ADL and its environments. The implementation presented in this Thesis was incremental, starting with the implementation with only one sensor, i.e., accelerometer, to the implementation with most common sensors in the smartphones, i.e., accelerometer, magnetometer, gyroscope, microphone and GPS receiver. The implementations are performed with three different configurations of ANN, such as MLP method with Backpropagation, FNN method with Backpropagation and DNN method with normalized and nonnormalized data. All implementations are based in groups of features extracted from the different sensors’ data available in each study. The normalization for MLP and FNN methods was performed with MIN/MAX normalizer and the normalization for DNN method was performed 153 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living with mean and standard deviation. The details about the configurations of the ANN methods are available in chapter 4. The novel framework performs the recognition of ADL and environments in three stages, these are the recognition of common ADL with motion and magnetic sensors, the recognition of environments with acoustic sensors, and the recognition of activities without motion with motion, magnetic and location sensors fused with the environment recognized. Following the results obtained with the accelerometer data, the best results were achieved with DNN method, reporting an accuracy of 85.89% with normalized data and 80.35% with non-normalized data using cross-validation in the recognition of ADL. Following the results obtained with the fusion of the accelerometer and the magnetometer data, the best results were achieved also with the DNN method, reporting an accuracy of 86.49% with normalized data and 70.43% with non-normalized data using cross-validation in the recognition of ADL. Following the results obtained with the fusion of the accelerometer, the magnetometer and the gyroscope data, the best results were achieved also with the DNN method, reporting an accuracy of 89.51% with normalized data and 74.47% with non-normalized data using crossvalidation in the recognition of ADL. Following the results obtained with the acoustic data, the best results were achieved with FNN method with Backpropagation, reporting an accuracy of 82.75% with normalized data and 86.50% with non-normalized data using cross-validation in the recognition of environments. Following the results obtained with the fusion of previously used sensors with the location sensors and/or the environment recognized, the best results were achieved with DNN method, reporting an accuracy of 100% with the different combinations of sensors in the recognition of activities without motion. Finally, the novel framework was implemented and validated in a smartphone, reporting an overall accuracy between 58.02% and 89.15%. The main objective of this Thesis was accomplished with the development of the novel framework for the recognition of ADL and environments, recognizing a greater number of ADL/environments than the previous works analysed and reporting an overall accuracy higher than the mean of the results, and a similar accuracy than the highest results reported by the previous works available in the literature. 2. Future Work The recognition of ADL and its environments may be performed with other approaches, and the results obtained by the framework may be improved with research and implementation of Data Imputation methods to be incorporated into the Data Processing module. The Data Processing 154 Multi-sensor data fusion in mobile devices for the identification of Activities of Daily Living module can be also improved with other approaches related to Data Validation and Data Cleaning methods. However, the most important module of the framework for the recognition of ADL and its environments is the Classification module, in which other types of machine learning methods may be implemented and tested with the datasets acquired during this Thesis, including Support Vector Machines (SVM), Decision Trees and Ensemble Learning Methods (e.g., Adaboost) in order to check the results are better with other machine learning methods. Taking into account the limited battery, processing and memory capabilities of the smartphones, the most important challenge in the development of the framework is to arrange a method that acquires the data continuously without effects in the performance of the smartphones in order to be able to recognize risk situations, e.g., falls of older people. Currently, the method developed is prepared for smartphones. The implementation of the framework in other devices (e.g., smartwatches) would require further experimentation, acquiring new data and developing new classification methods adapted to these devices. Consequently, the framework should be adapted to the specific characteristics of the mobile device in use. One of the most important differences in the data acquisition between the smartphone and the smartwatch is the position of them, because the smartphone can be placed in the front pocket of the user’s trousers, but the smartwatch is commonly placed on the user’s wrist. As future work, the number of ADL and environments recognized by the framework may be incremented, but this would also require further experiments. As final achievement, this framework may be used in the development of a personal digital life coach. 155