research-article

Modeling and simulation for unmanned and autonomous system test and evaluation

Authors:

Mauricio Castillo-Effen,

Nikita Visnevski,

Raj SubbuAuthors Info & Claims

PerMIS '09: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems

Pages 86 - 92

https://doi.org/10.1145/1865909.1865927

Published: 21 September 2009 Publication History

Abstract

Test and evaluation may be viewed as a technology enabler for the successful deployment of unmanned vehicles and robots in all their envisioned applications. It is however a challenging endeavor, considering that roboticists and developers are not used to thinking of test and evaluation as an integral component of robot development. Moreover, the community who has conducted test and evaluation up to this date does not possess the tools to cope with the growing complexity of unmanned and autonomous systems. This paper proposes a solution to one of the hardest problems in testing and evaluation of robots: test planning. The approach put forward relies on constructive simulation tools and on new techniques for searching in high dimensional spaces. The goal of the test planner is to generate a set of tests that make highly efficient use of resources to unveil weaknesses of the system under test. A secondary objective of the paper is to create reciprocal awareness between test and evaluation and robotics communities, who could benefit significantly from each other.

References

[1]

S. Balakirsky, C. Scrapper, S. Carpin, and M. Lewis. USARSim: Providing a Framework for Multi-robot Performance Evaluation. In Performance Metrics of Intelligent Systems PerMIS'06, 2006.

[2]

F. Bonsignorio, A. D. Pobil, and J. H. South. Defining the requisites of a replicable robotics experiment. In Workshop on Good Experimental Methodology In Robotics, at RSS 2009, Seattle, WA, June 2009.

[3]

M. L. Cohen, J. E. Rolph, and D. L. Steffey, editors. Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements. National Academy Press, 1998.

[4]

Defense Acquisition University. Glossary of Defense Acquisition Acronyms & Terms. Defense Acquisition University Press, 12th edition, July 2005.

[5]

Department of Defense. Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A). DoD Instruction 5000.61, May 2003.

[6]

Department of Defense. Interim Defense Acquisition Guidebook. https://acc.dau.mil/dag, June 2009.

[7]

P. A. Djang and F. Lopez. Unmanned and Autonomous Systems Mission Based Test and Evaluation. In Performance Metrics of Intelligent Systems PerMIS'09, Gaithersburg, MD, September 2009. NIST.

Digital Library

[8]

Henrik I. Christensen et al. A Roadmap for US Robotics -- From Internet to Robotics. Technical report, The Computing Community Consortium -- Computing Research Association, http://www.us-robotics.us, May 2009.

[9]

F. Macias. The Test and Evaluation of Unmanned and Autonomous Systems. ITEA Journal, 29(4):388--395, December 2008.

[10]

R. Madhavan, E. R. Messina, and J. S. Albus, editors. Intelligent Vehicle Systems: A 4D/RCS Approach. Nova Science Publishers, 2007.

[11]

MAK Technologies -- VT Systems, Cambridge, MA 02138 USA. B-HAVE Module for VR-Forces, User's Guide, revision bhv-1.2-1-080214 edition, 2008.

[12]

MAK Technologies -- VT Systems, Cambridge, MA 02138 USA. VR-Forces -- Developer's Guide, revision vrf-3.11-2-080103 edition, 2008.

[13]

MAK Technologies -- VT Systems, Cambridge, MA 02138 USA. VR-Forces User's Guide, revision vrf-3.11-1-080125 edition, 2008.

[14]

Maurits Butter et al. Robotics for Healthcare -- Final Report. Technical report, European Commission, DG Information Society, October 2008.

[15]

Office of the Secretary of Defense. FY2009-2034 Unmanned Systems Integrated Roadmap. Technical report, Department of Defense, United States of America, April 2009.

[16]

R. Subbu, N. A. Visnevski, and P. A. Djang. Evolutionary Framework for Test of Autonomous Systems. In Performance Metrics of Intelligent Systems PerMIS'09, Gaithersburg, MD, September 2009. NIST.

Digital Library

Cited By

Bocciarelli PD'Ambrogio AGiglio APaglia E(2015)A model-driven framework for distributed simulation of autonomous systemsProceedings of the Symposium on Theory of Modeling & Simulation: DEVS Integrative M&S Symposium10.5555/2872965.2872994(213-220)Online publication date: 12-Apr-2015
https://dl.acm.org/doi/10.5555/2872965.2872994
Castillo-Effen MVisnevski NMessina EMadhavan R(2010)Testing and evaluation aspects of integration of unmanned air systems into the national air spaceProceedings of the 10th Performance Metrics for Intelligent Systems Workshop10.1145/2377576.2377600(127-132)Online publication date: 28-Sep-2010
https://dl.acm.org/doi/10.1145/2377576.2377600

Recommendations

Testing and evaluation aspects of integration of unmanned air systems into the national air space
PerMIS '10: Proceedings of the 10th Performance Metrics for Intelligent Systems Workshop

Current developments show that the integration of Unmanned Aerial Systems (UAS) into the National Airspace System (NAS) is a process that will inevitably happen. Arguably, it may be viewed as one of the key milestones in the history of aviation. Whereas ...
Unmanned and autonomous systems mission based test and evaluation
PerMIS '09: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems

We propose to apply principles from the Army Evaluation Center's Mission Based Test and Evaluation (MBT&E) to Unmanned and Autonomous Systems (UAS) Test and Evaluation (T&E) in order to conduct rigorous, real-world testing based on anticipated military ...
Evolutionary framework for test of autonomous systems
PerMIS '09: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems

A DoD mission and challenge is to enable a high percentage of autonomous vehicles in the warfighter fleet by 2015. These systems will need to display a high degree of autonomous capabilities. The capabilities of these autonomous systems must be ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PerMIS '09: Proceedings of the 9th Workshop on Performance Metrics for Intelligent Systems

September 2009

322 pages

ISBN:9781605587479

DOI:10.1145/1865909

General Chair:
Raj Madhavan
Oak Ridge National Laboratory/NIST
,
Program Chair:
Elena Messina
Intelligent Systems Division, NIST

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

NIST: National Institute of Standards and Technology

In-Cooperation

SIGAI: ACM Special Interest Group on Artificial Intelligence

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

T&E/S&T Program through the White Sands Missile Range, NM

Conference

PerMIS '09

Sponsor:

NIST

PerMIS '09: Performance Metrics for Intelligent Systems

September 21 - 23, 2009

Maryland, Gaithersburg

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
252
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bocciarelli PD'Ambrogio AGiglio APaglia E(2015)A model-driven framework for distributed simulation of autonomous systemsProceedings of the Symposium on Theory of Modeling & Simulation: DEVS Integrative M&S Symposium10.5555/2872965.2872994(213-220)Online publication date: 12-Apr-2015
https://dl.acm.org/doi/10.5555/2872965.2872994
Castillo-Effen MVisnevski NMessina EMadhavan R(2010)Testing and evaluation aspects of integration of unmanned air systems into the national air spaceProceedings of the 10th Performance Metrics for Intelligent Systems Workshop10.1145/2377576.2377600(127-132)Online publication date: 28-Sep-2010
https://dl.acm.org/doi/10.1145/2377576.2377600

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents