short-paper

Free access

Lessons Learned while Trying to Reproduce the OpenRF Experiment

Authors:

Mohamed Naoufal Mahfoudi,

Thierry Turletti,

Thierry Parmentelat,

Walid DabbousAuthors Info & Claims

Reproducibility '17: Proceedings of the Reproducibility Workshop

Pages 21 - 23

https://doi.org/10.1145/3097766.3097772

Published: 11 August 2017 Publication History

PDF eReader

Abstract

Evaluating and comparing performance of wireless systems, like for any other scientific area, requires the ability to reproduce experimental results. In this paper, we describe the specific issues that we encountered when focusing on reproducing the experiments described in a paper related to wireless systems. We selected the OpenRF paper published in SIGCOMM 2013, a very interesting research work allowing to perform beamforming on commodity WiFi devices. We illustrate how reproducibility is strongly dependent on the used hardware, and why an extensive knowledge of the used hardware and its design is necessary. On the basis of this experience, we propose some recommendations and lessons for the design of reproducible wireless experiments.

References

[1]

Swarun Kumar, Diego Cifuentes, Shyamnath Gollakota, and Dina Katabi. Bringing cross-layer mimo to today's wireless lans. SIGCOMM Comput. Commun. Rev., 43(4):387--398, August 2013.

Digital Library

Google Scholar

[2]

IEEE Standard for Information technology (2009). Local and metropolitan area networks---Specific requirements---Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications Amendment 5: Enhancements for Higher Throughput, IEEE Std 802.11n-2009, Oct 2009.

Google Scholar

[3]

Daniel Chaim Halperin, Wenju Hu, Anmol Sheth, and David Wetherall. Tool Release: Gathering 802.11N Traces with Channel State Information. ACM CCR, 41(1):53--53, Jan 2011.

Digital Library

Google Scholar

Cited By

View all

Chwalisz MGeissdoerfer KWolisz A(2019)Walker: DevOps Inspired Workflow for ExperimentationIEEE INFOCOM 2019 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)10.1109/INFCOMW.2019.8845199(277-282)Online publication date: Apr-2019
https://doi.org/10.1109/INFCOMW.2019.8845199
Parmentelat TTurletti TDabbous WMahfoudi MBronzino FKumar SGarcia-Saavedra A(2018)nepi-ngProceedings of the 12th International Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization10.1145/3267204.3267213(56-65)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1145/3267204.3267213
Saucez DIannone L(2018)Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility WorkshopACM SIGCOMM Computer Communication Review10.1145/3211852.321186348:1(70-74)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3211852.3211863

Index Terms

Lessons Learned while Trying to Reproduce the OpenRF Experiment
1. Networks
  1. Network performance evaluation
    1. Network experimentation

Recommendations

Lessons learned from an XP experiment with students: test-first needs more teachings
PROFES'06: Proceedings of the 7th international conference on Product-Focused Software Process Improvement

For most XP techniques only a few experimental results on their effects are available. In October 2004 we started a medium-term experiment to investigate the impact of test-first compared to a classical-testing approach. We carefully designed a ...
Reconstructing the software environment of an experiment with kameleon
COMPUTE '12: Proceedings of the 5th ACM COMPUTE Conference: Intelligent & scalable system technologies

In the scientific experimentation process, an experiment result needs to be analyzed and compared with several others, potentially obtained in different conditions. Thus, the experimenter needs to be able to redo the experiment. Several tools are ...
Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project
HRI '24: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

We are witnessing increasing calls for reproducibility and replicability in HRI studies to improve reliability and confidence in empirical findings. One solution to facilitate this is using a robot platform that researchers frequently use, making it ...

Comments

Information & Contributors

Information

Published In

Reproducibility '17: Proceedings of the Reproducibility Workshop

August 2017

31 pages

ISBN:9781450350600

DOI:10.1145/3097766

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

Reproducibility

Qualifiers

Short-paper
Research
Refereed limited

Conference

SIGCOMM '17

Sponsor:

SIGCOMM

SIGCOMM '17: ACM SIGCOMM 2017 Conference

August 25, 2017

CA, Los Angeles, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
370
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)6

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chwalisz MGeissdoerfer KWolisz A(2019)Walker: DevOps Inspired Workflow for ExperimentationIEEE INFOCOM 2019 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)10.1109/INFCOMW.2019.8845199(277-282)Online publication date: Apr-2019
https://doi.org/10.1109/INFCOMW.2019.8845199
Parmentelat TTurletti TDabbous WMahfoudi MBronzino FKumar SGarcia-Saavedra A(2018)nepi-ngProceedings of the 12th International Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization10.1145/3267204.3267213(56-65)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1145/3267204.3267213
Saucez DIannone L(2018)Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility WorkshopACM SIGCOMM Computer Communication Review10.1145/3211852.321186348:1(70-74)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3211852.3211863

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Lessons learned from an XP experiment with students: test-first needs more teachings

Reconstructing the software environment of an experiment with kameleon

Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tag

Qualifiers

Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

PDF

eReader

Login options

Full Access

Abstract

References

Cited By

Index Terms

Recommendations

Lessons learned from an XP experiment with students: test-first needs more teachings

Reconstructing the software environment of an experiment with kameleon

Towards Reproducible Language-Based HRI Experiments: Open-Sourcing a Generalized Experiment Project

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tag

Qualifiers

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Login options

Full Access

Share

Share this Publication link

Share on social media

Affiliations