Machine learning (ML) algorithms are often assumed to be the most accurate way of producing pre-d... more Machine learning (ML) algorithms are often assumed to be the most accurate way of producing pre-dictive models despite problems with explainability and adverse impact. The 3rd annual Society for Industrial and Organizational Psychology Machine Learning Competition sought to find ML models for personnel selection that could balance the best of ML prediction with the constraint of minimizing selection bias based on race and gender. To test the possible advantages of simple rules over ML algorithms, we entered a simple and explainable rule-based model inspired by recent advances in model comparison. This simple model outperformed most ML models entered and was comparable to the top performers while retaining positive qualities such as explainability and transparency.
This initiative examined systematically the extent to which a large set of archival research find... more This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
The trust that humans place on recommendations is key to the success of recommender systems. The ... more The trust that humans place on recommendations is key to the success of recommender systems. The formation and decay of trust in recommendations is a dynamic process influenced by context, human preferences, accuracy of recommendations, and the interactions of these factors. This paper describes two psychological experiments (N=400) that evaluate the evolution of trust in recommendations over time, under personalized and non-personalized recommendations by matching or not matching a participant's profile. Main findings include: Humans trust inaccurate recommendations more than they should; when recommendations are personalized, they lose trust in inaccurate recommendations faster than when recommendations are not personalized; and participants report less trust and lower overall ratings of personalized but inaccurate recommendations compared to not-personalized inaccurate recommendations. We make connections to the possible implications of these psychological findings to the design of recommender systems.
Personality and Individual Differences, Jun 1, 2019
Abstract A concurrent, construct validity study was conducted to compare responses from a traditi... more Abstract A concurrent, construct validity study was conducted to compare responses from a traditional personality instrument to responses from a game-like instrument designed to assess personality. Personality assessment research demonstrates that misleading and careless responses typically stem from lack of internal motivation or interest in the task. To address the underlying motivation issue, we created a text-based fantasy game where participants can select a variety of choices to reflect corresponding personality characteristics. Results from three studies show consistent moderate-to-strong correlations between choices in the game and scores on a traditional Five-Factor Model personality inventory, indicating that game-like personality assessments can be developed with acceptable construct validity in a mentally stimulating way.
Digital maps are important for many decision-making tasks that require situational awareness, nav... more Digital maps are important for many decision-making tasks that require situational awareness, navigation, or location-specific data. Often, digital mapping tools must generate a map that displays labels near associated features in a visually appealing manner, without occluding important information. Automated label placement systems generally accomplish this nontrivial task through a combination of heuristic algorithms and cartography rules, but the resulting maps often do not reflect the preferences and needs of the map user. To achieve higher quality map views, research is needed to identify cognitive and computational approaches for generating high-quality maps that meet user needs and expectations. In this paper, we present a study that explores the visual preferences of map users and supports the development of a preference model for digital map displays. In particular, we found that participants demonstrated consistent preferences for how labels are placed near their point of ...
We examined the ability of game-like personality measures (GPMs) to reduce careless responding an... more We examined the ability of game-like personality measures (GPMs) to reduce careless responding and faking. Across two experiments (N = 279) participants completed both GPMs and the IPIP-50. In Experiment 1 participants completed both assessments initially and again 1 to 4 weeks later with instructions to make themselves appear as desirable as possible. Experiment 2 included careless responding identification items in each measure. Results support the hypothesis that game-like assessment reduces both careless responding and faking compared to traditional Likert type personality assessment.
Machine learning (ML) algorithms are often assumed to be the most accurate way of producing pre-d... more Machine learning (ML) algorithms are often assumed to be the most accurate way of producing pre-dictive models despite problems with explainability and adverse impact. The 3rd annual Society for Industrial and Organizational Psychology Machine Learning Competition sought to find ML models for personnel selection that could balance the best of ML prediction with the constraint of minimizing selection bias based on race and gender. To test the possible advantages of simple rules over ML algorithms, we entered a simple and explainable rule-based model inspired by recent advances in model comparison. This simple model outperformed most ML models entered and was comparable to the top performers while retaining positive qualities such as explainability and transparency.
This initiative examined systematically the extent to which a large set of archival research find... more This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
The trust that humans place on recommendations is key to the success of recommender systems. The ... more The trust that humans place on recommendations is key to the success of recommender systems. The formation and decay of trust in recommendations is a dynamic process influenced by context, human preferences, accuracy of recommendations, and the interactions of these factors. This paper describes two psychological experiments (N=400) that evaluate the evolution of trust in recommendations over time, under personalized and non-personalized recommendations by matching or not matching a participant's profile. Main findings include: Humans trust inaccurate recommendations more than they should; when recommendations are personalized, they lose trust in inaccurate recommendations faster than when recommendations are not personalized; and participants report less trust and lower overall ratings of personalized but inaccurate recommendations compared to not-personalized inaccurate recommendations. We make connections to the possible implications of these psychological findings to the design of recommender systems.
Personality and Individual Differences, Jun 1, 2019
Abstract A concurrent, construct validity study was conducted to compare responses from a traditi... more Abstract A concurrent, construct validity study was conducted to compare responses from a traditional personality instrument to responses from a game-like instrument designed to assess personality. Personality assessment research demonstrates that misleading and careless responses typically stem from lack of internal motivation or interest in the task. To address the underlying motivation issue, we created a text-based fantasy game where participants can select a variety of choices to reflect corresponding personality characteristics. Results from three studies show consistent moderate-to-strong correlations between choices in the game and scores on a traditional Five-Factor Model personality inventory, indicating that game-like personality assessments can be developed with acceptable construct validity in a mentally stimulating way.
Digital maps are important for many decision-making tasks that require situational awareness, nav... more Digital maps are important for many decision-making tasks that require situational awareness, navigation, or location-specific data. Often, digital mapping tools must generate a map that displays labels near associated features in a visually appealing manner, without occluding important information. Automated label placement systems generally accomplish this nontrivial task through a combination of heuristic algorithms and cartography rules, but the resulting maps often do not reflect the preferences and needs of the map user. To achieve higher quality map views, research is needed to identify cognitive and computational approaches for generating high-quality maps that meet user needs and expectations. In this paper, we present a study that explores the visual preferences of map users and supports the development of a preference model for digital map displays. In particular, we found that participants demonstrated consistent preferences for how labels are placed near their point of ...
We examined the ability of game-like personality measures (GPMs) to reduce careless responding an... more We examined the ability of game-like personality measures (GPMs) to reduce careless responding and faking. Across two experiments (N = 279) participants completed both GPMs and the IPIP-50. In Experiment 1 participants completed both assessments initially and again 1 to 4 weeks later with instructions to make themselves appear as desirable as possible. Experiment 2 included careless responding identification items in each measure. Results support the hypothesis that game-like assessment reduces both careless responding and faking compared to traditional Likert type personality assessment.
Uploads
Papers by Jason Harman