The Anatomy of Hardware Reverse Engineering: An Exploration of Human Factors During Problem Solving

Reverse engineering is defined as a process in which a human analyst examines an existing technical system to retrieve its underlying structures, components, and functionalities without access to its high-level description [49]. Reverse engineering is commonly employed in various fields such as machinery, software, and (micro)electronics like chips. Two principle types of reverse engineering exist in the context of digital systems relevant to cybersecurity: Software and Hardware Reverse Engineering.

Software Reverse Engineering (SRE) is regularly performed on machine code or binary files, which contain highly optimized bytecode (i.e., strings consisting of zeroes and ones) designed to run on specific devices [41]. Common applications for Software Reverse Engineering (SRE) encompass everyday use cases such as malware detection, security analyses, obsolescence management, or the establishment of interoperability [24].

Prior research that explored human factors in SRE focused on program comprehension of (obfuscated) source code: In 2014, Ceccato et al. conducted a user study with students to assess the impact of code obfuscation on the capability of humans to understand and modify source code in terms of time, efficiency, and co-factors such as prior experiences [9]. Hänsch et al. partially replicated these experiments and showed that program comprehension effectiveness is generally higher for experienced programmers than for inexperienced programmers, but that the comprehension gap between these groups narrows considerably when source code obfuscation is used. Later, Ceccato et al. extracted a detailed model for program comprehension of obfuscated source code based on three reports of professional penetration testers examining realistic applications [10].

Unlike program comprehension, however, reverse engineering of software usually does not take place at the source-code level, but at the binary level, which is much harder to comprehend and rarely contains high-level information such as variable names. Binary-level SRE shares similarities with HRE in that it involves highly optimized implementations using molecular low-level operations. Nevertheless, the two areas differ fundamentally in that software typically runs sequentially, while hardware usually runs in a highly parallelized manner. Against this background, the transferability of the above-described prior work on human factors in program comprehension to the HRE domain appears to be low.

Votipka et al. conducted an interview study with professional reverse engineers with the goal of gaining insights to improve the design of reverse engineering tools and divide the processes of binary analysis into three phases, to which they assign static and dynamic methods [58]. Beyond this, we are not aware of any research to date that examines the human processes of SRE at the binary level using empirical data.

In the case of software, tools such as decompilers, disassemblers, low-level debuggers, and packet sniffers that have been developed over the course of the past few decades are now in common use by analysts. Advanced SRE tool suites such as IDA Pro (proprietary) or Ghidra (open-source) even combine different (semi-)automated tools to provide analysts with a variety of functionalities suited to their individual needs. Long-standing tool development, an active research community, and well-established business use cases have over time all helped SRE advance from a mostly manual to a predominately automated process [37].

While the domain of HRE does not benefit from a comparable level of tool support, recent efforts have led to the development of the HAL HRE framework by Fyrbiak et al. [36]. HAL is a uniform platform, which provides support for a number of techniques needed for HRE (e.g., enabling visual inspection or interactive netlist exploration). Crucially, even when employing such tools, analysts usually have no choice but to create custom solutions or even perform detailed manual analysis due to the complex and proprietary nature of commercial hardware designs [6, 34]. However, the general technical methods for conducting HRE are well understood: Reverse engineering a microchip involves several steps until a 2-D model (the so-called netlist), consisting of individual logic gates and storage elements together with their interconnections, of the unknown chip design can be obtained [57]. In order to acquire information of interest (e.g., module boundaries, hierarchies, and high-level descriptions) from such netlists, different techniques are combined: Structural analysis interprets the netlist as a graph and allows the application of efficient graph-based algorithms while detailed functional analysis focuses on the recognition and verification of combinational logic [56]. Last but not least, dynamic analysis (i.e., simulation) helps to reveal temporal relations in netlists such as the behavior of Finite State Machines (FSMs) [35]. The human analyst generally only employs a subset of these techniques based on the specific technical conditions encountered during, as well as the goals of (e.g., detecting and removing an Intellectual Property (IP) watermark or finding a cryptographic key in a circuit) the task at hand.

HRE is commonly applied to achieve both legitimate (e.g., hardware Trojan detection, identification of counterfeit products, and analysis of a competitor’s product) as well as illegitimate (e.g., malicious manipulations, overproduction, and counterfeiting) ends [47]. As an example, we mention the well-known method for the detection of hardware counterfeits based on watermarks [1]. Hardware manufacturers protect their IP by embedding these watermarks directly into the netlist, e.g., for protecting against the theft of innovative developments. If analysts detect the watermark in an unauthorized clone of the original netlist, they can prove that it has been counterfeited. Conversely, an attacker has to detect and remove the watermark in the netlist of the original product to create a watermark-free counterfeit. In either case, the reverse engineer employs (a subset of) the previously described analysis techniques.

In summary, the general technical analysis methods for HRE are well understood. However, an HRE attack always requires custom solutions as well as analysts’ skills and knowledge.

Surprisingly little is known about the cognitive processes which most influence HRE performance.

In their theoretical model, Lee and Johnson–Laird [43] define reverse engineering of Boolean systems, i.e., binary logic expressions, as a specific type of problem solving. In general, problem solving is characterized as an essential cognitive ability and a key competence necessary to handle complex situations in our daily lives [33]. When confronted with a problem, the problem solver lacks the knowledge for producing an immediate and routine solution. In order to achieve a desired goal state (i.e., problem solution) the problem solver applies a series of cognitive operators (e.g., problem-solving strategies) [22, 28]. Prior psychological research has established a broad range of taxonomies to describe and define problems. A widely accepted and applied taxonomy distinguishes between simple and complex problems [48]. Simple problems can be represented in a problem space and are characterized by a clear set of means through which the problem solver can reach a clearly defined goal state [22]. An example of a simple problem is the Tower-of-Hanoi problem in which the problem solver receives three rods and several different-sized discs. The goal is to move the discs from the left to the right rod in ascending order of disc diameter. For reaching the desired goal, the problem solver has a clearly defined set of means (e.g., only one disc can be moved at a time; no larger discs on smaller ones). In contrast, complex problems lack a clear problem definition, goal state, or precisely defined means for solving the problem [22]. Funke [32] defines five characteristics of complex problems:

An example of a complex problem is the Lohhausen problem, where a problem solver is asked to assume the role of a mayor of a small town and has to manage more than 2,000 variables of a dynamically changing system [21]. It is an open question to which problem-solving taxonomy HRE can be assigned or to what extent HRE problem solving combines aspects of simple or complex problems.

Lee and Johnson–Laird [43] postulate that reverse engineering is a specific and poorly understood type of human problem solving. They define problem solving in reverse engineering as “[...] working out how to assemble components with known properties into a system that has the input-output relations of a target system.” ([43]; p 20). In other words, reverse engineering problem solving is defined as the process of inferring how underlying mechanisms of a given system work, how the components influence the outputs, and how they depend on each other. Lee and Johnson–Laird [43] conducted five experiments in a laboratory setting with students who had no domain-specific prior knowledge. In these experiments, the students were asked to draw Boolean circuits that would control electric lights or a water flow system. Based on their findings, the authors concluded that reverse engineering performance is influenced by three factors: (1) The number of variable components, (2) their number of settings, which yield the outputs, (3) the interdependence of components that influence the outputs. Based on interview data, Lee and Johnson–Laird [43] were able to draw conclusions about applied strategies in the reverse engineering of Boolean systems. The authors found that the participants chose one of two main initial strategies: focusing on a single output at a time or focusing on a single component at a time. After solving a sub-problem, they then extended their solutions to the next sub-problem.

It nevertheless remains unclear whether Lee and Johnson–Laird’s [43] results can be generalized to apply to the processes of HRE for two main reasons. First, Lee and Johnson–Laird describe the processes of reverse engineering of simple Boolean systems, which are hardly comparable to complex real-world HRE challenges where a reverse engineer must make sense of hundreds of thousands or millions of logic components. Second, the laboratory experiment that they conducted included students who lacked relevant background knowledge in digital circuits, Boolean algebra, and so on—knowledge that reverse engineers in real-world situations clearly possess. To systematically describe the problem-solving processes used by hardware reverse engineers, we believe it is necessary to conduct a qualitative study in a realistic setting that closely approximates real-world conditions.

Previously, we [6] conducted an exploratory investigation into the technical processes of HRE and proposed a three-phased model consisting of (1) Candidate Identification, (2) Candidate Verification, and (3) Realization. We argue that analysts pass through these three phases in order to conduct an HRE attack and further maintain that reverse engineers analyze chip designs through both manual analyses (e.g., visual identification of crucial components) and semi-automated steps (e.g., verification of components based on writing programs). Moreover, our prior findings suggest that working memory may play a role in solving an HRE problem task more efficiently in that participants with higher scores on this particular cognitive factor were able to more quickly produce solutions than participants with a lower score in working memory. Finally, we [6] conducted an initial descriptive analysis of the problem-solving behavior of two intermediates, which showed that the faster participant chose a strategy which can be described as divide-and-conquer, whereas the slower participant solved the problems through trial-and-error. Our previous work advanced the understanding of the technical and cognitive processes involved in HRE, but also emphasizes the need for further investigation—it is undoubtedly necessary to apply a systematic methodology of analysis to examine the problem-solving processes and applied strategies of more than two analysts.

It is highly likely that expert hardware reverse engineers have developed an extensive base of knowledge and set of skills through years of deliberate practice [27], and that differences in individual factors, such as level of expertise, play a role in HRE problem-solving performance. Such experts may have gone through skill acquisition processes [2, 29] and consequently may have been able to produce efficient solutions by leveraging their fast problem-solving and procedural workflows. To frame the point in more everyday terms, an expert reverse engineer might at once know how to overcome an HRE challenge similar to a situation in which a chess master immediately recognizes which moves are feasible (e.g., [14]) or in which a skilled medic is able to immediately render a diagnosis based on observable symptoms [18]. It is an open question as to whether problem solving of HRE experts is comparable to problem solving employed by HRE novices or intermediates. It is conceivable that HRE experts and intermediates apply similar problem-solving strategies since previous research in the domain of software development [3] has shown both novices and experts to employ a top-down strategy of deriving and solving sub-problems. Since other research [40] on solving programming problems demonstrated that experts chose a broader and more holistic approach, it is also conceivable that HRE experts may differ from novices in their problem-solving strategies. These open questions further establish the need to achieve a better understanding of this under-studied type of human problem solving by analyzing reverse engineers with different levels of expertise as they engage in a realistic problem-solving task.

Researchers aiming to conduct experimental studies in HRE must contend with the methodological problem posed by the paucity of HRE experts available for research. Because the worldwide population of ethical hardware hackers—experts engaged in legitimate HRE activities such as those presented in Section 2.1—is very small, efforts to recruit them are not only time-consuming but also often unsuccessful. This already small population is narrowed even further when one considers that such HRE experts often work for highly-specialized companies or government agencies that impose contractual restrictions that forbid them from disclosing details of their work up to and including their working processes. Therefore, even when if specialists are willing and very much suited, they are often unable to participate in research which explores HRE processes and underlying cognitive factors. While there certainly exists another population of HRE experts who use their skills for questionable practices, ethical considerations would not allow us to recruit them—nor could we expect them to be interested in supporting our research: By developing cognitive obfuscation techniques, we explicitly aim at impeding such illegitimate HRE attempts.

In prior research, we developed a methodological approach that allows addressing the methodological problem that HRE experts are unavailable to. The use of a 14-week HRE training has been proven effective in promoting HRE skill acquisition in novices with relevant background knowledge [6, 61, 62]. During the first six weeks of training, students acquire declarative knowledge (e.g., theoretical backgrounds in relevant topics such as chip design or Boolean algebra). In the subsequent eight weeks, students are asked to work on four practical HRE problem-solving tasks with the goal of transforming their declarative knowledge into procedural knowledge [2]. These HRE skill-training exercises encompass several HRE-specific challenges, such as the retrieval and analysis of crucial components in a netlist or the extraction of a cryptographic key. The practical phase of the training also imparts comprehensive skills in the use of the HRE tool HAL (see Section 3.2.2). We used this same methodology to train the experienced cybersecurity students in our present study as it has been demonstrated to enable experienced novices with relevant, domain-specific background knowledge to solve realistic HRE problems (see Section 3).

HRE enables the retrieval of crucial information from an unknown chip design and is often applied in the context of IP theft or the removal of security protection mechanisms such as watermarks. The cognitive processes and abilities of the hardware reverse engineer are the primary determinants of success as fully-automated tools for HRE do not yet exist. The exploration of the cognitive processes behind HRE has thus far been limited to a couple of prior works: Lee and Johnson–Laird [43] classified reverse engineering of Boolean systems as a specific type of human problem solving. In an initial investigation, we [6] developed a three-phase model of technical processes in HRE and studied the reversing procedures of two engineers with comparable levels of experience. HRE problem-solving processes—and in particular their time-efficiency and the ways and degree to which they are impacted by expertise—otherwise remain poorly understood. In the present work, we begin to fill this gap by systematically analyzing the problem-solving processes of reverse engineers with differing levels of expertise. Moreover, we conduct in-depth analyses of the time-efficiency of applied problem-solving strategies. Based on our findings and by including cognitive processes in our considerations, we offer recommendations and present ideas for the development of novel HRE-impeding countermeasures and strive to answer the following research questions:

In the present study, we investigated how eight students on intermediate levels of HRE expertise and one expert solved an HRE problem-solving task. In Section 3.1.1, we outline how we trained the student sample to acquire an intermediate level of HRE expertise. We elaborate on the sample of intermediates in Section 3.1.2 and provide information on the expert participant in Section 3.1.3.

All participants voluntarily participated in our study and provided written informed consent. We emphasized that the participants retained the right to withdraw and/or request the deletion of all study-related data at any time and without reason. All student participants received monetary compensation for the completion of study-related materials. We protected participants’ privacy by using randomly-assigned pseudonyms instead of participant names on all study-related materials and in all study-related activities. During data analysis, these pseudonyms were replaced by randomly-assigned numbers. The privacy-protection procedures of this study were reviewed and approved by the data protection officer of the university at which it was conducted.

Demographic Questions. Using two short questionnaires, we asked participants to provide information about their socio-demographic backgrounds. Of these, one was designed for the student participants and the other for the HRE expert. Students were asked to answer questions about their age, major, and target degree. To corroborate the reported expertise level of the expert, the expert was asked questions including age, the highest level of education, current job position, expertise level, years of relevant experience in HRE, and hours spent performing HRE per week (according to Votipka and colleagues [58]).

Behavioral Log Files. A behavioral log file was automatically generated by HAL for each participant solving the HRE task. Every log file contained between 148 and 467 entries consisting of a timestamp and one of the following events: (i) a script-based analysis step containing the executed Python script evaluated for syntactical correctness and the corresponding console output; (ii) a short Python console input evaluated for syntactical correctness and the corresponding output; (iii) a manual analysis step of a netlist component (e.g., selection of a gate or net), together with a unique identifier for the component; or (iv) an indicator for (in)activity phases and their duration. In sum, the log files of the nine participants contained 2,445 different events (1,141 executed scripts, 68 console inputs, 1,217 manual netlist interactions, and 19 idle events). Each participant’s log file was pre-processed in order to display the events together with their associated information in tabular form. An excerpt of such a pre-processed log file is shown in Table 1.

Iterative Open Coding based upon GT. We qualitatively analyzed each event recorded for each participant by applying an iterative open coding scheme based upon GT methodology [55]. Grounded Theory (GT) is a well-established and standard research method from the social sciences and is often applied in research areas in which no previous theories or models exist [12]. As prior research on cognitive processes in HRE is lacking, we decided to apply GT as it provides explicit guidelines which facilitate the analysis of problem-solving processes.

A detailed description of the iterative coding procedure we applied is as follows. First, we grouped one or several consecutive events into \(1{,}232\) segments so that related events could be annotated together. These groups mostly consisted of manual netlist interactions and only rarely consecutive console inputs or executed scripts. To create a basis for the assignment of open codes describing HRE problem-solving processes and strategies, we annotated each segment with the following:

We subsequently assigned one or several open codes (e.g., script-based inspection of watermark candidates, or successful correction of syntactical errors) capturing the most relevant annotations to each segment. Two of the authors began the open coding process by collaboratively annotating and encoding the log files of four participants to create an initial code book, making updates and re-coding previous segments as necessary. Using this initial code book, a third author annotated and encoded the log files of the remaining five participants independently. After completing this process of annotating and encoding each participant’s log file, we discussed the results, incorporated any new codes into the code book, and retroactively applied any such codes where applicable. The final code book contains 103 different unique codes, which are a condensed version of the segment annotations and reflect the observed HRE processes of our participants. These 103 unique open codes were assigned 1,887 times across the 1,232 annotated segments. After completing the iterative open coding for all participants, an external researcher independently coded 71 randomly selected segments from three participants on the basis of the existing code book with an acceptable inter-coder reliability of \(84.5\)%. Two of the coders had relevant backgrounds in HRE and a significant level of domain-specific prior knowledge in, for example, Boolean Algebra, chip architecture, hardware watermarks, and the handling of HAL, as well as scripting in Python. The third coder had relevant knowledge of cognitive psychology and qualitative data analysis, such as applying an iterative open coding procedure to analyze qualitative data and to derive a theoretical model from the data.

Taxonomy of Open Codes. Using the final code book as a starting point, we developed a taxonomy of observed HRE problem-solving processes during several rounds of discussions organized for that purpose. To do this, we first grouped related open codes into problem-solving clusters based on similarities. These clusters were then subordinated into nine principal actions representing the problem-solving processes exhibited by participants during the study (RQ1), thereby establishing a foundation for the later analysis of problem-solving strategies (RQ2). Eight of the nine principal actions could again be organized into two main categories, allowing HRE problem-solving processes to be subdivided into either programming-related or reversing-specific actions.

Once completed, this taxonomy enabled us to describe observed problem-solving processes and to highlight differences between participants with different levels of expertise (see Section 4).

Total Solution Time. We computed the total solution time, which is an important metric for performance analyses in psychological studies. In the particular setting of our study, participants were asked to solve a single HRE problem-solving task. While industry experts are likely to develop strategies that transfer to other HRE tasks, our participants were not specifically requested to develop generalizable solutions. We will return to this aspect in Section 5.4. As such, time is a central metric for assessing the efficiency of HRE and can provide important insights into the efficiency of participants’ solutions when combined with our in-depth qualitative analysis of applied problem-solving strategies.

Solution times were calculated per participant using the time stamps in the automatically generated log files. Idle events, which could be clearly identified as breaks unrelated to the task based on their annotations and wall-clock time (i.e., the time of day at which they occurred), were excluded from the total solution time. As the participants were already preselected based on their fully correct solutions, we analyzed total solution time to determine differences in the efficiency of observed problem-solving strategies (RQ2).

Incorporating Open Code Duration. Since the mere number of open codes assigned does not alone provide a granular metric with which to describe the problem-solving strategies of a reverse engineer (RQ2), we supplemented it through the incorporation of participant solution time. To do so, we first split the total solution time of participants into 50 time windows of 2% each.³ We then identified which open code(s) were assigned in each time window, thereby creating a fine-grained measure of the relative amount of time each participant spent on single actions (as described by the respective open codes). This metric has the added benefit of providing information about which open codes were assigned towards the beginning, middle, or end of a participant’s time-on-task.

The incorporation of participants’ time spent on assigned open codes, in summary, enables the analysis of their applied problem-solving strategies to answer RQ2.

At the beginning of the 14-week HRE training, participants signed the informed consent document and were assigned a randomly chosen pseudonym. Subsequently, all participants were asked to answer the questionnaires on socio-demographics via an online survey provider. After finishing the HRE training, participants received the HRE problem-solving task materials consisting of a short task description, the watermarked netlist, and a copy of the paper in which the implemented watermark was first proposed.

Participants were provided a two-week window in which to complete the task and were free to choose their time and place of work as well as whether they wanted to work through the problem in one sitting or in smaller steps with breaks in between. The participants individually worked on their submissions, which was clearly communicated to them at the beginning of the HRE training and the main study. Based on the submissions and the log files, we checked the students’ submissions for plagiarism and copies. We did not detect any signs of plagiarism in the participants’ submissions. Once finished, participants uploaded their log files to a server located at the university and received the stipulated monetary compensation for their participation in the study.

Following the methodology described in Section 3.4, we systematically analyzed and categorized the log files submitted by eight intermediates and one expert after completing the realistic HRE task presented to them in the study. We grouped the 103 unique codes into nine principal actions, enabling us to describe HRE problem-solving processes in detail and to create a model of HRE problem solving, which is presented in Figure 2 below. According to our analysis, eight of the nine principal actions could be assigned to one of the two main HRE problem-solving categories: Reversing Actions, and Code Development. These two main categories clearly differentiate actions which directly influence the success of problem-solving processes from actions pertaining to the development of programming code, which indirectly influences problem-solving processes. The ninth and final principal action, External Influences, encompasses codes that describe exogenous influences upon HRE problem-solving processes such as external interruptions of the analyst.

Codes from all nine principal actions were assigned to the events contained in the log files of eight participants; we did not observe any External Influences in the logs of the ninth participant. Table 2 provides a per-participant overview of assigned open codes aggregated at the principal-actions level. A detailed view of all open codes and the frequency with which they were assigned to each participant can be obtained from Appendix C. The HRE problem-solving model we developed is hierarchical and consists of four levels, progressing from general to specific: categories are represented at the highest level, principal actions at the second level, followed by clusters and finally by open codes at levels three and four, respectively. The number of unique open codes specifies how many different open codes a principal action or cluster contains, whereas the number of assigned open codes represents how often unique codes were assigned to participants’ actions.

As introduced in Section 3.4 and expanded upon in the preceding section, we applied an iterative open coding approach based upon GT [55] and systematically gathered and analyzed behavioral log files from nine participants tasked with solving a realistic HRE problem in order to develop a detailed model of HRE problem-solving processes. The model consists of two main categories and nine principal actions which were developed based on 103 assigned open codes. While we do not claim the model to be exhaustive—new open codes may emerge through the analysis of different tasks—we believe that it covers all of the essential actions hardware reverse engineers conduct in order to solve HRE tasks.

The problem-solving model has a hierarchical structure, ranging from more task-specific layers (open codes) to more general layers (clusters and principal actions). This structure was developed by the three coders in several rounds of discussion based on the final open codes after the coding process was completed. Accordingly, we have moved from data to the HRE problem-solving model, or in other words, from the specific to the general. Therefore, data at the open-code layer is sometimes task-specific, i.e., specific to removing a watermark from a Field Programmable Gate Array (FPGA) netlist (especially in the two principal actions Inspection and Information Gathering and Reversing Milestones and Sub-Goals). However, by applying an inductive analysis, we transferred the specific context of the open codes through the clusters to more general principal actions. Due to the exploratory nature of this work, we cannot draw definitive conclusions about the generalizability of our HRE problem-solving model. We consider our developed model as the first important step in describing problem-solving processes in HRE. However, this model still needs to be validated and possibly extended by future research with different realistic HRE tasks.

The HRE framework HAL is used by a growing number of researchers and professionals and provides a simple integrated development environment for script-based netlist analysis in Python, a path also taken by well-known tools from the field of SRE (e.g., IDA Pro). Code development skills, such as programming skills in Python, are more general abilities that are applied in a variety of contexts. However, in the case of HRE, code development skills are relevant to analyze an unknown netlist. Without programming skills, HRE would be very cumbersome or even impossible. Thus, the actions assigned to Code Development are also likely to be relevant in the context of HRE tasks.

The resulting HRE model allows us to conceptualize the observed HRE processes. This conceptualization in turn provides us with the basis, a language so to speak, with which we can describe the problem-solving strategies of the HRE-process. The above-presented model, which divides reversing-specific HRE problem-solving processes into four principal actions with additional granularity at the sub-cluster level, provides in-depth perspective as we explore fundamental expertise-related differences in participant problem-solving processes in our discussion and analysis in RQ1b. The principal actions are central to the HRE problem-solving processes of our participants as they indicate obstacles encountered while solving the task in the first case, and represent the variety of approaches applied to solve the task in the second. To identify and explore differences in the time efficiency of participants’ problem-solving strategies, these principal actions must be viewed in context with time-on-task as well as other actions which may have led to certain Reversing Problems or Reversing Strategy Decisions. We delve deeper into these principal actions and present this discussion of time-efficiency in RQ2.

In order to answer RQ1b, we searched for qualitative differences on an open-code level between the problem-solving processes of the eight intermediates and the one expert. An excerpt of a visualization of this comparison is shown in Figure 3. The left-hand side depicts three open codes which were only observed in the expert’s process, all stemming from the main category Reversing Actions. The right-hand side similarly shows the five most common codes assigned only to actions undertaken by the intermediates. Of these, three again stem from the Reversing Actions category, with the other two being related to External Influences. The middle of Figure 3 shows the five most common codes—all related to Code Development—assigned to both the expert and intermediates. In total, there are 52 unique codes which were only assigned to intermediates’ problem-solving processes. Of these 52 codes, only 10 codes fall into the Code Development category, whereas 39 are related to the intermediates Reversing Actions.

Open codes indicating a lack of experience such as introduction of repeated semantic errors (assigned 11 times) and introduction of redundant code (assigned 6 times) reflect major differences in intermediates’ Code Development. In this context, intermediates also inserted anticipatory documentation (assigned 8 times) to help keep better track of their work-in-progress. The expert and the intermediates otherwise had the majority of unique open codes related to Code Development in common.

Differences in regards to Reversing Actions were more marked, with the majority of unique open codes being assigned only to the problem-solving processes of the intermediates. Intermediates showed unique processes in the principal action Inspection and Information Gathering as reflected by several open codes related to manual information gathering, whereas two of the three open codes exclusively assigned to the expert are related to script-based information gathering. In the principal action Reversing Strategy Decisions, intermediates showed several diverse and unique processes as indicated by 13 exclusively assigned codes. Worthy of mention is that seven out of eight unique codes from the Reversing Problems principal action were assigned only to intermediates. Seven of the eleven open codes from the principal action Reversing Milestones and Sub-Goals were assigned to both the expert and to the intermediates, whereas four codes were assigned to intermediates exclusively.

In the preceding section, we compared the HRE problem-solving process of one expert and eight intermediates at the open-code level.

Our analysis shows that major differences in the problem-solving processes of the two groups exist within the Reversing Actions category, primarily concentrated in the principal actions Inspection and Information Gathering, Reversing Strategy Decisions, and Reversing Problems. While many of the unique open codes from those principal actions were only assigned to intermediates, it is plausible that each analyst only executed a subset of a broader universe of reversing-related actions available to them. The actions enumerated in the three aforementioned principal actions will allow us to develop a precise picture of strategies employed during the observed attack, and will therefore remain in focus as we analyze the efficiency of HRE problem-solving strategies in RQ2. As we tackle the topic of efficiency, we will expand our analysis beyond the number of assigned codes to include a fine-grained metric based on time (see Section 3.4).

Intermediates tended to be interrupted by External Influences as evidenced by the frequency with which they had to re-enter and re-orient themselves after experiencing external interruption. While we cannot draw a generally applicable conclusion from this observation, it is conceivable that the expert was better able to focus on the task than the intermediates.

The problem-solving processes of the expert and the intermediates in regard to Code Development differed only on the margins, with the expert exhibiting a higher level of programming experience—the many open codes assigned to both groups otherwise indicate a general similarity of process.

The question at the heart of the following analysis is twofold: Was the expert able to solve the HRE problem-solving task more quickly and in a more efficient way than the intermediates and are there differences in the applied problem-solving strategies?

Because a hardware reverse engineer will ultimately always succeed if given enough resources, efficiency is in this field defined as a function of time. This point was also borne out in the results of our study: all of the participants correctly completed the HRE task (as indicated by their high solution probabilities), but with solution times ranging from 163 to 528 minutes (see Table 3).

In Section 4.3, we established that the differences between the expert and the intermediates are most concentrated in the principal actions Inspection and Information Gathering, Reversing Strategy Decisions, and Reversing Problems. Thus, we used the open codes assigned to these principal actions to form the basis of our evaluation of the efficiency of the participants’ problem-solving strategies.

The absolute number of open codes assigned is broadly descriptive of the steps of which problem-solving processes consist but is an otherwise coarse metric. Folding in the actual time participants spent on the assigned open codes (see Section 3.4) added granularity to our analysis through the consideration of the relative time each participant spent on actions related to open codes within the principal actions Inspection and Information Gathering, Reversing Strategy Decisions, and Reversing Problems in Table 3. This table provides an overview of the most prevalent actions taken in each of the participants’ problem-solving strategies. It also shows the time-intensity of those actions as well as the point in the timeline of the attack at which they were taken. It therefore serves as a reference which complements the case-by-case descriptions of problem-solving strategies that follow. We proceed according to total solution time in ascending order, starting with the fastest participant.

Expert: Sub-steps and Test Cases. The expert achieved a solution in 163 minutes, the fastest time of all participants. The expert’s Reversing Strategy Decisions revolved around the (small-step) preparation of reversing sub-steps, indicating an ability to sub-divide the task into several isolated sub-problems which could then be overcome individually. The continuous development of test cases that helped the expert to test solutions to sub-problems in a practical manner further supports this conclusion. From the outset, the expert applied both manual and script-based Inspection and Information Gathering actions, demonstrating the ability to quickly translate manual methods into scripts, thereby automating information gathering from the beginning. Of note is that, after encountering a problem (lost track of the reversing approach) in the middle of the reversing attack, the expert reverted to manual selections. These manual selections seemed to be an important anchor which helped to overcome the reversing problem in just seven minutes.

P8: External Resources, Reversion, and Duplication. Participant 8 (P8) required 176 minutes to solve the HRE task, which is very close the expert’s solution time. P8’s problem-solving strategy was characterized by three overarching Reversing Strategy Decisions: using external resources, reversion to a proven approach, and the duplication of partial solutions. Especially at the beginning and in the middle of the attack, P8 applied knowledge and skills acquired from previous HRE training tasks (proven approaches; e.g., reversing methods to identify components of interest). Throughout the attack, P8 continuously referred to external resources such as the provided coding guide, pen-and-paper analyses, or online resources. In the further course of the attack, P8 divided the HRE task into several similar sub-tasks. P8 then solved one of those sub-tasks and adapted the solution to the other open sub-tasks (as reflected by the open code duplication of partial solutions). With respect to Inspection and Information Gathering, P8 performed only manual actions. While most of those actions took place during the beginning of the attack, in-depth manual inspection of watermark candidates was observed throughout the task. Moreover, P8 only showed very short-lasting Confusion (lost track of the reversing approach; lasting for 7 minutes), which was mainly caused by several simultaneous Code Development actions.

P3: External Resources and Sub-steps. At 187 minutes, Participant 3 (P3) was also very close to the expert’s solution time. In terms of Reversing Strategy Decisions, P3 often engaged in (small-step) preparation of reversing sub-steps during the first half of the attack, that is, P3 was able to divide the larger problem into smaller sub-problems. At the beginning of the task, P3 was using external resources and applied script-based methods for Inspection and Information Gathering. This bore fruit as the attack progressed, as P3 conducted only a few manual information-gathering actions toward the middle of the task and did not need to gather any additional information in the second half, having already automated essential information gathering the in beginning. Despite the short overall solution time, we observed a very large number of Reversing Problems, which were present for 75 minutes in P3’s problem-solving process. These problems arose from a Lack of Understanding of two basic HRE concepts which had to be resolved before correctly completing the task.

P5: Sub-steps, Generic Approach, and Test Cases. Participant 5 (P5) required 221 minutes to solve the HRE task, about an hour longer than the expert. P5 consulted external resources and applied script-based Inspection and Information Gathering methods at the beginning of the attack. A dominant strategy that could be observed upon review of P5’s Reversing Strategy Decisions was the evolution of a generic approach stemming from the selection of test candidates and development of test cases. A generic approach is characterized by its re-usability and adaptability, e.g., for similar tasks on other netlists. The other overarching strategy observed in P5’s problem-solving process was the (small-step) preparation of reversing sub-steps. Although similar development of test cases and sub-step preparation was observed in the problem-solving processes of other participants, a generic approach leading to re-usable solutions was unique to P5. Despite us identifying two dead ends as well as a few strategy changes for script-based analyses, P5 was able to work through the observed Lack of Understanding in a relatively short amount of time (12 minutes).

P2: Sub-steps and External Resources. During P2’s attack, which lasted 221 minutes, two overarching Reversing Strategy Decisions could be observed: small-step preparation of reversing sub-steps and using external resources. The several manual and script-based Inspection and Information Gathering actions which P2 took appeared only at the beginning of the attack. In the middle of the attack, P2 showed and resolved a Lack of Understanding, which accounted for roughly 35 minutes of the total solution time.

P4: External Resources. The attack of Participant 4 (P4) lasted 233 minutes. While using external resources could be identified as the overarching Reversing Strategy Decision especially in the middle and end of P4’s attack, other strategies, including the development of test cases, reversion to a proven approach, and preparation of a reversing sub-step were also observed. P4 applied several manual Inspection and Information Gathering actions toward the end of the attack, whereas no script-based information-gathering methods could be observed. In P4’s problem-solving process, several Reversing Problems totaling 65 minutes were annotated. These problems occurred mainly at the beginning and during the middle of the attack and encompassed Failed Attempts such as the unsuccessful transfer of an already known approach to a current problem and Confusion leading to an instance of lost track of the reversing approach.

P6: External Resources and Sub-steps. Participant 6 (P6) required a total of 261 minutes to reach a solution. The overarching Reversing Strategy Decisions of P6 were using external resources and small-step preparation of a reversing sub-step. In addition to these strategies, P6 also had reversion to a proven approach and applied a generic approach towards the end of the attack. With respect to Inspection and Information Gathering, P6 applied several manual and script-based techniques at the beginning and during the middle of the attack. P6 spent a total of 78 minutes on Reversing Problems, including several attempts to resolve observed Lack of Understanding.

P7: Fully Manual and Hardcoding Approach. Participant 7 (P7) was one of the slowest reverse engineers (351 minutes). P7’s strategy at the beginning of the attack was based on using external resources. As time progressed, P7 made several unique Reversing Strategy Decisions, first through a strategy change from script-based to manual analysis and then later to a fully manual, hardcoding approach during the middle of the attack, which resulted in duplication of partial solutions. The fully manual and hardcoding approach consisted of encoding fixed values into the solution, which had been read out through manual analyses. This approach was also comparable to the participant’s Inspection and Information Gathering processes, which consisted of in-depth manual inspection of watermark candidates. During the attack, P7 encountered a relatively small number of Reversing Problems (Lack of Understanding and Failed Attempts) and these were resolved comparably quickly (35 minutes).

P1: External Resources and Reversing Problem Shooting. Participant 1 (P1) was the slowest overall participant with a total solution time of 528 minutes. The dominant Reversing Strategy Decisions of P1 consist of using external resources and reversion to a proven approach. Notably, during the second half of the attack, P1 failed to recognize that a correct solution of a sub-problem had been reached, and inappropriately changed reversing strategies. No subsequent Reversing Strategy Decisions were observed. Reversing Problems cost P1 a total of 167 minutes, the highest overall time of all participants. P1’s initial Inspection and Information Gathering approach was based on manual actions and only very few further manual information gathering actions could be observed during the subsequent course of the attack.

In summary, our results in RQ2 revealed differences in the problem solver’s time efficiency and applied problem-solving strategies. These differences extended beyond total solution time to encompass the set of strategies applied as well as the (in)ability to solve Reversing Problems quickly. We discuss these findings in the following.

The case-by-case analysis of RQ2 revealed similarities and differences between our participants, which are discussed in greater detail below. The open code (small-step) preparation of reversing sub-steps was predominant throughout the complete expert’s problem-solving process. Besides the expert, P5 and P2 also strongly relied on small-step preparation during their respective HRE problem-solving processes. We also identified that P3, P4, and P6 included small-step preparation in their approaches. Thus, we assume that most of the participants divided the main HRE task into several smaller sub-tasks which they solved in a step-by-step manner.

Furthermore, our data suggest that the open code development of test cases was also predominant in the problem solving of the HRE expert. The development of the solution based on test cases could only be identified in two other participants’ strategies: P5 and P4 (both at the beginning and in the middle of the HRE process). We assume that development with test cases is a more specific open code that only appears in the problem solving of reverse engineers who may have a strong background in software development, as test case generation is a common tool there.

Our results showed that all intermediates except P5 used external resources. Relying on external resources (e.g., provided coding guide, pen-and-paper analyses, or online resources) to develop reversing strategies, may compensate for a lack of skills and knowledge, and obtaining reassurance seemed to be an efficient approach for most intermediates. We also observed that most intermediates chose proven approaches (i.e., solutions from previous working steps) in order to save time or to reduce efforts for developing new solution strategies.

Our data suggest individual solution approaches. Intermediates P5 and P7 settled upon unique approaches, the former leading to a more and the latter to a less efficient solution. P5’s generic approach lead to a solution that would have been useful for completing potential future reversing tasks but was a relatively inefficient way to complete this single and specific problem-solving task. Although P7’s fully-manual approach was beset by a relatively small number of Reversing Problems, it was so at the expense of taking inefficiently long to complete. In other words, step-by-step manual analysis and subsequent processing of manually-identified components in the script was mostly accurate, but also very time-consuming.

Overall, we found that the open code strategy change was not a code that was prevalent in participants’ problem solving. Only P5 and P7 had to change their strategy during the reversing process. Hence, we assume that most of the participants had a clear plan on how they wanted to solve the HRE task and therefore were not forced to change their strategy.

In terms of Inspection and Information Gathering, we found that manual netlist interactions (e.g., manual selection of netlist components) occurred in each participant’s problem solving—but to varying degrees. Whereas P5, P7, and P8 strongly focused on manual approaches for netlist inspection and information gathering, the expert, P1, P2, P3, P4, and P6 applied manual approaches only occasionally. Furthermore, we found that only four participants (expert, P2, P3, and P6) applied script-based inspection and information gathering. We assume that analysts preferred manual netlist exploration via the graphical representation of the netlist in HAL over script-based information gathering.

In general, our results showed that more Reversing Problems led to higher total solution times. The fastest two participants (the expert and P8) each needed only seven minutes to resolve their single reversing problem. P1, in contrast, experienced a large number of Reversing Problems, became confused, and stayed off track for a while, resulting in a long total solution time. We did, however, identify one exception in the Reversing Problems dataset: P3 was one of the fastest participants, but also demonstrated frequent episodes of reversing-related Lack of Understanding. Despite a high frequency of mistakes, P3 was able to achieve important Reversing Milestones, resolve Lack of Understanding, and abandon dead-end actions in the second half of the attack.

Based on our previous assumptions and the results of RQ1, it would have been conceivable that the HRE expert was the only participant who could solve the task efficiently. Contrary to these expectations, the in-depth analysis we conducted pursuant to RQ2 revealed that two intermediates were able to achieve very efficient solutions by applying different problem-solving strategies than the expert. We discuss this finding in the next section.

In summary, the case- by-case comparison to answer RQ2 reveals that the participants had several similarities in their problem-solving strategies (e.g., manual Inspection and Information Gathering; small-step preparation of reversing sub-steps; using external resources). However, our analysis also shows that despite these observed overlaps, none of the nine solutions can be described as congruent with any other solution because participants chose different foci that resulted in different compositions of assigned open codes. In other words, we found that participants’ sub-processes are comparable, but that they occurred at different times of the reversing process (beginning, middle, end) and to different extents (less dominant vs. predominant). Our results show that specific sub-processes of the nine hardware reverse engineers were comparable, but the overall strategy differed. Thus, our results suggest that there is no single optimal (“one and only” or “end all, be all”) problem-solving strategy for the most efficient solution to the HRE task at hand. Rather, our analysis showed that several participants achieved efficient solutions through the application of individually composed strategies.

Based on our findings, we make two main contributions. First, our results contribute to the ongoing debate in psychology on the role of cognitive abilities and prior domain-specific knowledge in problem-solving performance. Our findings suggest that both expertise and cognitive abilities may play a role in HRE, and that HRE tasks may involve aspects of both simple and complex problems. Second, our work contributes to the theoretical considerations of Lee and Johnson–Laird (2013) [43] and extends the understanding of problem solving in HRE.

Our study revealed that the HRE expert achieved the most time-efficient solution and applied a unique set of problem-solving strategies that could not be found in the problem solving of the intermediates. Through years of deliberate practice, experts build up a wealth of domain-specific knowledge and prior experience in solving domain-specific problems [27]. Experts’ efficient problem solving is based on strong domain-specific knowledge that significantly influences the perception and categorization of problems [16, 45]. This problem categorization and representation then lead to the selection of suitable problem-solving strategies enabling time-efficient solutions [16, 19, 46]. Against this background, it is reasonable to assume that in accordance with the research referenced above, the HRE expert selected strategies based on strong domain-specific knowledge. This profound level of well-structured domain-specific knowledge may have supported the expert in categorizing the problem efficiently and in selecting suitable strategies to solve the tasks efficiently. Ericsson and Kintsch (1995) suggested that experts’ efficient problem solving was based on effective long-term working memory that enabled experts to efficiently activate prior domain-specific knowledge (e.g., stored procedures or chunks) [26]. Thus, we assume that the time-efficient problem solving by the HRE expert may have been based on an efficient activation and retrieval of prior knowledge from the long-term working memory.

Furthermore, our results showed that the two intermediates also achieved time-efficient solutions, although they had less domain-specific knowledge and less problem-solving experience than the HRE expert. We presume that these intermediates’ high performance may have been influenced by cognitive abilities enabling them to compensate for their lack of domain-specific knowledge and experience. In a prior work [6] we showed that higher scores in the IQ sub-factor working memory may have supported reverse engineers to solve the HRE task quicker than analysts with lower scores in working memory. We hypothesize that cognitive abilities such as the working memory may explain how intermediates coped with gaps in HRE-specific knowledge and problem-solving experience. Our assumption aligns with the expert-performance approach that explains how novices and intermediates deal with gaps in domain-specific knowledge and problem-solving experience [27]. The expert-performance approach postulates that domain-specific performance is correlated with general cognitive abilities for novices and intermediates, but not for individuals with higher levels of expertise and specific cognitive structures [25]. Prior research supported the expert-performance approach by showing that superior performance of non-experts in chess was correlated with above-average IQs [8, 30]. We hypothesize that cognitive information processing sub-systems such as the working memory [4] may have helped the two high-performing intermediates in the current study to temporarily keep information in mind and to work on it—for example, by retaining and applying information gathered by inspection of netlist components. It is in particular thinkable that the intermediates’ general speed of information processing—a central basis of human intelligence [20]—may have influenced their performance in the HRE task, allowing them to compensate for a lack of prior domain-specific knowledge and problem-solving experience. During an HRE attack, reverse engineers try to achieve a desired goal state by making sense of thousands up to millions of single netlist components and their interconnections. From this mass of information, they formulate sub-goals and use intermediate results to plan subsequent steps in the attack without losing sight of the desired goal state. Furthermore, attackers have to ignore irrelevant components and draw their attention to relevant components to avoid potential missteps. Based on these results (superior performance by the expert and two intermediates), we suggest that superior performance in HRE may be a function of both expertise and cognitive abilities.

The role of prior knowledge and cognitive abilities is also at the heart of taxonomies that define human problem solving. Prior psychological research has established a broad range of taxonomies to define problems. A widely accepted and applied taxonomy distinguishes between simple and complex problems [22]. Prior research postulates, that solving simple and solving complex problems is based on different cognitive processes [22, 52]. Complex problems are usually described as knowledge-rich systems that activate large semantic networks of prior knowledge and potentially successful problem-solving strategies [22]. Simple problems are typically described as knowledge-free systems as they require less domain-specific knowledge [17] and more cognitive abilities (e.g., [54]) than complex problems. As previously outlined (see Section 2.2), it is an open question to which problem type HRE belongs.

From what we have found, HRE seems to combine aspects of both simple and complex problems. HRE attacks begin with the reverse engineer obtaining a netlist, which can be defined as the initial state of the problem. By selecting suitable operators (e.g., information gathering based on manual netlist analysis), the attacker attempts to achieve a desired goal state (e.g., removal of watermark from a protected circuit). The clear definition of the initial state, the means, and the goal state, are characteristics that are commonly assigned to simple problems [22]. A further characteristic that HRE and simple problems have in common is time-stability—they change only as a result of inputs from the problem solver [31]. In contrast to simple problems, however, a realistic HRE problem may be impossible to solve for someone with insufficient domain-specific knowledge and skills (on hardware circuits, chip design, etc.). Another hallmark of semantic-rich HRE problems is the huge amount of information which must be processed. The attacker analyzes netlists ranging from thousands up to millions of components and their interconnections. The complexity of an HRE problem increases with netlist size (i.e., the number of components), requiring reduction and abstraction on the part of the reverse engineer to remain manageable. These characteristics of complexity and connectivity of HRE problems are typically assigned to complex problems (e.g., [33]. Furthermore, a hardware reverse engineer applies a series of operations to retrieve needed information that are not directly apparent at the beginning of an HRE problem. Thus, an attacker has to handle a lack of transparency (at the beginning of the attack), that is comparable to those found in complex problem-solving settings (e.g., [32]). While these three aspects of complex problems are present in HRE problem solving, two others—dynamics and polytelic situations—are not. As described above and in contrast to complex problems which usually change dynamically, HRE problems are stable. Moreover, a reverse engineer analyzes the netlist components to achieve (sub-)goals that are clearly defined, non-competitive and not mutually exclusive. Our embedding of HRE in existing problem-solving taxonomies leads us to assume that HRE may combine aspects of both simple and complex problems.

In the following, we draw our first conclusion from the previous discussion points. Our findings seem to be in line with prior psychological research with experts who achieved superior performance based on their efficient problem categorization and representation that influences the selection of suitable operators. Moreover, our findings may support the expert-performance approach [27], which explains how non-experts deal with the lack of domain-specific knowledge and experience. Against this background, we assume that both HRE intermediates achieved time-efficient solutions based on their highly-efficient information processing systems. Our attempt to embed HRE in the context of taxonomies of human problem solving resulted in our proposal that HRE seems to combine aspects of both simple and complex problems. Although HRE and simple problems seem to have several core aspects in common, the huge amount of information pertaining to non-transparent, highly interconnected components that must be processed and analyzed creates a challenge that places HRE squarely outside of the realm of simple problems as typically defined and constructed. Therefore, HRE may combine aspects of simple and complex problems. In summary, our exploratory findings lead to the hypothesis that HRE may be a type of problem solving that requires both domain-specific expertise (more relevant for solving complex problems) as well as cognitive abilities (more relevant for solving simple problems).

This hypothesis relates to the theoretical considerations by Lee and Johnson–Laird (2013) [43], who suggest that reverse engineering of Boolean systems may be a specific type of human problem solving. As outlined in Section 2.2, it was unclear to what extent the results of Lee and Johnson–Laird (2013) [43] concerning applied problem-solving strategies and difficulties in reverse engineering are applicable to the domain of HRE problem solving. We hypothesize that the results by Lee and Johnson–Laird are limited in their applicability to the HRE domain. We found differences in applied problem-solving strategies in HRE, which may be due to the nature of the reversing tasks used in the studies. Lee and Johnson–Laird (2013) [43] reported that participants applied one of two main problem-solving strategies: focusing on a single output at a time, or focusing on a single component at a time. In contrast, our results revealed that none of the applied problem-solving strategies could be described as the single “best” or as the main strategy for solving the realistic HRE task. Instead, our results show that our participants applied individual strategies consisting of various combinations of problem-solving steps. Moreover, we also found unique problem-solving strategies (expert, P5, P7). Furthermore, Lee and Johnson–Laird (2013) [43] found that difficulties in reverse engineering depend on three factors: (a) number of components, (b) number of components influencing an output, and (c) dependencies of components influencing an output. In contrast to these more general assumptions, our results revealed specific HRE problems (e.g., confusion based on track losses in reversing approach; dead-ends based on misleading strategies), which in some cases led to difficulties and longer solution times (e.g., P1; P6).

We draw two main conclusions. The first was based on one of our main findings, that superior performance in HRE was achieved by the HRE expert and by two intermediates with above average-scores in the IQ sub-factor working memory [6]. We hypothesize that superior HRE performance may be based on both domain-specific expertise as well as on cognitive abilities. This aligns with our theoretical contribution that HRE tasks may involve aspects of both simple and complex problems. In general, solving simple problem relies more on cognitive abilities [54] and solving complex is based on domain-specific knowledge [22].

In this work, we contribute to the debate initiated by Lee and Johnson–Laird (2013) [43] by extending the understanding of human problem solving in HRE. In this context, we conducted an exploration of HRE-specific problem-solving strategies, suggesting that a single best strategy may not exist and reverse engineers rely more on individual problem-solving procedures. Furthermore, we extended the understanding of when hardware reverse engineers are struggling by describing HRE-specific problems (e.g., misleading strategies and dead-ends) that occurred during problem solving.

In light of these points, we feel that HRE is an interesting area for future research on human problem solving. It would be valuable to conduct controlled experiments in order to analyze influences and interactions of expertise and cognitive abilities in HRE performance as well.

Additionally, a promising investigation would be to analyze difficulties in the HRE problem-solving processes, and whether these could be avoided over time due to the acquisition of skills and domain-specific knowledge. It would also be interesting to analyze how reverse engineers solve other HRE problem tasks, such as reversing an (obfuscated) FSM or extracting a cryptographic key from an unknown netlist. In the light of developing countermeasures impeding HRE, future studies should focus on how reverse engineers analyze protected (obfuscated) netlists and which problem-solving strategies are applied in order to break the obfuscation.

In order to analyze HRE as a specific type of human problem solving, we suggest to include time pressure in future studies to investigate if higher levels of induced stress (based on time pressure) may influence the performance of HRE tasks. We suggest that future studies may conduct interviews with other HRE professionals or more experienced engineers to obtain feedback on the HRE problem-solving model and to probe their mental models of HRE in general. Furthermore, future studies may also investigate whether the model can be confirmed and extended to problem-solving in other HRE tasks.

Here we discuss how the understanding of reverse engineers’ problem-solving processes and strategies may contribute to the field of cybersecurity by supporting the achievement of our overarching goal—the development of novel forms of countermeasures (i.e., cognitive obfuscation) that impede HRE attacks. In general, countermeasures against reverse engineering can never provide absolute protection [5]. Instead, effective countermeasures raise the cost of an attack (i.e., time-on-task) to a prohibitively high level. However, previously proposed countermeasures were based solely upon technical aspects and no holistic measure for comparing the cost of HRE-based attacks yet exists [60].

Therefore, we propose several recommendations in pursuit of cognitive obfuscation. First, the HRE problem-solving model we have developed can serve as a framework with which to examine existing hardware obfuscation techniques across the following relevant dimensions.

Answering these questions would provide a granular view of the cognitive processes involved in defeating existing hardware obfuscation techniques. The most promising obfuscation techniques can then subsequently be used as a toolkit for building cognitive obfuscation. Since future work on cognitive obfuscation might focus on inducing specific Reversing Problems in order to confuse and force the attacker into time-consuming dead-ends, these studies should specifically focus on the different types of Reversing Problems and how they occur. Those studies may also reveal which Reversing Problems are better avoided by engineers who have completed a period of skill acquisition as well as those which are no less easily overcome even after acquiring additional domain-specific knowledge and skills. Second, novel countermeasures should impede efficient and recurring problem-solving strategies, if observed. This means that cognitively challenging tasks should lead reverse engineers down the wrong track, for example by combining physical and logic-level obfuscation techniques which disguise themselves as another known technique. In the event that no effective problem-solving strategies are observed, a combination of obfuscation techniques from the aforementioned toolkit may lead to countermeasures which effectively increase the cost of mounting an HRE attack. Lastly, we hypothesized that expertise alone might not be the only relevant factor in efficiently solving HRE problems. It is thinkable that the success and efficiency of an HRE attack also depend on the engineer’s general cognitive abilities (e.g., on the information processing capabilities). Thus, cognitive obfuscation challenges should try to overwhelm the attacker’s information processing capabilities, for example by designing highly complex challenges with a huge amount of non-transparent and strongly-interconnected information which has to be processed in several parallel tasks.

Our exploratory results on human problem solving in HRE are a first step to understand the underlying cognitive processes of reverse engineers. Nevertheless, future studies need to quantify whether cognitive abilities such as the working memory or the level of expertise significantly influence the problem solving in HRE tasks. Therefore, deriving concrete ideas for cognitive obfuscation at this stage would be too early and would exceed the exploratory nature of this article.

While our work provides a first exploration of HRE problem-solving processes and strategies and highlights several theoretical and technical implications of the same, it is not without limitations.

First, the present study was limited by the methodological challenge that, at this point, HRE experts are generally unavailable for research. To address this challenge, we developed and applied a comprehensive 14-week training, which enabled students to acquire intermediate levels of HRE expertise. We generated an appropriate sample for our study by selecting the top-performing students from the training. One HRE expert further complemented our sample. Comparing the sample of intermediates to the one expert provides some assurance for our findings, but it is not entirely certain to what extent they can be generalized to other HRE experts. This is a problem inherent in this type of research.

Second, we developed our problem-solving model based on a single realistic HRE problem-solving task. Although we expect that our model captures the most significant processes of HRE and might be applicable to other HRE tasks, it is conceivable that further studies examining different problem-solving tasks (e.g., finding a register that stores a cryptographic key in a netlist) would result in its extension or revision.

Third, the measurement of solution time in combination with qualitative analysis of problem-solving steps was a relevant metric for the analysis of efficiency in this study. However, the calculation of time in minutes might be limited in its transferability to industry, where HRE tasks may incur significant overhead due to the need for the engineering of specialized tools and the development of generic, reusable techniques. Here, time would need to be considered in days or weeks rather than hours or seconds.

Fourth, the open code using external resources includes several different actions such as working with pen and paper or consulting online resources. Due to our study design and with the privacy demands of our participants in mind, we collected only data in HAL. Thus, while we could determine that participants consulted external resources that have helped them move forward, we cannot say with certainty which resources they used. For future studies, HAL now includes a feature that allows us to specifically query the resources used in order to analyze how participants employ these resources to solve a task.

Despite the above limitations, our exploratory results and discussion aspects can be considered a first important step towards a better understanding of the unknown human factors in HRE.

Here, we list the final codebook used to annotate the content of each log file. The codebook is divided into nine parts, which reflect the nine principal actions of our HRE problem-solving model (see Figure 2). All 103 open codes are assigned to one corresponding principal action. In the following, we provide a short description of each open code.

Error Introduction.

Troubleshooting.

Test and Validation.

Code Adjustments.

Inspection and Information Gathering.

Reversing Strategy Decisions.

Reversing Milestones and Sub-Goals.

Reversing Problems.

External Influences.