Decompilation

This report gives an overview of the work performed by the Programming Research Group as part of the European collaborative ESPRIT II REDO project (no. 2487). This work covered the areas of reverse-engineering: redocumentation and re-engineering; validation: post-hoc verification and generation of correct code from specifications; maintenance: new languages and methods to support maintenance. Research in areas of concurrent programming and decompilation were also performed.

Decompiling is often used in conjunction with recovering lost source code, or in reverse-engineering code when we do not have access to the source. Here we describe a novel use: places where accurate position reporting, even in the presence of optimized code or where source code is not available, could be helpful. Examples include tracebacks, debuggers, core dumps and so on. Also new is using decompilation to assist debugging at run-time. We show how more conventional methods of source-code reporting are vague and ambiguous. Although eecting a pervasive change in a compiler is arduous and error-prone, a decompiler can be developed somewhat independently. However, for the vast number of programming languages, decompilers do not exist. This is true even for dynamic interpreted languages where there is little going on in the way of " compilation. " Traditionally, decompilers are labor intensive and ad hoc, and their construction might also be unfamiliar. So this paper shows how to ameliorate these problems by describing a pipeline for writing decompilers for dynamic languages that we hope will become standard in the same way that the pipeline from the Dragon Book [9] has. Our pipeline diers somewhat from the standard compiler pipeline and from earlier decompiler pipelines [13, 16]. The dierences may lead to further avenues of research and practice. We use a grammar-directed parsing of instructions with an ambiguous grammar to create an AST that, at the top-levels, resembles that of the source-code AST. This is helpful in error reporting. An ambiguous grammar can give an exponential number of derivations on certain likely input sequences, and we describe techniques to deal with this issue. Finally, we describe issues around developing and maintaining a Python bytecode decompiler, which handles the Python language spanning 15 releases over some 20 years. We hope to encourage the incorporation of decompila-tion into tracebacks, debuggers, core dumps, and so on. Reporting a line number as a position in a program can be vague. Consider the following Python code: x = prev[prev[p]] ... If we get the following Python error at runtime: IndexError: list index out of range a line number, and possibly method name or path, will be reported. But which index caused the problem? Or consider possible runtime errors in these lines: x = a / b / c # which divide? # which index? [x[0] for i in d[j] if got[i] == e[i]] return fib(x) + fib(y) # which fib? # code created at runtime exec(a=%s; b=%s % (y, z)) As seen in the last example, there are problems with functions like eval() which evaluate either a string containing a program-language fragment, or which evaluate some intermediate representation of a program such as an Abstract Syntax Tree or a code object. In these cases, the program is created at runtime, either from strings or from method calls which create AST objects and, either way, there is no le containing source code as a stream of characters. The Pos type in Go's AST is more informative in that it encodes the line number, le name and column number all very compactly as an opaque index [11]. Even better, though, would be to extend this to an interval or a beginning and ending position. Go's implementation cleverly takes advantage of a sparseness property: the number of interesting locations in a program, whether a single point or a pair of points, is much smaller than the program's total number of bytes. Therefore it is practical for single integer to index into a small table whose entries have a lot of information. Long experience has shown that trying to get compilers and interpreters to track location correspondences through the compilation process and then save them in the runtime system is an uphill battle. The Go implementers resist extending a Pos type to cover interval positions because of their fear that the line/column approach already slows down compilation.

Compiler compilers are in widespread use, but decompiler compilers are a more novel concept. This paper presents an approach for the decompilation of object code back to source code using a decompiler generator. An example decompilation is presented. Potential applications include reverse engineering, quality assessment, debugging and safety-critical code validation or verification.

"The Dictionary of Computing gives the following (abridged) definitions for a compiler and decompiler (or reverse compiler):

compiler A program that translates high-level language into absolute code . . .

decompiler A program that attempts to . . . translate back from machine code to something resembling the source language. The task is difficult and not often attempted.

The following sections are intended to show how a decompiler may be constructed given only a knowledge of or access to the specification for a forward compiler, and some facility with Prolog or other high-level languages. It is easy to do and
should be attempted more often.

Decompilation is only useful, of course, if the object code is available and it is
required to reproduce the higher level `original' as an intermediate stage to aid understanding
or restructuring. The decompiled code can then be abstracted to a specification
by other techniques developed by the REDO project,"

Compiler compilers are in widespread use, but decompiler compilers are a more novel concept. This paper sets out a technique for the decompilation of object code back to source code, and describes a working decompiler generator. Existing and potential applications include reverse engineering, quality assessment, debugging and safety-critical code validation or verification.

Compiler compilers are in widespread use, but decompiler compilers are a more novel concept. This paper presents an approach for the decompilation of object code back to source code using a decompiler generator. An example decompilation is presented. Potential applications include reverse engineering, quality assessment, debugging and safety-critical code validation or verification.

The Dictionary of Computing gives the following (abridged) definitions for a compiler and decompiler (or reverse compiler):

      compiler:  A program that translates high-level language into absolute  code . . .

      decompiler: A program that attempts to . . . translate back from machine code to something resembling the source language. The task is difficult and  not often attempted.

The following sections are intended to show how a decompiler may be constructed given only a knowledge of or access to the specification for a forward compiler, and some facility with Prolog or other high-level languages. It is easy to do and should be attempted more often.

A compiler may be specied by a description of how each construct of the source language is translated into a sequence of object code instructions. It is possible to produce a compiler prototype almost directly from this specification in the form of a logic program. This defines a relation between allowed high-level and low-level program constructs. Normally a high-level program is supplied as input to a compiler and object code is returned. Because of the declarative nature of a logic program,
it is possible for the object code to be supplied and the allowed high-level programs returned, resulting in a decompiler, provided enough information is available in the object code. This paper discusses the problems of adopting such an approach in
practice. A simple compiler and decompiler are presented in full as an example in the logic programming language Prolog, together with some sample output. The possible benefits of using constraint logic programming are also considered. Potential applications include reverse engineering in the software maintenance process, verification of safety-critical object code, quality assessment of code and program debugging tools.

Keywords: Reverse Engineering, Decompilation, Logic Programming, Prolog, Compiler Prototyping, Constraint Logic Programming

While a compiler produces low-level object code from high-level source code, a decompiler produces high-level code from low-level code and has applications in the testing and validation of safety-critical software. The decompilation of an object code provides an independent demonstration of correctness that is hard to better for industrial purposes (an alternative is to prove the compiler correct). But, although compiler compilers are in common use in the software industry, a decompiler compiler is much more unusual.It turns out that a data type specification for a programming-language grammar can be remolded into a functional program that enumerates all of the abstract syntax trees of the grammar. This observation is the springboard for a general method for compiling decompilers from the specifications of (non-optimizing) compilers.This paper deals with methods and theory, together with an application of the technique. The correctness of a decompiler generated from a simple occam-like compiler specification is demonstrated. The basic problem of enumerating the syntax trees of grammars, and then stopping, is shown to have no recursive solution, but methods of abstract interpretation can be used to guarantee the adequacy and completeness of our technique in practical instances, including the decompiler for the language presented here.

Decompiling is often used in conjunction with recovering lost source code, or in reverse-engineering code when we do not have access to the source. Here we describe a novel use: places where accurate position reporting, even in the presence of optimized code or where source code is not available, could be helpful. Examples include tracebacks, debuggers, core dumps and so on. Also new is using decompilation to assist debugging at runtime. We show how more conventional methods of source-code reporting are vague and ambiguous. Although effecting a pervasive change in a compiler is arduous and error-prone, a decompiler can be developed somewhat independently. However, for the vast number of programming languages, decompilers do not exist. This is true even for dynamic interpreted languages where there is little going on in the way of “compilation.” Traditionally, decompilers are labor intensive and ad hoc, and their construction might also be unfamiliar. So this paper shows how to ameli...

Since the advent of modern programming language compilers whereby a set of human readable instructions are syntactically and semantically parsed and then translated and optimized to a binary format readable by a machine or an interpreter, there has been a need for the reversal of the process which is generally known as decompilation. Yet wide gaps of knowledge have remained in decompilation given that it is can be modeled as an identical process to that performed by a compiler except the input and output take on a different appearance. The Von Neumann architecture which modern computers are still based on, requires that the code and data remain in memory and operate on that which is contained within that memory as well yielding the possibility for code to modify itself which is merely a form of compression or obfuscation of the original code. By analyzing self-modifying code, and its implications for declarative programming languages as well as temporally, a model for decompilation can be described which will generalize and completely match the problem description handling the most complicated and generalized situations which are possible.

Decompiling is often used in conjunction with recovering lost source code, or reverse engineering code when you don't have access to the source code. Here I describe another use: in tracebacks, debuggers, core dumps and similar places where accurate position-reporting is desired. I show how more conventional methods of source-code reporting are vague and ambiguous. Decompilation is the most accurate way to describe where the program currently is, in high-level terms familiar to the programmer. It is not in widespread use because it is unfamiliar and labor-intensive. This paper shows how to ameliorate these problems by giving a pattern for writing decompilers. I hope to encourage the incorporation of decompilation into tracebacks, debuggers, core dumps, and so on. I also show how decompilation advances the art of debug-ging optimized code. Although effecting a pervasive change in a compiler is arduous and error-prone, the decompiler can be developed somewhat independently. Next, I describe a pipeline for a Python decompiler, and show similarities with a conventional programming-language compiler. Finally, I describe issues around developing and maintaining a Python bytecode decompiler, which handles input spanning 15 releases of the language over some 20 years.