Living in the Matrix with Bytecode Manipulation

QCon NY 2014
Living in the Matrix with
Bytecode Manipulation
Wednesday, June 11, 14

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/asm-cglib-javassist

Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

Ashley Puls
Senior Software Engineer
New Relic, Inc.

Follow Along
http://slidesha.re/1kZwCXr

Outline
• What is bytecode?
• Why manipulate bytecode?
• What frameworks can be used to
manipulate bytecode?
• Examine two frameworks in depth

What is Java bytecode?
.java file .class file
Java compiler
(javac)
Class Loader
Bytecode
Verifier
Java Runtime
System
Native OS
Java Virtual Machine
Source: http://www.techlila.com/write-programs-linux/

.java file .class file
Java compiler
(javac)
Class Loader
Bytecode
Verifier
Java Runtime
System
Native OS
Java Virtual Machine
Source: http://www.techlila.com/write-programs-linux/
Instruction set of the Java Virtual Machine

Why learn about Java bytecode
manipulation frameworks ?

• A bytecode manipulation framework is likely already
on your stack
• Spring
• Hibernate
• Groovy
• Clojure
• Eclipse
• JRuby

• A bytecode manipulation framework is likely already
on your stack
• Spring
• Hibernate
• Groovy
• Clojure
• Eclipse
• JRuby
• Its really fun!!!

• program analysis
• ﬁnd bugs in code
• examine code complexity
• generate classes
• proxies
• remove access to certain APIs
• compiler for another language like Scala
• transform classes without Java source code
• proﬁlers
• optimization and obfuscation
• additional logging

Logging
Source: http://www.vforteachers.com/About_NetSupport.htm

Logging
Source: http://www.vforteachers.com/About_NetSupport.htm, http://inzolo.com/blog/tutorials/bank-accounts

Logging
Source: http://www.vforteachers.com/About_NetSupport.htm, http://upload.wikimedia.org/wikipedia/commons/d/d3/49024-SOS-ATM.JPG

Logging
Audit Log: a message should be
logged every time an important
method is called

Logging
log important input parameters
to the method
Audit Log: a message should be
logged every time an important
method is called

Logging
public class BankTransactions {
public static void main(String[] args) {
BankTransactions bank = new BankTransactions();
// login and add i dollars to account
for (int i = 0; i < 100; i++) {
String accountId = "account" + i;
bank.login("password", accountId, "Ashley");
bank.unimportantProcessing(accountId);
bank.finalizeTransaction(accountId, Double.valueOf(i));
}
System.out.println(“Transactions completed”);
}
}

Logging
public class BankTransactions {
public static void main(String[] args) {
BankTransactions bank = new BankTransactions();
// login and add i dollars to each account
for (int i = 0; i < 100; i++) {
String accountId = "account" + i;
bank.login("password", accountId, "Ashley");
bank.unimportantProcessing(accountId);
bank.finalizeTransaction(accountId, Double.valueOf(i));
}
System.out.println(“Transactions completed”);
}
}

Logging
/**
* A method annotation which should be used to indicate important methods whose
* invocations should be logged.
*
*/
public @interface ImportantLog {
/**
* The method parameter indexes whose values should be logged. For example,
* if we have the method hello(int paramA, int paramB, int paramC), and we
* wanted to log the values of paramA and paramC, then fields would be ["0",
* "2"]. If we only want to log the value of paramB, then fields would be
* ["1"].
*/
String[] fields();
}

Logging
public void login(String password, String accountId, String userName) {
// login logic
}
public void finalizeTransaction(String accountId, Double moneyToAdd) {
// transaction logic
}

Logging
@ImportantLog(fields = { "1", "2" })
// login logic
}
}

Logging
// login logic
}
}
A call was made to method "login" on class "com/example/qcon/mains/BankTransactions".
Important params:
Index 1 value: ${accountId}
Index 2 value: ${userName}

Logging
// login logic
}
}
A call was made to method "finalizeTransaction" on class "com/example/qcon/
mains/BankTransactions".
Important params:
Index 1 value: ${moneyToAdd}
A call was made to method "login" on class "com/example/qcon/mains/BankTransactions".
Important params:
Index 2 value: ${userName}

Java Agent
• Ability to modify the bytecode without modifying the
application source code
• Added in Java 1.5
• Docs: http://docs.oracle.com/javase/8/docs/api/
java/lang/instrument/package-summary.html

Typical Java Process
JVM
public static void main(String[] args)
BankTransactions.class
Classloader
java com/example/qcon/mains/BankTransactions
Transactions completed

Java Agent
JVMAgent
java -javaagent:/path/to/agent.jar com/example/qcon/mains/
BankTransactions

Java Agent
JVM
BankTransactions

Java Agent
JVM
void premain(String agentArgs, Instrumentation inst)
Agent.class
MyTransformer.class
byte[] transform( . . . , byte[] bankTransBytes)
Agent
1. call Agent premain in manifest
2. JVM registers my transformer
BankTransactions

Java Agent
JVM
Agent.class
MyTransformer.class
Agent
3. Give BankTransactions bytes to
MyTransformer
4. MyTransformer provides
bytes to load
BankTransactions

Java Agent
JVM
void main(String[] args)
BankTransactions.class
Agent.class
MyTransformer.class
Agent
3. Give BankTransactions bytes to
MyTransformer
4. MyTransformer provides
bytes to load
5. BankTransactions loaded and main
runs
BankTransactions
Transactions completed

lets code
Manifest:
Premain-Class: com.example.qcon.agent.Agent

lets code
package com.example.qcon.agent;
public class Agent {
public static void premain(String agentArgs, Instrumentation inst) {
System.out.println("Starting the agent");
inst.addTransformer(new ImportantLogClassTransformer());
}
}
Manifest:
Premain-Class: com.example.qcon.agent.Agent

lets code
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
public class ImportantLogClassTransformer implements
ClassFileTransformer {
public byte[] transform(ClassLoader loader, String className,
Class classBeingRedefined, ProtectionDomain protectionDomain,
byte[] classfileBuffer) throws IllegalClassFormatException {
//TODO
return null;
}
}

lets code
System.out.println(" Loading class: " + className);
return null;
}
}

lets code
Starting the agent
Loading class: java/lang/invoke/MethodHandleImpl
Loading class: java/lang/invoke/MemberName$Factory
Loading class: java/lang/invoke/LambdaForm$NamedFunction
Loading class: java/lang/invoke/MethodType$ConcurrentWeakInternSet
Loading class: java/lang/invoke/MethodHandleStatics
Loading class: java/lang/invoke/MethodHandleStatics$1
Loading class: java/lang/invoke/MethodTypeForm
Loading class: java/lang/invoke/Invokers
Loading class: java/lang/invoke/MethodType$ConcurrentWeakInternSet$WeakEntry
Loading class: java/lang/Void
Loading class: java/lang/IllegalAccessException
Loading class: sun/misc/PostVMInitHook
Loading class: sun/launcher/LauncherHelper
Loading class: java/util/concurrent/ConcurrentHashMap$ForwardingNode
Loading class: sun/misc/URLClassPath$FileLoader$1
Loading class: com/example/qcon/mains/BankTransactions
Loading class: sun/launcher/LauncherHelper$FXHelper

lets code
// manipulate the bytes here and then return them
return null;
}
}

Bytecode Manipulation Frameworks
• ASM: http://asm.ow2.org/
• BCEL: http://commons.apache.org/proper/commons-bcel/
• CGLib: https://github.com/cglib/cglib
• Javassist: http://www.csg.ci.i.u-tokyo.ac.jp/~chiba/javassist/
• Serp: http://serp.sourceforge.net/
• Cojen: https://github.com/cojen/Cojen/wiki
• Soot: http://www.sable.mcgill.ca/soot/

Javassist
• Java Programming Assistant
• Subproject of JBoss
• Actively updated
• Version 3.18.2 released May 2014
• Source code: https://github.com/jboss-javassist/
javassist
• Documentation: http://www.csg.ci.i.u-tokyo.ac.jp/
~chiba/javassist/tutorial/tutorial.html

Javassist
ArrayList
HashMap
CtClass: represents a compile time class

Javassist
ClassPool: container of CtClass objects
ArrayList
HashMap

Javassist
ArrayList
HashMap
ArrayList
HashMap

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap
ByteArrayClassPath: converts a byte[] to a CtClass

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap
Classpool.getDefault().insertClassPath(new ByteArrayClassPath(“BankTrans”, classBytes);

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap
BankTrans

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap
BankTrans
CtClass bank = Classpool.getDefault().get(“BankTrans”);

Javassist
ArrayList
HashMap
BankTrans
ArrayList
HashMap
BankTrans
BankTrans
CtClass bank = Classpool.getDefault().get(“BankTrans”);

Javassist
BankTrans

Javassist
CtConstructor: represents a compile time constructor
init
BankTrans

Javassist
init
BankTrans
CtMethod: represents a compile time class
login
process

Javassist
init
CtMethod: represents a compile time method
CtMethod loginMethod = loginCtClass.getDeclaredMethod(“login”);
BankTrans
login
process

Javassist
init
CtMethod loginMethod = loginCtClass.getDeclaredMethod(“login”);
login
BankTrans
login
process

Javassist
login

Javassist
Modiﬁcations:
login

Javassist
Modiﬁcations:
login
putMethod.insertAfter(“System.out.println(”after method”);”);
putMethod.insertBefore(“System.out.println(“before method”);”);

Javassist
Modiﬁcations:
System.out.println(“before method”);
putMethod.insertAfter(“System.out.println(”after method”);”);
putMethod.insertBefore(“System.out.println(“before method”);”);
System.out.println(“after method”);
login

Javassist
Modiﬁcations:
CtClass exceptionType = classpool.get(“java.io.IOException”);
putMethod.addCatch(“{System.out.println($e); throw $e}”, exceptionType);
login

Javassist
Modiﬁcations:
CtClass exceptionType = classpool.get(“java.io.IOException”);
System.out.println(“exception”);
throw exception;
putMethod.addCatch(“{System.out.println($e); throw $e}”, exceptionType);
login

Javassist
Modiﬁcations:
putMethod.insertAt(5, true, “System.out.println(“hello”);”);
login

Javassist
Modiﬁcations:
putMethod.insertAt(5, true, “System.out.println(“hello”);”);
System.out.println(“hello”);
login

Javassist
Modiﬁcations:
insertBefore insertAfter insertAt addCatch
throw exception;
login

Javassist
Modiﬁcations:
Freeze Class:
throw exception;
login

Javassist
Modiﬁcations:
Freeze Class:
throw exception;
writeFile() toClass() toBytecode()
login

lets code

lets code
public class ImportantLogClassTransformer implements ClassFileTransformer {
//TODO
return null;
}
}

lets code
byte[] cfbuffer) throws IllegalClassFormatException {
1. convert byte array to a ct class object
2. check each method of ct class for annotation @ImportantLog
3. if @ImportantLog annotation present on method, then
a. get important method parameter indexes
b. add logging statement to beginning of the method
return null;
}

lets code
pool.insertClassPath(new ByteArrayClassPath(className,
classfileBuffer));
CtClass cclass = pool.get(className.replaceAll("/", "."));
2. check each method of ct class for annotation @ImportantLog

return null;
}

lets code
classfileBuffer));
if (!cclass.isFrozen()) {
for (CtMethod currentMethod : cclass.getDeclaredMethods()) {
Annotation annotation = getAnnotation(currentMethod);
}
}
return null;
}

lets code
classfileBuffer));
if (annotation != null) {
List<String> parameterIndexes = getParamIndexes(annotation);
}
}
}
return null;
}

lets code
classfileBuffer));
if (annotation != null) {
List<String> parameterIndexes = getParamIndexes(annotation);
currentMethod.insertBefore(createJavaString(
currentMethod, className, parameterIndexes));
}
}
return cclass.toBytecode();
}
return null;
}

lets code
private Annotation getAnnotation(CtMethod method) {
MethodInfo mInfo = method.getMethodInfo();
// the attribute we are looking for is a runtime invisible attribute
// use Retention(RetentionPolicy.RUNTIME) on the annotation to make it
// visible at runtime
AnnotationsAttribute attInfo = (AnnotationsAttribute) mInfo
.getAttribute(AnnotationsAttribute.invisibleTag);
if (attInfo != null) {
// this is the type name meaning use dots instead of slashes
return attInfo.getAnnotation("com.example.qcon.mains.ImportantLog");
}
return null;
}

lets code
private List<String> getParamIndexes(Annotation annotation) {
ArrayMemberValue fields = (ArrayMemberValue) annotation
.getMemberValue(“fields”);
if (fields != null) {
MemberValue[] values = (MemberValue[]) fields.getValue();
List<String> parameterIndexes = new ArrayList<String>();
for (MemberValue val : values) {
parameterIndexes.add(((StringMemberValue) val).getValue());
}
return parameterIndexes;
}
return Collections.emptyList();
}

lets code
private String createJavaString(CtMethod currentMethod, String className,
List<String> indexParameters) {
StringBuilder sb = new StringBuilder();
sb.append("{StringBuilder sb = new StringBuilder");
sb.append("("A call was made to method '");");
sb.append("sb.append("");
sb.append(currentMethod.getName());
sb.append("");sb.append("' on class '");");
sb.append(className);
sb.append("");sb.append("'.");");
sb.append("sb.append("n Important params:");");
. . .

lets code
sb.append("{StringBuilder sb = new StringBuilder");
sb.append("("A call was made to method '");");
sb.append(currentMethod.getName());
sb.append("");sb.append("' on class '");");
sb.append("");sb.append("'.");");
sb.append("sb.append("n Important params:");");
. . .
Put multiple statements inside brackets {}

lets code
. . .
for (String index : indexParameters) {
try {
int localVar = Integer.parseInt(index) + 1;
sb.append("sb.append("n Index: ");");
sb.append(index);
sb.append("");sb.append(" value: ");");
sb.append("sb.append($" + localVar + ");");
} catch (NumberFormatException e) {
e.printStackTrace();
}
}
sb.append("System.out.println(sb.toString());}");
return sb.toString();
}

lets code
. . .
for (String index : indexParameters) {
try {
int localVar = Integer.parseInt(index) + 1;
sb.append("sb.append("n Index: ");");
sb.append(index);
sb.append("");sb.append(" value: ");");
sb.append("sb.append($" + localVar + ");");
} catch (NumberFormatException e) {
e.printStackTrace();
}
}
sb.append("System.out.println(sb.toString());}");
return sb.toString();
}
$0, $1, $2, ... can be used to access
“this” and the actual method parameters

Javassist
• Positives
• Simplicity
• Do not have to write actual bytecode
• Decent documentation
• Negatives
• Generally slower than ASM
• Less functionality

ASM
• Released in Open Source in 2002
• Actively updated
• Version 5.0.3 released May 2014
• Two ASM libraries
• Event based (SAX like)
• Visitor design pattern
• Object based (DOM like)
• Documentation: http://download.forge.objectweb.org/
asm/asm4-guide.pdf

ASM
Source: http://download.forge.objectweb.org/asm/asm4-guide.pdf

ASM
ClassReader: given a byte[], parses a compiled class
ClassReader:
event producer
BankTrans
ClassWriter:
event consumer
ClassVisitor:
event ﬁlter

ASM
ClassReader:
event producer
BankTrans
ClassWriter:
event consumer
visitField
visitMethod
ClassVisitor:
event ﬁlter

ASM
ClassVisitor: delegates class events, event ﬁlter
visitAttribute visitField visitMethod visitInnerClass visitEnd
ClassReader:
event producer
BankTrans
ClassWriter:
event consumer
visitField
visitMethod
ClassVisitor:
event ﬁlter

ASM
ClassWriter: produces output byte[]
ClassReader:
event producer
BankTrans
ClassWriter:
event consumer
visitField
visitMethod
BankTrans
ClassVisitor:
event ﬁlter

lets code
import java.lang.instrument.Instrumentation;
public class Agent {
public static void premain(String agentArgs, Instrumentation inst) {
System.out.println("Starting the agent");
inst.addTransformer(new ImportantLogClassTransformer());
}
}

lets code
Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
ClassReader cr = new ClassReader(classfileBuffer);
ClassWriter cw = new ClassWriter(cr, ClassWriter.COMPUTE_FRAMES);
cr.accept(cw, 0);
return cw.toByteArray();
}

lets code
Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
ClassReader cr = new ClassReader(classfileBuffer);
ClassWriter cw = new ClassWriter(cr, ClassWriter.COMPUTE_FRAMES);
ClassVisitor cv = new LogMethodClassVisitor(cw, className);
cr.accept(cv, 0);
return cw.toByteArray();
}

ASM
ClassWriter: produces output byte[]
ClassReader:
event producer
BankTrans
ClassWriter:
event consumer
visitMethod
BankTrans
LogMethod
ClassVisitor:
event ﬁlter

lets code
public class LogMethodClassVisitor extends ClassVisitor {
private String className;
public LogMethodIfAnnotationVisitor(ClassVisitor cv, String pClassName) {
super(Opcodes.ASM5, cv);
className = pClassName;
}
@Override
public MethodVisitor visitMethod(int access, String name, String desc,
String signature, String[] exceptions) {
// put our logic in here
}
}

ASM
MethodVisitor: method event ﬁlter
MethodVisitor MethodVisitor
visitCode
login method
visitor
login method
visitor
visitAnnotation visitEnd visitCode visitTryCatchBlock
visitEnd
visitAttribute

lets code
public class LogMethodClassVisitor extends ClassVisitor {
public LogMethodIfAnnotationVisitor(ClassVisitor cv, String pClassName) {
super(Opcodes.ASM5, cv);
className = pClassName;
}
@Override
public MethodVisitor visitMethod(int access, String name, String desc,
String signature, String[] exceptions) {
MethodVisitor mv = super.visitMethod(access, name, desc, signature,
exceptions);
return new PrintMessageMethodVisitor(mv, name, className);
}
}

lets code
public class PrintMessageMethodVisitor extends MethodVisitor {
private String methodName;
private boolean isAnnotationPresent;
private List<String> parameterIndexes;
public PrintMessageMethodVisitor(MethodVisitor mv, String methodName,
String className) {
}
@Override
public AnnotationVisitor visitAnnotation(String desc, boolean visible) {

}
@Override
public void visitCode() {
}
}

lets code
String className) {
// initialize instance variables
}
@Override
1. check method for annotation @ImportantLog
2. if annotation present, then get important method param indexes
}
@Override
3. if annotation present, add logging to beginning of the method
}
}

lets code

if ("Lcom/example/qcon/mains/ImportantLog;".equals(desc)) {
isAnnotationPresent = true;
return new AnnotationVisitor(Opcodes.ASM5,
super.visitAnnotation(desc, visible)) {
public AnnotationVisitor visitArray(String name) {
if (“fields”.equals(name)) {
return new AnnotationVisitor(Opcodes.ASM5,
super.visitArray(name)) {
public void visit(String name, Object value) {
parameterIndexes.add((String) value);
super.visit(name, value);
}
};
} else {
return super.visitArray(name);
}
}
};
}
return super.visitAnnotation(desc, visible);
}

lets code
String className) {
// initialize instance variables
}
@Override
1. check method for annotation @ImportantLog
2. if annotation present, then get important method param indexes
}
@Override
3. if annotation present, add logging statement to beginning of the method
}
}

lets code
if (isAnnotationPresent) {
// create string builder
mv.visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out",
"Ljava/io/PrintStream;");
mv.visitTypeInsn(Opcodes.NEW, "java/lang/StringBuilder");
mv.visitInsn(Opcodes.DUP);
// add everything to the string builder
mv.visitLdcInsn("A call was made to method "");
mv.visitMethodInsn(Opcodes.INVOKESPECIAL,
"java/lang/StringBuilder", "<init>",
"(Ljava/lang/String;)V", false);
mv.visitLdcInsn(methodName);
mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL,
"java/lang/StringBuilder", "append",
"(Ljava/lang/String;)Ljava/lang/StringBuilder;", false);
. . .

use javap to examine the bytecode

public class PrintMessage {
public PrintMessage(String mName, String cName) {
methodName = mName;
className = cName;
}
public void print() {
sb.append("A call was made to method "");
sb.append(methodName);
sb.append("" on class "");
sb.append("".");
System.out.println(sb.toString());
}

javap -c bin/com/example/qcon/mains/PrintMessage

javap -c bin/com/example/qcon/mains/PrintMessage
public void print();
Code:
0: new #38 // class java/lang/StringBuilder
3: dup
4: invokespecial #40 // Method java/lang/
StringBuilder."<init>":()V
7: astore_1
8: aload_1
9: ldc #41 // String A call was made to method "
11: invokevirtual #43 // Method java/lang/
StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
14: pop
15: aload_1
16: aload_0
17: getfield #14 // Field methodName:Ljava/lang/String;
. . .

each line represents one JVM instruction
Code:
3: dup
7: astore_1
8: aload_1
14: pop
15: aload_1
16: aload_0
. . .

JVM instruction = 1 opcode with 0 or more operands
Code:
3: dup
7: astore_1
8: aload_1
14: pop
15: aload_1
16: aload_0
. . .

Opcode Description
load
load a reference onto the stack from a local variable
iload, lload, ﬂoad, dload, aload
store
stores a value from the stack into a local variable
istore, lstore, fstore, dstore, astore
lcd push a constant from a constant pool onto the stack
return
method return instruction
ireturn, lreturn, freturn, dreturn, areturn, return
invokevirtual invoke an instance method on an object
invokespecial
invokes an instance method requiring special handling
such as an initialization method
See the spec for more info: http://docs.oracle.com/javase/specs/jvms/se8/jvms8.pdf

Opcode Description
invokestatic invoke a class (static) method on a named class
athrow throws an exception
new create a new class
newarray create a new array
if<cond>
branch if int comparison with 0 succeeds
ifeq, ifne, iﬂt, ifge, ifgt, iﬂe
dup duplicate the top operand stack value
See the spec for more info: http://docs.oracle.com/javase/specs/jvms/se8/jvms8.pdf

ASM
• Positives
• Speed and Size
• Decent documentation
• Depth of Functionality
• Number of people using framework
• Negatives
• Need to write actual byte code
• longer developer ramp-up

Source: http://www.kulfoto.com/dog-pictures/15799/mans-best-friend
Make bytecode manipulation frameworks your friend.
They might save your job one day.

Source: http://alangregerman.typepad.com/.a/6a00d83516c0ad53ef017c3329aba8970b-800wi
Questions?

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/asm-cglib-
javassist

Living in the Matrix with Bytecode Manipulation

More Related Content

Living in the Matrix with Bytecode Manipulation