(Do not be afraid of)

PHP Compiler Internals

Sebastian Bergmann
June 13th 2009
Who I Am

 Sebastian Bergmann
 Involved in the PHP
project since 2000
 Creator of PHPUnit
 Co-Founder and
Principal Consultant
with thePHP.cc
Under PHP's Hood

Extensions

(date, dom, gd, json, mysql, pcre, pdo, reflection, session, standard, …)

PHP Core Zend Engine

Request Management Compilation and Execution

File and Network Operations Memory and Resource Allocation

Server API (SAPI)

(mod_php, FastCGI, CLI, ...)

This slide contains material by Sara Golemon

How PHP executes code
 Lexical Analysis
Converts the source from a sequence of characters into a
sequence of tokens
How PHP executes code
 Lexical Analysis
 Syntax Analysis
Analyzes a sequence of tokens to determine their grammatical
structure
How PHP executes code
 Lexical Analysis
 Syntax Analysis
 Bytecode Generation
Generate bytecode based on the information gathered by
analyzing the sourcecode
How PHP executes code
 Lexical Analysis
 Syntax Analysis
 Bytecode Generation
 Bytecode Execution
Lexical Analysis
Scan a sequence of characters
1 <?php
2 if (TRUE) {
3 print '*';
4 }
5 ?>
Lexical Analysis
Scan a sequence of characters
1 <?php T_OPEN_TAG
2 if (TRUE) {
3 print '*';
4 }
5 ?>
Lexical Analysis
Scan a sequence of characters
1 <?php T_OPEN_TAG
2 if (TRUE) { T_IF
T_WHITESPACE
(
T_STRING
)
T_WHITESPACE
{
T_WHITESPACE
3 print '*';
4 }
5 ?>
Lexical Analysis
Scan a sequence of characters
1 <?php T_OPEN_TAG
2 if (TRUE) { T_IF
T_WHITESPACE
(
T_STRING
)
T_WHITESPACE
{
T_WHITESPACE
3 print '*'; T_PRINT
T_WHITESPACE
T_CONSTANT_ENCAPSED_STRING
;
4 }
5 ?>
Lexical Analysis
Scan a sequence of characters
1 <?php T_OPEN_TAG
2 if (TRUE) { T_IF
T_WHITESPACE
(
T_STRING
)
T_WHITESPACE
{
T_WHITESPACE
3 print '*'; T_PRINT
T_WHITESPACE
T_CONSTANT_ENCAPSED_STRING
;
T_WHITESPACE
4 } }
5 ?>
Lexical Analysis
Scan a sequence of characters
1 <?php T_OPEN_TAG
2 if (TRUE) { T_IF
T_WHITESPACE
(
T_STRING
)
T_WHITESPACE
{
T_WHITESPACE
3 print '*'; T_PRINT
T_WHITESPACE
T_CONSTANT_ENCAPSED_STRING
;
T_WHITESPACE
4 } }
T_WHITESPACE
5 ?> T_CLOSE_TAG
Lexical Analysis
Scan a sequence of characters
T_OPEN_TAG <?php
T_IF if
T_WHITESPACE
(
T_STRING TRUE
)
T_WHITESPACE
{
T_WHITESPACE
T_PRINT print
T_WHITESPACE
T_CONSTANT_ENCAPSED_STRING '*'
;
T_WHITESPACE
}
T_WHITESPACE
T_CLOSE_TAG ?>
Lexical Analysis
Scan a sequence of characters
Lexical Analysis
Scanner Generators

 You do not want to write a scanner by

hand
 At least when the code for the scanner should
be efficient and maintainable
 Tools such as flex or re2c generate the
code for a scanner from a set of rules

<ST_IN_SCRIPTING>"if"
"if" { {
return T_IF;
}
Lexical Analysis
PHP Tokens

 T_ABSTRACT  T_CONCAT_EQUAL  T_ELSE  T_FUNCTION

 T_AND_EQUAL  T_CONST  T_ELSEIF  T_FUNC_C

 T_ARRAY  T_CONSTANT_ENCAPSED_STRING  T_EMPTY  T_GLOBAL

 T_ARRAY_CAST  T_CONTINUE  T_ENCAPSED_AND_WHITESPACE  T_GOTO

 T_AS  T_CURLY_OPEN  T_ENDDECLARE  T_HALT_COMPILER

 T_BAD_CHARACTER  T_DEC  T_ENDFOR  T_IF

 T_BOOLEAN_AND  T_DECLARE  T_ENDFOREACH  T_IMPLEMENTS

 T_BOOLEAN_OR  T_DEFAULT  T_ENDIF  T_INC

 T_BOOL_CAST  T_DIR  T_ENDSWITCH  T_INCLUDE

 T_BREAK  T_DIV_EQUAL  T_ENDWHILE  T_INCLUDE_ONCE

 T_CASE  T_DNUMBER  T_END_HEREDOC  T_INLINE_HTML

 T_CATCH  T_DOC_COMMENT  T_EVAL  T_INSTANCEOF

 T_CHARACTER  T_DO  T_EXIT  T_INT_CAST

 T_CLASS  T_DOLLAR_OPEN_CURLY_BRACES  T_EXTENDS  T_INTERFACE

 T_CLASS_C  T_DOUBLE_ARROW  T_FILE  T_ISSET

 T_CLONE  T_DOUBLE_CAST  T_FINAL  T_IS_EQUAL

 T_CLOSE_TAG  T_DOUBLE_COLON  T_FOR  T_IS_GREATER_OR_EQUAL

 T_COMMENT  T_ECHO  T_FOREACH  T_IS_IDENTICAL

Lexical Analysis
PHP Tokens

 T_IS_NOT_EQUAL  T_OBJECT_CAST  T_SR_EQUAL

 T_IS_NOT_IDENTICAL  T_OBJECT_OPERATOR  T_START_HEREDOC

 T_IS_SMALLER_OR_EQUAL  T_OLD_FUNCTION  T_STATIC

 T_LINE  T_OPEN_TAG  T_STRING

 T_LIST  T_OPEN_TAG_WITH_ECHO  T_STRING_CAST

 T_LNUMBER  T_OR_EQUAL  T_STRING_VARNAME

 T_LOGICAL_AND  T_PAAMAYIM_NEKUDOTAYIM  T_SWITCH

 T_LOGICAL_OR  T_PLUS_EQUAL  T_THROW

 T_LOGICAL_XOR  T_PRINT  T_TRY

 T_METHOD_C  T_PRIVATE  T_UNSET

 T_MINUS_EQUAL  T_PUBLIC  T_UNSET_CAST

 T_ML_COMMENT  T_PROTECTED  T_USE

 T_MOD_EQUAL  T_REQUIRE  T_VAR

 T_MUL_EQUAL  T_REQUIRE_ONCE  T_VARIABLE

 T_NAMESPACE  T_RETURN  T_WHILE

 T_NS_C  T_SL  T_WHITESPACE

 T_NEW  T_SL_EQUAL  T_XOR_EQUAL

 T_NUM_STRING  T_SR
Syntax Analysis
Analyze a sequence of tokens
Syntax Analysis
Parser Generators

 You do not want to write a parser by hand

 At least when the code for the scanner should
be efficient and maintainable
 Tools such as bison or lemon generate
the code for a parser from a set of rules

T_IF '(' expr ')' { ... }

statement { ... }
elseif_list else_single { ... }
PHP Bytecode
Disassembling with vld
1 <?php
2 if (TRUE) {
3 print '*';
4 }
5 ?>
sb@thinkpad ~ % php -dextension=vld.so -dvld.active=1 -dvld.execute=0 if.php
filename: /home/sb/if.php
function name: (null)
number of ops: 8
compiled vars: none
line # op fetch ext return operands
-------------------------------------------------------------------------------
2 0 EXT_STMT
1 JMPZ true, ->6
3 2 EXT_STMT
3 PRINT ~0 '%2A'
4 FREE ~0
4 5 JMP ->6
6 6 EXT_STMT
7 RETURN 1
PHP Bytecode
Disassembling with bytekit-cli
1 <?php
2 if (TRUE) {
3 print '*';
4 }
5 ?>
sb@thinkpad ~ % bytekit if.php
bytekit-cli 1.0.0 by Sebastian Bergmann.

Filename: /home/sb/if.php
Function: main
Number of oplines: 8

line # opcode result operands

-----------------------------------------------------------------------------
2 0 EXT_STMT
1 JMPZ true, ->6

3 2 EXT_STMT
3 PRINT ~0 '*'
4 FREE ~0
4 5 JMP ->6

6 6 EXT_STMT
7 RETURN 1
PHP Bytecode
Bytecode visualization with bytekit-cli
1 <?php
2 if (TRUE) {
3 print '*';
4 }
5 ?>
sb@thinkpad ~ % bytekit --graph /tmp --format svg if.php
PHP Bytecode
Disassembling with bytekit-cli
1 <?php
2 $a = 1;
3 $b = 2;
4 print $a + $b;
5 ?>
sb@thinkpad ~ % bytekit add.php
bytekit-cli 1.0.0 by Sebastian Bergmann.

Filename: /home/sb/add.php
Function: main
Number of oplines: 10
Compiled variables: !0 = $a, !1 = $b

line # opcode result operands

-----------------------------------------------------------------------------
2 0 EXT_STMT
1 ASSIGN !0, 1
3 2 EXT_STMT
3 ASSIGN !1, 2
4 4 EXT_STMT
5 ADD ~2 !0, !1
6 PRINT ~3 ~2
7 FREE ~3
6 8 EXT_STMT
9 RETURN 1
PHP Bytecode
List of Opcodes

 NOP  IS_NOT_EQUAL  POST_INC  ADD_VAR  UNSET_DIM

 ADD  IS_SMALLER  POST_DEC  BEGIN_SILENCE  UNSET_OBJ

 SUB  IS_SMALLER_OR_EQUAL  ASSIGN  END_SILENCE  FE_RESET

 MUL  CAST  ASSIGN_REF  INIT_FCALL_BY_NAME  FE_FETCH

 DIV  QM_ASSIGN  ECHO  DO_FCALL  EXIT

 MOD  ASSIGN_ADD  PRINT  DO_FCALL_BY_NAME  FETCH_R

 SL  ASSIGN_SUB  JMPZ  RETURN  FETCH_DIM_R

 SR  ASSIGN_MUL  JMPNZ  RECV  FETCH_OBJ_R

 CONCAT  ASSIGN_DIV  JMPZNZ  RECV_INIT  FETCH_W

 BW_OR  ASSIGN_MOD  JMPZ_EX  SEND_VAL  FETCH_DIM_W

 BW_AND  ASSIGN_SL  JMPNZ_EX  SEND_VAR  FETCH_OBJ_W

 BW_XOR  ASSIGN_SR  CASE  SEND_REF  FETCH_RW

 BW_NOT  ASSIGN_CONCAT  SWITCH_FREE  NEW  FETCH_DIM_RW

 BOOL_NOT  ASSIGN_BW_OR  BRK  FREE  FETCH_OBJ_RW

 BOOL_XOR  ASSIGN_BW_AND  BOOL  INIT_ARRAY  FETCH_IS

 IS_IDENTICAL  ASSIGN_BW_XOR  INIT_STRING  ADD_ARRAY_ELEMENT  FETCH_DIM_IS

 IS_NOT_IDENTICAL  PRE_INC  ADD_CHAR  INCLUDE_OR_EVAL  FETCH_OBJ_IS

 IS_EQUAL  PRE_DEC  ADD_STRING  UNSET_VAR  FETCH_FUNC_ARG

PHP Bytecode
List of Opcodes

 FETCH_DIM_FUNC_ARG  INIT_STATIC_METHOD_CALL

 FETCH_OBJ_FUNC_ARG  ISSET_ISEMPTY_VAR

 FETCH_UNSET  ISSET_ISEMPTY_DIM_OBJ

 FETCH_DIM_UNSET  PRE_INC_OBJ

 FETCH_OBJ_UNSET  PRE_DEC_OBJ

 FETCH_DIM_TMP_VAR  POST_INC_OBJ

 FETCH_CONSTANT  POST_DEC_OBJ

 EXT_STMT  ASSIGN_OBJ

 EXT_FCALL_BEGIN  INSTANCEOF

 EXT_FCALL_END  DECLARE_CLASS

 EXT_NOP  DECLARE_INHERITED_CLASS

 TICKS  DECLARE_FUNCTION

 SEND_VAR_NO_REF  RAISE_ABSTRACT_ERROR

 CATCH  ADD_INTERFACE

 THROW  VERIFY_ABSTRACT_CLASS

 FETCH_CLASS  ASSIGN_DIM

 CLONE  ISSET_ISEMPTY_PROP_OBJ

 INIT_METHOD_CALL  HANDLE_EXCEPTION
Extending the Compiler
Test First!
Zend/tests/unless.phpt
--TEST--
unless statement
--FILE--
<?php
unless (FALSE) {
print 'unless FALSE is TRUE, this is printed';
}

unless (TRUE) {
print 'unless TRUE is TRUE, this is printed';
}
?>
--EXPECT--
unless FALSE is TRUE, this is printed
Extending the Compiler
 Add token for unless to the scanner
 Add rule for unless to the parser
 Generate bytecode for unless in the compiler
 Add token for unless to ext/tokenizer
Add unless scanner token
Zend/zend_language_scanner.l
<ST_IN_SCRIPTING>"if" {
return T_IF;
}

<ST_IN_SCRIPTING>"unless" {
return T_UNLESS;
}

<ST_IN_SCRIPTING>"elseif" {
return T_ELSEIF;
}

<ST_IN_SCRIPTING>"endif" {
return T_ENDIF;
}

<ST_IN_SCRIPTING>"else" {
return T_ELSE;
}
Add unless parser rule
Zend/zend_language_parser.y
%token T_NAMESPACE
%token T_NS_C
%token T_DIR
%token T_NS_SEPARATOR
%token T_UNLESS
.
.
unticked_statement:
'{' inner_statement_list '}'
| T_IF '(' expr ')' {
.
.
| T_UNLESS '(' expr ')' {
zend_do_unless_cond(&$3, &$4 TSRMLS_CC);
} statement {
zend_do_if_after_statement(&$4, 1 TSRMLS_CC);
} {
zend_do_if_end(TSRMLS_C);
}
.
.
How if is compiled
Zend/zend_compile.c
void zend_do_if_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
typedef struct _znode {
int op_type;
union {
zval constant;

zend_uint var;
zend_uint opline_num;
zend_op_array *op_array;
zend_op *jmp_addr;
struct {
zend_uint var;
zend_uint type;
} EA;
} u;
} } znode;

zend_do_if_cond() is called when an if statement is compiled

How if is compiled
Zend/zend_compile.c
void zend_do_if_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
int if_cond_op_number =
get_next_op_number(CG(active_op_array));
zend_op *opline =
get_next_op(CG(active_op_array) TSRMLS_CC);

struct _zend_op {
opcode_handler_t handler;
znode result;
znode op1;
znode op2;
ulong extended_value;
uint lineno;
zend_uchar opcode;
} };

Allocate a new opline in the current oparray

How if is compiled
Zend/zend_compile.c
void zend_do_if_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
int if_cond_op_number =
get_next_op_number(CG(active_op_array));
zend_op *opline =
get_next_op(CG(active_op_array) TSRMLS_CC);

opline->opcode = ZEND_JMPZ;

Set the opcode of the new opline to JMPZ (jump if zero)

How if is compiled
Zend/zend_compile.c
void zend_do_if_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
int if_cond_op_number =
get_next_op_number(CG(active_op_array));
zend_op *opline =
get_next_op(CG(active_op_array) TSRMLS_CC);

opline->opcode = ZEND_JMPZ;
opline->op1 = *cond;

Set the first operand of the new opline to the if condition

How if is compiled
Zend/zend_compile.c
void zend_do_if_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
int if_cond_op_number =
get_next_op_number(CG(active_op_array));
zend_op *opline =
get_next_op(CG(active_op_array) TSRMLS_CC);

opline->opcode = ZEND_JMPZ;
opline->op1 = *cond;
closing_bracket_token->u.opline_num =
if_cond_op_number;
SET_UNUSED(opline->op2);
INC_BPC(CG(active_op_array));
}

Perform book keeping tasks such as marking the second operand of the
new opline as unused or incrementing the backpatching counter for the
current oparray
Add unless to compiler
Zend/zend_compile.c
void zend_do_unless_cond
(const znode *cond, znode *closing_bracket_token TSRMLS_DC)
{
int unless_cond_op_number =
get_next_op_number(CG(active_op_array));
zend_op *opline =
get_next_op(CG(active_op_array) TSRMLS_CC);

opline->opcode = ZEND_JMPNZ;
opline->op1 = *cond;
closing_bracket_token->u.opline_num =
unless_cond_op_number;
SET_UNUSED(opline->op2);
INC_BPC(CG(active_op_array));
}

All we have to do to generate code for the unless statement, as

compared to generate code for the if statement, is to use the JMPNZ
(jump if not zero) opcode instead of the JMPZ (jump if zero) opcode
Add unless to compiler
The generated bytecode
1 <?php
2 unless (FALSE) {
3 print '*';
4 }
5 ?>
sb@thinkpad ~ % bytekit unless.php
bytekit-cli 1.0.0 by Sebastian Bergmann.

Filename: /home/sb/unless.php
Function: main
Number of oplines: 8

line # opcode result operands

-----------------------------------------------------------------------------
2 0 EXT_STMT
1 JMPNZ true, ->6

3 2 EXT_STMT
3 PRINT ~0 '*'
4 FREE ~0
4 5 JMP ->6

6 6 EXT_STMT
7 RETURN 1
Run the test
sb@thinkpad php-5.3-unless % make test TESTS=Zend/tests/unless.phpt

Build complete.
Don't forget to run 'make test'.

=====================================================================
PHP : /usr/local/src/php/php-5.3-unless/sapi/cli/php
PHP_SAPI : cli
PHP_VERSION : 5.3.0RC3-dev
ZEND_VERSION: 2.3.0
PHP_OS : Linux 2.6.28-11-generic #42-Ubuntu SMP Fri Apr 17 01:57:59 UTC 2009 i686 GNU/Linux
INI actual : /usr/local/src/php/php-5.3-unless/tmp-php.ini
More .INIs :
CWD : /usr/local/src/php/php-5.3-unless
Extra dirs :
VALGRIND : Not used
=====================================================================
Running selected tests.
PASS unless statement [Zend/tests/unless.phpt]
=====================================================================
Number of tests : 1 1
Tests skipped : 0 ( 0.0%) --------
Tests warned : 0 ( 0.0%) ( 0.0%)
Tests failed : 0 ( 0.0%) ( 0.0%)
Expected fail : 0 ( 0.0%) ( 0.0%)
Tests passed : 1 (100.0%) (100.0%)
---------------------------------------------------------------------
Time taken : 0 seconds
=====================================================================
Add unless to ext/tokenizer
ext/tokenizer/tokenizer_data.c

sb@thinkpad tokenizer % ./tokenizer_data_gen.sh

Wrote tokenizer_data.c
 The End

Thank you for your interest!

These slides will be linked soon from

http://sebastian-bergmann.de/

You can vote for this talk on

http://joind.in/582
Acknowledgements
 Thomas Lee, whose Python Language Internals presentation at
OSDC 2008 inspired this presentation
 Stefan Esser for creating the Bytekit extension that provides
PHP bytecode access and analysis features
 Derick Rethans, David Soria Parra, and Scott MacVicar for reviewing
these slides
References

 http://www.php.net/manual/en/tokens.php
 http://www.zapt.info/opcodes.html
 Sara Golemon: ”Extending and Embedding PHP”

 http://derickrethans.nl/vld.php
 http://bytekit.org/
 http://github.com/sebastianbergmann/bytekit-cli/
 License
  This presentation material is published under the Attribution-Share Alike 3.0 Unported
license.
  You are free:
✔ to Share – to copy, distribute and transmit the work.
✔ to Remix – to adapt the work.
  Under the following conditions:
● Attribution. You must attribute the work in the manner specified by the author or
licensor (but not in any way that suggests that they endorse you or your use of the
work).
● Share Alike. If you alter, transform, or build upon this work, you may distribute the
resulting work only under the same, similar or a compatible license.
  For any reuse or distribution, you must make clear to others the license terms of this
work.
  Any of the above conditions can be waived if you get permission from the copyright
holder.
  Nothing in this license impairs or restricts the author's moral rights.