P P: Training An Increasingly General Problem Solver by Continually Searching For The Simplest Still Unsolvable Problem

P OWER P LAY: Training an Increasingly
General Problem Solver by Continually Searching

for the Simplest Still Unsolvable Problem
arXiv:1112.5309v2 [cs.AI] 4 Nov 2012
Jürgen Schmidhuber
The Swiss AI Lab IDSIA, Galleria 2, 6928 Manno-Lugano
University of Lugano & SUPSI, Switzerland
22 December 2011 (arXiv:1112.5309v1), revised 4 November 2012
Abstract
Most of computer science focuses on automatically solving given computational problems. I focus on
automatically inventing or discovering problems in a way inspired by the playful behavior of animals and
humans, to train a more and more general problem solver from scratch in an unsupervised fashion. Con-
sider the infinite set of all computable descriptions of tasks with possibly computable solutions. Given a
general problem solving architecture, at any given time, the novel algorithmic framework P OWER P LAY
[46] searches the space of possible pairs of new tasks and modifications of the current problem solver,
until it finds a more powerful problem solver that provably solves all previously learned tasks plus the
new one, while the unmodified predecessor does not. Newly invented tasks may require to achieve a
wow-effect by making previously learned skills more efficient such that they require less time and space.
New skills may (partially) re-use previously learned skills. The greedy search of typical P OWER P LAY
variants uses time-optimal program search to order candidate pairs of tasks and solver modifications by
their conditional computational (time & space) complexity, given the stored experience so far. The new
task and its corresponding task-solving skill are those first found and validated. This biases the search
towards pairs that can be described compactly and validated quickly. The computational costs of validat-
ing new tasks need not grow with task repertoire size. Standard problem solver architectures of personal
computers or neural networks tend to generalize by solving numerous tasks outside the self-invented
training set; P OWER P LAY’s ongoing search for novelty keeps breaking the generalization abilities of
its present solver. This is related to Gödel’s sequence of increasingly powerful formal theories based
on adding formerly unprovable statements to the axioms without affecting previously provable theorems.
The continually increasing repertoire of problem solving procedures can be exploited by a parallel search
for solutions to additional externally posed tasks. P OWER P LAY may be viewed as a greedy but practical
implementation of basic principles of creativity [42, 45]. A first experimental analysis can be found in
separate papers [53, 52].
1
Contents
1 Introduction 3
1.1 Basic Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Outline of Remainder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Notation & Algorithmic Framework P OWER P LAY (Variant I) 5
3 TASK I NVENTION, S OLVER M ODIFICATION, C ORRECTNESS D EMO 5

3.1 Implementing TASK I NVENTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.1.1 Example: Pattern Recognition Tasks . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1.2 Example: General Decision Making Tasks in Dynamic Environments . . . . . . . 6
3.2 Implementing S OLVER M ODIFICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3 Implementing C ORRECTNESS D EMONSTRATION . . . . . . . . . . . . . . . . . . . . . . 7
3.3.1 Most General: Proof Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3.2 Keeping Track Which Components of the Solver Affect Which Tasks . . . . . . . 8
3.3.3 Advantages of Prefix Code-Based Problem Solvers . . . . . . . . . . . . . . . . . 8
4 Implementations of P OWER P LAY 9

4.1 Implementation Based on Optimal Ordered Problem Solver OOPS . . . . . . . . . . . . . 9
4.1.1 Building on Existing OOPS Source Code . . . . . . . . . . . . . . . . . . . . . . 10
4.1.2 Alternative Problem Solvers Based on Recurrent Neural Networks . . . . . . . . . 10
4.2 Adapting the Probability Distribution on Programs . . . . . . . . . . . . . . . . . . . . . 11
4.3 Implementation Based on Stochastic or Evolutionary Search . . . . . . . . . . . . . . . . 11
5 Outgrowing Trivial Tasks - Compressing Previous Solutions 11
6 Adding External Tasks 12

6.1 Self-Reference Through Novel Task Search as an External Task . . . . . . . . . . . . . . 12
7 Softening Task Acceptance Criteria of P OWER P LAY 12

7.1 P OWER P LAY Variant II: Explicitly Penalizing Time and Space Complexity . . . . . . . . 13
7.2 Probabilistic P OWER P LAY Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
8 First Illustrative Experiments 14
9 Previous Relevant Work 14

9.1 Existing Theoretically Optimal Universal Problem Solvers . . . . . . . . . . . . . . . . . 14
9.2 Connection to Traditional Active Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9.3 Greedy Implementation of Aspects of the Formal Theory of Creativity . . . . . . . . . . . 15
9.4 Beyond Algorithmic Zero-Sum Games [37, 38] (1997-2002) . . . . . . . . . . . . . . . . 16
9.5 Opposing Forces: Improving Generalization Through Compression, Breaking Generaliza-
tion Through Novelty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
9.6 Relation to Gödel’s Sequence of Increasingly Powerful Axiomatic Systems . . . . . . . . 17
10 Words of Caution 17
11 Acknowledgments 18
2
1 Introduction
Given a realistic piece of computational hardware with specific resource limitations, how can one devise
software for it that will solve all, or at least many, of the a priori unknown tasks that are in principle easily
solvable on this architecture? In other words, how to build a practical general problem solver, given the
computational restrictions? It does not need to be universal and asymptotically optimal [17, 13, 41, 44] like
the recent (not necessarily practically feasible) general problem solvers discussed in Section 9.1; instead it
should take into account all constant architecture-specific slowdowns ignored in the asymptotic optimality
notation of theoretical computer science, and be generally useful for real-world applications.
Let us draw inspiration from biology. How do initially helpless human babies become rather general
problem solvers over time? Apparently by playing. For example, even in the absence of external reward or
hunger they are curious about what happens if they move their eyes or fingers in particular ways, creating
little experiments which lead to initially novel and surprising but eventually predictable sensory inputs,
while also learning motor skills to reproduce these outcomes. (See [31, 30, 37, 42, 45, 62] and Section
9.3 for previous artificial systems of this type.) Infants continually seem to invent new tasks that become
boring as soon as their solutions become known. Easy-to-learn new tasks are preferred over unsolvable or
hard-to-learn tasks. Eventually the numerous skills acquired in this creative, self-supervised way may get
re-used to facilitate the search for solutions to external problems, such as finding food when hungry. While
kids keep inventing new problems for themselves, they move through remarkable developmental stages
[23, 2, 11].
Here I introduce a novel unsupervised algorithmic framework for training a computational problem
solver from scratch, continually searching for the simplest (fastest to find) combination of task and corre-
sponding task-solving skill to add to its growing repertoire, without forgetting any previous skills (Section
2), or at least without decreasing average performance on previously solved tasks (Section 7.1). New skills
may (partially) re-use previously learned skills. Every new task added to the repertoire is essentially de-
fined by the time required to invent it, to solve it, and to demonstrate that no previously learned skills got
lost. The search takes into account that typical problem solvers may learn to solve tasks outside the growing
self-made training set due to generalization properties of their architectures. The framework is called P OW-
ER P LAY because it continually [25] aims at boosting computational prowess and problem solving capacity,
reminiscent of humans or human societies trying to boost their general power/capabilities/knowledge/skills
in playful ways, even in the absence of externally defined goals, although the skills learned by this type of
pure curiosity may later help to solve externally posed tasks.
Unlike our first implementations of curious/creative/playful agents from the 1990s [30, 54, 37] (Section
9.3; compare [1, 5, 22, 20]), P OWER P LAY provably (by design) does not have any problems with online
learning—it cannot forget previously learned skills, automatically segmenting its life into a sequence of
clearly identified tasks with explicitly recorded solutions. Unlike the task search of theoretically optimal
creative agents [42, 45] (Section 9.3), P OWER P LAY’s task search is greedy, but at least practically feasible.
Some claim that scientists often invent appropriate problems for their methods, rather than inventing
methods to solve given problems. The present paper formalizes this in a way that may be more convenient
to implement than those of previous work [30, 37, 42, 45], and describes a simple practical framework for
building creative artificial scientists or explorers that by design continually come up with the fastest to find,
initially novel, but eventually solvable problems.
1.1 Basic Ideas

In traditional computer science, given some formally defined task, a search algorithm is used to search a
space of solution candidates until a solution to the task is found and verified. If the task is hard the search
may take long.
To automatically construct an increasingly general problem solver, let us expand the traditional search
space in an unusual way, such that it includes all possible pairs of computable tasks with possibly com-
putable solutions, and problem solvers. Given an old problem solver that can already solve a finite known
set of previously learned tasks, a search algorithm is used to find a new pair that provably has the following
properties: (1) The new task cannot be solved by the old problem solver. (2) The new task can be solved by
3
the new problem solver (some modification of the old one). (3) The new solver can still solve the known
set of previously learned tasks.
Once such a pair is found, the cycle repeats itself. This will result in a continually growing set of known
tasks solvable by an increasingly more powerful problem solver. Solutions to new tasks may (partially) re-
use solutions to previously learned tasks.
Smart search (e.g., Section 4.1, Algorithm 4.1) orders candidate pairs of the type (task, solver) by
computational complexity, using concepts of optimal universal search [17, 41], with a bias towards pairs
that can be described by few additional bits of information (given the experience so far) and that can be
validated quickly.
At first glance it might seem harder to search for pairs of tasks and solvers instead of solvers only, due
to the apparently larger search space. However, the additional freedom of inventing the tasks to be solved
may actually greatly reduce the time intervals between problem solver advances, because the system may
often have the option of inventing a rather simple task with an easy-to-find solution.
A new task may be about simplifying the old solver such that it can still solve all tasks learned so far,
but with less computational resources such as time and storage space (e.g., Section 3.1 and Algorithm 7.1).
Since the new pair (task, solver) is the first one found and validated, the search automatically trades
off the time-varying efforts required to either invent completely new, previously unsolvable problems, or
compressing/speeding up previous solutions. Sometimes it is easier to refine or simplify known skills,
sometimes to invent new skills.
On typical problem solver architectures of personal computers (PCs) or neural networks (NNs), while
a limited known number of previously learned tasks has become solvable, so too has a large number of
unknown, never-tested tasks (in the field of Machine Learning, this is known as generalization). P OWER -
P LAY’s ongoing search is continually testing (and always trying to go beyond) the generalization abilities of
the most recent solver instance; some of its search time has to be spent on demonstrating that self-invented
new tasks are not already solvable.
Often, however, much more time will have to be spent on making sure that a newly modified solver did
not forget any of the possibly many previously learned skills. Problem solver modularization (Section 3.3,
especially 3.3.2) may greatly reduce this time though, making P OWER P LAY prefer pairs whose validation
does not require the re-testing of too many previously learned skills, thus decomposing at least part of the
search space into somewhat independent regions, realizing divide and conquer strategies as by-products of
its built-in drive to invent and validate novel tasks/skills as quickly as possible.
A biologically inspired hope is that as the problem solver is becoming more and more general, it will
find it easier and easier to solve externally posed tasks (Section 6), just like growing infants often seem to
re-use their playfully acquired skills to solve teacher-given problems.
1.2 Outline of Remainder

Section 2 will introduce basic notation and Variant 1 of the algorithmic framework P OWER P LAY, which in-
vokes the essential procedures TASK I NVENTION, S OLVER M ODIFICATION, and C ORRECTNESS D EMON -
STRATION . Section 3 will discuss details of these procedures.
More detailed instantiations of P OWER P LAY will be described in Section 4.3 (an evolutionary method,
Alg. 4.3) and Section 4.1 (an asymptotically optimal program search method, Alg. 4.1).
As mentioned above, the skills acquired to solve self-generated tasks may later greatly facilitate so-
lutions to externally posed tasks, just like the numerous motor skills learned by babies during curious
exploration of its world often can be re-used later to maximize external reward. Sections 6 and 7.1 will
discuss variants of the framework (e.g., Algorithm 7.1) in which some of the tasks can be defined externally.
Section 7.1 will also describe a natural variant of the framework that explicitly penalizes solution costs
(including time and space complexity), and allows for forgetting aspects of previous solutions, provided
the average performance on previously solved tasks does not decrease.
Section 8 will point to illustrative experiments in separate papers [53, 52]. Section 9 will discuss
relations to previous work.
4
2 Notation & Algorithmic Framework P OWER P LAY (Variant I)
B ∗ denotes the set of finite sequences or bitstrings over the binary alphabet B = {0, 1}, λ the empty
string, x, y, z, p, q, r, u strings in B ∗ , N the natural numbers, R the real numbers, ǫ ∈ R a positive constant,
m, n, n0 , k, i, j, k, l non-negative integers, L(x) the number of bits in x (where L(λ) = 0), f, g functions
mapping integers to integers. We write f (n) = O(g(n)) if there exist positive c, n0 such that f (n) ≤ cg(n)
for all n > n0 .
The computational architecture of the problem solver may be a deterministic universal computer, or a
more limited device such as a finite state automaton or a feedforward neural network (NN) [3]. All such
problem solvers can be uniquely encoded [9] or implemented on universal computers such as universal
Turing Machines (TM) [56]. Therefore, without loss of generality, the remainder of this paper assumes a
fixed universal reference computer whose input programs and outputs are elements of B ∗ . A user-defined
subset S ⊂ B ∗ defines the set of possible problem solvers. For example, if the problem solver’s architecture
is itself a binary universal TM or a standard computer, then S represents its set of possible programs, or
a limited subset thereof—compare Sections 3.2 and 4.1. If it is a feedforward NN, then S could be a
highly restricted subset of programs encoding the NN’s possible topologies and weights (floating point
numbers)—compare Section 8 and the original SLIM NN paper [47].
In what follows, for convenience I will often identify bitstrings in B ∗ with things they encode, such
as integers, real-valued vectors, weight matrices, or programs—the context will always make clear what is
meant.
The problem solver’s initial program is called s0 . There is a set of possible task descriptions T ⊂ B ∗ . T
may be the infinite set of all possible computable descriptions of tasks with possibly computable solutions,
or just a small subset thereof. For example, a simple task may require the solver to answer a particular input
pattern with a particular output pattern (more formal details on pattern recognition tasks are given in Section
3.1.1). Or it may require the solver to steer a robot towards a goal through a sequence of actions (more
formal details on sequential decision making tasks in unknown environments are given in Section 3.1.2).
There is a particular sequence of task descriptions T1 , T2 , . . ., where each unique Ti ∈ T (i = 1, 2, . . .)
is chosen or “invented” by a search method described below such that the solutions of T1 , T2 , . . . , Ti can
be computed by si , the i-th instance of the program, but not by si−1 (i = 1, 2, . . .). Each Ti consists
of a unique problem identifier that can be read by si through some built-in input processing mechanism
(e.g., input neurons of an NN [47]), and a unique description of a deterministic procedure for determining
whether the problem has been solved. Denote T≤i = {T1 , . . . , Ti }; T<i = {T1 , . . . , Ti−1 }.
A valid task Ti (i > 1) may require solving at least one previously solved task Tk (k < i) more ef-
ficiently, by using less resources such as storage space, computation time, energy, etc., thus achieving a
wow-effect. See Section 3.1.
Tasks and problem solver modifications are computed and validated by elements of another appropriate
set of programs P ⊂ B ∗ . Programs p ∈ P may contain instructions for reading and executing (parts of)
the code of the present problem solver and reading (parts of) a recorded history T race ∈ B ∗ of previous
events that led to the present solver. The algorithmic framework (Alg. 2) incrementally trains the problem
solver by finding p ∈ P that increase the set of solvable tasks.
3 TASK I NVENTION, S OLVER M ODIFICATION, C ORRECTNESS D EMO

A program tested by Alg. 2 has to allocate its runtime to solve three main jobs, namely, TASK I NVENTION,
S OLVER M ODIFICATION, C ORRECTNESS D EMONSTRATION. Now examples of each will be listed.
3.1 Implementing TASK I NVENTION

Part of the job of pi ∈ P is to compute Ti ∈ T . This will consume some of the total computation
time allocated to pi . Two examples will be given: pattern recognition tasks are treated in Section 3.1.1;
sequential decision making tasks in Section 3.1.2.
5
Alg. 2: Algorithmic Framework P OWER P LAY (Variant I)
Initialize s0 in some way.
for i := 1, 2, . . . do
repeat
Let a search algorithm (examples in Section 4) create a new candidate program p ∈ P. Give p
limited time to do (not necessarily in this order):
* TASK I NVENTION: Let p compute a task T ∈ T . See Section 3.1.
* S OLVER M ODIFICATION: Let p compute a value of the variable q ∈ S ⊂ B ∗ (a candidate for si )
by computing a modification of si−1 . See Section 3.2.
* C ORRECTNESS D EMONSTRATION: Let p try to show that T cannot be solved by si−1 , but that
T and all Tk (k < i) can be solved by q. See Section 3.3.
until C ORRECTNESS D EMONSTRATION was successful
Set pi := p; Ti := T ; si := q; update T race.
end for
3.1.1 Example: Pattern Recognition Tasks

In the context of learning to recognize or analyze patterns, Ti could be a 4-tuple (Ii , Oi , ti , ni ) ∈ I ×
O × N × N, where I, O ⊂ B ∗ , and Ti is solved if si satisfies L(si ) < ni and needs at most ti discrete
time steps to read Ii and compute Oi and halt. Here Ii itself may be a pair (Ii1 , Ii2 ) ∈ B ∗ × B ∗ , where
Ii1 is constrained to be the address of an image in a given database of patterns, and Ii2 is a pi -generated
“query” that uniquely specifies how the image should be classified through target pattern Oi , such that
the same image can be analyzed in different ways during different tasks. For example, depending on the
nature of the invented task sequence, the problem solver could eventually learn that O = 1 if I 2 = 1001
(suppressing task indices) and the image addressed by I 1 contains at least one black pixel, or if I 2 = 0111
and the image shows a cow.
Since the definition of task Ti includes bounds ni , ti on computational resources, Ti may be about
solving at least one Tk (k < i) more efficiently, corresponding to a wow-effect. This in turn may also yield
more efficient solutions to other tasks Tl (l < i, l 6= k). In practical applications one may insist that such
efficiency gains must exceed a certain threshold ǫ > 0, to avoid task series causing sequences of very minor
improvements.
Note that ni and ti may be unnecessary in special cases such as the problem solver being a fixed
topology feedforward NN [3] whose input and target patterns have constant size and whose computational
efforts per pattern need constant time and space resources.
Assuming sufficiently powerful S, P, in the beginning, trivial tasks such as simply copying Ii2 onto Oi
may be interesting in the sense that P OWER P LAY can still validate and accept them, but they will become
boring (inadmissible) as soon as they are solvable by solutions to previous tasks that generalize to new
tasks.
3.1.2 Example: General Decision Making Tasks in Dynamic Environments

In the more general context of general problem solving/sequential decision making/reinforcement learn-
ing/reward optimization [21, 15, 55] in unknown environments, there may be a set I ⊂ B ∗ of possible task
identification patterns and a set J ⊂ B ∗ of programs that test properties of bitstrings. Ti could then encode
a 4-tuple (Ii , Ji , ti , ni ) ∈ I × J × N × N of finite bitstrings with the following interpretation: si must
satisfy L(si ) < ni and may spend at most ti discrete time steps on first reading Ii and then interacting with
an environment through a sequence of perceptions and actions, to achieve some computable goal defined
by Ji .
More precisely, while Ti is being solved within ti time steps, at any given time 1 ≤ t ≤ ti , the internal
state of the problem solver at time t is denoted ui (t) ∈ B ∗ ; its initial default value is ui (0). For example,
ui (t) may encode the current contents of the internal tape of a TM, or of certain addresses in the dynamic
storage area of a PC, or the present activations of an LSTM recurrent NN [12]. At time t, si can spend
a constant number of elementary computational instructions to copy the task dscription Ti or the present
6
environmental input xi (t) ∈ B ∗ and a reward signal ri (t) ∈ B ∗ (interpreted as a real number) into parts of
ui (t), then update other parts of ui (t) (a function of ui (t − 1)) and compute action yi (t) ∈ B ∗ encoded as
a part of ui (t). yi (t) may affect the environment, and thus future inputs.
If P allows for programs that can dynamically acquire additional physical computational resources
such as additional CPUs and storage, then the above constant number of elementary computational in-
structions should be replaced by a constant amount of real time, to be measured by a reliable physical
clock.
The sequence of 4-tuples (xi (t), ri (t), ui (t), yi (t)) (t = 1, . . . , ti ) gets recorded by the so-called trace
T racei ∈ B ∗ . If at the end of the interaction a desirable computable property Ji (T racei ) (computed by
applying program Ji to T racei ) is satisfied, then by definition the task is solved. The set J of possible
Ji may represent an infinite set of all computable tasks with solutions computable by the given hardware.
For practical reasons, however, the set J of possible Ji may also be restricted to bit sequences encoding
just a few possible goals. For example, Ji may only encode goals of the form: a robot arm steered by
program or “policy” si has reached a certain target (a desired final observation xi (ti ) recorded in T racei )
without measurably bumping into an obstacle along the way, that is, there were no negative rewards, that
is, ri (τ ) ≥ 0 for τ = 1 . . . ti .
If the environment is deterministic, e.g., a digital physics simulation of a robot, then its current state
can be encoded as part of u(t), and it is straight-forward for C ORRECTNESS D EMONSTRATION to test
whether some si still can solve a previously solved task Tj (j < i). However, what if the environment is
only partially observable, like the real world, and non-stationary, changing in unknown ways? Then C OR -
RECTNESS D EMONSTRATION must check whether si still produces the same action sequence in response
to the input sequence recorded in T racej (often this replay-based test will actually be computationally
cheaper than a test involving the environment). Achieving the same goal in a changed environment must
be considered a different task, even if the changes are just due to noise on the environmental inputs. (Sure,
in the real world sj (j > i) might actually achieve Ji faster than si , given the description of Ti , but C OR -
RECTNESS D EMONSTRATION in general cannot know whether this acceleration was due to plain luck—it
must stick to reproducing T racej to make sure it did not forget anything.)
See Section 7.2, however, for a less strict P OWER P LAY variant whose C ORRECTNESS D EMONSTRA -
TION directly interacts with the real world to collect sufficient problem-solving statistics through repeated
trials, making certain assumptions about the probabilistic nature of the environment, and the repeatability
of experiments.
3.2 Implementing S OLVER M ODIFICATION

Part of the job of pi ∈ P is also to compute si , possibly profiting from having access to si−1 , because only
few changes of si−1 may be necessary to come up with an si that goes beyond si−1 . For example, if the
problem solver is a standard PC, then just a few bits of the previous software si−1 may need to be changed.
For practical reasons, the set S of possible si may be greatly restricted to bit sequences encoding
programs that obey the syntax of a standard programming language such as LISP or Java. In turn, the
programming language describing P should be greatly restricted such that any pi ∈ P can only produce
syntactically correct si .
If the problem solver is a feedforward NN with pre-wired, unmodifiable topology, then S will be re-
stricted to those bit sequences encoding valid weight matrices, si will encode its i-th weight matrix, and P
will be restricted to those p ∈ P that can produce some si ∈ S. Depending on the user-defined program-
ming language, pi may invoke complex pre-wired subprograms (e.g., well-known learning algorithms) as
primitive instructions—compare separate experimental analysis [53, 52].
In general, p itself determines how much time to spend on S OLVER M ODIFICATION—enough time
must be left for TASK I NVENTION and C ORRECTNESS D EMONSTRATION.
3.3 Implementing C ORRECTNESS D EMONSTRATION

Correctness demonstration may be the most time-consuming obligation of pi . At first glance it may seem
that as the sequence T1 , T2 , . . . is growing, more and more time will be needed to show that si but not si−1
7
can solve T1 , T2 , . . . , Ti , because one naive way of ensuring correctness is to re-test si on all previously
solved tasks. Theoretically more efficient ways are considered next.
3.3.1 Most General: Proof Search

The most general way of demonstrating correctness is to encode (in read-only storage) an axiomatic system
A that formally describes computational properties of the problem solver and possible si , and to allow pi to
search the space of possible proofs derivable from A, using a proof searcher subroutine that systematically
generates proofs until it finds a theorem stating that si but not si−1 solves T1 , T2 , . . . , Ti (proof search
may achieve this efficiently without explicitly re-testing si on T1 , T2 , . . . , Ti ). This could be done like
in the Gödel Machine [44] (Section 9.1), which uses an online extension of Universal Search [17] to
systematically test proof techniques: proof-generating programs that may invoke special instructions for
generating axioms and applying inference rules to prolong an initially empty proof ∈ B ∗ by theorems,
which are either axioms or inferred from previous theorems through rules such as modus ponens combined
with unification, e.g., [7]. P can be easily limited to programs generating only syntactically correct proofs
[44]. A has to subsume axioms describing how any instruction invoked by some s ∈ S will change the
state u of the problem solver from one step to the next (such that proof techniques can reason about the
effects of any si ). Other axioms encode knowledge about arithmetics etc (such that proof techniques can
reason about spatial and temporal resources consumed by si ).
In what follows, C ORRECTNESS D EMONSTRATIONS will be discussed that are less general but some-
times more convenient to implement.
3.3.2 Keeping Track Which Components of the Solver Affect Which Tasks
Often it is possible to partition s ∈ S into components, such as individual bits of the software of a PC,
or weights of a NN. Here the k-th component of s is denoted sk . For each k (k = 1, 2, . . .) a variable
list Lk = (T1k , T2k , . . .) is introduced. Its initial value before the start of P OWER P LAY is Lk0 , an empty
list. Whenever pi found si and Ti at the end of C ORRECTNESS D EMONSTRATION, each Lk is updated as
follows: Its new value Lki is obtained by appending to Lki−1 those Tj ∈ / Lki−1 (j = 1, . . . , i) whose current
k
(possibly revised) solutions now need s at least once during the solution-computing process, and deleting
those Tj whose current solutions do not use sk any more.
P OWER P LAY’s C ORRECTNESS D EMONSTRATION thus has to test only tasks in the union of all Lki .
That is, if the most recent task does not require changes of many components of s, and if the changed bits
do not affect many previous tasks, then C ORRECTNESS D EMONSTRATION may be very efficient.
Since every new task added to the repertoire is essentially defined by the time required to invent it, to
solve it, and to show that no previous tasks became unsolvable in the process, P OWER P LAY is generally
“motivated” to invent tasks whose validity check does not require too much computational effort. That is,
P OWER P LAY will often find pi that generate si−1 -modifications that don’t affect too many previous tasks,
thus decomposing at least part of the spaces of tasks and their solutions into more or less independent
regions, realizing divide and conquer strategies as by-products. Compare a recent experimental analysis of
this effect [53, 52].
3.3.3 Advantages of Prefix Code-Based Problem Solvers

Let us restrict P such that tested p ∈ P cannot change any components of si−1 during S OLVER M OD -
IFICATION , but can create a new si only by adding new components to si−1 . (This means freezing all
used components of any sk once Tk is found.) By restricting S to self-delimiting prefix codes like those
generated by the Optimal Ordered Problem Solver (OOPS) [41], one can now profit from a sometimes
particularly efficient type of C ORRECTNESS D EMONSTRATION, ensuring that differences between si and
si−1 cannot affect solutions to T<i under certain conditions. More precisely, to obtain si , half the search
time is spent on trying to process Ti first by si−1 , extending or prolonging si−1 only when the ongo-
ing computation requests to add new components through special instructions [41]—then C ORRECTNESS
D EMONSTRATION has less to do as the set T<i is guaranteed to remain solvable, by induction. The other
half of the time is spent on processing Ti by a new sub-program with new components s′i , a part of si but
8
not of si−1 , where s′i may read si−1 or invoke parts of si−1 as sub-programs to solve T≤i — only then
C ORRECTNESS D EMONSTRATION has to test si not only on Ti but also on T<i (see [41] for details).
A simple but not very general way of doing something similar is to interleave TASK I NVENTION,
S OLVER M ODIFICATION, C ORRECTNESS D EMONSTRATION as follows: restrict all p ∈ P such that they
must define Ii := i as the unique task identifier Ii for Ti (see Section 3.1.2); restrict all s ∈ S such that the
input of Ii = i automatically invokes sub-program s′i , a part of si but not of si−1 (although s′i may read si−1
or invoke parts of si−1 as sub-programs to solve Ti ). Restrict Ji to a subset of acceptable computational
outcomes (Section 3.1.2). Run si until it halts and has computed a novel output acceptable by Ji that
is different from all outputs computed by the (halting) solutions to T<i ; this novel output becomes Ti ’s
goal. By induction over i, since all previously used components of si−1 remain unmodified, the set T<i is
guaranteed to remain solvable, no matter s′i . That is, C ORRECTNESS D EMONSTRATION on previous tasks
becomes trivial. However, in this simple setup there is no immediate generalization across tasks like in
OOPS [41] and the previous paragraph: the trivial task identifier i will always first invoke some s′i different
from all s′k (k 6= i), instead of allowing for solving a new task solely by previously found code.
4 Implementations of P OWER P LAY

P OWER P LAY is a general framework that allows for plugging in many differents search and learning algo-
rithms. The present section will discuss some of them.
4.1 Implementation Based on Optimal Ordered Problem Solver OOPS
Alg. 4.1: Implementing P OWER P LAY with Procedure OOPS [41]

(see text for details) - initialize s0 and u (internal dynamic storage for s ∈ S) and U (internal dynamic
storage for p ∈ P), where each possible p is a sequence of subprograms p′ , p′′ , p′′′ .
for i := 1, 2, . . . do
set variable time limit tlim := 1;
let the variable set H be empty;
set Boolean variable DONE := FALSE
repeat
if H is empty then
set tlim := 2tlim ; H := {p ∈ P : P (p)tlim ≥ 1}
else
choose and remove some p from H
while not DONE and less than P (p)tlim time was spent on p do
execute the next time step of the following computation:
1. Let p′ compute some task T ∈ T and halt.
2. Let p′′ compute q ∈ S by modifying a copy of si−1 , and halt.
3. Let p′′′ try to show that q but not si−1 can solve T1 , T2 , . . . , Ti−1 , T .
If p′′′ was successful set DONE := TRUE.
end while
Undo all modifications of u and U due to p. This does not cost more time than executing p in the
while loop above [41].
end if
until DONE
set pi := p; Ti := T ; si := q;
add a unique encoding of the 5-tuple (i, pi , si , Ti , T racei ) to read-only storage readable by programs
to be tested in the future.
end for
The i-th problem is to find a program pi ∈ P that creates si and Ti and demonstrates that si but not
si−1 can solve T1 , T2 , . . . , Ti . This yields a perfectly ordered problem sequence for a variant of the Optimal
9
Ordered Problem Solver OOPS [41] (Algorithm 4.1).
While a candidate program p ∈ P is executed, at any given discrete time step t = 1, 2, ..., its internal
state or dynamical storage U at time t is denoted U (t) ∈ B ∗ (not to be confused with the solver’s internal
state u(t) of Section 3.1.2). Its initial default value is U (0). E.g., U (t) could encode the current contents
of the internal tape of a TM (to be modified by p), or of certain cells in the dynamic storage area of a PC.
Once pi is found, pi , si , Ti , T racei (if applicable; see Section 3.1.2) will be saved in unmodifiable read-
only storage, possibly together with other data observed during the search so far. This may greatly facilitate
the search for pk , k > i, since pk may contain instructions for addressing and reading pj , sj , Tj , T racej (j =
1, . . . , k − 1) and for copying the read code into modifiable storage U , where pk may further edit the code,
and execute the result, which may be a useful subprogram [41].
Define a probability distribution P (p) on P to represent the searcher’s initial bias (more likely programs
p will be tested earlier [17]). P could be based on program length, e.g., P (p) = 2−L(p) , or on a probabilistic
syntax diagram [41, 40]. See Algorithm 4.1.
OOPS keeps doubling the time limit until there is sufficient runtime for a sufficiently likely program
to compute a novel, previously unsolvable task, plus its solver, which provably does not forget previ-
ous solutions. OOPS allocates time to programs according to an asymptotically optimal universal search
method [17] for problems with easily verifiable solutions, that is, solutions whose validity can be quickly
tested. Given some problem class, if some unknown optimal program p requires f (k) steps to solve a
problem instance of size k and demonstrate the correctness of the result, then this search method will need
at most O(f (k)/P (p)) = O(f (k)) steps—the constant factor 1/P (p) may be large but does not depend
on k. Since OOPS may re-use previously generated solutions and solution-computing programs, however,
it may be possible to greatly reduce the constant factor associated with plain universal search [41].
The big difference to previous implementations of OOPS is that P OWER P LAY has the additional free-
dom to define its own tasks. As always, every new task added to the repertoire is essentially defined by the
time required to invent it, to solve it, and to demonstrate that no previously learned skills got lost.
4.1.1 Building on Existing OOPS Source Code

Existing OOPS source code [40] uses a FORTH-like universal programming language to define P. It
already contains a framework for testing new code on previously solved tasks, and for efficiently undoing
all U -modifications of each tested program. The source code will require a few changes to implement the
additional task search described above.
4.1.2 Alternative Problem Solvers Based on Recurrent Neural Networks

Recurrent NNs (RNNs, e.g., [59, 61, 27, 32, 12]) are general computers that allow for both sequential
and parallel computations, unlike the strictly sequential FORTH-like language of Section 4.1.1. They can
compute any function computable by a standard PC [29]. The original report [46] used a fully connected
RNN called RNN1 to define S, where wlk is the real-valued weight on the directed connection between the
l-th and k-th neuron. To program RNN1 means to set the weight matrix s = hwlk i. Given enough neurons
with appropriate activation functions and an appropriate hwlk i, Algorithm 4.1 can be used to train s. P
may itself be the set of weight matrices of a separate RNN called RNN2, computing tasks for RNN1, and
modifications of RNN1, using techniques for network-modifying networks as described in previous work
[33, 35, 34].
In first experiments [53, 52], a particularly suited NN called a self-delimiting NN or SLIM NN [47] is
used. During program execution or activation spreading in the SLIM NN, lists are used to trace only those
neurons and connections used at least once. This also allows for efficient resets of large NNs which may use
only a small fraction of their weights per task. Unlike standard RNNs, SLIM NNs are easily combined with
techniques of asymptotically optimal program search [17, 48, 39, 41] (Section 4.1). To address overfitting,
instead of depending on pre-wired regularizers and hyper-parameters [3], SLIM NNs can in principle learn
to select by themselves their own runtime and their own numbers of free parameters, becoming fast and
slim when necessary. Efficient SLIM NN learning algorithms (LAs) track which weights are used for which
tasks (Section 3.3.2), to greatly speed up performance evaluations in response to limited weight changes.
LAs may penalize the task-specific total length of connections used by SLIM NNs implemented on the
10
3-dimensional brain-like multi-processor hardware to expected in the future. This encourages SLIM NNs
to solve many subtasks by subsets of neurons that are physically close [47].
4.2 Adapting the Probability Distribution on Programs

A straightforward extension of the above works as follows: Whenever a new pi is found, P is updated to
make either only pi or all p1 , p2 , . . . , pi more likely. Simple ways of doing this are described in previous
work [48]. This may be justified to the extent that future successful programs turn out to be similar to
previous ones.
4.3 Implementation Based on Stochastic or Evolutionary Search

A possibly simpler but less general approach is to use an evolutionary algorithm to produce an s-modifying
and task-generating program p as requested by P OWER P LAY, according to Algorithm 4.3, which refers to
the recurrent net problem solver of Section 4.1.2.
Alg. 4.3: P OWER P LAY for RNNs Using Stochastic or Evolutionary Search
Randomly initialize RNN1’s variable weight matrix hwlk i and use the result as s0 (see Section 4.1.2)
for i := 1, 2, . . . do
set Boolean variable DONE=FALSE
repeat
use a black box optimization algorithm BBOA (many are possible [24, 10, 60, 49]) with adaptive
parameter vector θ to create some T ∈ T (to define the task input to RNN1; see Section 3.1) and a
modification of si−1 , the current hwlk i of RNN1, thus obtaining a new candidate q ∈ S
if q but not si−1 can solve T and all Tk (k < i) (see Sections 3.3, 3.3.2) then
set DONE=TRUE
end if
until DONE
set si := q; hwlk i := q; Ti := T ; (also store T racei if applicable, see Section 3.1.2). Use the
information stored so far to adapt the parameters θ of the BBOA, e.g., by gradient-based search [60,
49], or according to the principles of evolutionary computation [24, 10, 60].
end for
5 Outgrowing Trivial Tasks - Compressing Previous Solutions

What prevents P OWER P LAY from inventing trivial tasks forever by extreme modularization, simply allo-
cating a previously unused solver part to each new task, which thus becomes rather quickly verifiable, as its
solution does not affect solutions to previous tasks (Section 3.3.3)? On realistic but general architectures
such as PCs and RNNs, at least once the upper storage size limit of s is reached, P OWER P LAY will start
“compressing” previous solutions, making s generalize in the sense that the same relatively short piece of
code (some part of s) helps to solve different tasks.
With many computational architectures, this type of compression will start much earlier though, be-
cause new tasks solvable by partial reuse of earlier discovered code will often be easier to find than new
tasks solvable by previously unused parts of s. This also holds for growing architectures with potentially
unlimited storage space.
Compare also P OWER P LAY Variant II of Section 7.1 whose tasks may explicitly require improving the
average time and space complexity of previous solutions by some minimal value.
In general, however, over time the system will find it more and more difficult to invent novel tasks
without forgetting previous solutions, a bit like humans find it harder and harder to learn truly novel behav-
iors once they are leaving behind the initial rapid exploration phase typical for babies. Experiments with
various problem solver architectures (e.g., [53, 52]) are needed to analyze such effects in detail.
11
6 Adding External Tasks
The growing repertoire of the problem solver may facilitate learning of solutions to externally posed tasks.
For example, one may modify P OWER P LAY such that for certain i, Ti is defined externally, instead of being
invented by the system itself. In general, the resulting si will contain an externally inserted bias in form
of code that will make some future self-generated tasks easier to find than others. It should be possible to
push the system in a human-understandable or otherwise useful direction by regularly inserting appropriate
external goals. See Algorithm 7.1.
Another way of exploiting the growing repertoire is to simply copy si for some i and use it as a starting
point for a search for a solution to an externally posed task T , without insisting that the modified si also
can solve T1 , T2 , . . . , Ti . This may be much faster than trying to solve T from scratch, to the extent the
solutions to self-generated tasks reflect general knowledge (code) re-usable for T .
In general, however, it will be possible to design external tasks whose solutions do not profit from those
of self-generated tasks—the latter even may turn out to slow down the search.
On the other hand, in the real world the benefits of curious exploration seem obvious. One should
analyze theoretically and experimentally under which conditions the creation of self-generated tasks can
accelerate the solution to externally generated tasks—see [30, 54, 37, 38, 19, 4, 28, 62] for previous simple
experimental studies in this vein.
6.1 Self-Reference Through Novel Task Search as an External Task

P OWER P LAY’s i-th goal is to find a pi ∈ P that creates Ti and si (a modification of si−1 ) and shows that
si but not si−1 can solve T≤i . As s itself is becoming a more and more general problem solver, s may
help in many ways to achieve such goals in self-referential fashion. For example, the old solver si−1 may
be able to read a unique formal description (provided by pi ) of P OWER P LAY’s i-th goal, viewing it as an
external task, and produce an output unambiguously describing a candidate for (Ti , si ). If s has a theorem
prover component (Section 3.3.1), si−1 might even output a full proof of (Ti , si )’s validity; alternatively pi
could just use the possibly suboptimal suggestions of si−1 to narrow down and speed up the search, one of
the reasons why Section 2 already mentioned that programs p ∈ P should contain instructions for reading
(and running) the code of the present problem solver.
7 Softening Task Acceptance Criteria of P OWER P LAY

The P OWER P LAY variants above insist that s may not solve new tasks at the expense of forgetting to
solve any previously solved task within its previously established time and space bounds. For example,
let us consider the sequential decision-making tasks from Section 3.1.2. Suppose the problem solver can
already solve task Tk = (Ik , Jk , tk , nk ) ∈ I × J × N × N. A very similar but admissible new task
Ti = (Ik , Jk , ti , nk ), (i > k), would be to solve Tk substantially faster: ti < tk − ǫ, as long as Ti is not
already solvable by si−1 , and no solution to some Tl (l < i) is forgotten in the process.
Here I discuss variants of P OWER P LAY that soften the acceptance criteria for new tasks in various
ways, for example, by allowing some of the computations of solutions to previous non-external (Section 6)
tasks to slow down by a certain amount of time, provided the sum of their runtimes does not decrease. This
also permits the system to invent new previously unsolved tasks at the expense of slightly increasing time
bounds for certain already solved non-external tasks, but without decreasing the average performance on
the latter. Of course, P OWER P LAY has to be modified accordingly, updating average runtime bounds when
necessary.
Alternatively, one may allow for trading off space and time constraints in reasonable ways, e.g., in the
style of asymptotically optimal Universal Search [17], which essentially trades one bit of additional space
complexity for a runtime speedup factor of 2.
12
7.1 P OWER P LAY Variant II: Explicitly Penalizing Time and Space Complexity
Let us remove time and space bounds from the task definitions of Section 3.1.2, since the modified cost-
based P OWER P LAY framework below (Algorithm 7.1) will handle computational costs (such as time and
space complexity of solutions) more directly. In the present section, Ti encodes a tuple (Ii , Ji ) ∈ I × J
with interpretation: si must first read Ii and then interact with an environment through a sequence of
perceptions and actions, to achieve some computable goal defined by Ji within a certain maximal time
interval tmax (a positive constant). Let t′s (T ) be tmax if s cannot solve task T , otherwise it is the time
needed to solve T by s. Let ls′ (T ) be the positive constant lmax if s cannot solve T , otherwise it is the
number of components of s needed to solve task T by s. The non-negative real-valued reward r(T ) for
solving T is a positive constant rnew for self-defined previously unsolvable T , or user-defined if T is
an external task solved by s (Section 6). The real-valued cost Cost(s, T SET ) of solving all tasks in
′ ′
Pset T SET through s is a real-valued function of: all ls (T ), ts (T ) (for all TP∈ T SET ),′ L(s),
a task
and T ∈T SET r(T ). For example, the cost function Cost(s, T SET ) = L(s) + α T ∈T SET [ts (T ) −
r(T )] encourages compact and fast solvers solving many different tasks with the same components of s,
where the real-valued positive parameter α weighs space costs against time costs, and rnew should exceed
tmax to encourage solutions of novel self-generated tasks, whose cost contributions should be below zero
(alternative cost definitions could also take into account energy consumption etc.)
Let us keep an analogue of the remaining notation of Section 3.1.2, such as ui (t), xi (t), ri (t), yi (t), T racei ,
Ji (T racei ). As always, if the environment is unknown and possibly changing over time, to test perfor-
mance of a new solver s on a previous task Tk , only T racek is necessary—see Section 3.1.2. As always,
let T≤i denote the set containing all tasks T1 , . . . , Ti (note that if Ti =Tk for some k < i then it will appear
only once in T≤i ), and let ǫ > 0 again define what’s acceptable progress:
Alg. 7.1: P OWER P LAY Framework (Variant II) Explicitly Handling Costs of Solving Tasks
Initialize s0 in some way

for i := 1, 2, . . . do
Create new global variables Ti ∈ T , si ∈ S, pi ∈ P, ci , c∗i ∈ R (to be fixed by the end of repeat)
repeat
Let a search algorithm (Section 4.1) set pi , a new candidate program. Give pi limited time to do:
* TASK I NVENTION: Unless the user specifies Ti (Section 6), let pi set Ti .
* S OLVER M ODIFICATION: Let pi set si by computing a modification of si−1 (Section 3.2).
* C ORRECTNESS D EMONSTRATION: Let pi compute ci := Cost(si , T≤i ) and c∗i :=
Cost(si−1 , T≤i )
until c∗i − ci > ǫ (minimal savings of costs such as time/space/etc on all tasks so far)
Freeze/store forever pi , Ti , si , ci , c∗i
end for
By Algorithm 7.1, si may forget certain abilities of si−1 , provided that the overall performance as
measured by Cost(si , T≤i ) has improved, either because a new task became solvable, or previous tasks
became solvable more efficiently.
Following Section 3.3, C ORRECTNESS D EMONSTRATION can often be facilitated, for example, by
tracking which components of si are used for solving which tasks (Section 3.3.2).
To further refine this approach, consider that in phase i, the list Lki (defined in Section 3.3.2) contains
all previously learned tasks whose solutions dependP on sk . This can be used to determine the current value
V al(ski ) of some component sk of s: V al(ski ) = − T ∈Lk Cost(si , T≤i ). It is a simple exercise to invent
i
P OWER P LAY variants that do not forget valuable components as easily as less valuable ones.
The implementations of Sections 4.1 and 4.3 are easily adapted to the cost-based P OWER P LAY frame-
work. Compare separate papers [53, 52].
13
7.2 Probabilistic P OWER P LAY Variants
Section 3.1.2 pointed out that in partially observable and/or non-stationary unknown environments C OR -
RECTNESS D EMONSTRATION must use T racek to check whether a new si still knows how to solve an
earlier task Tk (k < i). A less strict variant of P OWER P LAY, however, will simply make certain assump-
tions about the probabilistic nature of the environment and the repeatability of trials, assuming that a limited
fixed number of interactions with the real world are sufficient to estimate the costs c∗i , ci in Algorithm 7.1.
Another probabilistic way of softening P OWER P LAY is to add new tasks without proof that s won’t
forget solutions to previous tasks, provided C ORRECTNESS D EMONSTRATION can at least show that the
probability of forgetting any previous solution is below some real-valued positive constant threshold.
8 First Illustrative Experiments

First experiments are reported in separate papers [53, 52] (some experiments were also briefly mentioned
in the original report [46]). Standard NNs as well as SLIM RNNs [47] are used as computational problem
solving architectures. The weights of SLIM RNNs can encode essentially arbitrary computable tasks as
well as arbitrary, self-delimiting, halting or non-halting programs solving those tasks. These programs may
affect both environment (through effectors) and internal states encoding abstractions of event sequences.
In open-ended fashion, the P OWER P LAY-driven NNs learn to become increasingly general solvers of self-
invented problems, continually adding new problem solving procedures to the growing repertoire, some-
times compressing/speeding up previous skills, sometimes preferring to invent new tasks and correspond-
ing skills. The NNs exhibit interesting developmental stages. It is also shown how a P OWER P LAY-driven
SLIM NN automatically self-modularizes [52], frequently re-using code for previously invented skills, al-
ways trying to invent novel tasks that can be quickly validated because they do not require too many weight
changes affecting too many previous tasks.
9 Previous Relevant Work

Here I discuss related research, in particular, why the present work is of interest despite the recent advent
of theoretically optimal universal problem solvers (Section 9.1), and how it can be viewed as a greedy but
feasible and sound implementation of the formal theory of creativity (Section 9.3).
9.1 Existing Theoretically Optimal Universal Problem Solvers

The new millennium brought universal problem solvers that are theoretically optimal in a certain sense.
The fully self-referential [9] Gödel machine [43, 44] may interact with some initially unknown, partially
observable environment to maximize future expected utility or reward by solving arbitrary user-defined
computational tasks. Its initial algorithm is not hardwired; it can completely rewrite itself without essential
limits apart from the limits of computability, but only if a proof searcher embedded within the initial
algorithm can first prove that the rewrite is useful, according to the formalized utility function taking
into account the limited computational resources. Self-rewrites due to this approach can be shown to be
globally optimal, relative to Gödel’s well-known fundamental restrictions of provability [9]. To make sure
the Gödel machine is at least asymptotically optimal even before the first self-rewrite, one may initialize it
by Hutter’s non-self-referential but asymptotically fastest algorithm for all well-defined problems Hsearch
[13], which uses a hardwired brute force proof searcher and ignores the costs of proof search. Assuming
discrete input/output domains X/Y ⊂ B ∗ , a formal problem specification f : X → Y (say, a functional
description of how integers are decomposed into their prime factors), and a particular x ∈ X (say, an
integer to be factorized), Hsearch orders all proofs of an appropriate axiomatic system by size to find
programs q that for all z ∈ X provably compute f (z) within time bound tq (z). Simultaneously it spends
most of its time on executing the q with the best currently proven time bound tq (x). Hsearch is as fast as
the fastest algorithm that provably computes f (z) for all z ∈ X, save for a constant factor smaller than
1+ǫ (arbitrarily small real-valued ǫ > 0) and an f -specific but x-independent additive constant [13]. Given
14
some problem, the Gödel machine may decide to replace Hsearch by a faster method suffering less from
large constant overhead, but even if it doesn’t, its performance won’t be less than asymptotically optimal.
Why doesn’t everybody use such universal problem solvers for all computational real-world problems?
Because most real-world problems are so small that the ominous constant slowdowns (potentially relevant
at least before the first self-rewrite) may be large enough to prevent the universal methods from being
feasible.
P OWER P LAY, on the other hand, is designed to incrementally build a practical more and more general
problem solver that can solve numerous tasks quickly, not in the asymptotic sense, but by exploiting to
the max its given particular search algorithm and computational architecture, with all its space and time
limitations, including those reflected by constants ignored by the asymptotic optimality notation.
As mentioned in Section 6, however, one must now analyze under which conditions P OWER P LAY’s
self-generated tasks can accelerate the solution to externally generated tasks (compare previous experi-
mental studies of this type [30, 54, 37, 38]).
9.2 Connection to Traditional Active Learning

Traditional active learning methods [6] such as AdaBoost [8] have a totally different set-up and purpose:
there the user provides a set of samples to be learned, then each new classifier in a series of classifiers
focuses on samples badly classified by previous classifiers. Open-ended P OWER P LAY, however, (1) con-
siders arbitrary computational problems (not necessarily classification tasks); (2) can self-invent all com-
putational tasks. There is no need for a pre-defined global set of tasks that each new solver tries to solve
better, instead the task set continually grows based on which task is easy to invent and validate, given what
is already known.
9.3 Greedy Implementation of Aspects of the Formal Theory of Creativity

The Formal Theory of Creativity [42, 45] considers agents living in initially unknown environments. At any
given time, such an agent uses a reinforcement learning (RL) method [15] to maximize not only expected
future external reward for achieving certain goals, but also intrinsic reward for improving an internal model
of the environmental responses to its actions, learning to better predict or compress the growing history of
observations influenced by its behavior, thus achieving wow-effects, actively learning skills to influence the
input stream such that it contains previously unknown but learnable algorithmic regularities. I have argued
that the theory explains essential aspects of intelligence including selective attention, curiosity, creativity,
science, art, music, humor, e.g., [42, 45]. Compare recent related work, e.g., [1, 5, 22, 20].
Like P OWER P LAY, such a creative agent produces a sequence of self-generated tasks and their solu-
tions, each task still unsolvable before learning, yet becoming solvable after learning. The costs of learning
as well as the learning progress are measured, and enter the reward function. Thus, in the absence of ex-
ternal reward for reaching user-defined goals, at any given time the agent is motivated to invent a series of
additional tasks that maximize future expected learning progress.
For example, by restricting its input stream to self-generated pairs (I, O) ∈ I × O like in Section 3.1.1,
and limiting it to predict only O, given I (instead of also trying to predict future (I, O) pairs from previous
ones, which the general agent would do), there will be a motivation to actively generate a sequence of
(I, O) pairs such that the O are first subjectively unpredictable from their I but then become predictable
with little effort, given the limitations of whatever learning algorithm is used.
Here some cons and pros of P OWER P LAY are listed in light of the above. Its drawbacks include:
1. Instead of maximizing future expected reward, P OWER P LAY is greedy, always trying to find the
simplest (easiest to find and validate) task to add to the repertoire, or the simplest way of improving
the efficiency or compressibility of previous solutions, instead of looking further ahead, as a universal
RL method [42, 45] would do. That is, P OWER P LAY may potentially sacrifice large long-term gains
for small short-term gains: the discovery of many easily solvable tasks may at least temporarily
prevent it from learning to solve hard tasks.
On general computational architectures such as RNNs (Section 4.1.2), however, P OWER P LAY is
expected to soon run out of easy tasks that are not yet solvable, due to the architecture’s limited
15
capacity and its unavoidable generalization effects (many never-tried tasks will become solvable by
solutions to the few explicitly tested Ti ). Compare Section 5.
2. The general creative agent above [42, 45] is motivated to improve performance on the entire history
of previous still unsolved tasks, while P OWER P LAY may discard much of this history, keeping only a
selective list of previously solved tasks. However, as the system is interacting with its environment,
one could store the entire continually growing history, and make sure that T always allows for
defining the task of better compressing the history so far.
3. P OWER P LAY as in Section 2 has a binary criterion for adding knowledge (was the new task solv-
able without forgetting old solutions?), while the general agent [42, 45] uses a more informative
information-theoretic measure. The cost-based P OWER P LAY framework (Alg. 7.1) of Section 7,
however, offers similar, more flexible options, rewarding compression or speedup of solutions to
previously solved tasks.
On the other hand, drawbacks of previous implementations of formal creativity theory include:
1. Some previous approximative implementations [30, 54] used traditional RL methods [15] with the-
oretically unlimited look-ahead, but those are not guaranteed to work well in partially observable
and/or non-stationary environments where the reward function changes over time, and won’t neces-
sarily generate an optimal sequence of future tasks or experiments.
2. Theoretically optimal implementations [42, 45] are currently still impractical, for reasons similar to
those discussed in Section 9.1.
Hence P OWER P LAY may be viewed as a greedy but feasible implementation of certain basic principles
of creativity [42, 45]. P OWER P LAY-based systems are continually motivated to invent new tasks solvable
by formerly unknown procedures, or to compress or speed up problem solving procedures discovered
earlier. Unlike previous implementations, P OWER P LAY extracts from the lifelong experience history a
sequence of clearly identified and separated tasks with explicitly recorded solutions. By design it cannot
suffer from online learning problems affecting its solver’s performance on previously solved problems.
9.4 Beyond Algorithmic Zero-Sum Games [37, 38] (1997-2002)

This guaranteed robustness against forgetting previous skills also represents a difference to the most closely
related previous work [37, 38]. There, to address the computational costs of learning, and the costs of mea-
suring learning progress, computationally powerful encoders and problem solvers [36, 38] (1997-2002)
are implemented as two very general, co-evolving, symmetric, opposing modules called the right brain
and the left brain. Both are able to construct self-modifying probabilistic programs written in a universal
programming language. An internal storage for temporary computational results of the programs is viewed
as part of the changing environment. Each module can suggest experiments in the form of probabilistic
algorithms to be executed, and make predictions about their effects, betting intrinsic reward on their out-
comes. The opposing module may accept such a bet in a zero-sum game by making a contrary prediction,
or reject it. In case of acceptance, the winner is determined by executing the experiment and checking its
outcome; the intrinsic reward eventually gets transferred from the surprised loser to the confirmed winner.
Both modules try to maximize reward using a rather general RL algorithm (the so-called success-story
algorithm SSA [48]) designed for complex stochastic policies (alternative RL algorithms could be plugged
in as well). Thus both modules are motivated to discover novel algorithmic patterns/compressibility (=
surprising wow-effects), where the subjective baseline for novelty is given by what the opponent already
knows about the (external or internal) world’s repetitive patterns. Since the execution of any computational
or physical action costs something (as it will reduce the cumulative reward per time ratio), both modules are
motivated to focus on those parts of the dynamic world that currently make surprises and learning progress
easy, to minimize the costs of identifying promising experiments and executing them. The system learns a
partly hierarchical structure of more and more complex skills or programs necessary to solve the growing
sequence of self-generated tasks, reusing previously acquired simpler skills where this is beneficial. Exper-
imental studies exhibit several sequential stages of emergent developmental sequences, with and without
external reward [37, 38].
16
However, the previous system [37, 38] did not have a built-in guarantee that it cannot forget previously
learned skills, while P OWER P LAY as in Section 2 does (and the time and space complexity-based variant
Alg. 7.1 of Section 7 can forget only if this improves the average efficiency of previous solutions).
To analyze the novel framework’s consequences in practical settings, experiments are currently being
conducted with various problem solver architectures with different generalization properties. See separate
papers [53, 52] and Section 8.
9.5 Opposing Forces: Improving Generalization Through Compression, Breaking

Generalization Through Novelty
Two opposing forces are at work in P OWER P LAY. On the one hand, the system continually tries to improve
previously learned skills, by speeding them up, and by compressing the used parameters of the problem
solver, reducing its effective size. The compression drive tends to improve generalization performance,
according to the principles of Occam’s Razor and Minimum Description Length (MDL) and Minimum
Message Length (MML) [50, 16, 57, 58, 51, 26, 18, 14]. On the other hand, the system also continually
tries to invent new tasks that break the generalization capabilities of the present solver.
P OWER P LAY’s time-minimizing search for new tasks automatically manages the trade-off between
these opposing forces. Sometimes it is easier (because fewer computational resources are required) to
invent and solve a completely new, previously unsolvable problem. Sometimes it is easier to compress (or
speed up) solutions to previously invented problems.
9.6 Relation to Gödel’s Sequence of Increasingly Powerful Axiomatic Systems

In 1931, Kurt Gödel showed that for each sufficiently powerful (ω-) consistent axiomatic system there is a
statement that must be true but cannot be proven from the axioms through an algorithmic theorem-proving
procedure [9]. This unprovable statement can then be added to the axioms, to obtain a more powerful
formal theory in which new formerly unprovable theorems become provable, without affecting previously
provable theorems.
In a sense, P OWER P LAY is doing something similar. Assume the architecture of the solver is a universal
computer [9, 56]. Its software s can be viewed as a theorem-proving procedure implementing certain enu-
merable axioms and computable inference rules. P OWER P LAY continually tries to modify s such that the
previously proven theorems remain provable within certain time bounds, and a new previously unprovable
theorem becomes provable.
10 Words of Caution
The behavior of P OWER P LAY is determined by the nature and the limitations of T , S, P, and its algorithm
for searching P. If T includes all computable task descriptions, and both S and P allow for implementing
arbitrary programs, and the search algorithm is a general method for search in program space (Section 4),
then there are few limits to what P OWER P LAY may do (besides the limits of computability [9]).
It may not be advisable to let a general variant of P OWER P LAY loose in an uncontrolled situation,
e.g., on a multi-computer network on the internet, possibly with access to control of physical devices, and
the potential to acquire additional computational and physical resources (Section 3.1.2) through programs
executed during P OWER P LAY. Unlike, say, traditional virus programs, P OWER P LAY-based systems will
continually change in a way hard to predict, incessantly inventing and solving novel, self-generated tasks,
only driven by a desire to increase their general problem-solving capacity, perhaps a bit like many humans
seek to increase their power once their basic needs are satisfied. This type of artificial curiosity/creativity,
however, may conflict with human intentions on occasion. On the other hand, unchecked curiosity may
sometimes also be harmful or fatal to the learning system itself (Section 6)—curiosity can kill the cat.
17
11 Acknowledgments
Thanks to Mark Ring, Bas Steunebrink, Faustino Gomez, Sohrob Kazerounian, Hung Ngo, Leo Pape,
Giuseppe Cuccu, for useful comments.
References
[1] A. Barto. Intrinsic motivation and reinforcement learning. In G. Baldassarre and M. Mirolli, editors,
Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, 2012. In press.
[2] D. E. Berlyne. A theory of human curiosity. British Journal of Psychology, 45:180–191, 1954.
[3] C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[4] G. Cuccu, M. Luciw, J. Schmidhuber, and F. Gomez. Intrinsically motivated evolutionary search for
vision-based reinforcement learning. In Proceedings of the 2011 IEEE Conference on Development
and Learning and Epigenetic Robotics IEEE-ICDL-EPIROB. IEEE, 2011.
[5] P. Dayan. Exploration from generalization mediated by multiple controllers. In G. Baldassarre and
M. Mirolli, editors, Intrinsically Motivated Learning in Natural and Artificial Systems. Springer,
2012. In press.
[6] V. V. Fedorov. Theory of optimal experiments. Academic Press, 1972.
[7] M. C. Fitting. First-Order Logic and Automated Theorem Proving. Graduate Texts in Computer
Science. Springer-Verlag, Berlin, 2nd edition, 1996.
[8] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an applica-
tion to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997.
[9] K. Gödel. Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I.
Monatshefte für Mathematik und Physik, 38:173–198, 1931.
[10] F. J. Gomez, J. Schmidhuber, and R. Miikkulainen. Efficient non-linear control through neuroevolu-
tion. Journal of Machine Learning Research JMLR, 9:937–965, 2008.
[11] H. F. Harlow, M. K. Harlow, and D. R. Meyer. Novelty and curiosity as determinants of exploratory
behavior. Journal of Experimental Psychology, 41:68–80, 1950.
[12] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780,
1997.
[13] M. Hutter. The fastest and shortest algorithm for all well-defined problems. International Journal of
Foundations of Computer Science, 13(3):431–443, 2002. (On J. Schmidhuber’s SNF grant 20-61847).
[14] M. Hutter. Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability.
Springer, Berlin, 2005. (On J. Schmidhuber’s SNF grant 20-61847).
[15] L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: a survey. Journal of AI
research, 4:237–285, 1996.
[16] A. N. Kolmogorov. Three approaches to the quantitative definition of information. Problems of
Information Transmission, 1:1–11, 1965.
[17] L. A. Levin. Universal sequential search problems. Problems of Information Transmission, 9(3):265–
266, 1973.
[18] M. Li and P. M. B. Vitányi. An Introduction to Kolmogorov Complexity and its Applications (2nd
edition). Springer, 1997.
18
[19] M. Luciw, V. Graziano, M. Ring, and J. Schmidhuber. Artificial curiosity with planning for au-
tonomous perceptual and cognitive development. In Proceedings of the First Joint Conference on
Development Learning and on Epigenetic Robotics ICDL-EPIROB, Frankfurt, August 2011.
[20] U. Nehmzow, Y. Gatsoulis, E. Kerr, J. Condell, N. H. Siddique, and T. M. McGinnity. Novelty

detection as an intrinsic motivation for cumulative learning robots. In G. Baldassarre and M. Mirolli,
editors, Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, 2012. In press.
[21] A. Newell and H. Simon. GPS, a program that simulates human thought. In E. Feigenbaum and
J. Feldman, editors, Computers and Thought, pages 279–293. McGraw-Hill, New York, 1963.
[22] P.-Y. Oudeyer, A. Baranes, and F. Kaplan. Intrinsically motivated learning of real world sensori-
motor skills with developmental constraints. In G. Baldassarre and M. Mirolli, editors, Intrinsically
Motivated Learning in Natural and Artificial Systems. Springer, 2012. In press.
[23] J. Piaget. The Child’s Construction of Reality. London: Routledge and Kegan Paul, 1955.
[24] I. Rechenberg. Evolutionsstrategie - Optimierung technischer Systeme nach Prinzipien der biologis-
chen Evolution. Dissertation, 1971. Published 1973 by Fromman-Holzboog.
[25] M. B. Ring. Continual Learning in Reinforcement Environments. PhD thesis, University of Texas at
Austin, Austin, Texas 78712, August 1994.
[26] J. Rissanen. Modeling by shortest data description. Automatica, 14:465–471, 1978.
[27] A. J. Robinson and F. Fallside. The utility driven dynamic error propagation network. Technical
Report CUED/F-INFENG/TR.1, Cambridge University Engineering Department, 1987.
[28] T. Schaul, Yi Sun, D. Wierstra, F. Gomez, and J. Schmidhuber. Curiosity-Driven Optimization. In
IEEE Congress on Evolutionary Computation (CEC), New Orleans, USA, 2011.
[29] J. Schmidhuber. Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem.
Dissertation, Institut für Informatik, Technische Universität München, 1990.
[30] J. Schmidhuber. Curious model-building control systems. In Proceedings of the International Joint
Conference on Neural Networks, Singapore, volume 2, pages 1458–1463. IEEE press, 1991.
[31] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural
controllers. In J. A. Meyer and S. W. Wilson, editors, Proc. of the International Conference on
Simulation of Adaptive Behavior: From Animals to Animats, pages 222–227. MIT Press/Bradford
Books, 1991.
[32] J. Schmidhuber. A fixed size storage O(n3 ) time complexity learning algorithm for fully recurrent
continually running networks. Neural Computation, 4(2):243–248, 1992.
[33] J. Schmidhuber. Learning to control fast-weight memories: An alternative to recurrent nets. Neural
Computation, 4(1):131–139, 1992.
[34] J. Schmidhuber. On decreasing the ratio between learning complexity and number of time-varying
variables in fully recurrent nets. In Proceedings of the International Conference on Artificial Neural
Networks, Amsterdam, pages 460–463. Springer, 1993.
[35] J. Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference on
Artificial Neural Networks, Amsterdam, pages 446–451. Springer, 1993.
[36] J. Schmidhuber. What’s interesting? Technical Report IDSIA-35-97, IDSIA, 1997.
ftp://ftp.idsia.ch/pub/juergen/interest.ps.gz; extended abstract in Proc. Snowbird’98, Utah, 1998; see
also [38].
19
[37] J. Schmidhuber. Artificial curiosity based on discovering novel algorithmic predictability through co-
evolution. In P. Angeline, Z. Michalewicz, M. Schoenauer, X. Yao, and Z. Zalzala, editors, Congress
on Evolutionary Computation, pages 1612–1618. IEEE Press, 1999.
[38] J. Schmidhuber. Exploring the predictable. In A. Ghosh and S. Tsuitsui, editors, Advances in Evolu-
tionary Computing, pages 579–612. Springer, 2002.
[39] J. Schmidhuber. Bias-optimal incremental problem solving. In S. Becker, S. Thrun, and K. Ober-
mayer, editors, Advances in Neural Information Processing Systems 15 (NIPS 15), pages 1571–1578,
Cambridge, MA, 2003. MIT Press.
[40] J. Schmidhuber. OOPS source code in crystalline format: http://www.idsia.ch/˜juergen/oopscode.c,
2004.
[41] J. Schmidhuber. Optimal ordered problem solver. Machine Learning, 54:211–254, 2004.
[42] J. Schmidhuber. Developmental robotics, optimal artificial curiosity, creativity, music, and the fine
arts. Connection Science, 18(2):173–187, 2006.
[43] J. Schmidhuber. Gödel machines: Fully self-referential optimal universal self-improvers. In B. Go-
ertzel and C. Pennachin, editors, Artificial General Intelligence, pages 199–226. Springer Verlag,
2006. Variant available as arXiv:cs.LO/0309048.
[44] J. Schmidhuber. Ultimate cognition à la Gödel. Cognitive Computation, 1(2):177–193, 2009.
[45] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans-
actions on Autonomous Mental Development, 2(3):230–247, 2010.
[46] J. Schmidhuber. P OWER P LAY: Training an Increasingly General Problem Solver by Continually
Searching for the Simplest Still Unsolvable Problem. Technical Report arXiv:1112.5309v1 [cs.AI],
2011.
[47] J. Schmidhuber. Self-delimiting neural networks. Technical Report IDSIA-08-12, arXiv:1210.0118v1
[cs.NE], IDSIA, 2012.
[48] J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adap-
tive Levin search, and incremental self-improvement. Machine Learning, 28:105–130, 1997.
[49] F. Sehnke, C. Osendorfer, T. Rückstieß, A. Graves, J. Peters, and J. Schmidhuber. Parameter-exploring

policy gradients. Neural Networks, 23(4):551–559, 2010.
[50] R. J. Solomonoff. A formal theory of inductive inference. Part I. Information and Control, 7:1–22,
1964.
[51] R. J. Solomonoff. Complexity-based induction systems. IEEE Transactions on Information Theory,
IT-24(5):422–432, 1978.
[52] R. K. Srivastava, B. R. Steunebrink, and J. Schmidhuber. First Experiments with P OWER P LAY.
Technical Report arXiv:1210.8385v1 [cs.AI], 2012.
[53] R. K. Srivastava, B. R. Steunebrink, M. Stollenga, and J. Schmidhuber. Continually adding self-
invented problems to the repertoire: First experiments with POWERPLAY. In Proceedings of the
2012 IEEE Conference on Development and Learning and Epigenetic Robotics IEEE-ICDL-EPIROB.
IEEE, 2012.
[54] J. Storck, S. Hochreiter, and J. Schmidhuber. Reinforcement driven information acquisition in non-
deterministic environments. In Proceedings of the International Conference on Artificial Neural Net-
works, Paris, volume 2, pages 159–164. EC2 & Cie, 1995.
20
[55] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge,
MA, 1998.
[56] A. M. Turing. On computable numbers, with an application to the Entscheidungsproblem. Proceed-
ings of the London Mathematical Society, Series 2, 41:230–267, 1936.
[57] C. S. Wallace and D. M. Boulton. An information theoretic measure for classification. Computer
Journal, 11(2):185–194, 1968.
[58] C. S. Wallace and P. R. Freeman. Estimation and inference by compact coding. Journal of the Royal
Statistical Society, Series ”B”, 49(3):240–265, 1987.
[59] P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model.
Neural Networks, 1, 1988.
[60] D. Wierstra, T. Schaul, J. Peters, and J. Schmidhuber. Natural evolution strategies. In Congress of
Evolutionary Computation (CEC 2008), 2008.
[61] R. J. Williams and D. Zipser. Gradient-based learning algorithms for recurrent networks and their
computational complexity. In Back-propagation: Theory, Architectures and Applications. Hillsdale,
NJ: Erlbaum, 1994.
[62] S. Yi, F. Gomez, and J. Schmidhuber. Planning to be surprised: Optimal Bayesian exploration in
dynamic environments. In Proc. Fourth Conference on Artificial General Intelligence (AGI), Google,
Mountain View, CA, 2011.
21

P P: Training An Increasingly General Problem Solver by Continually Searching For The Simplest Still Unsolvable Problem

Uploaded by

Copyright:

Available Formats

P P: Training An Increasingly General Problem Solver by Continually Searching For The Simplest Still Unsolvable Problem

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

P P: Training An Increasingly General Problem Solver by Continually Searching For The Simplest Still Unsolvable Problem

Uploaded by

Copyright:

Available Formats

P OWER P LAY: Training an Increasingly

General Problem Solver by Continually Searching

2 Notation & Algorithmic Framework P OWER P LAY (Variant I) 5

3 TASK I NVENTION, S OLVER M ODIFICATION, C ORRECTNESS D EMO 5

4 Implementations of P OWER P LAY 9

5 Outgrowing Trivial Tasks - Compressing Previous Solutions 11

6 Adding External Tasks 12

7 Softening Task Acceptance Criteria of P OWER P LAY 12

8 First Illustrative Experiments 14

9 Previous Relevant Work 14

1.1 Basic Ideas

1.2 Outline of Remainder

3 TASK I NVENTION, S OLVER M ODIFICATION, C ORRECTNESS D EMO

3.1 Implementing TASK I NVENTION

3.1.1 Example: Pattern Recognition Tasks

3.1.2 Example: General Decision Making Tasks in Dynamic Environments

3.2 Implementing S OLVER M ODIFICATION

3.3 Implementing C ORRECTNESS D EMONSTRATION

3.3.1 Most General: Proof Search

3.3.3 Advantages of Prefix Code-Based Problem Solvers

4 Implementations of P OWER P LAY

4.1 Implementation Based on Optimal Ordered Problem Solver OOPS

Alg. 4.1: Implementing P OWER P LAY with Procedure OOPS [41]

4.1.1 Building on Existing OOPS Source Code

4.1.2 Alternative Problem Solvers Based on Recurrent Neural Networks

4.2 Adapting the Probability Distribution on Programs

4.3 Implementation Based on Stochastic or Evolutionary Search

5 Outgrowing Trivial Tasks - Compressing Previous Solutions

6.1 Self-Reference Through Novel Task Search as an External Task

7 Softening Task Acceptance Criteria of P OWER P LAY

Initialize s0 in some way

8 First Illustrative Experiments

9 Previous Relevant Work

9.1 Existing Theoretically Optimal Universal Problem Solvers

9.2 Connection to Traditional Active Learning

9.3 Greedy Implementation of Aspects of the Formal Theory of Creativity

9.4 Beyond Algorithmic Zero-Sum Games [37, 38] (1997-2002)

9.5 Opposing Forces: Improving Generalization Through Compression, Breaking

9.6 Relation to Gödel’s Sequence of Increasingly Powerful Axiomatic Systems

[20] U. Nehmzow, Y. Gatsoulis, E. Kerr, J. Condell, N. H. Siddique, and T. M. McGinnity. Novelty

[49] F. Sehnke, C. Osendorfer, T. Rückstieß, A. Graves, J. Peters, and J. Schmidhuber. Parameter-exploring

You might also like