University of Computer Studies, Mandalay

(UCSM)

Faculty of Computer Science

Chapter 7 Integration of Heterogeneous
Sources of Partial Domain Knowledge
 A domain transforming neural network model that starts
from a single source of prior- knowledge and grows
incrementally as needed is introduced next.
 In this model two integration techniques are explored:
 (1) converting the expert system rule base into a corresponding
neural network and then extending this network by constructive
learning; and
 (2) embedding the pre-existing expert system directly into a
constructive neural network learning system.
Introduction
 Automatically classifying data into categories is an
important problem in many real life domains.
 There are two traditional ways for a computer system to
acquire knowledge required to perform classification
tasks.
 The knowledge based approach translates information obtained
from human domain experts into a form that is interpretable by a
computer system. This knowledge base is the core of an expert
system, which emulates human decision-making skills.
 The alternative approach, called machine learning, is an attempt to
extract knowledge directly from data.
Cont’d
 Each of these two approaches has its advantages and
disadvantages.
 An expert system represents knowledge in symbolic
form allowing relatively easy manipulation and
incorporation of newly obtained knowledge.
 A machine learning system is less dependent on human
understanding of the phenomena and can, in principle,
be applied to any domain with sufficient amount of
available data.
Cont’d
 Expert systems assume human understanding of the
phenomena and availability of an expert capable of explaining
domain knowledge to a computer programmer. The
knowledge used to build an expert system is typically
acquired from the expertise of many individuals and thus, it
can be inconsistent and incomplete.
 Similarly, a data set used to build a machine learning system
can be noisy, conflicting and sparse. Even if these problems
are not present in a given data set, extracting complex
nonlinear relationships directly from data through machine
learning can still be a difficult nonlinear optimization task.
Cont’d
 Hybrid intelligent systems that integrate knowledge
extraction from data and the use of existing alternative
sources of domain specific knowledge have had
considerable practical success.
Cont’d
 the most popular approaches for integrating multiple
learning components are:
 (1) combining expert modules through various averaging scenarios;
and
 (2) selecting the most competent local expert for any given example.
 The combining approach can potentially be used to
reduce the variance of an unstable predictor without
increasing its bias.
Cont’d
 Neural network machine learning techniques are
combined with expert systems in order to complement
and enhance their capabilities. This combination has
been carried out in several directions including:
 Transformational models
 Loosely-coupled models
 Tightly-coupled models
Transformational models
 one type of system is transformed into another, i.e. either
an expert system is transformed into a neural network or
vice versa.
 Neural nets are transformed into expert systems
whenever knowledge documentation and justification
facilities are needed.
 Conversely, an expert system can be transformed into a
neural network when speed, adaptability and robustness
are a priority.
Fully-integrated models
 Several systems share data structures and knowledge
representation.
 The most common variation of this model is the
connectionist expert system, in which symbolic nodes
are connected by weighted links.
 These systems are robust and have improved problem
solving characteristics, but the complexity of the system
makes it difficult to develop and maintain.
Loosely-coupled models
 This is an integrational model in which neural network
and knowledge- based systems interact through shared
data files.
 These systems are easier to develop than more
integrated models, allowing for the use of commercial
software for both expert systems and neural networks,
but at the cost of slower operation, redundancy in
computational capabilities, overlap in data input
requirements and high communication cost.
Tightly-coupled models
 This type of architecture uses independent neural net
and expert systems.
 The difference is that here the interaction is by means of
memory resident data structures, so communication and
operation velocities are vastly improved.
 System development is not much more difficult than that
for loosely-coupled systems, but redundancy is also an
issue.
Integrate Neural Network Learning
 Embedding transformed experts. This is a
transformational approach in which the expert is
converted into a neural network that serves as a starting
point for learning from examples.
 Embedding experts directly. In this approach the
experts are embedded without modifications in the
hybrid classification system, which means that no
knowledge of their internal mechanisms is required.
The Hybrid System
 Single expert expansion systems. In this approach, all
expert knowledge comes from a single source, which is
further expanded through neural network learning.
 Multiple experts integration systems. Here multiple,
possibly heterogeneous, experts are integrated into a
hybrid classification system.
Experts Intégration: Domain or
Range Transformation Dilemma
 A classification problem can be viewed as identification
of an appropriate mapping from a given domain to a
range of categories.
Domain Transformation
 When using the outputs of several experts as inputs to a
new classifier (called a combiner.), the original input
space is transformed to an entirely new one, its
dimensionality being defined by the number of experts
used and their output representation.
Cont’d
 Reduced domain dimensionality. This results in
models with less parameters, which, due to the curse of
dimensionality, can be a significant advantage when
designing a model from a limited data set.
 Input preprocessing. The domain transformation can
be regarded as a feature extraction process. This can
help eliminate the effect of irrelevant or noisy input
variables.
 Simpler decision regions. Even if the original
dimensionality is not significantly reduced through a
transformation, classes in the new domain may be easier
to separate due to more favorable clustering of the
patterns.
Range Transformation
 A classifier is trained to integrate the various experts by
partitioning the domain and assigning disjoint regions to
individual experts.
 The original problem domain does not change, but the
integrating classifier has a different class range defined
by the number of experts.
Cont’d
 Hybrid systems relying on range transformation can provide various
advantages
 over original classifiers, specifically the following:
 Simpler decision regions. As in the domain transformation case, it
is anticipated that the transformed problem will have simpler
decision regions than the original one, thus being easier to solve
and requiring a less complex classifier to do
 Restricted domain for local training. Assigning a portion of the
domain to a specific classifier allows local training of machine
learning based experts on the remaining region or regions.
Insensibility to output representation. Range transformation
works in the same way regardless of the experts’ output
representation. Indeed, the technique can even work with experts
whose output representations differ from one another.
Incremental Single Expert
Expansion
 A knowledge-based artificial neural network approach
called KBANN, that generates neural networks from
hierarchically-structured rules. In these neural networks
units correspond to rules, while connection weights and
thresholds correspond to rule dependencies.
 Each layer is then fully connected with the weights of the
new links set to small random values.
 Finally, the initial knowledge is refined by neural network
training from examples using the backpropagation
algorithm
The HDE Algorithm
 Construction of the HDE neural network is
performed in three phases:
 1. Determination of points on the decision boundary;
 2. Generation of a pool of candidate hyperplanes from
the obtained points; and
 3. Selection of the final separating hyperplanes from the
candidate pool and creation of hidden units from
selected hyperplanes.
Embedding of Transformed Prior
Knowledge
 the HDE algorithm is used to add neurons to the
last layer of the initial neural network.
 The output weights of the starting network are
modified after each new hidden unit is added
through HDE construction.
Direct Integration of Prior
Knowledge
 A hybrid architecture is constructed with the expert
system directly embedded into the neural network
 The three-phase process of the HDE algorithm is
followed with certain small exceptions.
 An initial network consisting of the input layer, a single hidden unit
and an output unit is created.
 The hidden unit is designated as an expert unit, which, instead of
computing the usual activation function, calls the expert system to
determine the unit’s output.
 The expert unit, then, acts as the initial hidden unit, contributing with
its decision boundary, such as the simple one.