AID98 Workshop on Machine Learning in Design
Evaluating the Impact of Distributed Learning
in Real-World Design Problems
Dan L. Grecu, David C. Brown
AI in Design Group
Department of Computer Science
Worcester Polytechnic Institute
Worcester, MA 01609, USA
dgrecu@cs.wpi.edu, dcb@cs.wpi.edu
1. Learning in real-world design problems
Design systems are starting
to move from addressing small and `neat' design problems towards more real-world
design tasks. The machine learning techniques that have been incorporated
in recent design systems have significantly contributed to this shift in
the capabilities of design systems (Duffy 1997; Reich 1998; Sim & Duffy
1998).
One of the main difficulties
encountered by design systems that approach complex design problems is
the distance between the problem solving process that's imagined when the
design system is developed, and the way the problem solving unfolds when
the system is facing an actual design task. The more complex the design
problems the larger this distance is likely to be.
From this perspective, the
mission of machine learning in intelligent design systems is to make the
design system satisfy a set of requirements that cannot be achieved at
the time the system is developed. In other words, learning has to bridge
the gap between the design system initially developed and a system that
successfully handles the design problems of the real world.
This gap is characterized
by four different aspects:
-
Design quality: Given a design
problem to which the system is repeatedly exposed, it is expected that
the system will be able to provide solutions of increasing quality.
-
Design process performance: A
learning design system is supposed to be able to reduce the costs of the
design process for achieving a solution. Such costs can be measured in
time, or in the size of the search space that has to be explored, or in
the number of internal conflicts that have to be resolved on the path to
a solution.
-
System functionality: Learning
can extend a design system's capabilities with functionality that was not
initially available. Providing explanations or rationale, constructing
analogies, or avoiding pitfalls are capabilities gained by a design system
as it acquires experience by learning during design problem solving.
-
Problem range: Every problem-solver
has a range of problems for which it has been designed. Learning may extend
this range beyond its initial limits without the explicit intervention
of the developers.
Design systems that learn
have focused primarily on the first two aspects. They reflect the goal
of doing better, either in terms of the generated solution or in terms
of the path used to reach that solution, in a
quasi-static problem
space. The last two aspects (3 and 4 above) are new challenges raised in
real-world design that are caused by
unpredictable variations in
the context that defines the search for a design solution. The functionality
shift (aspect 3) requires the ability to respond to increased demands on
what is produced as a result of a search for a solution and on
how
this search should be performed. The problem variability (aspect 4) exposes
the design system to a problem space that is potentially larger and may
have different characteristics than originally considered.
2. Distributing learning throughout the design
system
Most of the current design
systems use one well defined learning process. This has been motivated
by two factors:
-
Learning has been included in design systems
to focus on one single aspect that has to be improved (usually design
quality or design process performance).
-
The design problems approached were relatively
simple, compared to the scale and complexity of the problems usually faced
by human designers. Given the reduced problem scale, the learning models
were conceived as a direct link between the information to be acquired
(usually associated with a predetermined place in the design system) and
the objective (i.e., how the aspect or parameter needs to be changed).
Real-world domains and problem
complexity will require design systems to change these factors:
-
Learning will have to be distributed
throughout the design system. It is likely that this will be achieved through
multiple learning processes rather than implementing one all-encompassing
learning model.
-
Learning will have to pursue several objectives.
An objective may have to be satisfied by more than one learning process.
Learning processes should be able to operate in parallel.
-
Given the size of the design problems, a learning
process may be remote from the objective(s) it will respond to.
Learning will potentially influence different factors than the ones specified
in the objectives, but the change propagation will lead to achieving the
desired objective.
Under these circumstances,
implementing learning becomes a design problem itself! Developers will
have to decide where to place learning in a design system, what methods
to use, what information to include in the learning processes and how these
processes will be able to respond at run-time to the criteria for quality,
performance, functionality and coverage (Grecu & Brown 1998).
3. Evaluating the impact of learning
Given this real-world view
of learning in design, the analysis of learning's impact in a design system
requires more than simply monitoring the change of a parameter in response
to the use of newly acquired or synthesized information. Knowing the possible
consequences of this enhanced learning will impact the decision of how
to distribute learning throughout the design system. This, in turn, will
place requirements on the design system itself, as information that can
facilitate various learning processes must be made explicitly available,
as must the results of the learning processes.
In the following we discuss
some of the issues that are likely to arise when we attempt to do
evaluations
in this new learning context. We need to carefully consider evaluation
in advance, as the conclusions drawn from experiments with distributed
learning in design systems will determine future efforts in developing
design system architectures.
-
The response to various sets of objectives:
A design system that learns will evolve depending on the objectives and
criteria that have been specified for the learning. Therefore it's important
to test how the system changes in response to various combinations of criteria.
Of interest are the degree to which various criteria are satisfied, and
how the presence of some criteria can amplify or inhibit the satisfaction
of other criteria. For example, evaluation tests should include criteria
that refer: only to the design quality; only to the design process; to
both types of factors. The convergence of the learning processes is one
of the most important of the aspects that have to be analyzed in each case.
As with every learning process, the analysis should provide information
about the degree to which individual criteria are satisfied, about the
order in which they are satisfied, and about the level of accuracy of the
information learned in each process.
-
The `decomposition' of objectives:
Given that the objectives of the design system may be remote from the learning
processes, design systems will need to decompose design objectives to determine
which learning process(es) respond to each objective. This will require
the system to be able to establish supporting criteria that have to be
satisfied through learning on the path to the final objectives. An evaluation
of the learning models will have to take into consideration the possible
alternatives that exist both in decomposing objectives and in associating
them with the learning processes.
-
Learning processes shared by multiple
objectives: Several objectives may need to make use of the same
learning process. This can lead to conflicting or at least interfering
demands being placed on the learning process. Different objectives would
create different biases and relevancy criteria for the features used in
learning and for the results of the learning itself. The evaluation needs
to determine the separate effects resulting from the different objectives,
and to compare them to the combined effect of the simultaneous presence
of all the objectives.
-
The sets of features used in the learning
processes: Some of the learning processes will associate learned
information with descriptions of the design problem-solving processes.
The learning result will significantly depend on the features used to define
these descriptions. The choice of features is not always obvious. Imagine
a learning process that learns to associate a design context with a particular
decision in the design system. If the feature set describing the design
context includes irrelevant features the design system can unnecessarily
spend a processing overhead to compute those features before selecting
a decision. Alternatively, if the feature set does not include essential
features, the design system may be unable to discriminate between distinct
design contexts that call for different decisions.
-
Feedback sources: Given the potential
distance between the learning processes and the measurements and evaluations
of their effects, as well as the numerous dependencies that characterize
design problems, providing evaluation feedback for the learning
processes is a non-trivial task. System developers will have to select
relevant paths along which feedback is propagated, possibly by using the
decomposition methods already mentioned. It is likely that there will be
several ways that evaluations of objectives and criteria should be traced
back to their originating learning processes. As feedback represents one
of the most important shaping factors for learning, a careful analysis
of feedback sources and paths will be a necessary part of a serious analysis
of the learning effects.
-
Correlations between learning accuracy
and objective attainment: Learning in real-world design problems
will focus on multiple objectives. Developers will have to establish levels
of satisfaction for the individual objectives, such that they can be simultaneously
satisfied to an acceptable degree. These thresholds have to be translated
into the requirements for the individual learning processes. Evaluation
of the learning in the design system has to determine how sensitive the
variations in the global objectives are to the individual learning processes.
It is also necessary to determine the learning limits that are required
from an individual learning process, beyond which learning is not necessary
or may even have undesired effects.
-
Interference of learning processes:
Two learning processes can interfere by placing conflicting demands in
a certain area of the design system. In such cases it is important to determine
whether a compromise is possible that satisfies each of the improvement
goals to a certain degree, or whether the two goals will actually cancel
themselves out. In another scenario, two learning processes can act such
that one of the processes creates a "moving target" for the other process.
This happens when the information that the first process uses for learning
(the target) is itself modified as a result of a second learning process.
Such situations need to be identified and addressed in order to determine
whether the first learning process converges and whether the resulting
learned information is reliable.
-
Cross-talk resulting from training on
several classes of design problems: One of the evaluation targets
has to be the behavior of the system after being trained on several classes
of design problems. After the system has been trained or used on a class
of problems, it is desirable for the system to preserve the qualities it
has achieved while it is being trained on a different problem class. Interference
(cross-talk) is possible in such situations, as various types of problems
can generate opposite changes. The analysis should provide insight into
the characteristics of design problems that can lead to cross-talk phenomena,
and determine how learning processes can preserve the acquired performance
level under these circumstances.
-
Degradation of performance aspects that
were not specified by the user: There may be aspects of the design
solution and design process that are important, but which are not covered
by the objectives that guide the learning processes. Learning can negatively
influence such aspects if the learned knowledge interferes with them (e.g.,
by generating contradictions, or by altering search processes). It is important
that system developers establish a set of monitoring criteria for critical
design aspects, that allow them to detect when, and to what extent, the
monitored aspects are affected by the changes occurring in the system as
a result of learning.
4. Conclusions
The list above is not intended
to be an exhaustive set of evaluation criteria for real-world design systems
with distributed learning abilities. It is, however, meant to indicate
that as design problem complexity increases, the evaluation of the impact
of learning becomes a multi-faceted issue. The wide variety of factors
that need to be analyzed to assess the benefits of learning suggest that
there is a trade-off between these benefits and the potential cost of implementing
the learning methods and of having them run concurrently in the design
system.
5. References
[Duffy 1997]
A.H.B. Duffy (1997). "The `What' and `How' of Learning in Design,"
IEEE Expert, 12 (3): 71-77.
D.L. Grecu & D.C. Brown (1998). "Dimensions
of Machine Learning in Design," Artificial Intelligence for Engineering
Design, Analysis and Manufacturing, special issue on "Machine Learning
in Design," edited by A.H.B. Duffy, D.C. Brown, & A.K. Goel, 12
(2): 117-122.
Y. Reich (1998). "Learning in Design: From Characterizing
Dimensions to Working Systems," Artificial Intelligence for Engineering
Design, Analysis and Manufacturing, special issue on "Machine Learning
in Design," edited by A.H.B. Duffy, D.C. Brown, & A.K. Goel, 12
(2): 161-172.
S.K. Sim & A.H.B. Duffy (1998). "A Foundation
for Machine Learning in Design," Artificial Intelligence for Engineering
Design, Analysis and Manufacturing, special issue on "Machine Learning
in Design," edited by A.H.B. Duffy, D.C. Brown, & A.K. Goel, 12
(2): 193-209.
13 July 1998