Back
of the envelope reasoning involves generating quantitative answers in
situations where exact data and models are unavailable, where available data is
often incomplete and/or inconsistent. A rough estimate generated quickly is
more valuable and useful than a detailed analysis, which might be unnecessary,
impractical, or impossible because the situation does not provide enough time,
information, or other resources to perform one. Such reasoning is a key
component of commonsense reasoning about everyday physical situations. This
paper presents a similarity-based approach to such reasoning. In a new scenario
or problem, retrieving a similar example from experience, so to say, sets the
stage for solving the new problem by borrowing relevant modeling assumptions
and reasonable values for parameters. This provides us with a very useful class
of problems, which involve tightly interwoven qualitative and analogical
reasoning. Incorporating effects of quantitative dimensions in similarity
judgments and generalizations, hitherto unexplored, raises very interesting
questions.
Qualitative
reasoning (QR) aims to understand and model common sense. Forbus and Gentner
(1997) proposed a hybrid model of QR where analogical reasoning and qualitative
reasoning are tightly interwoven. In this paper, we look at quantitative
estimation (also called rough estimation, back of the envelope analysis, etc),
which we believe highlights some of the very important questions at the
intersection of analogical and qualitative reasoning. Back of the envelope
(BotE) analysis involves the estimation of rough but quantitative answers to
questions where the models and the data might be incomplete. In domains like
engineering, design, or experimental science, one often comes across situations
where a rough answer generated quickly is more valuable than waiting for more
information or resources. Some domains like environmental science [Harte, 1988]
and biophysics [O’Connor and Spotila, 1992] are so complex that BotE analysis
is the best that can be done with the available knowledge and data. BotE reasoning is ubiquitous in daily life
as well. A lot of common sense
reasoning hinges upon the ability to come up with quick, approximate, but
fine-grained enough for the task, estimates. We live in a world of quantitative
dimensions, and reasonably accurate estimation of quantitative values is
necessary for understanding and interacting with the world. Our life is full of
evaluations and rough estimates of all sorts. How long will it take to get
there? Do I have enough money with me? How much of the load can I carry at
once? These everyday, common sense estimates utilize our ability to draw a
quantitative sense of world from our experiences. We believe that the same processes underlie both these common
sense estimates and expert BotE, ballpark estimates. Specifically, the drawing upon experience to make such estimates,
and the achievement of expertise in part by accumulating, organizing, and
abstracting from experience to provide the background for such estimates, are
the same fundamental processes. We
claim that qualitative reasoning is essential for such analyses for two
reasons:
However,
some of the central assumptions of QR in practice must be rethought when
considering common sense knowledge, as opposed to narrow domain expertise. It is commonplace in QR to assume that a
domain theory is complete. This
assumption is implausible for common sense reasoning, whether or not one views
QR purely in terms of a component in a performance system or as a psychological
model. The closer one looks at human
knowledge, the more it appears that it is fragmentary, and more concrete than
abstract. It may be that such an
organization is a necessity for human-level performance, whether or not one is
making psychological claims. Let us
call this approach common sense QR (CSQR) for concreteness. Here are the constraints we currently think
guide CSQR:
BotE
reasoning (even when the problems are from non common-sense domains) operates
under similar constraints. A combination of QR and experiential knowledge seems
to be the key to BotE reasoning. QR helps us determine what phenomena are
relevant, and experiential knowledge supplies useful default and pre-computed
information, including both numeric values and relevant modeling assumptions,
as well as knowledge about similar situations that can serve as a reality check
for the estimates. To compare parameters, and to be able to make estimates
guided by similarity, and use the knowledge acquired from generalizations to
make estimates raises very interesting questions about how what role do
quantitative dimensions play in our judgments of similarity, and how we develop
our quantitative sense of a domain with experience.
Section
2 presents a brief review of relevant research. Section 3 explains what we
think BotE reasoning is, and our approach towards building such a
reasoner. Section 4 contains two
extended examples that illustrate our arguments. Section 5 presents some open
research issues, and we wind up with the summary.
Peterson
and Beach (1967) review a set of psychological studies to test people’s
abilities to derive statistical measures of populations and samples such as
proportions, means, variances, correlations, etc. Although some of the studies
have conflicting results, the key result that people are quite good at
abstracting measures of central tendency, and there are systematic differences
in intuitive judgments and objective statistical values. For example, people
don’t weigh all deviations equally in computing variance. Instead, they are hasty to believe in a
distribution even from a few samples, and tend to be conservative in revising
their measures on the basis of new data points. Tversky and Kahneman (1974)
reported people’s assessment of probabilities of uncertain events. In a very
important set of results, they show that people make systematic errors because
of a set of heuristics that they employ.
In
the 1960s, a popular psychological model for similarity was to represent
objects as points in a psychological space of stimulus dimensions, where
similarity is defined as the distance between points. Multidimensional scaling
(Shepard, 1962; Torgerson, 1965) is a technique designed to uncover this
psychological space by analyzing people’s similarity judgments. This work drew
a distinction between integral and separable dimensions, and explored how this
distinction affects our similarity judgments. Tversky’s set-theoretic account
(1977), where feature commonalities and feature differences both affect the
similarity between two concepts, raised many questions about the metric space
model. Gentner’s (1983) structure-mapping theory provides an account of analogy
and similarity that better fits the psychological data than either feature
space or feature set models. For
example, structure-mapping handles relationships as well as features, which is
crucial for the use of similarity in reasoning. The idea of structural alignment also provides deeper insights
into the comparison process that has led to many new predictions. For example, Markman and Gentner (1993)
proposed a structure-based model that makes three distinctions –commonalities,
alignable differences and non-alignable differences. Alignable differences are
differences along the same roles in two representations, whereas non-alignable
are differences along different roles. So, a hotel and motel have a lot of
alignable differences, whereas a hotel and motorbike has a lot of non-alignable
differences. In their studies since then, they have showed that people value
alignable differences more than non-alignable while making similarity
judgments.
Brown
and Siegler (1993) proposed a framework for real-world quantitative estimation
called the metrics and mappings framework. They make a distinction
between the quantitative knowledge and distributional properties of parameters
(metric knowledge), and ordinal information (mapping knowledge). Through a set
of experiments they showed that the ways people revise and assimilate
quantitative and ordinal information are quite different. Their experiments
involved subjects making quantitative estimates of populations of ninety-nine
countries. Afterwards participants were
told the correct value for populations of 24 of the countries, and then they
went through and re-estimated the full set of 99 populations (the 24 seed
countries and 75 transfer countries). Metric properties (as measured by sum of
absolute value of errors for all of the estimates) improved, but ordinal
knowledge (the order of different population, as measured by the rank-order
correlation) remained unchanged. On the other hand, telling them laws like
“Population of European countries are generally overestimated”, and “Population
of Asian countries are generally underestimated”, improved their ordinal
knowledge.
In a
recent study, Linder (1999) studied quantitative estimation in the context of
engineering education. Based on responses to real world questions, he tried to
build a framework for how people do rough estimations. About a hundred
mechanical engineering seniors at MIT, and fifty each at five other
universities attempted these estimation questions. He also compiled responses
from a hundred professionals, out of which about there were about thirty each
of electrical and mechanical engineers, and the rest from other engineering and
science background. It gives us a chance to look at people’s back of the
envelope reasoning. His focus was how to improve engineering curricula, and
thus his framework is informal and not couched in computational terms;
nevertheless, it provides an interesting source of data.
Back
of the envelope (BotE) analysis involves the estimation of rough but
quantitative answers to questions. Most of the questions are real-world
problems, where usually one does not have complete or accurate models or model
parameters. But yet one can get a lot out of approximate estimates. There is a
large variety of such questions, such as
Questions
1 to 4 are questions that might arise in engineering circumstances, whereas
Questions 5 to 7 are questions that arise in daily life. Question 5 seems more
based on direct observation than others.
For example, you might have earlier noticed how much time it takes for
you to arrive, or what were your best/worst times, and you recall those, and
might employ some measure of central tendency to come up with a time estimate.
In Question 6 (and others), it seems that one must build a simple estimation
model, and use this model to answer the question by estimating in turn
values for the parameters in the model.
This
type of reasoning is particularly common in engineering practice and
experimental sciences, including activities like evaluating the feasibility of
an idea, planning experiments, sizing components, and setting up and double-checking
detailed analyses. There is a tradeoff between specificity (resolution and
certainty in the answer) and economy. As we try to increase the specificity in
the answer, the analysis requires might require more resources in form of time,
information, formalization, and computation; and one might not have one or more
of these at hand.
Essentially,
BotE reasoning involves coming up with a numeric estimate for a parameter. This
can be decomposed this into two distinct (but not independent) processes –
Lets
look at a small example to make this distinction clear. Consider the question –
How many pieces of popcorn would fill the room you are now sitting in? The
parameter, num-popcorn is not one that one can recall a value from the
memory – so one way to derive it would be
num-popcorn =
volume-room/volume-popcorn …(1)
Approximating
room to a cuboid, and popcorn to a cube (considering the voids left after
packing in popcorn kernels this is a reasonable assumption),
num-popcorn = l*b*h / a^3 …(2)
where
l, b, h are length, breadth and height of the room and a is the edge of the
cube that describes a popcorn. In (2),
we have built an estimation-model for the number of popcorn kernels which we
have now described in terms of a set of parameters that can be estimated by
direct parameter estimation. Estimation-model building can be recursive (after
our initial model in (1), we had to build sub-models for the volumes of the
room and popcorn).
Linder
(1999) studied people’s abilities to make quantitative estimates in the context
of a mechanical engineering education. In one experiment, when people were
asked to estimate dimensions of an aluminum bar, more than 50% came up with
correct estimates and all the answers were in the correct order of
magnitude. However, in the same
experiment, when people were asked to estimate the power of a DC motor, only
about 30% got it right and the responses varied by six orders of magnitude! The
subjects were mechanical engineering seniors and mechanical and electrical
engineering professionals.
What
makes someone good at BotE reasoning? Experience with similar estimation tasks,
ability to compare a parameter with other known values, ease of access to
estimation models seem to be some of the important factors in numeric
estimation skill. Some parameters are clearly more accessible than others, and
there are strong domain expertise effects, too. One of the important things as
one learns a domain is extensive familiarity with the quantitative aspects of a
domain: when is a parameter value to be reasonable/typical, or high, or on the
conservative side, etc. And so it comes as no surprise that the intuitions of
an electrical engineer about motors and batteries is more accurate than an a
mechanical engineer, or that mechanical engineers’ answers about drag force and
tension are more accurate than those of electrical engineers. What is this
experiential knowledge, and how exactly does that help in BotE?
Thus we see analogical reasoning about within-domain
experience as being central both to building estimation models and to selecting
reasonable values for model parameters.
To make these ideas clearer, we turn to some extended examples for
illustration.
In
this section we look at two examples that illustrate various points that we
made earlier. Both the questions in this section were used in Linder’s study.
Q3
Estimate the drag force on a bicycle and rider traveling at 20 mph (9 m/s).
One
of the things to note about this problem (which is the case with most of
real-world estimation tasks), is that it is not complete. The basic description of the physical
situation is very abstract, and most of the quantitative information that is
needed to solve the problem is not provided.
Several subjects, given this problem, indicated that they pictured a
person on a bicycle from a distance from the side and/or the front; and often
they made sketches of these views [Linder, 1999]. This so beautifully argues
for model formulation phase itself involving retrieving a similar known
scenario, to fill in the details.
This
is a very simple solution. All of the power generated by the human is used up
in propelling the bicycle at the given speed, and that all of it goes to
overcoming the drag force. Since the estimate of the power that the human is
producing while cycling under given conditions is the only parameter that it
uses, the estimate strongly depends upon how representative the estimate of
power is in the circumstances of the problem.
|
Model |
Power = Force * Velocity |
|
Parameters |
Power
(produced by the human during cycling) = 200 Watts Velocity
(given) = 9m/s Force
(to be estimated) |
|
Solution |
Fdrag
= 200/9 ≈ 22 N |
In
the direct parameter estimation for power, it is key that we look for human
power output during similar activity. It turns out that humans can comfortably
produce 100 watts of power, and up to 1500 watts in spurts.
This
is the more standard solution that a mechanical engineer would come up with.
The drag equation (1), which helps calculate the drag force on a moving object
due to surrounding fluid, is definitely relevant to the problem. The difficulty
though is that it has a bunch of other parameters that we don’t know of, e.g.,
the drag coefficient, density of air, reference area of the body. The drag
coefficient (Cdrag) itself captures all complex dependencies (on the
viscosity and compressibility of air, geometry of the body, and the inclination
to flow) and is usually derived empirically. We look for similar scenarios, and
indeed there is one, of human falling with terminal velocity (maybe in context
of skydiving, and this is not a rare piece of information, considering that
quite a few people did use this). In the free-fall scenario, the terminal
velocity is known, and the drag force is known (as it counterbalances gravity,
it equals the weight of the person). This allows us to estimate the constant of
proportionality in the drag equation (2), and thus the drag force during
cycling.
|
Model |
Fdrag
= Cdrag (1/2 ρV2) A …(1) Or, Fdrag = KV2 for same sized objects in the same density
fluid. …(2) Plugging the value of K back into (2)
gives us Fdrag. |
|
Similar scenario: Free-fall, known
terminal velocity, VT = 50 m/s Here,
Fdrag_free_fall =
Weight. …(3) K
= Fdrag_free_fall /VT 2 = Weight/ VT 2 …(4) |
|
|
Parameters |
[A, Cdrag, ρ (density
of air)] can be lumped into K, V (velocity), VT = 50 m/s |
|
Calculations |
K = 750/ 50^2 = 0.3, Fdrag = 0.3 * 9 *9 ≈
25 N |
Q2
Estimate the energy stored in a new 9-volt transistor battery.
This
problem is an interesting example, where first principles reasoning from the
chemistry of energy generation in the battery involves complicated domain
knowledge, and none of the people asked even attempted to reason that way. What
most of the people did, was to imagine scenarios where such a battery was being
used, and try to think from there. And the thing that is beautiful is the fact that
this calculation gives us an estimate that is just as good as the more complex
method. This is a nice example of where, for the purposes of BotE estimates,
ability to successfully reason from known scenarios and examples buys us as
much as far more first principles knowledge would. The solution below presents
reasoning with very little knowledge about the battery. If I don’t know
anything about 9-volt battery, what is the next similar thing? A lot of people
thought about car batteries, 1.5-volt AA batteries, etc.
|
Model |
Suppose
I did not know anything about the 9v battery except its size, but I knew
examples of where 1.5v AA batteries were being used. If I make the assumption
that these two batteries are fundamentally the same, and only the difference
in volume should be responsible for difference in energies stored. Etransistor/EAA = Vtransistor/VAA …(1) |
|
In
a small hand-held flashlight, all the power provided by the batteries is used
up in lighting the bulb. N * EAA = Pbulb * Life …(2) Where
Pbulb is power rating of the flashlight bulb, and Life is the time
that a new set of batteries will take before they die out, and N is the
number of batteries in a flashlight. |
|
|
Parameters
and Calculations |
N
= 2 (number of batteries) Pbulb
= 1 Watts Life
= 2 hours EAA
= 1 * 2 * 3600 * 0.5 = 3600 J Vtransistor/VAA
= 2 Etransistor
= 7200 J |
This
example also demonstrates that using examples allows us to transform the
problem into ways that parameter estimation, or model building become more
intuitive or accessible. For example, knowledge of parameters like the rated
capacity of the battery, or, resistive load of the bulb would have led us to
solutions, but we think in terms of parameters that are more accessible to us.
Besides helping understand common sense qualitative reasoning, this is a great
problem solving strategy for scientific and engineering reasoning as well.
Our
approach is to use similarity to guide the estimation process. In this section
we look at our current models of similarity and generalization, and discuss the
interesting issues that BotE reasoning raises. Structure-mapping theory (SMT)
(Gentner, 1983) is a widely accepted model of analogy and similarity.
Structure-mapping engine (SME) (Falkenhainer et al 1989) is a
computational model of SMT. Given two structured propositional representations
as inputs, the base (about which we know more) and a target, SME
computes a mapping (or a handful of them). Each mapping is a set of correspondences
that align particular items in the base with items in the target, and candidate
inferences which are statements from the base that are hypothesized to hold
in the target by the virtue of these correspondences. Another important
component is making generalizations based on experiential knowledge, and SEQL
(Skorstad, Gentner and Medin, 1988) provides a framework for such reasoning.
With a large number of examples, generalizations will serve to ease the
organization of information, and also help in defining typicality and
representativeness with respect to parameter values, e.g., the number of
cylinders in a sports car, the weight of a truck, etc.
In
achieving our goal of being able to use experiential knowledge to guide BotE
reasoning, SME and SEQL have to be extended so that they can make sense of quantitative
information. That is, they already can
handle representations with numerical parameters, but similarity in aligned
numerical parameter values is does not affect the perceived similarity of the
descriptions compared. Given a problem,
or a scenario, we can retrieve similar examples from the corpus using MAC/FAC
(Forbus et al, 1995). Going from here to doing BotE reasoning raises
some important and open research questions –
The
above emphasis on quantitativeness is not to say that our internal
representations of quantity are necessarily numeric. A large class of
real-world tasks involves coming up with fine-grained estimates quickly and the
ability to do naďve arithmetic, e.g., how many people can get into the
elevator, etc. Numbers are a very powerful representation that can capture as
much fine-graininess as one wants, and support operations like the ability to
compare quantities and arithmetic across different quantity spaces. A
representation of quantity that captures common-sense reasoning will have to
support these types of tasks. One of the important things in estimation is the
ability to compare quantities. If the parameter itself is not known, then
finding a comparable parameter, e.g., one might think of the ceiling as 1.5
times the height of a person, so about 10ft is a reasonable estimate. Guerrin
(1995) presents a scheme to map a quality space onto the set of integers so
that one can define arithmetic, and with the refinement and abstraction operator,
symbols from different quality spaces can be compared.
We
would like to make a distinction between the notion of quantitativeness in
semi-quantitative reasoning (Berleant and Kuipers, 1997) and BotE reasoning. In
semi-quantitative reasoning, functional uncertainty is represented by defining
envelopes within which functional constraints must lie, and parametric
uncertainty is represented by numeric intervals. Clearly, this is still in the
spirit of first-principles reasoning, in contrast to our similarity-based
approach to model formulation and parameter estimation. Also, the
quantitativeness that we are proposing is based on our belief that common-sense
reasoning is indeed able to come up with fine-grained estimates, and we are
able to develop our notions of scale and quantitative dimensions with
experience in a domain.
In
this paper we have proposed a similarity-based model of back of the envelope
reasoning. We propose that the same
processes are used in both everyday common sense reasoning and in scientific
and engineering reasoning. We also
propose that these processes are highly experience-based, using within-domain
analogical reasoning and similarity to retrieve, apply, use, and generalize
from specific examples and previous problem-solving experience. This model of qualitative reasoning that
relies heavily on analogical reasoning, and equipped with a strong sense of
quantitative dimensions, is we suspect at the heart of common sense reasoning about
the physical world.
We
are currently exploring this model by using our analogical processing software
(SME, MAC/FAC, and SEQL) to create a BotE problem solver. This involves developing a corpus of
examples, including descriptions of objects, situations, and behaviors with
quantitative parameters. The BotE
problem solver we are building will store the solutions it derives in its
memory, to model the accumulation of problem-solving expertise.
Berleant,
D., and Kuipers, B. 1997. Qualitative and quantitative simulation: bridging the
gap. Artificial Intelligence Journal, 95(2): 215—255
Brown,
N. R., & Siegler, R. S. (1993). Metrics and mappings: A framework for
understanding real-world quantitative estimation. Psychological Review, 100(3), 511-534.
Falkenhainer,
B., Forbus, K. D., & Gentner, D. (1989). The structure-mapping engine:
Algorithm and examples. Artificial
Intelligence, 41, 1-63.
Forbus,
K. D. (1984). Qualitative process theory. Journal
of Artificial Intelligence, 24, 85-168.
Forbus,
K. and Gentner, D. (1997). Qualitative mental models: Simulations or memories? Proceedings
of the Eleventh International Workshop on Qualitative Reasoning, Cortona,
Italy.
Forbus,
K. D., Gentner, D., & Law, K. (1995). MAC/FAC: A model of similarity-based
retrieval. Cognitive Science, 19(2),
141-205.
Gentner,
D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7, 155-170.
Guerrin,
F. (1995). Dualistic algebra for qualitative analysis. In Proceedings of the
9th international workshop on Qualitative Reasoning, Amsterdam,
Holland.
Harte,
J. (1988). Consider a spherical cow: A course in environmental problem solving,
University Science Books, Sausalito, CA.
Linder,
B.M. (1991). Understanding estimation and its relation to engineering
education, Ph.D. Thesis, Department of Mechanical Engineering, Massachusetts
Institute of Technology.
Markman,
A. B., & Gentner, D. (1993c). Structural alignment during similarity
comparisons. Cognitive Psychology, 25,
431-467.
O’Connor,
M.P., and Spotila, J.M. (1992). Consider a spherical lizard: Animals, models
and approximations, American Zoologist, 32, pp 179-193.
Peterson,
C.R., and Beach, L.R. (1967). Man as an intuitive statistician, Psychological
Bulletin, 68(1), pp 29-46
Shepard,
R. N. (1962). The analysis of proximities: Multidimensional scaling with an
unknown distance function, I. Psychmetrika,
27(2), 125-140.
Skorstad,
J., Gentner, D., & Medin, D. (1988). Abstraction processes during concept
learning: A structural view. Proceedings
of the Tenth Annual Conference of the Cognitive Science Society, 419-425.
Torgerson,
W. S. (1965). Multidimensional scaling of similarity. Psychometrika, 30(4).
Tversky, A. (1977). Features of Similarity, Psychological
Review 84(4), pp 327 - 352.
Tversky,
A., and Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases,
Science, 185, pp 1124-1131