The proposition that people rely on mental models was first put forward by the Scottish psychologist Kenneth Craik in 1943. In his book The Nature of Exploration (Craik 1943), he wrote that the mind constructs "small-scale models" of reality that it uses to reason, to anticipate events and to underlie explanation.
Mental models constructed from comprehension of discourse have been Philip Johnson-Laird's (1989) subject of study. He posited that a reader creates a mental model of the text being read, which simulates the 'world' being described, as the reader understands/interprets it. According to Johnson-Laird, ambiguous passages of text can lead to more than one competing mental model, which is something authors deliberately use to keep the reader confused about certain aspects of, say, a story in a novel. However, passages of text that unambiguously produce a single mental model are easier to comprehend.
Relevance for interaction design
The slower processing of ambiguous sentences is one area where mental model research is relevant to interaction design, as interaction designers are, among other things, interested in measures of learnability and ease of use. In this respect the notion of mental models has applicability and explanatory power.
Another area of relevance are discussions of 'system causality conveyance'. The use of mental models in this sense was popularised in the HCI and interaction design community by Donald Norman in his book "The Design of Everyday Things" (Norman 1988). In his book, he used mental models to describe how a system is designed and implemented on the basis of the designer's mental model. Similar to a reader of a passage of text, the user develops a mental model of how he/she thinks the system works through interaction of the system. This model is used to reason about the system, to anticipate system behaviour and to explain why the system reacts as it does. In other words, the designer reifies (materialises) his mental model of a given design, e.g. a computer system, which becomes the only means of conveying his mental model to the user.
Figure 1 (adapted from Norman (1988) p. 16): The problem of ensuring that the user's mental model corresponds to the designer's model arises because the designer does not talk directly with the user. The designer can only talk to the user through the "system image" - the designer's materialised mental model. The system image is, like a text, open to interpretation.
Norman also speaks of conceptual models. In the article "Some Observations on Mental Models" Norman (1983) distinguishes between mental models and conceptual models: "Conceptual models are devised as tools for the understanidng or teaching of physical systems. Mental models are what people really have in their heads and what guides their use of things." In other words, the designer designs a conceptual model into the system in order for it to appear graspable and coherent to the user. If he/she manages to get the conceptual model right, the correct mental model (in the mind of the user) will follow. At least in theory.
Norman's account is of course an over-simplification, but it captures some central dilemmas of interaction design by using mental models as a vehicle for explanation and illustration.
Whether people create mental models or use other forms of vehicles for reasoning is of course an open research question. In any case, we humans are compulsive interpreters in that we cannot help making sense of what we perceive. Thus, users do not only interpret the visible parts of the system (the "system image" in figure 1) but also make guesses as to what goes on behind the scenes ("the system" in figure 1). This makes the stringent separation of the user interface from the underlying technical processes a poor strategy for achieving ease of use (e.g. Gentner and Nielsen 1996) as users cannot help trying to understand the 'intestines' of the system anyway. This compulsion toward making a mental model of the 'internal logic' of the system may be part of the reason why many efforts in the HCI community in the early 80s were wasted since it appeared that "The aim of interface design was to invent metaphors that users could understand, but which had not much to do with the internal workings of the system" (Andersen, p.4).
If you would like to learn more about mental model research and its applicability, see Payne (2000)
"All our ideas and concepts are only internal pictures".
-- Ludwig Boltzmann (1899)