An intelligent agent is intrinsically motivated to act if the information content alone, of the experience resulting from the action, is the motivating factor. Information content in this context is measured in the information-theoretic sense of quantifying uncertainty. A typical intrinsic motivation is to search for unusual, surprising situations, in contrast to a typical extrinsic motivation such as the search for food. Extrinsic motivations are typically described in artificial intelligence as task-dependent or goal-directed.
Origins in psychology
The study of intrinsic motivation in psychology and neuroscience began in the 1950s with some psychologists explaining exploration through drives to manipulate and explore, however, this homeostatic view was criticised by White. An alternative explanation from Berlyne in 1960 was the pursuit of an optimal balance between novelty and familiarity. Festinger described the difference between internal and external view of the world as dissonance that organisms are motivated to reduce. A similar view was expressed in the '70s by Kagan as the desire to reduce the incompatibility between cognitive structure and experience. In contrast to the idea of optimal incongruity, Deci and Ryan identified in the mid 80's an intrinsic motivation based on competence and self-determination.
Computational models
An influential early computational approach to implement artificial curiosity in the early 1990s by Schmidhuber, has since been developed into a "Formal theory of creativity, fun, and intrinsic motivation”. Intrinsic motivation is often studied in the framework of computational reinforcement learning, where the rewards that drive agent behaviour are intrinsically derived rather than externally imposed and must be learnt from the environment. Reinforcement learning is agnostic to how the reward is generated - an agent will learn a policy from the distribution of rewards afforded by actions and the environment. Each approach to intrinsic motivation in this scheme is essentially a different way of generating the reward function for the agent.
Curiosity vs. exploration
Intrinsically motivated artificial agents exhibit behaviour that resembles curiosity or exploration. Exploration in artificial intelligence and robotics has been extensively studied in reinforcement learning models, usually by encouraging the agent to explore as much of the environment as possible, to reduce uncertainty about the dynamics of the environment and how best to achieve its goals. Intrinsic motivation, in contrast, encourages the agent to first explore aspects of the environment that confer more information, to seek out novelty. Recent work unifying state visit count exploration and intrinsic motivation has shown faster learning in a video game setting.
Types of models
and Kaplan have made a substantial contribution to the study of intrinsic motivation. They define intrinsic motivation based on Berlyne's theory, and divide approaches to the implementation of intrinsic motivation into three categories that broadly follow the roots in psychology: "knowledge-based models", "competence-based models" and "morphological models". Knowledge-based models are further subdivided into "information-theoretic" and "predictive". Baldassare and Mirolli present a similar typology, differentiating knowledge-based models between prediction-based and novelty-based.
Information-theoretic intrinsic motivation
The quantification of prediction and novelty to drive behaviour is generally enabled through the application of information-theoretic models, where agent state and strategy over time are represented by probability distributions describing a markov decision process and the cycle of perception and action treated as an information channel. These approaches claim biological feasibility as part of a family of bayesian approaches to brain function. The main criticism and difficulty of these models is the intractability of computing probability distributions over large discrete or continuous state spaces. Nonetheless a considerable body of work has built up modelling the flow of information around the sensorimotor cycle, leading to de-facto reward functions derived from the reduction of uncertainty, including most notably active inference, but also infotaxis, predictive information, empowerment.
Competence-based models
autotelic principle is an attempt to formalise flow.
Intrinsically Motivated Learning
Intrinsically motivated learning is an emerging research topic in artificial intelligence and developmental robotics that aims to develop agents that can learn general skills or behaviours, that can be deployed to improve performance in extrinsic tasks, such as acquiring resources. Intrinsically motivated learning has been studied as an approach to autonomous lifelong learning in machines. Despite the impressive success of deep learning in specific domains, many in the field have pointed out that the ability to generalise remains a fundamental challenge in artificial intelligence. Intrinsically motivated learning, although promising in terms of being able to generate goals from the structure of the environment without externally imposed tasks, faces the same challenge of generalisation – how to reuse policies or action sequences, how to compress and represent continuous or complex state spaces and retain and reuse the salient features that have been learnt.
