Semantic folding


Semantic folding theory describes a procedure for encoding the semantics of natural language text in a semantically grounded binary representation. This approach provides a framework for modelling how language data is processed by the neocortex.

Theory

Semantic folding theory draws inspiration from Douglas R. Hofstadter's Analogy as the Core of Cognition which suggests that the brain makes sense of the world by identifying and applying analogies. The theory hypothesises that semantic data must therefore be introduced to the neocortex in such a form as to allow the application of a similarity measure and offers, as a solution, the sparse binary vector employing a two-dimensional topographic semantic space as a distributional reference frame. The theory builds on the computational theory of the human cortex known as hierarchical temporal memory, and positions itself as a complementary theory for the representation of language semantics.
A particular strength claimed by this approach is that the resulting binary representation enables complex semantic operations to be performed simply and efficiently at the most basic computational level.

Two-dimensional semantic space

Analogous to the structure of the neocortex, Semantic Folding theory posits the implementation of a semantic space as a two-dimensional grid. This grid is populated by context-vectors in such a way as to place similar context-vectors closer to each other, for instance, by using competitive learning principles. This vector space model is presented in the theory as an equivalence to the well known word space model described in the Information Retrieval literature.
Given a semantic space a word-vector can be obtained for any given word Y by employing the following algorithm:
For each position X in the semantic map
if the word Y is contained in the context-vector at position X
then add 1 to the corresponding position in the word-vector for Y
else
add 0 to the corresponding position in the word-vector for Y
The result of this process will be a word-vector containing all the contexts in which the word Y appears and will therefore be representative of the semantics of that word in the semantic space. It can be seen that the resulting word-vector is also in a sparse distributed representation format & . Some properties of word-SDRs that are of particular interest with respect to computational semantics are:
Semantic spaces in the natural language domain aim to create representations of natural language that are capable of capturing meaning. The original motivation for semantic spaces stems from two core challenges of natural language: Vocabulary mismatch and ambiguity of natural language.
The application of semantic spaces in natural language processing aims at overcoming limitations of rule-based or model-based approaches operating on the keyword level. The main drawback with these approaches is their brittleness, and the large manual effort required to create either rule-based NLP systems or training corpora for model learning. Rule-based and machine learning-based models are fixed on the keyword level and break down if the vocabulary differs from that defined in the rules or from the training material used for the statistical models.
Research in semantic spaces dates back more than 20 years. In 1996, two papers were published that raised a lot of attention around the general idea of creating semantic spaces: latent semantic analysis from Microsoft and Hyperspace Analogue to Language from the University of California. However, their adoption was limited by the large computational effort required to construct and use those semantic spaces. A breakthrough with regard to the accuracy of modelling associative relations between words was achieved by explicit semantic analysis in 2007. ESA was a novel based approach that represented words in the form of vectors with 100,000 dimensions. However practical applications of the approach are limited due to the large number of required dimensions in the vectors.
More recently, advances in neural networking techniques in combination with other new approaches led to a host of new recent developments: Word2vec from Google and GloVe from Stanford University.
Semantic folding represents a novel, biologically inspired approach to semantic spaces where each word is represented as a sparse binary vector with 16,000 dimensions in a 2D semantic map. Sparse binary representation are advantageous in terms of computational efficiency, and allow for the storage of very large numbers of possible patterns.

Visualization

The topological distribution over a two-dimensional grid lends itself to a bitmap type visualization of the semantics of any word or text, where each active semantic feature can be displayed as e.g. a pixel. As can be seen in the images shown here, this representation allows for a direct visual comparison of the semantics of two linguistic items.
Image 1 clearly demonstrates that the two disparate terms "dog" and "car" have, as expected, very obviously different semantics.
Image 2 shows that only one of the meaning contexts of "jaguar", that of "Jaguar" the car, overlaps with the meaning of Porsche. Other meaning contexts of "jaguar" e.g. "jaguar" the animal clearly have different non-overlapping contexts.
The visualization of semantic similarity using Semantic Folding bears a strong resemblance to the fMRI images produced in a research study conducted by A.G. Huth et al., where it is claimed that words are grouped in the brain by meaning.