On Why Machines Can Think

Generally, there are three main types of reasoning we employ when “thinking”: deduction, induction, and abduction.

In the 17th century, René Descartes introduced a relatively new idea — the dictum “cogito ergo sum” (“I think, therefore I am”). This simple formulation served as the basis of Western philosophy and defined for centuries our ideas on what constitutes the essence of being a human.

deduction is the ability to reach a conclusion from a given rule and a case that are assumed to be true.

Deduction is fundamental for our ability to do science. It’s also the type of reasoning that’s the easiest to reproduce by a machine.

Induction is the ability to generalise rules from a given set of observations. It’s central for our ability to do science since it allows us to quantitatively identify new patterns/rules.

Induction is much more challenging for machines. Your calculator, of course, cannot perform it. Machine learning models, however, can. In fact, that’s their primary objective: generalise from a set of given results.

Say, we have a supervised classification model that we’ll use for spam detection. First, we have the labelled training dataset — spam or not spam (a.k.a. the result). Within that dataset, we’ve compiled multiple cases for each result. Based on these, the model induces its own rules that can, later on, be applied to a case it has never seen before.

when dealing with unsupervised models such as recommendation systems, the process follows a similar beat. We first provide the model with a dataset about what people tend to buy when they go to the supermarket (result). Once we start with the model training, we’ll expect it to first cluster repeating patterns (cases) and then, induce its own rules that can be later on applied to similar contexts.

Abduction is the ability to generate new hypotheses from a single surprising observation (i.e., result). We do that every time we rely on our experiences to come to an explanation of sorts.

Abduction is the least reliable of the reasoning types. Chances are that the hypothesis you reached through abduction is not correct. For instance, the result of “wet street” might have had nothing to do with the rain — perhaps a pipe had bursted somewhere on the street during the night, or someone diligently sprayed the street with water. The rain, however, seems like a plausible explanation.

Those of you familiar with rule-based systems from the 1960s and 1970s, of course, can point at MYCIN, XCON and SHRDLU and claim that they’re capable of abduction. Others might bring up the examples of abduction cited by the Stanford AI index in 2022 and 2023 as one of the most promising areas for future research (i.e., abductive natural language inference).

Generally, abduction covers the act of generating a novel case (where learnings can be transferred from one context to another). IBE, on the other hand, is a very special and more context-specific form of induction that doesn’t necessarily require you to identify patterns quantitatively (i.e., you don’t need to observe a pattern 10.000 times to formulate a rule).

By definition, ML models are an induction-generating machines. This inclination is further strengthen by their so-called inductive bias.

Inductive bias is an integral concept in ML referring to the inherent assumptions or preferences a model possesses regarding the types of functions it should learn. The bias helps guide the learning process by restricting the set of possible hypotheses, making learning more efficient and accurate.

For example, decision trees focus on hierarchical structures and simple decision boundaries. Support Vector Machines aim to find wide margins between classes. Convolutional Neural Networks emphasise translation invariance and hierarchical feature learning in images. Recurrent Neural Networks are biased towards sequential patterns, Bayesian Networks model probabilistic relationships, regularised linear models prefer simpler models by penalising large coefficients, and general transformers like GPT-4 are characterised by their ability to capture sequential dependencies and relationships in data. These biases shape the models’ behaviour and suitability for different tasks. They also make it difficult to transfer learnings from one context to another.

OK, by now we discussed a primer on reasoning and we saw that machines can indeed reason. They perform both deduction and induction. However, what we tend to intuitively term as “thinking” is facilitated by abduction, which continues to be elusive due to conflation and architecture.

Even the philosopher who introduced the notion of abduction, Charles Peirce, doesn’t refer to it in a consistent manner.