AI Reasoning

Underline Science Inc.

Abstract

The ability to reason has been a salient characteristic of human intelligence, and there is a critical need for verifiable reasoning in AI systems. Main Takeaways * Reasoning has always been seen as a core characteristic of human intelligence. Reasoning is used to derive new information from given base knowledge; this new information is guaranteed correct when sound formal reasoning is used, otherwise it is merely plausible. * AI research has led to a range of automated reasoning techniques. These reasoning techniques have given rise to AI algorithms and systems, including SAT, SMT, and constraints solvers as well as probabilistic graphical models, all of which play a key role in critical real-world applications. * While large pre-trained systems (such as LLMs) have made impressive advancements in their reasoning capabilities, more research is needed to guarantee correctness and depth of the reasoning performed by them; such guarantees are particularly important for autonomously operating AI agents Context & History Reasoning is a core component of human intelligence. From the dawn of humanity, abductive reasoning has been used to predict danger and inductive reasoning made it possible to learn regularities governing the world. Beginning in Ancient Greece, deductive reasoning techniques were developed to draw valid conclusions that follow logically from premises known to be true. The development of reasoning methods with such a priori guarantees was a key factor in the advancement of modern science, mathematics, and engineering; notably, according to philosophers such as Charles Sanders Peirce, the interplay between abduction, deduction, and induction forms the basis of the scientific method and hence all modern science. Attempts to mechanize logical reasoning can be traced back to 13th-century philosopher Ramon Lull and lie at the heart of the concept of computation. Probabilistic reasoning and inference have also profoundly impacted reasoning, often relying on the celebrated theorem by Thomas Bayes on inverse probability that also forms the basis for many machine learning and statistics approaches. Finally, the evaluation of correct (sound) reasoning lies at the heart of most quantitative assessments of human cognition. Not surprisingly, reasoning has been central to the AI enterprise. Indeed, the earliest research in AI – from Logic Theorist onwards [1] – had a strong focus on reasoning [2]. Since the 1960s, AI has also embraced probabilistic reasoning and models, initially for medical diagnosis [3]. Since then, the reasoning tasks addressed in AI research have covered the gamut from planning and temporal reasoning to diagnosis and explanation. While early AI has paid attention to both plausible reasoning (case-based, analogical, qualitative) and sound formal reasoning with guarantees (logical, probabilistic, constraint-based), over the years, the focus has shifted more towards reasoning with formal guarantees. There are good reasons for this when designing AI systems and techniques that compensate for human limitations and weaknesses since reasoning with guarantees is challenging for humans. This has led to practically impactful applications of AI systems such as SAT, SMT, and constraints solvers, including the verification of correctness properties of computer hardware and software, the safety of communications protocols, the design of new proteins, and, more recently, the robustness of neural networks against adversarial attacks. It has also resulted in probabilistic graphical models [4, 5], which are powerful modeling and inference tools that have found their way into numerous applications of reasoning in medicine, robotics, and beyond. Current State & Trends The emergence of the Internet and the associated technology that made it possible to capture the human digital footprint at scale, as well as the leaps in computing power, have made possible novel approaches to learning bottom up from data. Of particular interest are large pre-trained models, such as LLMs, that have shown surprising abilities in plausible reasoning. Unlike the earlier research on reasoning in AI, LLMs have focused on plausible reasoning patterns as they emerge automatically after large-scale training on petabyte corpora. While the results have been quite remarkable so far, the reasoning in this context has been of the “plausible” variety with no guarantees. Meanwhile, sound formal reasoning techniques remain key to important and impactful applications of cuttingedge AI technology for the verification of computer hardware and software, as well as for real-world planning and resource allocation problems. They are also increasingly recognized as a crucial basis for the formal verification of machine learning techniques such as neural networks, e.g., in the context of local robustness against adversarial attacks [6]. Significant research activity takes place in these areas, focusing on improving various types of reasoning algorithms (notably with respect to their computational complexity), leveraging learning within sound formal reasoning, and combining reasoning and learning techniques [7, 8]. Research Challenges Bringing some of the rigorous a priori or post hoc guarantees back into plausible reasoning patterns turbocharged by the pre-trained models has become an active and promising area of research – especially where AI systems need to work autonomously in safety-critical domains. Research on so-called “large reasoning models” as well as on neurosymbolic approaches is addressing these challenges. Furthermore, even though formal reasoning with correctness guarantees is currently considerably less in vogue than the use of generative AI techniques for plausible reasoning, formidable and essential challenges also remain in that area. In this context, the combination of machine learning techniques with formal reasoning techniques holds considerable promise for economically and socially valuable breakthroughs, notably in the area of AI safety and transparency. The questions and challenges we face range from the philosophical: * What exactly is “reasoning”? to the practical: * Can LLM ‘reasoning’ be trusted? and include: What does the future hold for the advancement and role of symbolic reasoning? * To what extent can LLMs or other generative models reproduce or replace symbolic reasoning? * To what degree will symbolic reasoning be necessary or sufficient to overcome the current limitations of LLMs? * How well can AI reasoning, especially LLM ‘reasoning,’ be explained and understood? * How can computers better understand and simulate human reasoning? * What is the role of collaborative reasoning between humans and computers? * How best can LLMs and symbolic reasoning be integrated into “neurosymbolic reasoning”? * Are further breakthroughs, beyond both LLMs and traditional symbolic reasoning, required to achieve AGIlevel reasoning? * What forms of reasoning can best support humans when dealing with various challenges, e.g., in medical, scientific, engineering, and legal domains? Community Opinion The AAAI community appears to strongly agree on the importance of reasoning in AI systems. In our community survey, slightly over 55% of the respondents chose to answer specific questions related to the topic of reasoning. Of these, 79% indicated that the topic of reasoning is relevant to their research (with 44.7% marking it as “very relevant”). Of the properties required for referring to a process as reasoning, 77.5% of the survey participants marked “Knowledge can be incorporated”, 72.5% “Explanations can be provided,” and 56.9% “Involves multiple steps to arrive at a conclusion”. Interestingly, merely 37.4% indicated “Guaranteed correctness of inference results/outcomes”, and only 23.7% that “A formal system and solver is used,” which reflects the recent focus on informal, plausible reasoning, likely in the context of generative AI methods. This suggests that an effort may be warranted to better communicate the importance and success of formal, sound reasoning techniques. Finally, 44.7% of respondents agreed that “Reasoning involves a search process.” There was broad agreement among survey participants that focusing reasoning research in AI on humanlevel reasoning is valuable (41.6%) or even essential (47%); similarly, a focus on domain-specific reasoning abilities was seen by 49.6% of respondents as valuable, and by 42.8% as essential. This clearly reflects the importance attributed to a research focus on reasoning. The community also sees an exciting potential of synergy offered by logical and probabilistic models of reasoning that were developed in AI prior to large pre-trained models. This is clearly reflected in the fact that 76.9% of survey participants marked the integration of learning and reasoning approaches as very important (6 or 7 on a scale of 7); interestingly, the percentage of respondents that considered Explainability and verifiability as very important was similarly high (at 71.7%). Finally, 61.8% of survey participants estimated the minimal percentage of symbolic AI techniques required for reaching human-level reasoning to be at least 50% (with 24.8% estimating it at 75% or more, compared to 38.2% estimating it at 25% or below). What remains unclear is the degree to which AI researchers and practitioners realize that decidedly superhuman levels of reasoning are required for and displayed in the prominent and successful applications of formal AI reasoning techniques for scientific and mathematical discovery and engineering applications, as well as in AI safety. 1. Newell, A. & Simon, H. (1956). The logic theory machine: A complex information processing system. IRE Transactions on Information Theory 2: 61-79. 2. Brachman, R. and Levesque, H. (2004) Knowledge Representation and Reason

Authors 0

1. Author list not loaded yet.