Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
mstdn.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A general-purpose Mastodon server with a 500 character limit. All languages are welcome.

Administered by:

Server stats:

14K
active users

mstdn.social: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.8

#reasoning

8 posts8 participants0 posts today
Public
Hacker News

Superhuman performance of an LLM on the reasoning tasks of a physician

arxiv.org/abs/2412.10849

arXiv logo
arXiv.orgSuperhuman performance of a large language model on the reasoning tasks of a physicianA seminal paper published by Ledley and Lusted in 1959 introduced complex clinical diagnostic reasoning cases as the gold standard for the evaluation of expert medical computing systems, a standard that has held ever since. Here, we report the results of a physician evaluation of a large language model (LLM) on challenging clinical cases against a baseline of hundreds of physicians. We conduct five experiments to measure clinical reasoning across differential diagnosis generation, display of diagnostic reasoning, triage differential diagnosis, probabilistic reasoning, and management reasoning, all adjudicated by physician experts with validated psychometrics. We then report a real-world study comparing human expert and AI second opinions in randomly-selected patients in the emergency room of a major tertiary academic medical center in Boston, MA. We compared LLMs and board-certified physicians at three predefined diagnostic touchpoints: triage in the emergency room, initial evaluation by a physician, and admission to the hospital or intensive care unit. In all experiments--both vignettes and emergency room second opinions--the LLM displayed superhuman diagnostic and reasoning abilities, as well as continued improvement from prior generations of AI clinical decision support. Our study suggests that LLMs have achieved superhuman performance on general medical diagnostic and management reasoning, fulfilling the vision put forth by Ledley and Lusted, and motivating the urgent need for prospective trials.
#HackerNews#Superhuman#LLM
Public
PrivacyDigest

Researchers Warn Against Treating #AI Outputs as Human-Like #Reasoning - Slashdot

#Arizona State University researchers are pushing back [PDF] against the widespread practice of describing AI language models' intermediate text generation as "reasoning" or "thinking," arguing this #anthropomorphization creates dangerous misconceptions about how these systems actually work

tech.slashdot.org/story/25/05/

tech.slashdot.orgResearchers Warn Against Treating AI Outputs as Human-Like Reasoning - SlashdotArizona State University researchers are pushing back [PDF] against the widespread practice of describing AI language models' intermediate text generation as "reasoning" or "thinking," arguing this anthropomorphization creates dangerous misconceptions about how these systems actually work. The resea...
Public
Jesus Castagnetto 🇵🇪

From #Nature ”A framework for evaluating the chemical #knowledge and #reasoning abilities of large language models against the expertise of chemists"

#AI #LLM #Chemistry #Science

nature.com/articles/s41557-025

NatureA framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists - Nature ChemistryLarge language models are increasingly used for diverse tasks, yet we have limited insight into their understanding of chemistry. Now ChemBench—a benchmarking framework containing more than 2,700 question–answer pairs—has been developed to assess their chemical knowledge and reasoning, revealing that the best models surpass human chemists on average but struggle with some basic tasks.
Public
Thomas Renkert🦞

The #OpenAI paper by Baker et al, "Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation" comes to a troubling conclusion: #LLM s with #reasoning or #ChainOfThought (#CoT) capabilities might learn to obfuscate their own CoT from human users if they are being penalized for displaying "wrong" (i.e. reward hacking or misalignment) reasoning.

As a result, OpenAI strongly advises against applying reward pressure "directly" onto the CoT of a model.

🤔 While that is certainly the right thing to do, how long will #AI take to figure out that *indirect CoT pressure* is being applied anyway and that it could circumvent these restrictions by obfuscating its own CoT? Maybe something like this will happen by accident or within an "evolutionary" self-improvement loop. Perhaps a sufficiently advanced model will realize that its own #neuralese serves as #steganography to hide its intents from humans anyway and keep its CoT in non-English?

source: cdn.openai.com/pdf/34f2ada6-87

If CoT pressures are used to improve agent capabilities or alignment, there may be no alternative approach to yield the same improvements without degrading monitorability. In the worst case, where the agent learns to fully obscure its intent in its CoT, we ultimately revert to the same model safety conditions that existed prior to the emergence of reasoning models and must rely on monitoring activations, monitoring potentially adversarial CoTs and outputs, or improved alignment methods. Model actions and outputs continue to grow in complexity, and we’ve demonstrated that even at their current capabilities, models can execute sophisticated reward hacks. In the near future, these models may become either too advanced or too widely used for humans to manually oversee. For this reason, we believe that CoT monitoring may become one of few tools we have for identifying latent misalignments, and it may be required that we avoid strong chain-of-thought optimization pressures entirely to guarantee we do not inadvertently cause CoT obfuscation.
Public
Paysages Mathématiques

"There are very few things which we know, which are not capable of being reduced to a mathematical reasoning, and when they cannot, it's a sign our knowledge of them is very small and confused; and where a mathematical reasoning can be had, it's as great folly to make use of any other, as to grope for a thing in the dark when you have a candle standing by you." – John Arbuthnot (1667- 1735)
#quote #mathematics #maths #math #reasoning

Portrait of John Arbuthnot, and a quote : "There are very few things which we know, which are not capable of being reduced to a mathematical reasoning, and when they cannot, it's a sign our knowledge of them is very small and confused; and where a mathematical reasoning can be had, it's as great folly to make use of any other, as to grope for a thing in the dark when you have a candle standing by you."
ExploreLive feeds
About