))
“Contra papers claiming superhuman AI forecasting ” by nikos, Peter Mühlbacher, Lawrence Phillips, dschwarz
Manage episode 439877925 series 3364758
コンテンツは LessWrong によって提供されます。エピソード、グラフィック、ポッドキャストの説明を含むすべてのポッドキャスト コンテンツは、LessWrong またはそのポッドキャスト プラットフォーム パートナーによって直接アップロードされ、提供されます。誰かがあなたの著作物をあなたの許可なく使用していると思われる場合は、ここで概説されているプロセスに従うことができますhttps://ja.player.fm/legal。
[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]
Widespread, misleading claims about AI forecasting
Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.
These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:
Outline:
(00:24) Widespread, misleading claims about AI forecasting
(03:02) What does human-level or superhuman forecasting mean?
(04:08) Red flags for claims to (super)human AI forecasting accuracy
(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)
(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)
(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)
(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)
(15:19) Takeaways
(16:17) So how good are AI forecasters?
The original text contained 1 footnote which was omitted from this narration.
The original text contained 6 images which were described by AI.
---
First published:
September 12th, 2024
Source:
https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting
---
Narrated by TYPE III AUDIO.
---
…
continue reading
Widespread, misleading claims about AI forecasting
Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.
These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:
- Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:
- A post on Marginal Revolution with the title and abstract [...]
Outline:
(00:24) Widespread, misleading claims about AI forecasting
(03:02) What does human-level or superhuman forecasting mean?
(04:08) Red flags for claims to (super)human AI forecasting accuracy
(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)
(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)
(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)
(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)
(15:19) Takeaways
(16:17) So how good are AI forecasters?
The original text contained 1 footnote which was omitted from this narration.
The original text contained 6 images which were described by AI.
---
First published:
September 12th, 2024
Source:
https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting
---
Narrated by TYPE III AUDIO.
---
Images from the article:
365 つのエピソード
シェア
