2024 AI Development Report Summary by Fei‑Fei Li’s Team

01 Core Information

In 2024, the field of artificial intelligence (AI) made significant advances while also facing notable challenges.

AI has surpassed human performance on specific tasks such as image classification and language understanding, yet it remains limited on more complex tasks.

The industry continues to dominate AI research, especially in the production of machine‑learning models.

The cost of training large AI models keeps rising; for example, training GPT‑4 cost about $78 million, while Gemini cost roughly $191 million.

The United States, China, and Europe are the main contributors to AI models, with China leading in AI patents.

There is a lack of unified standards for AI model reliability assessment. Generative AI investment has surged, AI has improved worker efficiency and quality, accelerated scientific and medical progress, and the number of AI‑related regulatory provisions in the US has increased dramatically, raising global awareness of AI’s potential impact.

02 AI Research and Development

2.1 Key Points

AI research remains industry‑driven, with an increasing number of open‑source models, rising training costs, and the United States, China, and Europe leading large‑model development. China leads in AI patents, and the volume of AI research resources and related papers on GitHub continues to grow.

2.2 Comparative Information

From 2010 to 2022, the number of AI‑related papers has increased annually, with the United States maintaining the top position in publishing high‑quality machine‑learning models.

2.3 Will Models Exhaust Data?

AI model development relies on massive data, raising concerns that high‑quality data may be exhausted soon. Synthetic data can mitigate this, but models trained on synthetic data may suffer performance degradation.

2.4 Foundation Model Development

Foundation models are trained on broad datasets, are versatile, and applicable to many downstream tasks. Their deployment in real‑world scenarios is increasing, with varying numbers of releases across countries and organizations.

2.5 Training Model Costs

Training large AI models continues to cost tens of millions to hundreds of millions of dollars, reflecting growing resource investment in the AI field.

03 Technical Performance

3.1 Key Points

AI has outperformed humans on specific tasks; multimodal models such as Google’s Gemini and OpenAI’s GPT‑4 demonstrate capabilities in processing both images and text. New benchmark suites like SWE‑bench and HEIM, as well as human‑evaluated leaderboards for chatbots, reflect improvements in AI performance.

3.2 Major Model Releases

2023 saw the release of several important AI models, including Anthropic’s Claude, OpenAI’s GPT‑4, and Stability AI’s Stable Diffusion v2, many of which surpassed human levels on various benchmarks.

3.3 AI Performance

AI surpasses humans in image classification, English comprehension, and natural‑language inference, but still lags on competition math, multilingual understanding, and visual commonsense reasoning.

3.4 Multidisciplinary, High‑Difficulty Benchmarks (MMMU, GPQA, ARC)

New evaluation suites such as MMMU, GPQA, and ARC aim to assess AI’s multidisciplinary reasoning and abstract inductive abilities; while AI achieves some success, a gap remains compared with human experts.

3.5 Agents

LLM‑based AI agents have improved in automatically handling tasks in specific scenarios, as demonstrated by the 25 agents evaluated in AgentBench.

3.6 RLHF & RLAIF

Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with AI Feedback (RLAIF) are methods for training AI models to better align with human preferences; RLAIF shows superior performance on harmless dialogue generation.

3.7 LLM Performance Over Time

LLM performance evolves over time; some studies indicate that with new data and user feedback, performance on certain tasks may decline.

3.8 Techniques to Boost LLM Effectiveness

Techniques such as prompting, OPRO, and fine‑tuning are employed to enhance LLM performance by optimizing task description or reducing memory requirements.

3.9 Environmental Impact of Training AI Systems

Training large AI models consumes substantial resources and emits CO₂, impacting the environment. Nevertheless, AI can also be used to predict urban air quality and optimize energy usage, yielding positive environmental effects.

04 AI Reliability

AI reliability assessment involves privacy, data governance, transparency, explainability, security, and fairness. Comprehensive standards for evaluating large language models are lacking; political misinformation generation and detection, as well as political bias in ChatGPT, have become concerns.

4.1 Definition of AI Reliability

AI reliability is defined and evaluated across dimensions such as data governance, explainability, fairness, privacy, security, and transparency.

05 Economic Impact of AI

AI’s economic impact is multifaceted, improving production efficiency, reshaping labor markets, and influencing investment trends. Generative AI investment has grown rapidly; AI‑related job numbers have decreased, yet AI reduces corporate expenses and boosts revenue. China leads in industrial robotics; AI enhances worker productivity. Fortune 500 companies increasingly discuss AI, especially generative AI.

5.1 Major AI News

2023 notable AI news includes BioNTech’s acquisition of InstaDeep, Microsoft’s investment in OpenAI, the release of GitHub Copilot, integration of Einstein GPT with Microsoft Office, and Bloomberg’s use of LLMs for financial data analysis.

5.2 Job Market Information

AI‑related job market share has shifted; demand for AI skills in the US has declined, while Hong Kong shows relatively high demand. The number of newly founded AI companies worldwide continues to rise.

5.3 Developer Usage of AI Tools

Developers most frequently use AI tools such as GitHub Copilot and ChatGPT, with cloud service platforms also seeing widespread adoption.

06 AI Advances in Medicine and Education

AI applications in medicine and education have made significant progress. AI accelerates scientific advancement, exemplified by AlphaDev and GNoME. In healthcare, systems like EVEscape and AlphaMissence improve disease prediction and gene classification. The number of FDA‑approved AI medical devices has increased, and AI‑related academic programs are rapidly expanding worldwide.