Gartner releases key trends shaping the future direction of data science and machine learning

Gartner today released important trends affecting the future direction of data science and machine learning (DSML). With the rapid development and evolution of the DSML industry, data is increasingly important for the development and application of artificial intelligence (AI), especially the focus of investment is shifting to the field of generative artificial intelligence. 

Peter Krensky, research director at Gartner, said: “As the application of machine learning continues to rapidly expand across various industries, DSML is shifting from a pure focus on predictive models to a more pervasive, dynamic and data-centric technology field, and generative artificial intelligence. The craze for artificial intelligence (AI) is also driving this trend, and new capabilities and use cases are emerging for data scientists and their organizations.”

Gartner research shows that important trends affecting the future direction of the DSML industry include:

Trend 1: Cloud Data Ecosystem

Data ecosystems are transitioning from standalone software or hybrid deployment models to fully cloud-native solutions. Gartner predicts that by 2024, 50% of new cloud deployment systems will be based on a consistent cloud data ecosystem rather than manually integrated point solutions.

Gartner recommends that enterprise organizations evaluate two capabilities of the data ecosystem: one is to solve the problem of data fragmentation; the other is to access and integrate data outside their own environment.

Trend 2: Edge AI

Enterprise organizations increasingly need to create and process data at edge locations through edge AI, which will help them gain real-time insights, discover new business models, and meet strict data privacy requirements. Edge AI can also help enterprise organizations improve their AI development, orchestration, integration, and deployment capabilities.

Gartner predicts that by 2025, more than 55% of deep neural network data analysis will occur at the data capture point of edge systems, compared with less than 10% in 2021. Organizations should determine which applications, AI training, and inference capabilities need to be moved to edge environments near IoT endpoints.

Trend 3: Responsible AI

Responsible AI makes AI a positive force rather than a threat to society and AI itself. When enterprises need to use AI to make the right choices in terms of business logic and ethics, they will encounter many issues, such as business and social value, risk, credibility, transparency and accountability, etc. Responsible AI can help solve these problems. Gartner predicts that by 2025, 1% of AI service providers will use pre-trained AI models on a large scale, making responsible AI a focus of social attention.

Gartner recommends that enterprises should consider the risk factor when exploring the value of AI and remain cautious when using AI solutions and models. Suppliers should be given assurances that they are managing their own risks and compliance obligations to prevent potential financial loss, legal action and reputational damage to the organization.

Trend 4: Data-centric AI

This approach is no longer centered on models and code, but rather centered on data to build a more powerful AI system. Organizations will adopt solutions such as AI-specific data management, synthetic data, and data labeling technologies to address many data challenges such as data accessibility, volume, privacy, security, complexity, and scope.

Using generative AI to create synthetic data is a rapidly growing field that reduces the burden of obtaining real-world data to train machine learning models more efficiently. Gartner predicts that by 2024, 60% of AI data will be synthetic data, used to simulate reality, future scenarios and reduce AI risks, compared with only 1% in 2021.

Trend 5: Accelerating AI investment

Enterprise organizations entering the solution implementation stage, as well as industries hoping to achieve growth through AI technology and related businesses, will continue to accelerate investment in AI. Gartner predicts that by the end of 2026, AI startups that rely on basic models (large models trained through massive amounts of data) will receive more than $10 billion in investment.

In a recent Gartner survey of more than 2,500 corporate executives, 45% of respondents said that the recent ChatGPT craze prompted them to increase their investment in AI. 70% of the respondents said that their companies are in the stage of researching and exploring generative AI, and 19% said that their companies have entered the pilot or production stage.