Marketing has been one of the most significantly impacted functions by the rapid adoption and advancement of Large Language Models (LLMs). These powerful Generative AI tools are actively changing the way we approach various marketing tasks, from content creation and copywriting to strategic planning and campaign development. However, despite the growing interest and investment in LLMs, there has been a lack of comprehensive, unbiased studies evaluating their performance against real-world marketing use cases.
To address this gap, our team of experienced marketers conducted the first impartial study assessing the leading LLMs in the market. We recognized the need for a study that was not only conducted for marketers by marketers but also one that focused on the practical applications and challenges faced by marketing professionals in their daily work.
Background
The global AI market has already been valued at $241.8 billion in 2023 as compared to $135 billion in 2022, and it is expected to surge to a staggering $740 billion by 2030, reflecting a CAGR of 17.3%, according to Statista. This surge is being spearheaded by the United States, boasting the world’s strongest AI research capabilities, followed closely by Europe. Enterprises are seeing a steep adoption rate. According to OpenAI, 90% of Fortune 500 brands are actively using their tools and API internally.
Generative AI Adoption in Marketing
The marketing and advertising industry has emerged as a frontrunner in AI adoption, with a remarkable 37% of professionals already utilizing this technology in their daily tasks.
This rapid embrace is not surprising, considering the unique blend of creativity and data analysis inherent to marketing. AI seamlessly integrates with this blend, offering powerful tools for both creative exploration and data-driven decision-making.
In a 2023 study conducted with marketers in the United States, 73% of respondents reported using generative artificial intelligence tools, such as chatbots, as a part of their company’s work. This widespread adoption indicates a growing comfort level and understanding of the potential benefits these tools offer.
Generative AI, particularly Large Language Models (LLMs), has revolutionized the way people work across industries, providing innovative solutions to complex tasks. It has found prominent usage in various job functions, notably in industries such as IT and Technology, Finance, Healthcare, and Marketing.
Marketing Evaluation to Leading LLMs Report
The study involved a panel of 20 seasoned marketers, each with a minimum of 6 years of experience in the field. These experts were tasked with reviewing the output of anonymized leading LLMs across a range of the most common marketing use cases, including copywriting, marketing strategy development, content creation, creative ideation, and campaign planning. To ensure an unbiased evaluation, we carefully crafted prompts that were highly practical, realistic, and consistent across each model. This approach guaranteed that no single model was favored or engineered to outperform others, a common issue in LLM evaluations published by the creators of the models themselves.
The LLMs covered in the study:
- LLama 2 by Meta
- Gemini 1.5 by Google
- Perplexity by Perplexity
- ChatGPT (publicly available GPT 3.5) by OpenAI
- Claude 2.1 by Anthropic
- Claude 3 – Sonnet by Anthropic
By conducting this study, we aim to provide marketers with valuable insights into the strengths and weaknesses of each LLM, empowering them to make informed decisions when incorporating these tools into their marketing strategies and workflows. Our findings shed light on the variability in performance across different aspects, such as creativity, thoroughness, and coherence, highlighting the importance of selecting the right LLM for specific marketing tasks. The data presented in this report demonstrates the significant potential of AI applicability use case in marketing as the industry continues to evolve.
The graphs below offer a broad overview of how various Large Language Models (LLMs) performed across different marketing tasks. For a more in-depth analysis and breakdown of these findings, download the free report by clicking the download button below.
As the use of Generative Artificial Intelligence (Gen AI) continues to grow, a critical question emerges: Which Gen AI tool is best suited for what purpose, and how do the leading solutions compare? This report addresses this question by evaluating the performance of six Large Language Models (LLMs) across various marketing tasks and against diverse evaluation criteria.
