The rapid adoption and advancement of Large Language Models (LLMs) changed the way we approach marketing tasks, from content creation and copywriting to strategic planning and campaign development. LLMs are supporting and simulating human creativity for more efficient and effective output. Despite the growing interest and investment, there has been a lack of comprehensive, unbiased studies evaluating their performance against real-world marketing use cases.
To address this gap, our team at NoGood conducted a comprehensive study in March 2024. We recognized the need for a study that was not only conducted for marketers by marketers but also focused on the practical applications and challenges faced by marketing professionals in their daily work.
In November 2024, we conducted an additional study to capture the most recent LLMs. This impartial study asked expert marketers how they would rank the leading LLMs answers compared to their own responses to the marketing prompts.
Background
Since 2023, the global AI market has grown exponentially. Statista projected that Generative AI (Gen AI) is likely to reach a market size evaluation of $62.72 billion in 2025 and continue to grow at a staggering rate of 41.53%, resulting in an evaluation of $356.10 billion by 2030.
This surge is being spearheaded by the United States, which boasts the world’s largest Gen AI market with a projected valuation of $20.29 billion in 2025. In marketing alone, the global Gen AI market size is estimated to be worth around 4.5 billion USD in 2025 and is expected to grow at a CAGR rate of 31.8%, reaching 41.1 billion USD in 2033.
Generative AI Adoption Per Industry
Gen AI has impacted product development and internal processes across industries. Studies by McKinsey found that global AI adoption has jumped from 50% in 2023 to 72% in 2024. The majority of respondents reported use of generative AI in their marketing, sales, service-development, and IT functions.
Even with the large jump in adoption of AI across the board, there’s still a level of selectivity depending on the industry. The March 2024 LLM Report noted that the marketing and advertising industry has emerged as a frontrunner in AI adoption, with 37% of professionals already utilizing this technology in their daily tasks. Marketers are seamlessly integrating AI tools and platforms to streamline their creative processes and glean additional insights from their data to make informed decisions.
Within these industries, adoption across employee functions is also divided. Unsurprisingly, those in marketing and sales positions are most likely to use Generative AI at 34%. The least likely to use Gen AI are those working in manufacturing positions at 15%.
To better conceptualize the complete market size evaluation, it’s helpful to compare the breakdown of monthly traffic across major LLMs in October 2024 compared to January/February 2024. Although not all numbers are disclosed, we can infer an approximation of traffic numbers based on market share.
Nearly all LLM platforms experienced significant growth in traffic over this period with the exception of Google’s Gemini. While Gemini saw a slight 6% decline, this is likely attributed to the launch of Gemini as an inbuilt feature on Android phones, which may have made the web version less attractive for users. If this is true, maintaining stable traffic on Gemini’s web platform despite the shift is an impressive achievement in itself.
Perplexity showed the highest growth with traffic surging by 113%, while ChatGPT continued to dominate the space by driving both the highest traffic of all the LLMs and an 83% increase in traffic.
Purely from a data standpoint, it’s quite impressive. However, this study fails to answer a critical question. How effective are these LLMs at actually doing the job they’re tasked with completing? We aim to find the answer to this small piece of the puzzle with our innovative study that compares LLMs and their real answers to marketing tasks.
Generative AI Adoption in Marketing
Brands are eager to leverage the potential of AI-powered technology, and they’re turning to new and innovative tools and platforms to support this pursuit. This is driving adoption of AI technologies across various sectors. A comprehensive report by McKinsey found that B2B organizations saw a 10-20% revenue increase with the implementation of advanced analytics and AI.
Large companies like IBM are leveraging LLM marketing applications to generate brand content and tailor marketing campaigns to specific regions. Innovative companies like Goodie are helping brands develop AI brand narratives, focusing on positioning brands on the new digital shelf of AI answer engines.
Capabilities previewed by ChatGPT’s o1 imply that LLMs are shifting into problem solving models that use analytical and critical human-like thought processes to arrive at an answer. Marketers are eager to find out what this means in the broader scheme of their work. By analyzing the most recent and most popular models of LLMs, the below study offers marketers unique insight into the capabilities of each of these platforms and which they can trust.
Marketing Evaluation Study
The study involved a panel of 40 seasoned marketers across a variety of sectors (creative, advertising, brand development, and more) with a minimum of 6 years of experience in the field. The experts were tasked with reviewing the anonymized outputs from leading LLMs across a range of common marketing use cases, including copywriting, marketing strategy development, content creation, creative ideation, and campaign planning.
Prompts were crafted to be realistic and based on real case examples. The prompts asked across models were consistent to ensure all LLMs could be fairly evaluated. No single model was favored to outperform others; this is a common issue when LLM evaluations are conducted by the companies running the LLMs themselves.
LLMs Covered in the Study
- ChatGPT 3.5 by OpenAI
- ChatGPT 4 by OpenAI
- ChatGPT 4o by OpenAI
- ChatGPT o1 Preview by OpenAI
- Claude 2.1 by Anthropic
- Claude 3 by Anthropic
- Claude 3 Sonnet by Anthropic
- Gemini 1.5 by Google
- Llama 2 by Meta
- Llama 3.1 by Meta
- Perplexity by Perplexity
This study aims to provide marketers with valuable insights into the strengths and weaknesses of each LLM, empowering them to make informed decisions when incorporating these tools into their marketing strategies and workflows.
The graphs below show a broad overview of how the various LLMs performed across six different marketing tasks. For a more in-depth analysis and breakdown of the findings, download the free report by filling out the form below.
With the increased use of Gen AI across industries, it’s more important than ever to understand which tool is best suited for what purpose and how one tool’s solutions compare to another’s. This report addresses this question by evaluating the performance of ChatGPT 3.5, ChatGPT 4, ChatGPT 4o, ChatGPT o1 Preview, Claude 2.1, Claud 3, Claude Sonnet 3, Google Gemini 1.5, Llama, Llama 3.1, Perplexity LLMs across six different marketing tasks using a tested evaluation criteria.
