Concerns about generative AI bubbles are growing. To protect yourself from disillusionment, measure your level of disillusionment. … [+]
Meta just launched its AI assistant on Instagram, WhatsApp, Messenger, and Facebook like a missile into the corporate battlefield. The media is also following suit, reporting in detail on the “battle against ChatGPT.” But this chatbot war won’t go down in history like the actual robot wars. When Meta, OpenAI, Microsoft, and others make ostensibly competitive moves, they are not fighting to control a large source of power. Rather, it’s mostly theatrical, a battle for attention and stature.
At the end of the day, the field of generative AI is just cosmetic. Instead of demonstrating concrete, proven value, we all too often promote grandiose visions of endless possibilities.
But while chatbots like Meta’s AI assistant and ChatGPT are easier to use than other forms of AI, they are harder to use. good— in a way that creates measurable value for the company. Other types of AI may not enjoy the same degree of ease of use, but some AI, such as predictive AI, often provide higher benefits than genAI.
Considering today’s situation where steaks are sizzling because they don’t have enough to eat, anxiety is growing. washington post “AI hype bubble is deflating, says journalist”have a hard time finding examples of innovative change” investors recognize. Backlash against genAI’s over-promises Many other agree.
Even studies aimed at demonstrating the value of genAI can prove to be failures. Researchers at Stanford University are bullish on the technology, studying the productivity of teams using genAI to tackle business problems such as how to increase B2B sales and working without an AI assistant. compared to other teams. Much to their surprise, the researchers found that using genAI yielded more average ideas. One reason for this is that most of the data used for training naturally reflects common “inside the box” thinking, and also because humans using genAI overdo it. Trust and your own cognitive effort is reduced.
But these researchers remain bullish and suggest new guidelines for making the most of this new technology.
Benchmarking GenAI to establish tangible value
In the case of genAI, we usually don’t know what the returns will be because the genAI project simply ignores the benchmark. But if you’re not measuring value, you’re not chasing value. Only by evaluating the business benefits, or lack thereof, can you receive the feedback you need to successfully move your project forward. Companies can measure benefits as efficiency gains, such as time savings and increased productivity.
One of the things that slows down benchmarking is that stress testing a project can make you look like a party loser. AI hype is intoxicating, but nothing kills the buzz of daydreaming about its immense promise like a sober reading of today’s values. If you evaluate genAI’s performance, you may find that genAI is better than better and far from the dream of revolutionary AI.
But proven victories are better than pipe dreams. For example, adhere to the rare standards of Ally Financial, the largest all-digital bank in the United States. “Marketers can reduce the time required to produce creative campaigns and content by up to two to three weeks, with an average time savings of 34%.”
Or you can follow the lead of a rare Fortune 500 software company studied by MIT Sloan. By using conversational assistants, the company’s customer support team has increased the number of issues they can resolve per hour by 14%. The growth rate for entry-level and low-skilled employees was even higher, by 34%. This benchmark is unusual. Other companies such as Airbnb, Intuit, and Motorola report that they are just beginning to measure the value of genAI, but have not yet reported their findings.
Such success comes from careful application of genAI. For example, it is often possible to generate a first draft that is useful for mechanical tasks such as certain customer support activities. In contrast, it often produces content that is too obvious and generic to be useful for advanced writing tasks such as journalism, and is best used for copy editing or preliminary research instead ( (as long as it is manually fact-checked). Attempts to leverage genAI typically involve an ad hoc experimentation process. We live in the wild and unpredictable genAI West. Its value is not guaranteed.
But even if genAI’s efforts prove worthwhile, they will likely prove not delivering the revolutionary wins that industry leaders would have us believe possible. The current wave of hype continues a long tradition of AI theater. AI has always hung its hat on the alluring but relentlessly vague term “intelligence.” AI in general, and more specifically his genAI, taps into the ingrained excitement bred by decades of science fiction and breathless AI speculation. And genAI offers instant and broader appeal than perhaps any other technology. Anyone can use it intuitively in English and other languages (although critics say it should be supported more). However, the common narrative that technology is reaching common human-level capabilities is unfounded.
The best way to avoid genAI disillusionment is to benchmark genAI’s performance. Rather than indulging in glamorous stories about machine “intelligence”, focus on reliable use cases that deliver tangible value and measure that value to keep your project on track.
