Facing Google Search Disruption: Artificial Intelligence Optimization (AIO) — Embrace or Resist?

As AI-driven search tools like OpenAI’s SearchGPT gain traction, they’re raising questions about the future of search and how it might impact Google’s long-standing dominance. SearchGPT and other AI models are moving towards a search experience that provides direct, conversational answers rather than a list of links. This shift toward AI-powered search could fundamentally change what we expect from search engines—especially if AI’s results become faster, more accurate, and free from ad influence.

Here’s an exploration of how this shift works, why it’s happening, and what it could mean for Google and the search landscape as a whole.

Google’s Hold on Search—and Where It’s Vulnerable

For the past two decades, Google has been synonymous with search. As of October 2024, it owns 89.33% of the global search engine market share (gs.statcounter.com), generating significant ad revenue through traditional search-based advertising.

In Q3 2024 alone, Google’s ad revenue totaled $65.9 billion, up from $59.6 billion year-over-year. Within this, Google Search and Other contributed $49.4 billion, a 12% increase year-over-year (Alphabet Q3 2024 Earnings). Meanwhile, YouTube Ads generated $8.9 billion, also up 12%, and Google Network, which covers ads on third-party sites, was slightly down, coming in at $7.5 billion versus $7.7 billion in Q3 2023.

These numbers highlight Google’s reliance on an ad-driven model that, while profitable, depends on users clicking through to various sites. The ad-centric approach has served Google well for years, but as AI-based search models like SearchGPT bypass traditional search ads by summarizing content directly, the shift could gradually draw users—and their engagement—away from Google’s paid placements.

Google’s Strategic Recalibrations and the Push for Diversification

Alphabet’s Q3 2024 results reveal that Google’s parent company is making significant moves beyond its core search revenue. Google Cloud, for instance, posted $1.9 billion in operating income, marking it as a profitable segment for Alphabet. This comes as Cloud revenue grew 35% year-over-year, illustrating how Alphabet’s investments in cloud infrastructure and AI services are paying off. Although Google Services remains the primary revenue driver, Cloud’s profitability and scale are giving Alphabet some breathing room and revenue diversity.

Meanwhile, YouTube’s ad and subscription revenue exceeded $50 billion over the past four quarters, underscoring Alphabet’s success in making YouTube a standalone media powerhouse through premium subscriptions, creator content, and live-streaming. Alphabet’s continued cost-cutting measures—over 1,000 fewer employees since last year, restructuring costs totaling $607 million—also indicate a focus on driving efficiency while still investing heavily in AI and R&D.

To boost investor confidence, Alphabet allocated $2.5 billion in dividends and committed an additional $0.20 per share for December while continuing an aggressive stock buyback strategy, spending $15.3 billion in Q3 alone.

Ads vs. Subscription: Is the Future of Search Ad-Free?

A key differentiator between SearchGPT and Google Search is the absence of ads in OpenAI’s model. SearchGPT uses a subscription-based approach, bypassing ads and allowing users to access an ad-free, information-focused experience. Sam Altman, CEO of OpenAI, even praised this model on Twitter, noting the appeal of a search tool unencumbered by sponsored results and suggesting that a subscription model might be a viable alternative for AI-powered tools.

The idea of subscription-driven search raises questions about whether users might prefer an ad-free experience that emphasizes content relevance over sponsored rankings. This model could provide a level of objectivity that ad-funded models struggle to maintain, giving users the answers they seek without commercial influence.

The Monetization Challenge — Can Websites Still Make Money in an AI-Driven Search Landscape?

The rise of AI-driven chat interfaces poses a real challenge for websites that rely on traffic to generate ad revenue or monetize through other means. Traditional search engines send users directly to content sites, where ads and other revenue-generating models support the creation of content. However, with chat-based interfaces like ChatGPT and similar tools that summarize or directly answer user questions, users may not feel the need to click through to the source sites—potentially disrupting the ecosystem that funds content creation.

Here are some key issues and evolving solutions around this tension

1. Reduced Traffic and Ad Revenue — If users get answers directly from chat interfaces, websites receive less traffic, which means fewer ad impressions and lower revenue. This creates a dilemma: websites bear the cost of creating and hosting high-quality content, while the benefits (in terms of user engagement and revenue) may shift to the chat-based platform.
2. Content Licensing Agreements — In response to this issue, companies like OpenAI have started to enter licensing agreements with major publishers (e.g., Dotdash Meredith, News Corp). These deals offer content creators some form of compensation, as the AI models pay for access to their content. However, such agreements are selective, and smaller or independent publishers may not benefit from these arrangements.
3. Exploring Subscription Models — Some AI platforms, like SearchGPT, are experimenting with subscription-based models rather than ads. If a significant portion of revenue comes from subscriptions rather than ad impressions on websites, there may be an opportunity for AI companies to share revenue directly with content creators. However, it’s still unclear whether subscriptions can fully replace the ad-based ecosystem.
4. Increased Emphasis on Brand Loyalty and Exclusive Content — As AI summarization becomes more popular, publishers may pivot towards offering exclusive or gated content accessible only through direct site visits or subscriptions. Brands with loyal audiences or high-value content could find new ways to sustain revenue by focusing on members-only access, newsletters, or direct engagement strategies.
5. Creating Value-Added Content That AI Can’t Replicate — Another possible solution is for publishers to create interactive, multimedia, or community-driven content that’s harder for AI to summarize effectively. Interactive data visualizations, real-time discussions, and exclusive video content are examples of assets that might drive users back to websites directly, bypassing the limitations of a text-based summary from an AI interface.

Will AI-Powered Search Replace Google?

Let’s be realistic: for now, AI-driven search engines are not a one-to-one replacement for traditional search, and Google isn’t going anywhere. But the AI search experience, with its focus on conversational responses and ad-free subscription options, has the potential to challenge Google’s search model on certain fronts.

Here are a few key areas where this competition could heat up:

• AI-based search provides immediate, conversational answers, which could attract users looking for straightforward information without the need to click through. For example, in customer service applications or casual browsing, AI search might edge out traditional search for convenience.
• Bing’s backing of SearchGPT highlights a growing alliance between AI and traditional search engines, one that’s positioning itself as an alternative to Google. While Google has launched its own AI initiatives like the Search Generative Experience (SGE), it’s still very much tied to its ad-driven model, which AI search can bypass altogether.
• Users have become increasingly concerned about privacy and ad-tracking, and some see AI search as a less invasive option. Additionally, publishers blocking AI bots may signal a push for new standards around data use, with AI companies having to adapt their practices to comply with publishers’ demands.

Even with these advantages, there’s no certainty that AI search will replace Google in the foreseeable future. But it’s clear that user behavior is evolving, and with it, the expectations for search. AI-driven tools like SearchGPT might not dethrone Google, but they will continue to reshape how people engage with search platforms, sparking innovation and competition.

Sam Altman, CEO of OpenAI, highlighted this reimagining of search in his conversation with Lex Fridman:

I find that boring. I mean, if the question is if we can build a better search engine than Google or whatever, then sure, we should go, people should use the better product, but I think that would so understate what this can be. Google shows you 10 blue links, well, 13 ads and then 10 blue links, and that’s one way to find information.

But the thing that’s exciting to me is not that we can go build a better copy of Google search, but that maybe there’s just some much better way to help people find and act on and synthesize information. Actually, I think ChatGPT is that for some use cases, and hopefully we’ll make it be like that for a lot more use cases.

Sam Altman on the Lex Friedman Podcast

Altman’s vision emphasizes that AI-driven models like ChatGPT are not simply trying to outperform Google on its own turf but to create a fundamentally different, more direct approach to information retrieval.

This vision raises a crucial question: Does AI-driven search like SearchGPT actually drive meaningful clicks to source sites, or does it bypass them altogether? Google’s model, while ad-heavy, still sends users directly to content sites. But with AI tools providing comprehensive answers in-app, users may skip visiting the original source entirely, impacting traffic for publishers.

For comparison sites and affiliate-focused publishers, this could be even more challenging. When ChatGPT can deliver product details and summaries in-app, users may see less reason to click through to comparison or review sites, potentially redirecting engagement directly to brands. Convincing publishers to grant AI access will likely require assurance that this new model can offer them valuable, engaged traffic rather than siphoning off users without benefit.

The New York Times and Its Blocking of AI Bots Like SearchGPT and Others

As AI-driven tools like SearchGPT, Perplexity, and Claude gain traction, major publishers, including The New York Times, are choosing to restrict access by these AI crawlers. Through their robots.txt file, The New York Times effectively blocks a broad range of AI-based tools and other bots from accessing their content, aiming to control how and where their material is used.

Here’s an excerpt from The New York Times’ robots.txt file, illustrating their blocking approach for various AI tools and web crawlers:

Disallow Rules

User-agent: anthropic-ai
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Google-CloudVertexBot
Disallow: /
Allow: /wirecutter/

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Guarding the Gates: Why Publishers Are Blocking AI Bots

This comprehensive block restricts various AI and data-scraping bots from accessing The New York Times’ content, marking a strategic move to prevent unauthorized use and repurposing by AI tools without explicit permission. This tactic is increasingly common among publishers seeking tighter control over how AI models access and use their content, especially for training purposes.

By updating their robots.txt files to block bots like OpenAI’s GPTBot, Google’s Google-Extended, and Perplexity’s PerplexityBot, The Times and other media giants aim to safeguard their proprietary material, intellectual property rights, and the unique value of their original reporting. As the influence of AI-driven content grows, the decision to restrict AI bots reflects broader concerns over potential misuse of content in AI development.

This block underscores the caution with which major outlets are approaching AI access, recognizing that AI’s rapid advancement requires stricter boundaries to protect content assets in an era of increasing demand for high-quality, responsibly-sourced information.

Originality.ai have produced an AI bot block tracker to monitor across the top 1,000 websites AI based bots being blocked:

List of OpenAI’s bots:

• GPTBot: This bot is associated with OpenAI’s Generative Pre-trained Transformer (GPT) models, which are designed to understand and generate human-like text. GPTBot is utilized for tasks such as drafting emails, writing code, and answering questions.
• OAI-SearchBot: This bot is used for accessing OpenAI’s search capabilities, enabling users to retrieve information from the web through AI-driven search models. It provides conversational responses to user queries, enhancing the search experience.
• ChatGPT-User: This represents the user interface for ChatGPT, a conversational AI model that engages in dialogue with users. ChatGPT-User is designed to assist with a wide range of tasks, from answering questions to providing recommendations.
• OpenAI API Bot: This refers to various API integrations using OpenAI’s technology, allowing developers to incorporate AI capabilities into their applications. The OpenAI API Bot facilitates interactions between external applications and OpenAI’s models.
• DALL-E Bot: Associated with OpenAI’s DALL-E model, this bot generates images from textual descriptions. Users can input a description, and DALL-E Bot will create a corresponding image, enabling creative visual content generation.
• Whisper Bot: This bot is linked to OpenAI’s Whisper model, which transcribes and understands audio. Whisper Bot can convert spoken language into text, facilitating applications in transcription, translation, and voice-controlled interfaces.

What This Means for the Future of Search

As AI-powered search grows, we’re likely to see two search ecosystems emerging, one based on traditional link-driven results, anchored by Google’s ad model, and another that’s conversational and content-focused, where AI search models like SearchGPT thrive. If AI-driven models become more popular, search could shift towards a hybrid system where both traditional search engines and AI-based responses co-exist, each serving different user needs.

One thing is for sure, the future of search is no longer Google’s alone. AI-driven search engines have thrown a new set of rules into the game, and whether they take a significant share or not, they’re pushing traditional search engines to adapt or lose ground. We’re witnessing the evolution of search, and while it’s too early to call the game, AI’s role in shaping the future of information retrieval has already begun.

Can AI Search Be Gamed?

The Bing Dependency in AI Search Visibility

AI-driven models like SearchGPT heavily rely on Bing’s index, making content visibility in this new search landscape both an opportunity and a challenge. Testing confirms that visibility within SearchGPT’s results is directly tied to Bing’s indexing status. If Bing doesn’t index a site—due to bugs, penalties, or other issues—it won’t appear in SearchGPT’s search results (I’ve been able to verify this myself), even if it performs well on Google. This highlights a crucial dependency on Bing that many may overlook.

Could OpenAI eventually build a complex, algorithm-driven index that continuously crawls the web, with weighted rankings? Maybe, but it seems unlikely when they can leverage Bing’s Search API. This partnership allows OpenAI to leverage Bing’s extensive search history, established index of billions of websites, and a complex algorithm some might say (despite market share issues) comparable to Google’s, making the need for a separate index less pressing.

This dependency is significant, but not all AI search engines follow the same model. For instance, Perplexity AI initially relied on Bing and other search indexes until 2022, but it has since started developing its own index and language models. This strategic shift allows Perplexity to gain more control over its search capabilities and reduce reliance on third-party services. By building its own systems, Perplexity aims for tailored search results and fosters a more competitive search ecosystem.

We’ve started building our own index, allowing us more control over search results and reducing reliance on other technologies.

Aravind Srinivas, CEO of Perplexity

Prometheus is the engine powering the collaboration between Microsoft and OpenAI, effectively managing how user queries interact with Bing’s index. By seamlessly integrating your input with Bing’s indexing, ranking, and answer features.

Navigating the Complexities of AI-Driven Search Visibility

The dependency of AI-driven models like SearchGPT on Bing’s index introduces new tactical opportunities but also warrants careful consideration.

Recent observations by SEO specialist Ivan Hristov illustrate this point: he noted in a LinkedIn post that when a site experiences a Bing penalty, it may disappear from ChatGPT’s search results as well, while still performing well on Google (LinkedIn Post by Ivan Hristov). This finding suggests that AI-powered search models reliant on Bing can indeed be influenced by changes in Bing’s indexing and ranking systems.

However, this is far from a straightforward “SEO hack.” Any attempts to manipulate visibility within these AI-driven tools would require a sophisticated understanding of Bing’s algorithms and the potential repercussions.

The Role of Content Licensing Deals in AI Search

Adding complexity, OpenAI has been signing licensing deals with major publishers, including Dotdash Meredith and News Corp. For instance, in May 2024, News Corp agreed to a five-year content licensing deal with OpenAI valued at over $250 million, which includes access to current and archived materials from publications such as The Wall Street Journal, Barron’s, and The Times of London. Similarly, Dotdash Meredith, the publisher of brands like People, Better Homes & Gardens, and Investopedia, signed a deal with OpenAI for content licensing, further enhancing the depth of material available to these models.

These partnerships give SearchGPT and similar models privileged access to high-quality, proprietary content, adding exclusivity to what these AI models can retrieve and present. This means that while manipulating Bing’s indexing might control visibility to a degree, the presence of licensed content deals introduces a selective filter that limits AI-driven models’ dependence on broader web crawls. As AI-powered search evolves, only content providers with solid Bing visibility or direct licensing agreements with OpenAI may find consistent visibility on platforms like SearchGPT.

To Embrace or Not Embrace, That Is the Question?

For the majority of sites, embracing this shift might not just be an option—it might be the only viable path forward. We’ve built up a kind of “dependency” on Google, a reliance so ingrained that it’s almost like an addiction, making it tough to imagine an alternative. Google’s dominance has given it immense control over who sees our content, when, and how often. But with the rise of AI-driven search models, we have a chance—maybe even an obligation (after all you don’t want to “get Chegged“)—to explore new ways of reaching audiences.

Adapting to AI-powered search won’t be easy. It demands new skills, fresh perspectives, and a willingness to step into uncharted territory. But for those of us willing to embrace the change, there’s potential for something bigger, a more balanced ecosystem where no single entity holds all the power, where content is more accessible, and where we’re free to innovate without the looming shadow of a single algorithm dictating our success.

In the end, the choice is ours. We can cling to what’s familiar, or we can embrace the future of search with all its uncertainty—and perhaps find new opportunities that are yet to be discovered.