Axioma News feed

RAG vs “Live” AI Search - people keep using these interchangeably.

If you’re a publisher, the difference isn’t academic. It directly impacts whether your content shows up in AI answers… or quietly disappears.

RAG (Retrieval-Augmented Generation) is, at its core, about grounding a model in a predefined set of sources. Before the model answers, it is pointed to a specific corpus - a knowledge base, a curated dataset, or a closed list of trusted sites - and retrieves relevant pieces from there.

It’s controlled, predictable, and bounded. And from a publisher perspective, that has a very clear implication: you either made it into that dataset, or you don’t exist in that system at all.

Live AI search works very differently. Instead of relying on a fixed set of sources, the system goes out to the open web in real time. It generates queries, hits search APIs, pulls the top results, visits those pages, and then synthesizes an answer on the fly.

It’s dynamic, competitive, and constantly changing. Which means that every time a query is made, you’re effectively competing again to be one of the sources selected.

So while both are forms of “retrieval,” the underlying mechanics, and the implications, are very different.

With RAG, the problem is inclusion.

With live search, the problem is ranking.

For publishers, this translates directly into how you think about distribution.

In a RAG world, visibility comes from being part of curated datasets, partnerships, or trusted source lists. In a live search world, visibility comes from being ranked high enough to be selected in real time.

If you’re not included in the first, you’re invisible. If you’re not ranked in the second, you’re also invisible.

Which means this isn’t a question of choosing one or the other. You need to think about both layers at the same time: how your content becomes a trusted, included source in closed systems, and how it remains discoverable and competitive in open ones.

Because AI isn’t a single traffic channel. It’s multiple access layers to your content.

And if you’re only optimizing for one of them, you’re missing a big part of how your content will be used going forward.

Most people think AI search works like magic...

It doesn’t. It’s actually a pretty aggressive, structured pipeline.

And importantly - this whole flow usually kicks in when the model decides it needs fresh information. Things that aren’t safely sitting in its training data. News, changing facts, long-tail queries, anything where being up-to-date actually matters.

When that happens, the model doesn’t just “figure it out.” It rewrites your prompt into multiple search queries, slightly different angles of the same intent, just to increase the chances of finding relevant information.

Those queries are then sent to a search API (often a mix of big web indexes like Google or Bing, depending on the product and partnership). From there, the agent pulls the top ~10–20 results (sometimes more) that are most likely to contain the answer.

Then comes the part most people don’t realize: it actually goes and visits those pages. In real time. Multiple HTTP requests firing in parallel, pulling content from across the web in a fraction of a second.

Each page gets parsed, cleaned, and compressed down to just the relevant bits. And then the model stitches everything together into a single response that feels coherent and complete - even if, occasionally, it’s slightly hallucinated.

So what looks like one clean answer…

…is really a swarm of parallel visits happening behind the scenes.

And this has a pretty big implication:

If you’re not in those top 10–20 results, you’re basically invisible to the agent.

AI doesn’t browse. It samples. And it samples from the same pool SEO has always been competing over - just much more aggressively.

Which means SEO isn’t going away. If anything, it just became the gating layer for whether you show up in AI answers at all. At least until the AI companies build their own index...

No ranking → no crawl → no mention → no traffic.

AI search isn’t replacing the web.

It’s compressing it.

Publishers are starting to see a new type of visitor on their websites: AI agents.

Not just one type - three different kinds of LLM agents are now crawling publisher content, each with a different purpose.

Understanding them is becoming critical for publishers.

1) The LIVE SEARCH Agent: These agents retrieve information in real time to answer user questions. When someone asks ChatGPT, Perplexity or Claudea question, the model may send an agent to fetch fresh information from publisher websites before generating the answer.

Purpose:

• Get the most up-to-date information

• Cite sources

• Improve response accuracy

Think of them as the AI equivalent of a user clicking a search result.

2) The TRAINING Agent: These agents collect content to improve future versions of models. They gather large amounts of data that can be used for:

• training new models

• fine-tuning models

• improving reasoning and knowledge coverage

This traffic is not about answering a question right now — it's about making future models smarter.

3) The INDEXING Agent: Indexing agents crawl sites to build structured knowledge indexes that LLMs can access quickly. They organize information so models can:

• find relevant documents faster

• perform retrieval-augmented generation (RAG)

• reduce hallucinations

You can think of them as the AI equivalent of a search engine index.

Why this matters for publishers: For the first time, publishers are being visited not just by humans and search engines - but by AI agents with different goals.

Each type of agent creates different value:

• Live search agents drive visibility and citations

• Training agents contribute to model intelligence

• Indexing agents enable AI knowledge systems

The big question for the next few years:

How should publishers control, measure, and monetize these new AI visitors?

Because whether we like it or not, AI agents are becoming a major class of internet traffic.

AI Agents traffic is small, but growing fast. And nobody sees this coming.

AI browsing agents still represent a small share of total web traffic (~0.1-0.2%), but the growth curve is steep and the behavioral impact is outsized.

Unlike crawlers, agents perform real-time task execution and information retrieval, often bypassing traditional analytics and attribution mechanisms.

The shift is subtle, but it signals a deeper change: the web is transitioning from human navigation to machine-mediated access.

The companies that build visibility and control layers early will have a significant advantage.

[Source: AI Agents: The Hidden Undercurrent of the Web - Oct’ 25 by Topaz Hurvitz]

Cost per crawl isn’t the end game. It’s leverage.

For the past year, publishers have debated charging AI companies per request - simple, programmable, enforceable. And technically, it works. Controls like pay-per-crawl turn bot management into an economic gate.

But look at where the real money is flowing. The biggest deals aren’t metered per fetch - they’re multi-year licensing agreements tied to training, product integration and commercial terms. Negotiated. Strategic. Not transactional.

In a world of real-time retrieval, RAG and agent workflows, a single crawl is a weak proxy for value. One request might power nothing - or a commercial answer seen by millions.

The shift is clear: the monetizable event isn’t the page fetch. It’s the downstream utility.

Cost per crawl may create leverage. But the market is already moving toward something bigger.

You’re not blocking AI traffic - you’re blocking your own visibility.

AI visibility matters even if your end decision is “deny everything,” because the traffic itself is the signal. Which models are hitting you, what they’re pulling, how often, and from where tells you what your content is worth in the new AI supply chain. If you can’t see it, you’re not making a policy decision - you’re guessing.

And this isn’t “bot traffic” in the old sense. We’re moving from periodic indexing to real-time retrieval, agent workflows, and automated content extraction at scale. That shift changes everything: cost, infrastructure load, and most importantly, the economics of who benefits from your content.

Blocking might be the right move - but without observability, you don’t know what you’re blocking. You can’t quantify demand, you can’t identify high-value endpoints, and you can’t separate opportunistic scraping from legitimate product integrations. You’re treating every request like the same threat, and leaving strategy on the table.

The publishers who win this next era won’t be the ones who block the fastest. They’ll be the ones who measure first, understand the patterns, and then choose control, monetization, or selective access from a position of leverage.

In an AI world, visibility isn’t a nice-to-have. It’s power. If you want to democratize visibility across the org, drop me a comment or PM.

Everyone’s talking about Claude’s “anti‑ads” Super Bowl spot.

I get the sentiment: people are tired of aggressive tracking and low‑quality advertising. But I worry this framing may backfire – both for Anthropic and for the AI ecosystem.

The current economics of frontier AI simply don’t work on subscriptions alone. Even if every user happily paid $20 a month, it’s hard to see how that covers the massive, ongoing spend on GPUs, data centers, and R&D at scale.

At some point, either prices need to go way up, products need to get much more constrained, or additional revenue streams have to appear.

That’s where advertising, in a broad sense, comes in. Ads can be intrusive, manipulative, and misaligned – but they can also fund access, create real entertainment, and add value. The Super Bowl itself is the perfect example: we tune in for the ads as much as for the game.

The real question isn’t “ads or no ads.” It’s:

* What kind of monetization aligns with user trust?

* How do we design incentives so that AI products stay affordable, sustainable, and open?

* Can we build a new generation of “good ads” and value‑based sponsorships that respect attention instead of exploiting it?

If AI is going to be a general‑purpose utility, we need honest conversation about the business model. Pretending we can scale limitless compute on $20 subscriptions and vibes alone feels like a short‑term narrative that won’t survive the unit economics.

The UK CMA saying that publishers and online businesses should be allowed to opt out of Google AI Overviews (= block Google AI agents without blocking Google Search crawler) feels like one of those quiet moments that actually matters a lot.

This isn’t about being anti-AI. It’s about control and choice.

AI Overviews sit on top of publishers’ work and often answer the question before anyone clicks through. The journalism still gets funded, edited, hosted, and legally defended by publishers, but the value capture increasingly happens somewhere else. Until now, the “choice” has mostly been theoretical: participate or risk disappearing.

What the CMA is really signaling is that publishers deserve a real say in how their content is used, summarized, and surfaced - not just a default setting decided by the platform.

Opt-out isn’t the destination. It’s leverage. The endgame is opt-in on clear terms, with transparency around usage and a value exchange that actually makes sense for the people creating the content in the first place.

If AI is becoming the primary interface to information, then control over how journalism flows into that interface can’t be an afterthought.

This won’t stay a UK-only conversation for long.

‍

The article on PressGazette (Jan 22nd, see in first comment) is not surprising. We see it on every search on our LLM of choice, wondering how come the big media brands are missing from the sources list.

The data indicates that nearly 8 in 10 of the world’s top news websites are blocking AI training agents, due to a perceived lack of reciprocal value from these models utilizing their content.

Publishers are asserting control over how AI systems interact with their content, reacting to unpaid data usage and a lack of transparent value exchange.

Blocking agents is not merely a technical switch; it represents a statement about value, rights, and sustainability:

- Publishers seek fair compensation for their journalism.

- They are safeguarding brand equity and editorial integrity.

- They are resisting business models that extract content without providing traffic, revenue, or licensing fees.

AI will increasingly dominate the world’s information landscape. If news publishers feel excluded from the economic benefits and sidelined in visibility through AI answer citations, this fragmentation will only accelerate.

Instead of defaulting to defensive blocks, the future of publishing should focus on strategic partnerships, where:

- Publishers reclaim control over how their content is accessed

- LLMs contribute fairly - either by paying-per-crawl or increasing referral traffic, for example by adopting a UX more similar to Perplexity's.

- Users receive trustworthy, cited, and up-to-date information

The choice is not between AI or publishers; it is between extractive arrangements and sustainable cooperation.

Some new details are quietly emerging about OpenAI’s ad rollout, and they’re pretty revealing.

This isn’t a big-bang ads platform launch. It’s a tightly controlled experiment.

Early partners look to be large brands like Klarna, Shopify and Expedia. Budgets are intentionally small (under $1M) and there’s no self-serve interface. Everything is hands-on with OpenAI.

The most interesting part: ads are being measured on views, not clicks.

That tells you a lot.

This isn’t about driving traffic. It’s about shaping perception inside the AI answer itself. Brand presence at the moment a decision is being formed, not a link you might click later. Click on the ad sometimes will lead to a conversation, not a landing page: "you might see an ad and be able to directly ask [The Brand] the questions you need to make a purchase decision" (from OpenAI's Jan 16th announcement).

Access, for now, is invite-only: driven by relationships, brand safety, and categories where recommendations already matter.

If clicks defined the old internet, influence inside AI reasoning may define the next one.

Early days - but the direction is already clear.

The contrast between Walmart and Amazon reveals the central dilemma in the new AI economy. Walmart is actively partnering with OpenAI and Google, ensuring their product data is visible when AI agents like ChatGPT and Gemini are asked for shopping recommendations. Amazon, conversely, is reportedly blocking everything, treating other companies' AI agents as hostile crawlers (while expecting others to let their own AI agent in!).

One is leaning into AI as the "front door to commerce", the other is trying to protect their (retail media) moat.

The defensive approach is flawed. AI agents will compare your business regardless, using data from cooperative competitors and public review sites. Blocking only guarantees you're invisible in the critical, high-leverage answers. The brands that will control the next decade of distribution won't be the ones who blocked the most - they'll be the ones who shaped the narrative by participating early.

This isn't just about retail. It's about every brand, publisher, and platform.

The fundamental question is: Do you want to be cited, or do you want to be ignored?

Companies that figure out how to engage with the AI infrastructure - on their own terms, with clear economics - will own the next decade of distribution.

Playing defense means waking up to a world that moved without you.

OpenAI putting ads into ChatGPT feels like a real fork in the road moment.

On one hand, this is the first time in years anyone has a credible shot at putting a dent in Google’s search dominance - something Bing, even with all the AI firepower, never really managed to do.

ChatGPT already is search for a lot of people. On the other hand, ads change the contract with users. If AI search starts to feel like Google Search circa 2024, do people quietly drift to cleaner, ad-free alternatives like Perplexity or Claude?

The big question isn’t whether OpenAI can build an ad business - it’s whether they can do it without breaking the thing that made people switch in the first place.

Very interesting to look into the landscape of deals vs lawsuits in the publishers vs AI companies world.

To show where the gravity really is I created this visualization using the Tow Center’s AI Deals and Disputes Tracker (https://lnkd.in/e9pgs4_Y), to show the density of deals (green) vs lawsuits (red) vs grants (yellow) per AI company.

Over the past year, publishers have been forced into a new game: do you sue the AI companies using your work, sign a licensing deal, or hope a grant buys you time to experiment?

What stands out is the asymmetry. Relatively a small number of (large) publishers get deals and a few AI companies concentrate most of the activity - some with dense clusters of deals, others with a growing web of litigation, and a few sitting in both camps at once.

This isn’t just a legal story! It is a power map of who is getting paid, who is fighting in court, and who is still on the sidelines.

For publishers, the takeaway is that “wait and see” is not a strategy. Collective action (lawsuits, alliances, or licensing frameworks like RSL) and better data about AI traffic and usage will be critical to move from reactive deals or one‑off grants to a sustainable economic model.

Curious how others are reading this landscape - and whether you see more leverage in the courtroom, the negotiating room, or in new technical standards. You can play with this interactive mind map here: https://lnkd.in/e4aGBfHX

What if an AI Agent was a person - what would they be like?

Instead of asking how to use large language models (LLMs), ask: What if humans acquired LLM-like capabilities? What kind of persona would that create? What kind of content would resonate with them?

Call it the Human-LLM Native Persona: an audience that thinks in vectors, not headlines.

Here’s what I think defines this emerging cognitive profile:

How They Think:

* Pattern-oriented, ambiguity-tolerant, and non-linear.

* More interested in possibilities than fixed conclusions.

* They thrive on synthesis, analogies, and conceptual depth.

What They Seek:

* Thinking frameworks, not definitive answers.

* Insight that reorganizes what they already suspected or half-knew.

* Mental shortcuts that are cognitively elegant - not intellectually lazy.

What They Avoid:

* Simplistic tips and listicles.

* Oversold certainty and shallow persuasion.

* Content optimized for SEO rather than thoughtfulness.

What Keeps Them Engaged:

* Content that feels like a continuation of their own thinking.

* Writers who think out loud rather than deliver “truths.”

* Depth, coherence, and a clear intellectual identity.

Why They Return:

* Each interaction upgrades their mental model of the world.

* The content leaves a cognitive residue- they think differently because of it.

* They trust the source as a thinking partner, not a broadcaster.

Implications for Content Creators:

Avoid:

- Didactic teaching content

- Linear how-to guides

- Final, closed-off conclusions

Prioritize:

- Thinking artifacts (frameworks, tradeoffs, mental models)

- Content that can be explored from multiple perspectives

- Writing that expresses curiosity, ambiguity, and evolution

If LLM-level cognition becomes the norm, content will no longer compete for attention - it will compete for alignment. The creators and brands who thrive won’t simply provide information. They will help people think better.

They will elevate cognition.

Blocking AI might feel like protection - but it may actually accelerate traffic decline.

A new research paper (Zhao & Berman, Dec 2025, see below) tracked online news publishers’ traffic + behavior as LLMs became part of the discovery layer.

Here’s the surprising part:

Many major publishers started blocking AI crawlers via robots.txt (starting mid-2023). But for large publishers, blocking was followed by a ~23% drop in total traffic and a ~14% drop in human traffic compared to publishers who didn’t block.

That second number matters most.

Because it suggests blocking doesn’t just reduce bots — it reduces real discovery and demand.

In other words: if AI is becoming the front door to information, disappearing from it may be worse than being summarized by it.

This is the part I keep thinking about:

When you block AI systems, you may not just lose referral clicks…

you may lose brand presence - the subtle, repeated exposures that keep you in the consideration set.

And in a world where AI search answers are replacing browsing, fewer users click external links, and discovery is shifting away from “blue links”…

…being absent may compound into faster decline.

The best strategy probably isn’t “block vs. don’t bloc

Amazon’s ‘Buy for Me’ blowup last week is a reminder that in the age of AI agents, merchants (or any online business for that matter) can’t afford to be passive about how platforms represent and resell their business.

Every major platform will try some version of this - with different levels of disintermediation risk for the brands on the other side.

If you didn't hear about it: Over the last few weeks, Amazon has been trialing its new “Shop Direct” / “Buy for Me” AI shopping tools, which let customers browse third‑party sites and complete purchases inside Amazon – even for brands that never chose to sell on Amazon at all.

Dozens (and now hundreds) of small Shopify, Squarespace, WooCommerce and Wix merchants suddenly found their products scraped, re‑listed on Amazon, and auto‑ordered via a “buyforme amazon” flow, without any explicit opt‑in.

Merchants are angry for good reasons: this effectively turns them into involuntary drop‑shippers on a platform they may have deliberately avoided, with all the knock‑on effects to pricing power, brand positioning and direct‑to‑consumer relationships. At the same time, Amazon argues it is only using public data, helping brands “reach new customers,” and that anyone who doesn’t like it can email support and opt out.

Strip away the PR and you get the real question that will define agentic commerce: who actually has to consent when an AI agent acts on your behalf – the user, the platform, or the merchant?

All of the big platforms are going to run this play: use AI agents to expand their surface area, keep the user inside their own interface, and treat “public” merchant data as fair game until someone forces a new norm. The only variable is how aggressively they disintermediate you – from “helpful new channel” at one end, to “you discover the experiment only when bad orders start showing up in your warehouse” at the other.

If you are a merchant that cares about how AI agents “see” your website, your policies and your brand, you will need your own opinionated layer:

* How should agents interpret stock, price and promotions?

* What is not allowed (bundles, shipping promises, returns)?

* How do you signal consent (or non‑consent) in a machine‑readable way to different agents and platforms?

Merchants that want to actively design and control how AI agents experience their site, instead of waiting for the next surprise experiment, are welcome to reach out to us (@axioma). This is exactly the sort of agent‑aware content, policy and integration layer that we're building.

‍

For publishers and online businesses, “Zero‑click” AI summaries feel like an existential threat when all you measure is clicks and pageviews.

But if you zoom out, they are also proof that your content is valuable enough to be the substrate of an entirely new interface for how people get information.

The websites and publishers who survive will be the ones who do two things in parallel: protect their rights, and build proactive monetization for the moments when AI reads, cites, and depends on their work.

Talking with many publishers and online businesses these days feels like many are sitting on the beach watching the AI tsunami wave approach. Some are building higher sandcastles (more paywalls, more blocking), others are pretending the tide is not rising.

There is a third option: treat AI agents as a new distribution channel, and build or use products that let you see them, while controlling the response you give them.

Survival will come from treating AI as infrastructure you can design for - not a black box that just “happens” to your traffic.

Publishers should absolutely support RSL and similar standards for AI-era micropayments, but this is only the starting point for a real revenue and control strategy, not the end state.​

Arena Group, BuzzFeed, USA Today and Vox Media have joined the Really Simple Licensing (RSL) Collective, pushing toward a common standard for how AI systems can use and pay for content (Read on Digiday below).

Turning robots.txt into a machine-readable rights and royalty layer is a huge step toward making micro-payments for AI training and inference possible at scale.​

But here’s the tension: all of these frameworks still depend on voluntary compliance. They only work if AI companies choose to honor the rules, which means “standards” alone won’t guarantee publishers actually get paid.

That’s why publishers need to do two things at once: converge on shared protocols like RSL to send clear signals to the market, and, in parallel, actively reclaim control over their content, negotiate real deals, and design a long-term strategy for how AI visits translate into measurable value.​

"Hope is not a strategy". The publishers that win in this next phase won’t be the ones who simply flip the RSL switch and wait; they’ll be the ones who treat AI as a new distribution and monetization channel, build enforcement and leverage into their stack, and experiment aggressively with how licensed AI usage can drive sustainable revenue rather than uncompensated extraction.

The AI search results page has no page 2.

In most AI answer engines, if you are not in the top 10–20 sources, you effectively do not exist.

For publishers, the game is shifting from “rank as many URLs as possible” to “be reliably cited as a trusted source when an agent answers.” That means structuring content, signals, and brand authority so AI systems see you as a default citation, not just another blue link.

The winners will be the sites that deliberately optimize for AI visibility—tracking how often they’re cited, tuning content to be “answer‑ready,” and treating AEO/AIO as a core channel—while everyone else keeps chasing yesterday’s SEO playbook.

Many publishers are asking “when will AI search start to really matter for my traffic?” I think the better question is: “what happens when it does?”. Today, Google still sends hundreds of times more (human) traffic than AI systems, but AI agents already generate a growing share of the crawling, scraping and infrastructure cost for publishers. That gap is exactly where a new economic model will emerge: AI as the primary consumer of your content, and humans as the minority. Are you ready for that world?