A recent investigation has uncovered a problem for websites relying on JavaScript for structured data.
This data, often in JSON-LD format, is difficult for AI crawlers to access if not in the initial HTML response.
Crawlers like GPTBot (used by ChatGPT), ClaudeBot, and PerplexityBot can’t execute JavaScript and miss any structured data added later.
This creates challenges for websites using tools like Google Tag Manager (GTM) to insert JSON-LD on the client side, as many AI crawlers can’t read dynamically generated content.
Key Findings About JSON-LD & AI Crawlers
Elie Berreby, the founder of SEM King, examined what happens when JSON-LD is added using Google Tag Manager (GTM) without server-side rendering (SSR).
He found out why this type of structured data is often not seen by AI crawlers:
Initial HTML Load: When a crawler requests a webpage, the server returns the first HTML version. If structured data is added with JavaScript, it won’t be in this initial response.
Client-Side JavaScript Execution: JavaScript runs in the browser and changes the Document Object Model (DOM) for users. At this stage, GTM can add JSON-LD to the DOM.
Crawlers Without JavaScript Rendering: AI crawlers that can’t run JavaScript cannot see changes in the DOM. This means they miss any JSON-LD added after the page loads.
In summary, structured data added only through client-side JavaScript is invisible to most AI crawlers.
Why Traditional Search Engines Are Different
Traditional search crawlers like Googlebot can read JavaScript and process changes made to a webpage after it loads, including JSON-LD data injected by Google Tag Manager (GTM).
In contrast, many AI crawlers can’t read JavaScript and only see the raw HTML from the server. As a result, they miss dynamically added content, like JSON-LD.
Google’s Warning on Overusing JavaScript
This challenge ties into a broader warning from Google about the overuse of JavaScript.
In a recent podcast, Google’s Search Relations team discussed the growing reliance on JavaScript. While it enables dynamic features, it’s not always ideal for essential SEO elements like structured data.