Link Unfurling Explained: The Crawler-to-Card Pipeline

Link unfurling looks simple from the outside — paste a URL, get a card — but the pipeline behind it involves a specific sequence of decisions, each of which can fail independently and each of which explains a different category of broken preview. This is a mechanism-level walkthrough of that pipeline, useful if you're building something that generates metadata programmatically, debugging a preview at the request level, or just want to understand precisely what's happening rather than treating it as a black box.

Step one: URL detection and normalization

Before any network request happens, the platform has to detect that a string in the content being posted is a URL, then normalize it — resolving relative fragments if any exist, stripping or preserving certain query parameters depending on the platform's own rules, and in many cases checking it against an internal cache keyed on the normalized form. This normalization step is why URLs that look identical to a human can sometimes hit different cache entries: a trailing slash, a capitalization difference in the domain, or a tracking parameter can all be treated as a distinct URL by a platform's cache key even when they resolve to the same content.

Step two: the crawler request

Assuming no cached entry exists (or a refresh was explicitly triggered), the platform issues an HTTP request to the URL using a dedicated crawler user agent — distinct from a regular browser's user agent string, which matters because servers sometimes serve different content based on user agent, intentionally or not. This request is typically a simple GET with a limited timeout, generally in the range of a few seconds, and critically, it does not execute JavaScript. This single fact is the root cause of the most common unfurling failure: any metadata added to the page by client-side code after the initial HTML loads is invisible to this request, even though a browser rendering the same page would show it correctly.

Step three: parsing the response for metadata

Once the HTML response comes back, the crawler parses the head section for a known set of meta tags — primarily Open Graph properties (og:title, og:description, og:image, og:url, og:type) and, on platforms that support it, Twitter Card properties (twitter:card, twitter:title, twitter:image). This parsing is generally forgiving of minor markup issues but strict about a few specific things: the property attribute (not name, which is a different HTML attribute conventionally used for other kinds of meta tags) is what Open Graph parsing looks for, and a tag placed outside the head element, or duplicated with conflicting values, produces inconsistent results across different parsers rather than a clean, predictable failure.

Step four: resolving the image

If an og:image or twitter:image value was found, the crawler generally makes a second request specifically to fetch that image, separate from the initial page fetch. This is its own point of failure, independent of whether the tag itself was well-formed: the image URL has to be absolute, publicly reachable without authentication, not blocked by robots.txt, and within whatever size and format constraints the platform enforces. Some platforms fetch and validate the image synchronously as part of building the card; others may defer this and cache a placeholder if the image fetch is slow, which is part of why intermittent, load-dependent image failures are harder to reproduce reliably than a straightforwardly broken tag.

Step five: assembling and caching the card

With title, description, and image resolved (or explicitly absent, for any that failed or weren't provided), the platform assembles a card according to its own layout rules — this is where platform-specific differences in appearance come from, since the underlying data can be identical while the rendered card looks different across platforms. The assembled result is then stored, keyed on the normalized URL from step one, so that future references to the same URL skip the entire pipeline and reuse this stored card directly. This caching step is what creates every stale-preview problem: nothing in the pipeline re-runs automatically when the source page changes, because from the platform's perspective, the URL hasn't changed, and the URL is the only thing it's keying on.

Where each documented failure mode maps to a pipeline step

• No preview at all, on a page with visibly correct tags: usually step three — tags added client-side, invisible to a crawler that doesn't run JavaScript
• Text renders, image doesn't: usually step four — relative URL, auth-gated image, or a size/format rejection
• Preview shows old content after a real change: step five — the cache never invalidated because nothing told it the source changed
• Wrong content entirely, unrelated to the shared page: often step one — a redirect chain resolving to an unexpected destination before the crawler ever reaches your intended page

A note on crawler identification and why blocking it accidentally happens

Because the crawler request in step two arrives with a distinct user agent and none of the cookies, headers, or session state a real visitor's browser would carry, it's easy for infrastructure designed to block bots — rate limiters, bot-detection middleware, aggressive WAF rules — to catch link-preview crawlers as an unintended side effect. This is a genuinely common cause of a preview that fails intermittently or fails only in production while working fine in a staging environment with looser security rules. If a page unfurls correctly when you fetch it manually but fails when an actual platform tries to unfurl it, checking your server or CDN's access logs for the specific crawler user agent (Facebook's, for instance, identifies itself explicitly) is one of the more reliable ways to catch this, since the failure won't show up in any tag-level validation — the request never got far enough to reach step three at all.

Why this matters for anything generating metadata programmatically

If you're building a system that generates pages, images, or metadata dynamically — a marketplace with thousands of listing pages, a content platform with user-generated posts, an API that renders images on request — understanding this pipeline changes how you design URLs and caching, not just how you debug a single broken preview after the fact. Specifically, it argues strongly for giving distinct content distinct URLs rather than serving different content from a fixed URL over time: since the pipeline's entire caching model keys on the URL string, content that changes without a URL change is invisible to any cache invalidation short of a manual forced re-scrape, on every platform, every time.

How useopengraph's rendering pipeline is built around this

useopengraph's REST API and MCP server expose OG image rendering with exactly this mechanism in mind: publishing a template produces a new, immutable version at its own URL (/api/render/{templateId}/v{version}/...), so every distinct piece of rendered content genuinely has a distinct URL for the unfurl pipeline to cache against — there's no scenario where a platform's step-five cache goes stale relative to your content, because the URL itself changes whenever the content does. For teams building programmatic generation — marketplaces, listing pages, per-user content — this means the caching layer at every platform works with your system instead of fighting it, and the API and MCP server let that rendering be triggered directly from a build pipeline or an AI agent rather than a manual template edit each time.