Making multi-market sitemaps reliable from catalogue to CDN
An 8,925-product crawl took 486 seconds, while CDN cache identity collapsed every shard. I bounded generation and made each regional shard independently cacheable.
Disclosure: Client and internal project details have been anonymized.
Overview
I inherited sitemap routes that had mostly shipped but had not yet been proven against the full catalogue and deployed cache behavior. Dynamic Next.js handlers used a 24-hour cache at the content delivery network (CDN) to regenerate the sitemaps daily without a separate scheduled job.
Once the routes reached staging, deeper failures appeared.
The sitemap could return valid XML with no products, time out while reading the catalogue, omit an entire market because of a language mismatch, or serve the same cached products for every shard. For search engines, each failure reduced or corrupted the set of product pages the storefront could advertise. Resolving them meant tracing the route handlers, backend layer, commerce API, content system, locale configuration, and CDN.
Generation and delivery boundaries
Valid XML, missing content
Reducing the page size to 100 preserved cursor-driven traversal of the complete catalogue. The product sitemap changed from zero URLs to 8,925.
The static sitemap had a different problem: its query selected only the homepage instead of the full page tree. Correcting the root and filtering private and authentication pages—without changing site-wide access rules—took the sitemap from one public URL to nine.
Replacing a full-catalogue request with bounded shards
With the full catalogue, the product sitemap returned a 502. Generating it required about 90 sequential API calls for roughly 8,925 products and took around 486 seconds.
The gateway ended the request before it finished, so nothing reached the CDN cache. Every new request repeated the full crawl and timed out again.
Sharding the catalogue into groups of 2,500 URLs capped generation at 25 commerce API pages per shard. A small count query tells the sitemap index how many shards to list for each locale. If that query fails or omits its total, the index advertises the maximum supported shard count instead of silently listing none.
Pagination needed special care. Regional and product-lifecycle filters run after the commerce API returns a page, so the sitemap may write fewer products than the cursor consumed. Advancing by the filtered count would put the local offset out of step with the commerce cursor, causing gaps, overlaps, or an early stop.
Each shard advances by the API page's unfiltered count, while writing only the filtered products to XML.
A mid-shard API error now returns an uncacheable 503 instead of a 200 with a partial sitemap. A temporary dependency failure can no longer cache incomplete results for 24 hours.
Discovering a locale that contained no localized content
The New Zealand product sitemap remained empty even though products existed for that market.
The first hypothesis was country-specific pricing. I tested language and price country independently. Removing the price country made no difference; changing the search language did.
The commerce project declared both en-AU and en-NZ, so using en-NZ for New Zealand requests looked correct. But product names, slugs, descriptions, and search keywords existed only in en-AU. Empty product searches used a wildcard in the requested language, so the New Zealand query matched nothing.
Localized matching now searches across all languages declared by the project. In the local environment, the New Zealand sitemap went from zero URLs to 6,817.
The same locale mismatch also affected live product-name searches: New Zealand users could find exact SKUs, but name searches used a language with no content.
Localized sorting and facets remained unchanged in this change because those API fields accept only one language. Choosing one safely required confirming that no custom attributes held New Zealand-specific values; expanding the fix without that evidence would have introduced a new assumption. A separate data review provided that evidence before the single-language paths changed.
Finding the final failure outside the application
After the localization change reached the deployed site, all shards loaded—but every shard contained the same products.
Local verification against the deployed commerce project produced distinct shards with no overlapping URLs. Because only the deployed version failed, I moved the investigation to the platform layer.
Two deployed shard URLs returned identical bodies and CDN cache-hit headers. The CDN cache key ignored the custom shard query parameter.
Whichever shard reached the origin first therefore filled one cache entry that every shard URL reused for 24 hours.
The cache key now includes the shard identifier for successful responses, while temporary failures remain uncacheable.
Verification
Regression coverage exercised the API page-size cap, static page traversal, filtered-window pagination, empty-window continuation, mid-shard failures, missing catalogue totals, and multi-language query construction.
The first deployed check confirmed that the product route no longer returned 502 and that the sitemap index and robots.txt remained available. Post-deployment checks then confirmed distinct Australian and New Zealand shard content across repeated opens and hard refreshes—the failure had not returned from the 24-hour CDN cache.
The evidence establishes URL coverage and deployed cache behavior, not SEO uplift. No final per-shard generation time, crawl-coverage change, indexing result, or organic-visibility metric was recorded.
Operating envelope
The offset-based design supports the commerce API's 10,000-product random-access envelope. At roughly 8,925 products, it covered the measured catalogue without truncation. A larger catalogue would require a separate cursor-token pagination design rather than silently exceeding that contract.
Searching all declared languages is correct for the current content model, where localized product text exists in one language. Introducing New Zealand-specific product copy would change that data contract and require a different localization strategy.
Temporary upstream errors deliberately return retryable 503 responses. That availability policy protects sitemap and cache integrity instead of presenting partial data as success.
Outcome
The product sitemap went from zero URLs to 8,925, making the full measured Australian catalogue available through sitemap documents. The static sitemap went from one public URL to nine. The New Zealand product sitemap went from zero URLs to 6,817 in local verification, and the deployed regional sites ultimately served distinct shard content.
Operationally, sharding replaced a repeated 502 loop with bounded requests that could reach the CDN, while uncacheable 503 responses prevented a transient commerce failure from persisting as a partial sitemap for 24 hours.