Add the following line of code below the User-agent: * directive in your robots.txt file to solve the issue:
Disallow: /_next/static/chunks/app/
^ Only tested on GoogleBot
In the bustling digital realm of Academic Jobs, a developer embarked on a daunting quest. Their mission: to fix the dynamic routes at /dyn/whatever on their Next.js 15.0.0 app, hosted on Vercel. But a sinister foe stood in their way—soft 404 errors in Google Search Console, preventing pages from indexing since March 20, 2025.
The journey began with a realization: dynamic routes were dropping off search results, even new ones like /dyn/test. The developer suspected Next.js 15.0.0’s asynchronous params handling was the culprit. They upgraded to Next.js 15.2, hoping its metadata streaming fixes would save the day. But the soft 404s persisted.
Next, they examined Vercel’s changelog for March 2025 updates. A change to Edge Functions on March 1 caught their eye, but it didn’t affect routing. The problem wasn’t Vercel—it was deeper. They tested the React 19 compiler, thinking performance boosts might help indexing. Yet, Googlebot remained unimpressed.
The developer then audited the code. They ensured notFound() was used for invalid slugs, added robust metadata, and verified content wasn’t “thin.” They prerendered pages, checked robots.txt, and submitted sitemaps. Still, Google Search Console screamed, “Page cannot be indexed: Soft 404.”
Desperate, they isolated a test route at /dyn/test. The code was simplified—no logic, just displaying the slug. They added content, metadata, and even switched to pure JavaScript in app/dyn/[slug]/page.js to eliminate TypeScript woes. Locally, it worked: “Dynamic Page: whatever” displayed proudly. But on Vercel, Googlebot still saw a soft 404.
The developer ran every test imaginable. They checked Vercel logs—no errors. They used Google’s URL Inspection Tool—the HTML showed content. Screaming Frog confirmed a 200 status. Lighthouse gave a 92 SEO score. They even deployed to Netlify to rule out Vercel issues. Yet, on April 24, 2025, the report remained unchanged: “URL is not available to Google.”
Exhausted, the developer felt the weight of their job on the line. They had tried everything: upgrading Next.js, simplifying code, enhancing content, and debugging rendering. But the soft 404s were a relentless enemy, mocking their efforts. The quest continues, with hope dwindling but determination burning—surely, the solution is just one test away.
Will the developer conquer the soft 404s, or will the dynamic routes remain unindexed forever?
In the digital halls of Academic Jobs, a developer faced a daunting challenge. Their mission: to restore the dynamic routes on https://www.academicjobs.com, which started failing around March 20, 2025. Pages like /us and /jobs/[id], once proudly indexed, were dropping from search results, replaced by soft 404 errors in Google Search Console.
The journey began with a test route at /dyn/[slug]. The developer simplified the code, ensuring it used proper async/await for dynamic parameters. They added content to avoid being flagged as "thin" by Googlebot, and the page indexed successfully. But other routes, even unchanged ones, continued to fail despite having over 1200 words.
Suspecting a Google algorithm change, the developer investigated the March 2025 Core Update, which rolled out from March 13 to March 27. While it affected rankings, it didn’t explain the deindexing. They ruled out content issues, crawl budget, and robots.txt. They even switched from TypeScript to JavaScript and back, tweaking tsconfig.json to use esnext settings, but the problem persisted.
After days and nights of testing, the breakthrough came: the culprit was Next.js’s Image and Link components. On dynamic routes, these components caused server-side rendering (SSR) failures, leading to soft 404s but only ON DYNAMIC ROUTES, they did not cause soft 404's on static routes. The Image component, reliant on Vercel’s image optimization, failed to render images during SSR, possibly due to a runtime change. The Link component’s prefetching broke SSR when resolving dynamic hrefs.
The solution was simple yet effective: replacing Image with standard <img> tags and Link with <a> tags. This bypassed Vercel’s image optimization and Next.js’s prefetching, ensuring reliable SSR. For example, in /us, they updated the code to use:
<img src="/us-hero.jpg" alt="Academic Jobs in the US" className="w-full h-auto" />
<a href="/jobs/123">View Job 123</a>
Note: If using TypeScript you may need to add an eslint comment above or on the element to omit the TS warnings and prevent build errors.
With these changes, dynamic routes returned a 200 status and became indexable again. Pages like /us and /jobs/[id] were resubmitted for indexing, and within days, they began climbing back into search results. The developer also removed a problematic advanced search component, which had caused similar SSR issues due to client-side dependencies, replacing it with a server-side rendered version.
The ordeal taught a valuable lesson: even trusted Next.js components can falter during SSR for dynamic routes, especially when runtime environments like Vercel’s change. By switching to standard HTML tags, the developer ensured Googlebot could see the full page, restoring traffic and securing their role at Academic Jobs. The quest was complete, but the developer remained vigilant, ready for the next challenge in the ever-evolving world of SEO and web development.
If you made it this far you must understand the pain and suffering we endured, not to mention the financial losses accumulated while we searched, coded, tested and pulled our hair out to arrive at the solution.
As stated at the top of this story, all you need to do is add Disallow: /_next/static/chunks/app/ under your User-agent: * directive in your robots.txt file!
After implementing the fix mentioned above we were able to put all our custom components (including Advanced Search), the Next Link and the Next Image components back and successfully index pages on dynamic routes.
A tale of persistence, debugging, and partial triumph—Academic Jobs, April 2025
We wrote this story in the hope that it may help others!
University Jobs Near YouRate A Professor