robots.txt - which generated directories should have a Disallow rule?

[mbifulco]

Hi there,

I've been doing some inspection of my site (mikebifulco.com's SEO using the ahrefs webmaster tools to occasionally run audits on my site. In doing so, I noticed that their crawler is checking the _next directory on my site:

My understanding is that I probably don't want /_next and /api to be indexed for search, but I haven't found any real guidelines in the docs for next on this. (this is all I found: static file serving and "what is a robots.txt file?"

Also, I checked robots.txt for some sites that I thought might show recommended practices... and was a bit surprised.

vercel.com

link

User-Agent: *
Disallow:
Disallow: /api
Disallow: /oauth
Disallow: /confirm
Disallow: /notifications
Disallow: /old-browser.html

nextjs.org

link

(404 page)

mikebifulco.com (for comparison)

link

# *
User-agent: *
Allow: /

# Host
Host: https://mikebifulco.com

# Sitemaps
Sitemap: https://mikebifulco.com/sitemap.xml

should I add a disallow rule for crawlers on _next? What about /api, presuming I don't want any of my APIs to show up in search results? Nothing special there, but there's no reason anyone would ever want to browse directly to one of those routes.

[vpaflah]

Did you find anything useful?

[HamedMP]

Same question here, Claude 3.5 recommends having _vercel but not sure if that will impact showing the images on the search results