Server Islands get indexed by crawlers

[louisescher]

Astro Info

Astro                    v5.1.1
Node                     v22.5.1
System                   Linux (x64)
Package Manager          pnpm
Output                   static
Adapter                  @astrojs/node
Integrations             none

If this issue only occurs in one browser, which browser is a problem?

Describe the Bug

Since v5.0.0, Server Islands are retrieved via GET request. This introduces an issue where crawlers (Googlebot and others) think that the server islands are pages, and try to index that page, which they fail to do. See this support thread on Discord for more information.

What's the expected result?

Server Islands should set the noindex HTTP header to indicate to crawlers that they shouldn't be indexed by default.

Link to Minimal Reproducible Example

https://stackblitz.com/edit/github-fmzsdm7p?file=src%2Fpages%2Findex.astro

Participation

• I am willing to submit a pull request for this issue.

[JLarky]

I was able to set noindex header from middleware, another option would be to be able to force POST method for server islands, maybe server:defer="POST"?

[louisescher]

@JLarky That was the default behavior in Astro v4, but it introduces issues when trying to cache server islands, as the cache headers are ignored by all major browsers on POST requests. It would be easier to set the header anyways imo

[Fakerko]

I have the same problem, Google indexed my server islands as pages.

@louisescher how did you fixed it?

[louisescher]

I have the same problem, Google indexed my server islands as pages.

@louisescher how did you fixed it?

Hi there! You can either:

1. Add the server islands to your robots.txt file or
2. Add the X-Robots-Tag header using Astro.response.headers.set in the frontmatter of the island and set it to noindex