Redirect URLs

[jkumz]

Hi,

Is there a way to prevent crawling domains if a URL re-directs to a different URL?

For me right now, if it hits a URL that redirects to a different domain it proceeds to crawl that domain as well, even using a strategy of enqueueing links.

For example, www.somelink.com/github redirects to their github profile, which then leads to crawling every URL on page, which leads to endless crawling of GitHub.

My code:
`async def main() -> None:
crawler = PlaywrightCrawler()

urls_found = []

# Define a request handler and attach it to the crawler using the decorator.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
    # See BeautifulSoup documentation for API docs.
    url = context.request.url
    context.log.info(f"On URL: {url}")
    urls_found.append(url)

    await context.enqueue_links(strategy=EnqueueStrategy.SAME_ORIGIN)

await crawler.run(["some url here"])

print(f"Found {len(urls_found)} URLs")`

Thanks in advance

[jkumz]

I made a workaround via putting enqueue links in a conditional block so that it will only start crawling if the domain is the same.
Still redirects to URLs outside of original domain, but doesn't start crawling any URLs under different domains which is sufficient.