[🐛 Bug]: Failed to initialize Firefox WebDriver: Message: Not Supported

[Milosevic-Uros]

What happened?

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36

Steps to reproduce:

Created a python project for scraping content from various websites. The project uses Firefox version 133.0, gecko driver 0.35.0, selenium 4.27.1, scrapy-selenium 0.0.7, selenium-wire 5.1.0 and webdriver-manager 4.0.2. The project runs fine locally however when deploying to azure I get the error Failed to initialize Firefox WebDriver: Message: Not Supported when calling the API for scraping.

Actual results:
Deployment goes through successfuly, however, when calling API request to scrape I get this
Failed to initialize Firefox WebDriver: Message: Not supoorted

Code for creating firefox drivers
class FirefoxWebDriverPool:
proxy_url = ""

def __init__(self, size=CONCURRENT_SELENIUM_DRIVERS):

    self.pool = Queue(maxsize=size)
    self.proxy_url = PROXY_URL
    self.ua = UserAgent()

    for _ in range(size):
        try:
            driver = self.create_new_driver()
            self.pool.put(driver)
        except Exception as e:
            logging.info(f"Failed to initialize Firefox WebDriver: {e}")

def create_new_driver(self):
    options = webdriver.FirefoxOptions()

    options.add_argument("--headless")
    #options.add_argument("--width=1600")
    #options.add_argument("--height=900")

    firefox_desktop_ua = self.get_firefox_desktop_user_agent()
    options.set_preference("general.useragent.override", firefox_desktop_ua)
    options.set_preference("log.level", "trace")
    options.set_preference("remote.log.level", "trace") 

    try:
        driver = webdriver.Firefox(service=FirefoxService(GeckoDriverManager().install(),
                log_output="./geckodriver.log",
                service_args=['--log', 'trace']),
            seleniumwire_options=(
                {
                    'connection_timeout': 180,
                    'request_timeout': 180,
                    'response_timeout': 180,
                    "proxy": {
                        "http": self.proxy_url,
                        "no_proxy": "localhost,127.0.0.1,dev_server:8080",
                    }
                }
                if self.proxy_url
                else {}
            ),
            options=options,
        )

        # Intercept requests to modify headers
        driver.request_interceptor = self.interceptor

        return driver
    except Exception as e:
        logging.info(f"Failed to create new Firefox WebDriver: {e}")
        raise           
    

def get_firefox_desktop_user_agent(self):
    while True:
        user_agent = self.ua.firefox
        parsed_ua = parse(user_agent)
        if not parsed_ua.is_mobile and not parsed_ua.is_tablet:
            return user_agent

# noinspection PyMethodMayBeStatic
def interceptor(self, request):
    request.headers['sec-fetch-site'] = 'same-origin'
    request.headers['sec-fetch-mode'] = 'navigate'
    request.headers['sec-fetch-user'] = '?1'
    request.headers['sec-fetch-dest'] = 'document'

def get_driver(self):
    return self.pool.get()

def return_driver(self, driver):
    try:
        logging.info(driver.title)  # Simple operation to check driver health
        driver.delete_all_cookies()
        driver.get("about:blank")
        self.pool.put(driver)
    except Exception:
        try:
            driver.quit()
        except Exception as e:
            logging.info(f"Error quitting driver: {e}")

        try:
            new_driver = self.create_new_driver()
            self.pool.put(new_driver)
        except Exception as e:
            logging.info(f"Failed to create new Firefox WebDriver: {e}")

def close_all_drivers(self):
    while not self.pool.empty():
        driver = self.pool.get()
        driver.quit()

As for the output of geckodriver.log only thing I get is this
1733390107346 geckodriver INFO Listening on 127.0.0.1:49355

Expected results:

Webdriver should initialize correctly like it works when running locally

How can we reproduce the issue?

Use following driver setup, deploy to azure and start the app

Relevant log output

INFO:WDM:====== WebDriver manager ======
2024-12-04T16:25:10.736346779Z INFO:WDM:Get LATEST geckodriver version for 133.0 firefox
2024-12-04T16:25:10.760316492Z INFO:WDM:Get LATEST geckodriver version for 133.0 firefox
2024-12-04T16:25:10.777195812Z INFO:WDM:Driver [/root/.wdm/drivers/geckodriver/linux64/v0.35.0/geckodriver] found in cache
2024-12-04T16:25:10.806607596Z DEBUG:selenium.webdriver.remote.remote_connection:POST http://localhost:34195/session {'capabilities': {'firstMatch': [{}], 'alwaysMatch': {'browserName': 'firefox', 'acceptInsecureCerts': True, 'moz:debuggerAddress': True, 'pageLoadStrategy': <PageLoadStrategy.normal: 'normal'>, 'proxy': {'proxyType': 'manual', 'httpProxy': '127.0.0.1:41237', 'sslProxy': '127.0.0.1:41237'}, 'moz:firefoxOptions': {'prefs': {'remote.active-protocols': 3, 'general.useragent.override': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0', 'log.level': 'trace', 'network.proxy.allow_hijacking_localhost': True}, 'args': ['--headless']}}}}
2024-12-04T16:25:10.823550017Z DEBUG:selenium.webdriver.remote.remote_connection:Remote response: status=501 | data=Not Supported
2024-12-04T16:25:10.823613118Z | headers=HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'Date': 'Wed, 04 Dec 2024 16:25:10 GMT', 'Content-Length': '14'})
2024-12-04T16:25:10.823625018Z DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
2024-12-04T16:25:10.849677858Z INFO:root:Failed to create new Firefox WebDriver: Message: Not Supported
2024-12-04T16:25:10.849726158Z
2024-12-04T16:25:10.849732658Z INFO:root:Failed to initialize Firefox WebDriver: Message: Not Supported

Operating System

Windows 11 locally, ubuntu-22.04 on azure

Selenium version

Python 4.27.1

What are the browser(s) and version(s) where you see this issue?

Firefox 133.0

What are the browser driver(s) and version(s) where you see this issue?

GeckoDriver 0.35.0

Are you using Selenium Grid?

Not using

[github-actions]

@Milosevic-Uros, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!