Skip to content

[py] speed up Service launch (sleep depending on the number of attempts to check if the service has started) - #13321

Merged
diemol merged 2 commits into
SeleniumHQ:trunkfrom
deedy5:dev
Dec 18, 2023
Merged

[py] speed up Service launch (sleep depending on the number of attempts to check if the service has started)#13321
diemol merged 2 commits into
SeleniumHQ:trunkfrom
deedy5:dev

Conversation

@deedy5

@deedy5 deedy5 commented Dec 17, 2023

Copy link
Copy Markdown
Contributor

Description

Python client. Problem file: py/selenium/webdriver/common/service.py
When the driver is initialized, the service starts, and then a loop checks to see if it has started.
But there is a constant delay of 0.5 seconds in the loop. Here the delay logic changes - instead of the constant 0.5 seconds, it increases in increments of 0.05, starting at 0.01 and going up to 0.5.

Motivation and Context

When a driver with many options is started, there is a 0.5 second delay. For one request it is not noticeable, but for many requests, when it is necessary to initialize the driver each time, it results in significant time loss.

Line_profiler

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    90                                               @profile
    91                                               def start(self) -> None:
    92                                                   """Starts the Service.
    93                                           
    94                                                   :Exceptions:
    95                                                    - WebDriverException : Raised either when it can't start the service
    96                                                      or when it can't connect to the service
    97                                                   """
    98         1        911.6    911.6      0.2          self._start_process(self._path)
    99         1          4.3      4.3      0.0          t0 = time()
   100         1          1.4      1.4      0.0          count = 0
   101         2          4.7      2.4      0.0          while True:
   102         2        460.8    230.4      0.1              logger.warning(f"start() {count=} {time() - t0=}")
   103         2         44.1     22.0      0.0              self.assert_process_still_running()
   104         2       5225.4   2612.7      1.0              if self.is_connectable():
   105         1          1.8      1.8      0.0                  break
   106                                                       
   107         1          1.9      1.9      0.0              count += 1
   108         1     500111.7 500111.7     98.7              sleep(0.5)
   109         1          5.2      5.2      0.0              if count == 60:
   110                                                           raise WebDriverException(f"Can not connect to the Service {self._path}")

Reproduce the problem

test.py

from itertools import cycle
from random import choice, randint
from time import time

from selenium import webdriver

DEBUG_PORT = (x for x in cycle(range(9000, 9999)))


class Driver:
    def __init__(self, headless=False):
        self.options = webdriver.ChromeOptions()
        if headless:
            self.options.add_argument("--headless=new")
        self.options.add_experimental_option("excludeSwitches", ["enable-automation"])
        self.options.add_experimental_option("useAutomationExtension", False)
        self.options.add_argument("--disable-blink-features=AutomationControlled")
        self.options.add_argument("--no-sandbox")
        self.options.add_argument("--disable-gpu")
        self.options.add_argument(f"--window-size={randint(1024, 1920)},{randint(768, 1080)}")
        self.options.add_argument(f"--remote-debugging-port={next(DEBUG_PORT)}")
        self.options.add_argument("--disable-dev-shm-usage")
        self.options.add_argument("--lang=en")
        self.options.add_argument("--incognito")
        self.options.add_argument("--log-level=3")
        self.options.add_argument("--no-default-browser-check")
        self.options.add_argument("--no-first-run")
        self.options.add_argument("--ignore-certificate-errors")
        self.options.page_load_strategy = "eager"
        prefs = {"profile.managed_default_content_settings.images": 2}  # disable images
        self.options.add_experimental_option("prefs", prefs)
        self.driver = webdriver.Chrome(options=self.options)
        
    def quit(self):
        self.driver.quit()

    def __enter__(self):
        return self.driver

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.driver.quit()


def get_url_selenium(url):
    with Driver(headless=False) as driver:
        driver.set_page_load_timeout(100)
        driver.get(url)
        return driver.page_source


if __name__ == "__main__":
    url = "https://bot.sannysoft.com/"
    for _ in range(10):
        r = get_url_selenium(url)

Edit selenium/webdriver/common/service.py - remove sleep and logging where service will be started:

def start(self) -> None:
        """Starts the Service.

        :Exceptions:
         - WebDriverException : Raised either when it can't start the service
           or when it can't connect to the service
        """
        self._start_process(self._path)

        count = 0
        t0 = time()  # Add start time
        while True:
            self.assert_process_still_running()
            if self.is_connectable():
                logger.warning(f"{count=} elapsed={time() - t0}")  # Add logging message
                break

            count += 1
            #sleep(0.5)  # Remove sleep
            if count == 60000:  # Increase count
                raise WebDriverException(f"Can not connect to the Service {self._path}")

run test.py

count=68 elapsed=0.010582923889160156
count=84 elapsed=0.008266687393188477
count=55 elapsed=0.005543947219848633
count=61 elapsed=0.006617546081542969
count=36 elapsed=0.0072443485260009766
count=114 elapsed=0.010538339614868164
count=82 elapsed=0.008222818374633789
count=43 elapsed=0.008531808853149414
count=113 elapsed=0.010515928268432617
count=49 elapsed=0.005129098892211914

Here we see that the startup time is <= 0.01.
So there is no point in waiting for 0.5 seconds, it is just a waste of time.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • I have read the contributing document.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@CLAassistant

CLAassistant commented Dec 17, 2023

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@diemol diemol left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good improvement, thank you, @deedy5!

@diemol
diemol merged commit 98ea560 into SeleniumHQ:trunk Dec 18, 2023
@deedy5 deedy5 changed the title Python client: sleep depending on the number of attempts to check if the service has started [py] speed up Service launch (sleep depending on the number of attempts to check if the service has started) Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants