Move to w3c WebDriver spec

We've been using a `capabilities` syntax for create session that has long been
deprecated. Firefox finally forced our hand with geckodriver 0.35.0,
which has removed support for the obsolete syntax.

All supported WebDrivers understand the newer syntax.

We used to pass capabilities in as `desiredCapabilities`,
we now pass them in as a single item in the `firstMatch` vector.
This should match existing behaviour.

Moving to the new capabilities syntax tells `chromedriver`
(and `msedgedriver`) that we are running in "w3c mode". This has benefits on
Etaoin code. Our many customizations for these WebDrivers are no longer
necessary, they now support the W3C WebDriver endpoint spec.

In "w3c mode", chromedriver disallows any endpoints that could be served
by W3C WebDriver endpoints.

Notable examples are mouse and touch operations. These are now
handled by the W3C WebDriver action endpoints. WebDriver actions are not
stand-alone, they are submitted as transaction of steps. As such the
following Chrome-specific fns are no longer needed and would be confusing
if not deleted:

- `mouse-btn-down`
- `mouse-btn-up`
- `with-mouse-btn`
- `mouse-move-to` (was also available in Firefox)
- `mouse-click`
- `right-click`
- `left-click`
- `touch-down`
- `touch-move`
- `touch-up`

Users will instead express these manipulations via the existing
`perform-actions`.

Fns that represent a transaction of actions in themselves remain:
- `drag-and-drop` (from element to element)
- `double-click*` (on an element)
- `*-click-on` (on an element)
- `touch-tap` (on an element)

Verifying that everything worked required a full review of the API.

Some sweeping changes:
- Our recent deletion of PhantomJS support allowed me to streamline many
`defmulti`s into `defn`s.
- Added links to W3C WebDriver Spec endpoints to docstrings

Added some fns to expose at the W3C WebDriver Spec granularity:
- `get-timeouts`
- `set-timeouts`
- `get-element-rect`
- `get-element-rect-el`
- `get-element-rect`
- `set-window-rect`
- `get-window-rect`

We have finer grained versions of the above (ex. `get-element-position`,
`get-element-size`, etc). We've kept these for backwards compatibility,
but if we were starting today, we would have just matched today's W3C
WebDriver spec.

Minor fixes/changes:
- Uncomment and fix the maximize test
- Screenshots on elements work on safari, enable them
- Added notes on displayedness to docstrings
- Remove internal support for undocumented `:desired-capabilities`, the implementation
was either ultra legacy or misunderstood legacy APIs.

Add `bb test-coverage` task to check what we are not covering with tests

New tests include coverage for:
- `back`
- `forward`
- `set-cookie`
- `release-actions`
- `*-timeout*`
- `double-click`

Closes #467

Incidentally closes #522 via better docstrings on getting properties.

Closes #484 with W3C WebDriver spec endpoint links.

CHANGELOG.adoc

@@ -22,8 +22,38 @@ A release with an intentional breaking changes is marked with:
 * Technically breaking
 ** {issue}613[#612]: Remove all support for long obsolete and long untested PhantomJS
 ({lread})
+** {issue}467[#467]: Move to W3C WebDriver spec.
+({lread})
+*** The impetus was Firefox no longer supporting legacy `:capabilities` syntax.
+The breaking impact was on Chrome/Edge, when in "w3c mode" it will fail on WebDriver endpoints where there is a viable w3c alternative.
+This means some custom Chrome specific fns should now be expressed as w3c WebDriver actions.
+The following Chrome-specific fns have been deleted:
+**** `mouse-btn-down`
+**** `mouse-btn-up`
+**** `with-mouse-btn`
+**** `mouse-move-to` (was also available in Firefox)
+**** `mouse-click`
+**** `right-click`
+**** `left-click`
+**** `touch-down`
+**** `touch-move`
+**** `touch-up`
+*** Remove internal support for undocumented `:desired-capabilities`.
+The implementation was either ultra legacy or misunderstood legacy APIs.
 
 * Other changes
+** Add new fns that more lightly abstract W3C WebDriver Spec (as part of {issue}467[#467] API review sweep)
+({lread})
+*** `get-timeouts` - as alternative to `get-*-timeout`
+*** `set-timeouts` - as alternative to `set-*-timeout`
+*** `get-element-rect` - as alternative to `get-element-size`, `get-element-location`
+*** `get-element-rect-el` - as alternative to `get-element-size-el`, `get-element-location-el`
+*** `get-window-rect` - as alternative to `get-window-size`, `get-window-position`
+*** `set-window-rect` - as alternative to `set-window-size`, `set-window-position`
+** Review tests and add some missing coverage (as part of {issue}467[#467] API review sweep)
+({lread})
+** {issue}467[#467]: Required a full sweep of the API, so also includes:
+({lread})
 ** {pr}552[#552]: Add support for wide characters to input fill functions
 ({person}tupini07[@tupini07])
 ** {issue}566[#566]: Recognize `:driver-log-level` for Edge
@@ -48,6 +78,8 @@ A release with an intentional breaking changes is marked with:
 ({lread})
 *** {issue}602[#602]: Document all `:fn/*` query pseudo-functions in a definitive list
 ({person}dgr[@dgr])
+*** {issue}484[#484]: Add W3C WebDriver Spec links to docstrings
+*** {issue}522[#522]: Via better docstrings on getting properties
 
 == v1.0.40 - 2023-03-08 [[v1.0.40]]
 

bb.edn

@@ -66,8 +66,10 @@
                   :task test-server/-main}
   test:bb        {:doc "Runs tests under Babashka [--help]"
                   :task test/test-bb}
-  test-doc       {:doc "test code blocks in user guide"
+  test-doc       {:doc "Test code blocks in user guide"
                   :task test-doc/test-doc}
+  test-coverage  {:doc "Run doc and unit tests on JVM and generate ./target/clofidence code coverage report"
+                  :task test-coverage/-main}
   test-matrix    {:doc "Returns a test matrix for CI [--help]"
                   :task test-matrix/-main}
   drivers        {:doc "[list|kill] any running WebDrivers"

deps.edn

@@ -12,7 +12,7 @@
  :aliases
  {;; we use babashka/neil for project attributes
   ;; publish workflow references these values (and automatically bumps patch component of version)
-  :neil {:project {:version "1.0.40" ;; describes last release and is template for next release
+  :neil {:project {:version "1.1.40" ;; describes last release and is template for next release
                    :name  etaoin/etaoin
                    ;; not neilisms - could potentially conflict with new neilisms
                    :github-coords clj-commons/etaoin}}
@@ -51,6 +51,16 @@
                                           :vars [:symbol]}}
               :main-opts ["-m" "babashka.cli.exec"]}
 
+  :clofidence {:classpath-overrides {org.clojure/clojure nil}
+               :extra-deps {com.github.flow-storm/clojure {:mvn/version "LATEST"} ; >= 1.11.1-15
+                            com.github.flow-storm/clofidence {:mvn/version "LATEST"}}
+               :exec-fn clofidence.main/run
+               :exec-args {:report-name "Etaoin Test Coverage"
+                           :output-folder "target/clofidence"
+                           :test-fn cognitect.test-runner.api/test
+                           :test-fn-args [{:dirs ["test" "target/test-doc-blocks/test"]}]}
+               :jvm-opts ["-Dclojure.storm.instrumentOnlyPrefixes=etaoin"]}
+
   ;; for consistent linting we use a specific version of clj-kondo through the jvm
   :clj-kondo {:extra-deps {clj-kondo/clj-kondo {:mvn/version "2024.05.24"}}
               :main-opts ["-m" "clj-kondo.main"]}

doc/01-user-guide.adoc

@@ -173,7 +173,7 @@ Edge and `msedgedriver` must match so you might need to specify the version:
 `scoop install edgedriver@101.0.1210.0`
 ** Download: link:{url-edge-dl}[Official Microsoft download site]
 
-Check your WebDriver installations launching by launching these commands.
+Check your WebDriver installations by launching these commands.
 Each should start a process that includes its own local HTTP server.
 Use Ctrl-C to terminate.
 
@@ -190,7 +190,7 @@ You can optionally run the Etaoin test suite to verify your installation.
 TIP: Some Etaoin API tests rely on ImageMagick.
 Install it prior to running test.
 
-From a clone of the https://github.com/clj-commons/etaoin[Etaoin GitHub repo]
+From a clone of the https://github.com/clj-commons/etaoin[Etaoin GitHub repo]:
 
 * To check tools of interest to Etaoin:
 +
@@ -333,7 +333,7 @@ A portion of the above rewritten with `doto`:
 ----
 
 == Playing Along in your REPL
-We encourage you to try the examples in from this user guide in your REPL.
+We encourage you to try the examples from this user guide in your REPL.
 
 The Interwebs is constantly changing.
 This makes testing against live sites impractical.
@@ -385,7 +385,7 @@ into:
 ;; => ["username2" "pass2" "some text2"]
 ----
 
-If any exception occurs during a browser session, the WebDriver process might hang until you kill it manually.
+If any exception occurs during a browser session, the WebDriver process might hang around until you kill it manually.
 To prevent that, we recommend the `with-<browser>` macros:
 
 [source,clojure]
@@ -939,7 +939,7 @@ The `query-shadow-root-el` and `query-all-shadow-root-el` allow you to specify t
 (def shadow-root (e/get-element-shadow-root-el driver host))
 (e/get-element-text-el driver (e/query-shadow-root-el driver shadow-root {:css "#in-shadow"}))
 ;; => "I'm in the shadow DOM"
- 
+
 (->> (e/query-all-shadow-root-el driver shadow-root {:css "span"})
      (map #(e/get-element-text-el driver %)))
 ;; > ("I'm in the shadow DOM" "I'm also in the shadow DOM")
@@ -1069,17 +1069,7 @@ It can be used, for example, to check your handling of disallowing multiple form
 (e/double-click driver {:tag :button :name "submit"})
 ----
 
-There are also "blind" clicking functions.
-They trigger mouse clicks on the current mouse position:
-
-[source,clojure]
-----
-(e/left-click driver)
-(e/middle-click driver)
-(e/right-click driver)
-----
-
-Another set of functions do the same but move the mouse pointer to a specified element before clicking on them:
+Functions that move the mouse pointer to a specified element before clicking on it:
 
 [source,clojure]
 ----
@@ -1575,7 +1565,7 @@ The `:eager` option only works with Firefox at the moment.
 
 Etaoin supports link:{actions}[Webdriver Actions].
 They are described as "virtual input devices".
-They act as little device input scripts that run simultaneously.
+They act as little device input scripts that can even be run simultaneously.
 
 Here, in raw form, we have an example of two actions.
 One controls the keyboard, the other the pointer (mouse).
@@ -2315,6 +2305,8 @@ From least to most verbose:
 * `:debug`
 * `:all` for all messages.
 
+Applies to Chrome and Edge only.
+
 See <<console-logs>>
 
 [id=opt-driver-log-level,reftext=`:driver-log-level`]
@@ -2473,9 +2465,9 @@ _Default:_ <not set>
 
 _Example:_ See <<http-proxy>> for an example usage.
 
-A *WebDriver*'s capabilities can be vendor specific and define preferred options.
+A *WebDriver*'s capabilities can be vendor-specific and define preferred options.
 Read WebDriver vendor docs before setting anything here.
-While reading docs, note that Etaoin passes along `:capabilities` as `desiredCapabilties`.
+While reading docs, note that Etaoin passes along `:capabilities` under `firstMatch`.
 
 === Using Headless Drivers [[headless]]
 
@@ -2489,7 +2481,7 @@ Running without a UI is helpful when:
 Ensure your browser supports headless mode by checking if it accepts `--headless` command-line argument when running it from the terminal.
 
 When starting a driver, pass the `:headless` boolean flag to switch into headless mode.
-This flag is ignored for Safari which, as of June 2022, still does not support headless mode.
+This flag is ignored for Safari which, as of August 2024, still does not support headless mode.
 
 //{:test-doc-blocks/test-ns user-guide-headless-test}
 [source,clojure]
@@ -2566,7 +2558,7 @@ To specify a directory where the browser should download files, use the `:downlo
 ----
 
 Now, when you click on a download link, the file will be saved to that folder.
-Currently, only Chrome and Firefox are supported.
+Currently, only Chrome, Edge and Firefox are supported.
 
 Firefox requires specifying MIME-types of the files that should be downloaded without showing a system dialog.
 By default, when the `:download-dir` parameter is passed, the library adds the most common MIME-types: archives, media files, office documents, etc.
@@ -2598,7 +2590,7 @@ Set a custom `User-Agent` header with the `:user-agent` option when creating a d
 ----
 
 Setting this header is important when using <<headless,headless browsers>> as many websites implement some sort of blocking when the User-Agent includes the "headless" string.
-This can lead to 403 response or some weird behavior of the site.
+This can lead to 403 responses or some weird behavior of the site.
 
 === HTTP Proxy [[http-proxy]]
 
@@ -2622,7 +2614,7 @@ To set proxy settings use environment variables `HTTP_PROXY`/`HTTPS_PROXY` or pa
 NOTE: A `:pac-url` is for a https://en.wikipedia.org/wiki/Proxy_auto-config#The_PAC_File[proxy autoconfiguration file].
 Used with Safari as other proxy options do not work in Safari.
 
-To fine tune the proxy you use the original https://www.w3.org/TR/webdriver/#proxy[object] and pass it to capabilities:
+To fine tune the proxy you use the original https://www.w3.org/TR/webdriver/#proxy[object] and pass it as capabilities:
 
 //:test-doc-blocks/skip
 [source,clojure]

doc/02-developer-guide.adoc

@@ -237,6 +237,20 @@ Run `bb cljdoc-preview --help` for help.
 * `bb cljdoc-preview view` opens a view to your imported docs in your default web browser
 * `bb cljdoc-preview stop` stops the docker image
 
+=== Test Coverage
+Sometimes it's nice to get an idea of what parts of Etaoin its unit and doc tests (or more importantly, don't) cover.
+
+[source,shell]
+----
+bb test-coverage
+----
+
+The intent is not to strive for some percentage of coverage, just information on what is not covered.
+
+When possible, run from macOS, the only OS where we hit all supported browsers (you'll need all browsers and WebDrivers installed and up to date).
+
+It will take a while, but after tests are complete, crack open `./target/clofidence/index.html` for results.
+
 == Other Notes
 
 === Logging

env/test/resources/static/test.html

@@ -6,16 +6,16 @@
     </head>
     <body>
 
-
         <h3>Click section</h3>
         <script>
             function fill() {
                 document.getElementById("baz").innerHTML = "clicked";
             }
-        </script>
+
+       </script>
         <button id="foo" onclick="fill()">Click me</button>
         <span id="baz"></span>
-        <hr>
+       <hr>
 
         <h3>CSS section</h3>
         <style>
@@ -257,7 +257,48 @@ <h3>Shadow DOM</h3>
             </template>
         </div>
 
+        <hr>
+
+        <h3>Pointer Click Target</h3>
+        <div id="pointerClickTarget" style="border: 1px solid black;">
+            [not clicked]
+        </div>
+
+        <h3>Mouse Button State</h3>
+        <pre id="mouseButtonState">[no clicks yet]</pre>
+
         <h3 id="document-end">Document end</h3>
+        <script>
+            // Mouse button state, taken from:
+            // https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/buttons
+            const buttonNames = ["left", "right", "wheel", "back", "forward"];
+            function mouseButtonPressed(event, buttonName) {
+               // Use binary `&` with the relevant power of 2 to check if a given button is pressed
+               return Boolean(event.buttons & (1 << buttonNames.indexOf(buttonName)));
+            }
 
+            function format(event) {
+              const { type, buttons } = event;
+              const obj = { type };
+              for (const buttonName of buttonNames) {
+                obj[buttonName] = mouseButtonPressed(event, buttonName);
+              }
+              return JSON.stringify(obj, null, 2);
+            }
+
+            const log = document.getElementById("log");
+            function logButtons(event) {
+              // Avoid interfering with form elements
+              mouseButtonState.textContent = format(event);
+            }
+            document.addEventListener("mouseup", logButtons);
+            document.addEventListener("mousedown", logButtons);
+
+            // pointer click target
+            const pointerClickTarget = document.getElementById("pointerClickTarget");
+            pointerClickTarget.addEventListener("dblclick", (e) => {
+                pointerClickTarget.textContent = "[double clicked]";
+            });
+        </script>
     </body>
 </html>

script/test_coverage.clj

@@ -0,0 +1,17 @@
+(ns test-coverage
+  (:require [babashka.fs :as fs]
+            [helper.shell :as shell]
+            [lread.status-line :as status]))
+
+(defn generate-doc-tests []
+  (status/line :head "Generating tests for code blocks in documents")
+  (shell/command "clojure -X:test-doc-blocks gen-tests"))
+
+(defn run-all-tests []
+  (status/line :head "Running unit and code block tests under Clojure for coverage report")
+  (shell/command "clojure" "-X:test:test-docs:clofidence"))
+
+(defn -main [& _args]
+  (fs/delete-tree "./target/clofidence")
+  (generate-doc-tests)
+  (run-all-tests))