How to interact with shadow DOM in Selenium?

In one of my recent automation projects, I was writing code to click on a certain element on the webpage. It was a very simple element with an id avatar. To my surprise, Selenium failed to find that element and threw an exception NoSuchElementException . I inspected that element more carefully and found out that the element is inside some weird element shadow-root. In fact, there was a tree of elements including avatar inside that element shadow-root .

A quick google search revealed that shadow-root is not a regular DOM element, it is part of a shadow DOM. And as of now, Selenium WebDriver cannot interact with it. There is a proposal pending with w3c to support it.

What is DOM?

Before understanding the shadow DOM you should first get familiar with DOM.

The Document Object Model (DOM) connects web pages to scripts or programming languages by representing the structure of a document—such as the HTML representing a web page—in memory

https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model#HTML_DOM

In simple words, when the web browser fetches the HTML page it parses the document and turns it into DOM that is loaded into the memory. Here is a very simple HTML document:

<html lang="en">
 <head>
   <title>A simple web page</title>
  </head>
 <body>
    <h1>Hello world</h1>
    <p>I am rendered!</p>
  </body>
</html>

The HTML DOM representation of this document will look something like this:

HTML representation in memory as DOM

As you see the DOM is a tree-like structure. Please feel free to read more about DOM in this article.

What is a shadow DOM?

The shadow DOM is a way to achieve encapsulation in the HTML document. By implementing it you can keep the style and behavior of one part of the document hidden and separate from the other code of the same document so that there is no interference.

Shadow DOM allows hidden DOM trees to be attached to elements in the regular DOM tree — this shadow DOM tree starts with a shadow root, underneath which can be attached to any elements you want, in the same way as the normal DOM.

There are some bits of shadow DOM terminology to be aware of:

  • Shadow host: The regular DOM node that the shadow DOM is attached to.
  • Shadow tree: The DOM tree inside the shadow DOM.
  • Shadow boundary: the place where the shadow DOM ends, and the regular DOM begins.
  • Shadow root: The root node of the shadow tree.

The above section is taken from the MDN. You can read more about shadow DOM here.

How to access the shadow DOM?

You can access the shadow DOM by running a regular JavaScript in the main page context if its mode is open.

Let us take the following HTML code snippet:

<div>
  <div id="shell">
    #shadow-root (open)
      <div id="avatar"></div>
  </div>
  <a href="./logout.html">Logout</a>
</div>

As you can see it has both a shadow DOM (avatar) as well as regular DOM elements( Logout link).

To access the shadow DOM elements using JavaScript you first need to query the shadow host element and then can access its shadowRoot property. Once you have access to the shadowRoot, you can query the rest of the DOM like regular JavaScript.

var host = document.getElementById('shell');
var root = host.shadowRoot;
var avatar = root.getElementById('avatar');

How to write a Selenium code to access shadow DOM?

You probably know that Selenium WebDriver provides a mechanism to inject any JavaScript code into the browser. Please feel free to read this tutorial if you want to learn more about it.

We can take advantage of that feature and inject a piece of JavaScript into the browser to get the target element inside the shadow DOM. Once we have the target element we can parse it into a WebElement and can perform any valid operation on that element.

Java Code:

WebElement host = driver.findElement(By.id("shell"));
JavascriptExecutor js = (JavascriptExecutor)driver;
WebElement shadowRoot = (WebElement)(js.executeScript("return arguments[0].shadowRoot", host));
shadowRoot.findElement(By.id("avatar")).click();

Python Code:

host = driver.find_element_by_id("shell"))
shadowRoot = driver.execute_script("return arguments[0].shadowRoot", host)
shadowRoot.find_elemen_by.id("avatar")).click()

Sometimes I have observed that the WebDriver’s click() method throws some Exceptions. In this case, we can directly use JavaScript’s click() method.

Java Code:

WebElement host = driver.findElement(By.id("shell"));
JavascriptExecutor js = (JavascriptExecutor)driver;
js.executeScript("arguments[0].shadowRoot.getElementById('avatar').click()", host);

Python Code:

host = driver.find_element_by_id("shell"))
driver.execute_script("return arguments[0].shadowRoot.getElementById('avatar').click()", host)

Nested shadow DOM

Sometimes there is a more complex DOM structure where we have multiple shadow DOMs nested inside each other. If you inspect your Chrome browser’s Downloads page – chrome://downloads/ you’ll find the following DOM structure:

Nested shadow DOMs

As you can see there are three levels of shadow DOMs nested inside each other. What if you want to access the target element <div id="leftContent"> that is inside the third shadow DOM?

Well, you can apply the same principals we learned so far in this tutorial – Write JavaScript to first access the shadow host, then get the shadow DOM by accessing the shadowRoot property on the host. Once you have access to the first shadow DOM you can traverse it and try to access the root of the second shadow DOM and so on.

document.getElementsByTagName('downloads-manager')[0]
.shadowRoot
.getElementById('toolbar')
.shadowRoot
.getElementById('toolbar')
.shadowRoot
.getElementById('leftContent')

If we want to click on that element we can inject the complete JavaScript to the browser:

Java Code:

WebElement host = driver.findElement(By.tagName("downloads-manager"));
JavascriptExecutor js = (JavascriptExecutor)driver;
js.executeScript("arguments[0].shadowRoot.getElementById('toolbar').shadowRoot.getElementById('toolbar').shadowRoot.getElementById('leftContent').click()", host);

Python Code:

firstHost = driver.find_element_by_tag_name("downloads-manager")
driver.execute_script("return arguments[0].shadowRoot.getElementById('toolbar').shadowRoot.getElementById('toolbar').shadowRoot.getElementById('leftContent').click()", host)

We can optimise the above code and write a helper method that can return the shadow DOM for any shadow host:

Java Code:

public WebElement getShadowRoot(WebElement host) {
  JavascriptExecutor js = (JavascriptExecutor)driver;
  WebElement shadowRoot = (WebElement) js.executeScript("return arguments[0].shadowRoot", host);
  return shadowRoot;
}

Python Code:

def getShadowRoot(host):
    shadowRoot = driver.executeScript("return arguments[0].shadowRoot", host)
    return shadowRoot

We can use this helper method every time we need access to the shadow DOM:

Java Code:

//Get first shadow host and access its shadow root
WebElement host1 = driver.findElement(By.tagName("downloads-manager"));
WebElement root1 = getShadowRoot(host1);
//Get second shadow host and access its shadow root
WebElement host2 = shadowRoot1.findElement(By.id("toolbar"));
WebElement root2 = getShadowRoot(host2);
//Get third shadow host and access its shadow root
WebElement host2 = shadowRoot2.findElement(By.id("toolbar"));
WebElement root3 = expandRootElement(host2);
//Get the target element inside the third shadow DOM
WebElement downloads = root3.findElement(By.id("leftContent")).getText();
assert downloads.getText().contains('Downloads');

Python Code:

# Get first shadow host and access its shadow root
host1 = driver.find_element_by_tag_name("downloads-manager")
root1 = getShadowRoot(host1)
# Get second shadow host and access its shadow root
host2 = shadowRoot1.find_element_by_id("toolbar")
root2 = getShadowRoot(host2)
# Get third shadow host and access its shadow root
host2 = shadowRoot2.find_element_by_id("toolbar")
root3 = expandRootElement(host2)
# Get the target element inside the third shadow DOM
downloads = root3.find_element_by_id("leftContent")
assert 'Downloads' in downloads.text

Challenge

Can you apply the learnings of this tutorial and type some text in the search bar for Chrome’s downloads page? If you inspect the search bar and observe carefully you’ll find that it is inside the nested third shadow DOM. Please feel free to ask for help or post your solution in the comments below.

6 thoughts on “How to interact with shadow DOM in Selenium?”

    • Hi Cristian,

      Maybe you can use page factory @FindBy to find the host and then have a helper method somewhere in your PageObject or BaseClass (as per your framework implementation) that take care of interacting with the shadow DOM elements. For instance, if you take the example I provided in this tutorial and modify it like this

      @FindBy(id = “shell”)
      private WebElement host;

      //Somehwhere inside a helper method
      doClickOnShadowDOM() {
      JavascriptExecutor js = (JavascriptExecutor)driver;
      WebElement shadowRoot = (WebElement)(js.executeScript(“return
      arguments[0].shadowRoot”, host));
      shadowRoot.findElement(By.id(“avatar”)).click();
      }

      Reply

Leave a Reply