This post is about how I implemented a simple Visual Regression Test on my test playground, which interacts with elements inside an iFrame. This is something that can be a challenge for testing, since many of the most popular tools for automated testing like Gemini and Cypress don’t seem to support this. I’m adding to an existing project, so you might want to skim some previous posts for context. You can see the files for this project in this repo, and I’ve tried to screenshot anything relevant along the way. (Note: I also added a .gitignore file, which told git to ignore and not upload the node modules.
As usual, I’m trying to write the blog I would have wanted to read before starting the work. For this reason, it has some assumed knowledge – particularly, anything I covered in previous posts in the series – but also has a lot of basic information you might already know.
Making an iFrame inside the recipe page
An iFrame is simply a page inside a page. That is, it’s a html file containing all the stuff you want for a web page, which is then served inside another html file for a web page. Since the Recipe tab on my test site was pretty much empty, I added an iFrame here. iFrames are used for many ads, as well as by online banking sites among others. The one I made contained only a checkbox with some text saying “Tick me!”:
I added some internal CSS (where you include the style in the head of the html document instead of referring to a separate css file) to give the iFrame a white background so that is stands out on the page. You can see this in the head section of the html doc above.
To load one page as an iFrame in another, you need to create an iFrame element in the main page using
Since I’m serving local files to localhost instead of using a hosted site, I entered the file name instead of a url. I also added a height just in case this could change, causing tests on the frame to be flaky.
The process of adding an iFrame to the project took a lot less time and effort than I had previously thought so I’m tempted to make a little Bruce advert to sit here. For now, here is the frame in Recipes:
Capturing the frame in Gemini
Now that I had an iFrame on my site, I wanted to make a simple test to screenshot it. The test didn’t attempt to interact with it at all, or look for any elements inside of it. This test is meant to be a baseline to work from, the barest bones a Gemini test can get to:
Since it captured the frameArea (a section of the page that contains the iFrame) instead of the iFrame itself, the screenshot took up the whole width of the page, but that was fine for this purpose.
Next, I wanted to try selecting the checkbox that’s inside the iFrame. This was a lot trickier than you might think, since Selenium only accepts certain types of selectors. Despite this limitation, I tried a few variations on this, trying for about an hour to find something that worked:
This line says “inside the document, look for a class called frameyFrame, then return the Window object so you can access the iFrame’s internal DOM, and find document, then body, then the class tickbox”. However, when that was entered into the find() method used in my previous vrts, the test failed with a long error that boils down to “An invalid or illegal selector was specified”.
I tried my best, searched in stackoverflow for solutions and eventually got an experienced dev at work on board to help out. We spent half a day trying to get this to work through various methods, but in the end we decided it simply wasn’t possible. (Please let me know if you find a solution we missed though!)
Switching to Puppeteer
There are probably a bunch of options I could have chosen to use, but I’d just watched a few videos on puppeteer a week before, and decided to give this a shot.
I don’t want to make this into a super long post about how I set up Puppeteer, especially since I did that before Christmas. I don’t remember having any particular problems with setting it up so I will link the repos from which I got the instructions. I installed Puppeteer (which includes Headless Chrome), and used Differencify for the image diffing. I then added Jest to the project, which is a framework for running test suites, and finally added the Differencify Jest Reporter to get html reports for the tests.
Puppeteer documentation: https://github.com/GoogleChrome/puppeteer
Differencify documentation, including instructions for using with Jest: https://github.com/NimaSoroush/differencify
When it came to writing a test, I pretty much followed the docs. The first thing I did before starting was think through the steps the test would need to complete. In this case – open a new browser -> go to the link -> wait for the page to load -> look through the contents of the iFrame -> find the tickbox -> tick the tickbox -> take a screenshot -> close the page -> make an assertion that can pass/fail.
At the top of the test, there are a couple of lines required for Differencify to work. These simply get the Differencify package into the test so that it can understand particular keywords etc, which are the same words that you would use with Puppeteer without Differencify. I have debug:true turned on, since this allows you to see a log of each action taken and how long it took at each step, in the console. This was useful for, well – debugging, as well as checking which steps took longer than others to complete.
Under that, we have the start of the test itself. I can’t explain why the syntax is how it is, but you should follow the docs and it will be fine. There are two pieces of descriptory text in these lines, I used “tests differencify” and “ticks the tickbox”. When this test is run, these are combined into the name of the test in the Differencify report, so it’s important to make it clear here what each test does here. You can also have multiple tests in the suite using multiple it()s. Then each of these would be “tests differencify ticks the tickbox” and “tests differencify eats some cheese” for example.
I’ve annotated the test here to show what each line does. Sorry about the super large image!
As you can see, it does everything I thought it should – opens the browser, navigates to the page and waits for it to load. I definitely could have looked up a better solution than ‘wait 500ms’, as there should be some command for waiting until the network is idle, which would be more effective. Differencify then finds the iFrame and clicks on the tickbox, then takes a screenshot to prove it. If this is the first time running the test then it will use this screenshot as the ‘golden’ image, aka the holy source of truth and correctness against which future screenshots will be compared. If it’s not the first, then it will compare the new screenshot with the old one and highlight any differences.
Of course, all of this is totally useless unless we confirm that it does indeed tick the tickbox. I was super excited to find that it does – after trying to work with both Cypress and Gemini when it comes to our products, finding something that really does interact with them was totally worth my time.
After happily running this thing a few times, I decided to see it fail so that I could check out how the diffing shows up. This is where I discovered a limitation. Even after taking out the two lines that find the iFrame and tick the tickbox, saving and rerunning – the test passed! With nothing in the code asking it to tick the tickbox, my test came back as a pass – and because it passed, it didn’t bother saving a ‘current’ snapshot for me to check by eye, since according to Differencify they were the same. The only image I had to look at in the report was telling me it had ticked the box and passed:
No. No you did not pass, you little- rrrrRRRR. It might be that the tickbox is such a small change in terms of overall % difference, that it passed under the default tolerance threshold. I was reluctant to lower the tolerance, since this can mean extra flakiness.
Originally, Differencify was screenshotting the entire page in this test, and I changed this to shoot only the DOM element containing the iFrame. I thought this might help by making the tickbox take up a larger percentage of the overall image so that it doesn’t slip under the tolerance. This didn’t work. and I decided that a vrt is probably just not a good way of checking such a small change – but I will be experimenting later with some of the other things you can do in Puppeteer, ie checking the state of elements directly to see if a checkbox is ticked or a box contains the expected type of output (string/number/etc) without taking any screenshots at all. I don’t have eventEmitters or anything like that, but from my reading it looks like Puppeteer can grab pretty much anything you want.
I ditched the checkbox in that case, and went for something more obvious – a piece of text that changes colour from black to red on click. Lo and behold, not only did it successfully click on the text, but the test also recognised a failure when I commented out the lines exocuting the click action:
What did I learn?
I also learned something fundamental to test writing, which I had not encountered before. Write a test that fails, then make changes so that it passes, instead of making changes and then writing a test to pass. If that makes sense… What I did was write a passing test, and if I had not wanted to see what colour Differencify highlights diffing in, then I might not have taken out the lines that make it work and I wouldn’t have found out that my test was ineffective. The test I wrote checked the criteria effectively only once – the very first time I ran it. After then, it could have failed any number of times without my knowing. It’s also just good to work out the limitations of your testing tools or methods, to test the tests as it were.
It’s not perfect, but I have a simple test showing that Puppeteer can be used to create tests that interact with elements inside of an iFrame. The next thing I’d like to do is play around with pulling data out of the iFrame elements, eg text or colour, and checking that those match expectations without using screenshots. For example, checking that the text next to the checkbox says ‘Tick me!’ or that pressing a button causes a change in another element. I might also do some experimenting around this with Protractor, as we’re looking to implement these types of test at work soon. We’ll be hooking them into the BrowserStack API for some cross-browser testing too, so I’ll be looking to document some of that process if I have the time.
Additionally, I’d like to make it so that the Visual Regression Tests send their screenshots to a bucket instead of my local drive, though I currently have no idea how that would work (which is why it’s a good idea to try!)
I’ve moved experimentation with ReactJS onto the back burner for now, simply because I don’t have time to develop my skills both in webdev and testing at the same time.
In an ideal world I’d like to be updating this blog once a week or so, but progress is slow on all personal development fronts. We’re taking measures at the moment, switching up our style at work to try and free up more time so that I can be writing automated tests for our product during work hours. Then outside of work hours, I can find the time to transfer those findings onto my personal project in as simple a way as possible, then write about it. I also have a post coming up about the app Toggl and how I use it to track my time on different types of tasks. Until then, see ya!