Category Archives: Automatic Tests

Automatic visual tests with Episerver: Prepare your conditions with Content Providers

In my previous post I talked about how to set up automatic visual tests – and for a time, it was good. Then editors and other team members starts changing the content and your tests starts showing deviations based on content changes rather than code changes.

Yes, you can set up rules on not changing areas included in the tests, but can you really trust everyone?

To get around this, I started working on a set of Content Providers that will give a static set of content. Kind of like when you’re creating unit tests and mocking implementations for interfaces:

public interface IService {
   string Get(int id);

   string[] List();
var mockService = new Mock<IService>(); // Start mocking the interface "IService"

   .Setup(s => s.Get(It.IsAny<int>())) // Parameter is any integer
   .Returns((string)null); // Return null
   .Setup(s => s.Get(10)) // But if parameter is 10
   .Returns("alf"); // Return "alf"

   .Setup(s => s.List()) // When calling List()
   .Returns(new[] { "alf" }); // Return an array that only contains "alf"

IService service = mockService.Object; // Done mocking the interface and create an instance of it

Now let’s create a Test Class to verify that everything Works before building the real implementation of IService.

// Assert that the mocked instance returns what we expect
Assert.AreEqual("alf", service.Get(10));

var list =service.List();
Assert.AreEqual(1, list.Length);

Based on the content types in Alloy Templates I have a sample of the Content Provider that sets up examples of the pages and blocks that I want to test.

Separate project for Models

First of all when I create an Episerver solution, I want to have all Episerver Model related in a separate project. That way I can easier separate code that belongs to presentation (the actual MVC web project) from code that belongs to business logic.

After I have created a standard Alloy Template solution called Toders.VisualTests.Web I move all Content Types and Episerver specific assets that are tightly connected to the Content Types (Attributes, or similar) into a specific Models project.

For Alloy there are also some interfaces, Global constant strings and similar that I need to move so that the project compiles properly. Check my commit in GitHub to see the moved files.

Creating the Content Providers

Once I have moved the Models into a separate project, I can create a separate project that will contain my Content Providers.

As the Episerver content structures for Assets (Blocks and Media) and Pages are separate. I will need to create two separate Content Providers, the AssetProvider and PageProvider.


In the AssetProvider I create the structure for Media and Blocks that will be used on the Pages. It contains an AssetModel so that I can use the created static assets on my Pages.


In the PageProvider I create the structure of the pages that are to be tested. I can create pages that contain Blocks and Media from the ContentProvider.

As you can see I am using the AssetModel to deliver the created assets into the Page Provider.

Initiating everything

To get everything working, I need to create some new Content Types that will act as roots for my Content Providers – StaticStartPage for the Pages and StaticAssetsRootFolder. This is to easier identify which content to use as roots for my Content Providers.

Once the Content Types are created, I create a Start Page for my static content and . With the current implementation, I’ll need to restart my website because I will set up the static website on initialization.

At startup, I have an InitializationModule that checks whether a Page of the type StaticStartPage is created, ensures that there is a Folder of the type StaticAssetsRootFolder and initializes the AssetProvider and PageProvider. The reason I let them implement the original Start Page and ContentFolder is to

After initialization, the module also ensures that there is a SiteDefinition created that is using the static start page as start page.

Finally I need to set up my IIS server to listen to the url for the SiteDefinition (per default is it set to localhost:2335, but can be updated to anything in the Admin Mode).

Testing it

Finally I need to create some tests. This part is what I show in my previous blog post about Automatic visual tests.

For this article I’m using some other tests where I have split the implementation between just doing screenshots with PhantomJS and an implementation that user AppliTools Eyes.


Currently I’m testing what some content with various Display Options looks like what in a Content Area. I also look what text in different formats looks like.

If I would receive a bug report from my editors that bold text within a bullet list looks weird, I can in a TDD behaviour add it to the Page Provider and verify it with my tests:


var mainBody = "<h2>Header 2</h2>" +
    "<h3>Header 3</h3>" +
    "<p>Paragraph with <strong>strong</strong> and <em>italic</em> text. Also a <strong><em>combination</em></strong>.</p>" +
    "<li>Bullet 1</li>" +
    "<li>Bullet 2</li>" +
    "<li>Bullet 3</li>" +
    "</ul>" +
    "<li>Numbered 1</li>" +
    "<li>Numbered 2</li>" +
    "<li>Numbered 3</li>" +

    page.MainBody = new XhtmlString(mainBody);


var mainBody = "<h2>Header 2</h2>" +
    "<h3>Header 3</h3>" +
    "<p>Paragraph with <strong>strong</strong> and <em>italic</em> text. Also a <strong><em>combination</em></strong>.</p>" +
    "<li>Bullet 1</li>" +
    "<li>Bullet 2</li>" +
    "<li>Bullet 3</li>" +
    "<li><strong>Fat text</strong></li>" +
    "</ul>" +
    "<li>Numbered 1</li>" +
    "<li>Numbered 2</li>" +
    "<li>Numbered 3</li>" +

    page.MainBody = new XhtmlString(mainBody);

I hope this gives you some inspiration on how to improve your QA and tests on your website. Of course having a separate website requires that you have a License that allows for multiple websites. This can be achieved using sub pages to your already existing websites, but there’s a risk that you can’t have a fully testable copy of your original website.

You can download the code from GitHub, and set it up with the following steps:

  • Set up a standard Alloy Templates database and adjust the connectionstrings.config
  • Set up an IIS server pointing to the src\Alloy.MVC.Template folder on your drive, make sure that it has two bindings – one for the ordinary Alloy Website and one for your static content
  • Make sure that the InitializationModule are using the correct url for the static website in the field StaticWebsiteUrl in the top of the class
  • Create a page of the type “Static Start Page” under the root page
  • Unfortunately you’ll need to restart the website so that the Site Definition will come in place

Automatic visual tests with Episerver: Test after a Deploy

A common reason I have heard that Episerver developers don’t write automatic tests are that they’re “only doing content websites”.

First of all I suggest that you have an “API first” approach, even if you’re only going to make content based functionality such as listing and filtering News Entries.

This falls well into the classic three tier layer where you separate data, logic and presentation.


But that’s not what I want to share today. I’m here to talk about Visual Tests!

Visual tests

With visual tests I mean that you’re comparing what your website looks like before and after a release. This is normally done just by performing a test routine – following steps such as clicking links or buttons, entering form fields and so on.

This is something that can take far too much time and energy and within a DevOps mindset you want to automate these tasks as much as possible.



The first tool I use for automatic visual tests is Selenium which is a tool where you can script a web browser to do basically anything.

You can use various techniques to create your automated scripts, either coding them with many different languages (C#, Python, Ruby or even Perl, JavaScript and PHP).

You can also use your scripts with different web drivers which are available for most modern browsers on the most popular Operating Systems out there. This is very good when you want to make sure that the same function works and looks the same on Microsoft Edge, Firefox, Chrome and so on.

A simple piece of code to browse to a website using the Chrome browser and take a screenshot.

using (IWebDriver driver = new ChromeDriver())
    driver.Manage().Window.Size = new Size(1280, 980);
    driver.Url = "";
    ((ITakesScreenshot)driver).GetScreenshot().SaveAsFile(@"c:\temp\", ScreenshotImageFormat.Png);

Simple as that. Next problem is that it would take too much time to compare screenshots and there is a risk of the human error. Missing that the font might be wrong or that the margin is a few pixels too large or small.

Simplify comparison with Applitools Eyes

For one of our customers we’re using Applitools Eyes to help us compare the screenshots.

It is a tool where you can create test cases and compare screenshots between different test rounds and get a warning if there is something different between each round.

If you get a warning you can say yes or no whether the change was OK, and if you want to use this screenshot as base for an upcoming comparison.

So what we do in our code is instead of creating a screenshot with the WebDriver, we wrap our WebDriver with the Eyes API.

So with this code we’re doing the same test as previously, but instead of saving the screenshot to disk, we’re starting a test in Eyes and sends the screenshot of our start page.

using (IWebDriver driver = new ChromeDriver())
    var eyes = new Eyes
        ApiKey = "--APIKEY--"

    eyes.Open(driver, "My Application", "My Test");

    driver.Manage().Window.Size = new Size(1280, 980);
    driver.Url = "";
    eyes.CheckWindow("Start Page");

If there’s any differences, the API will throw an Exception with information on where to verify the screenshots. We can also adjust the code to not throw Exceptions in case we want to.

Automatic Visual Tests after a Release

At the latest Episerver Partner Close Up earlier this year, Eric Herlitz and I had a presentation about DevOps where we showed how you can use your automatic tests as an automated step while deploying.

We implemented these visual tests with the simple tests. Note that some of the screenshots and code are not best practice in a release and test setup but this was enough for demo cases.

1: Create a normal Alloy Website solution

Just use the Episerver Visual Studio Templates to create a standard Alloy Website.


2: Switch the WebDriver to use Phantom JS

As the visual tests will be performed on a TFS agent outside a user context, it is better to use the Headless browser “Phantom JS”. It’s a browser that does not have a UI, but can render and take screenshots of websites as if it was a normal browser.

Simply install the NuGet packages required for Phantom JS and use the OpenQA.Selenium.PhantomJS.PhantomJSDriver instead.

3: Put the code inside a Test Project

Create a Test Project in your solution and start scripting.

In my tests I am performing the following steps

  1. Browse to the start page
  2. Send a screenshot to Eyes
  3. Browe to the contact page which contains a contact form built with Episerver Forms
  4. Send a screenshot to Eyes
  5. Submit the form, expecting that some validation messages should show for the required fields
  6. Send a screenshot to Eyes
  7. Fill in the field for name and enter an invalid e-mail address into the e-mail field.
  8. Submit the form, expecting that some validation messages should show for the invalid e-mail
  9. Send a screenshot to Eyes
  10. Finish the e-mail address, check some checkboxes and a comment.
  11. Send a screenshot to Eyes
  12. Submit the form, expecting that the form should be submitted and a confirmation message to be displayed
  13. Send a screenshot to Eyes

I have uploaded our tests scripts to my GitHub. In these steps I have made sure that the tests does not throw Exceptions if Eyes notices any differences.

4: Push the code to VSTS and setup VSTS Build to Publish the Artifacts for the test project

Just push your code to a repository in your VSTS, or to a repository that is accessible by VSTS (such as GitHub).

Setup VSTS to build your solution.

vsts build.png

Looking closer at the task for publishing the test project you can see that the artifacts are called “VisualTests”. We will use this later.vsts publush tests.png

5: Deploy to Azure using VSTS

Create the Azure Resources required for running an Episerver website. this blog post from Erik Henningson contains some Powershell scripts that makes it very easy to get everything up and running.

Me and Eric are more used to using Octopus Deploy for Release Management but we decided to use the VSTS Release Management for our demo.

The steps are quite simple:

vsts release 1.png

  1. Get the code from the Artifacts
  2. Put the code to the Azure website that represents my Dev environment (including some configuration transformation)

In a real project I would also add environments for Test, Pre-prod and Production (or simply the Integration environment if I was running DXC Services).

6: Add the Release Step to run your Tests from the artifact “VisualTests”

I added the step to also run the tests from the Artifact “VisualTests”. It will simply run the tests which in turn will start pushing screenshots to AppliTools Eyes.vsts release 2.png

After the deploy, I let my Release Managers approve the release before it can continue on in the Release Life Cycle.

vsts release 3.png

This is so that the manager can verify that everything works as intended without manually performing the test procedures.


These are the basics of setting up Automatic Visual Tests using Selenium, AppliTools Eyes, VSTS and Azure. As I mentioned everything also doesn’t follow best practices but I imagine that there are experts out in the open on these specific tools, Feel free if you have any hints to share.

Keep in mind that even if Visual Tests is a simple step if you don’t have any automatic tests, but they do NOT replace a good suit of Unit Tests.

Unit Tests can still give you a better best coverage when you have a more complex application and/or have integrations against other systems.

What’s next?

Our next issue is that the tests often fails if someone plays around with content on the dev server for other tests. One approach for this could be to spin more environments for each test purpose, or reset the database.

I have another idea that might help with this and can help with other challenges we’ve experiences. Stay tune for next blog post about Automatic Visual Tests.