You have finished you application, and you have created a lot of unit tests, it should be ok, no? Not quite. There are a lot of issues that can arise from the UI. You cannot unit test the UI and, if you done it right, you have a full set of tests for your Model and ViewModel, but not for the View.
For web applications, you can test the UI with something like Selenium (http://www.seleniumhq.org/), a framework that automates browsers and can test the UI of your web apps. You can even check my article (in portuguese) here, where I show how to automate a web search and store the results in a WPF application.
But and what can be done for desktop applications? You can use UI automation and create a Coded UI Test (https://msdn.microsoft.com/en-us/library/dd286726.aspx), but that only works with Visual Studio Enterprise. You could also use something like UI Automation PowerShell Extensions, but this is somewhat cumbersome: you must program your tests using PowersSell and test the results – there is no test runner, except the PowerShell window.
Wouldn’t it be good to create our tests the same way we create our unit tests, using the test framework we are used to (that can be MS-Test, NUnit, XUnit, and so on?). That’s not out of reality. Now, there is something available for us: Appium (http://appium.io/).
Appium is an open source framework that uses the same technology as Selenium to test Android and iOS apps, but Microsoft has created a WebDriver for Appium that allows you to test also Windows apps. With it, you can test any Windows app, and it can be a Win32 app, .NET app or even Windows UWP apps. And the best thing is that you can youse any test framework you want to create your tests. Nice, no? And did I say that you don’t need to have the app’s source code to create the tests?
Installing the Microsoft WinDriver
To install the Microsoft WinDriver, you can go to https://github.com/Microsoft/WinAppDriver/releases and download WindowsApplicationDriver.msi and run it. Once it is installed, just go to the installation path and run WinAppDriver.exe and that’s it. You are up and running.
The web server is listening at http://127.0.0.1:4723. You can test this by opening a web browser window and typing the address: http://127.0.0.1:4723/status. You should get something like this:
{"build":{"revision":"30002","time":"Wed Nov 30 16:48:11 2016","version":"0.7.1611"},"os":{"arch":"amd64","name":"windows","version":"10"}}
Now we are ready to create our first tests. To show you I don’t need any source code, we will create tests for the Windows Calculator.
Testing the Windows Calculator
Open Visual Studio and create a new Test Project for the .NET Framework and name it CalcTests. Then right-click in the References node in the Solution Explorer and select “Manage NuGet packages”. Select the Appium.WebDriver.
Then let’s create our first test. In TestMethod1 method in UnitTest1.cs, type the following:
public void TestMethod1() { DesiredCapabilities appCapabilities = new DesiredCapabilities(); appCapabilities.SetCapability("app", "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"); appCapabilities.SetCapability("deviceName", "WindowsPC"); var calculatorSession = new WindowsDriver(new Uri("http://127.0.0.1:4723"), appCapabilities); Assert.IsNotNull(calculatorSession); }
Then run it. If everything goes fine, the test should pass and the calculator should be open in your desktop. If you read the code, you may ask: “where do I find this Id in the second line of the method?”. If you designed the app, you can find it in the app manifest, in the Packaging tab:
If you haven’t designed it, you can check your installed packages with PowerShell with a command like this:
Get-AppxPackage | where {$_.Name -like "*calc*"}
And get the family name:
With this, we can interact with our tests. In order to don’t have to open the calculator after each test, we can create a method to initialize the calculator at the beginning of the tests and close it at the end:
public class CalculatorTests { private static WindowsDriver _calculatorSession; [TestMethod] public void CalculatorIsNotNull() { Assert.IsNotNull(_calculatorSession); } [ClassInitialize] public static void StartCalculator(TestContext context) { DesiredCapabilities appCapabilities = new DesiredCapabilities(); appCapabilities.SetCapability("app", "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"); appCapabilities.SetCapability("deviceName", "WindowsPC"); _calculatorSession = new WindowsDriver(new Uri("http://127.0.0.1:4723"), appCapabilities); } [ClassCleanup] public static void CloseCalculator() { _calculatorSession.Dispose(); _calculatorSession = null; } }
The next step is to be sure that the calculator is in a know state at the start of the test. This is done in the method marked with the TestInitialize attribute:
[TestInitialize] public void ResetCalculator() { _calculatorSession.FindElementByAccessibilityId("NavButton").Click(); _calculatorSession.FindElementByName("Standard Calculator").Click(); _calculatorSession.FindElementByName("Clear").Click(); }
This code clicks the navigator button and then selects the “Standard Calculator” menu item and clears the display. As you can see from the code, I am using the FindElementByAccessibilityId and FindElementByName. You should be asking yourself “And where do I find these Ids”. You can inspect all apps running in your machine using the Inspect program that comes with the SDK. Open a Visual Studio Command Prompt and type inspect. A program like this should open:
You can inspect the calculator and get the Ids and names for the buttons and menu items. Note that in some cases (like in the CalculatorResults textbox), the Name property changes and you cannot find it by name, so you must use the AccessibilityId, that doesn’t change.
With that, we can create our second test, that will ensure that clicking in a key will display the corresponding number in the results. For this test, I will use a new feature introduced in MSTest V2, Data Row Tests – you can create a single method that will run different tests with the data entries you specify. This is different that using many assertions in the same test: when you use a Data Row Test, you have independent tests for each row, and you can see which ones fail (and the TestInitialize and TestCleanup are run for each one). When you join all assertions in a single test, if one fails, the remaining ones are not run, so you don’t know if one of the following tests also fails. And, besides that, the TestInitialize and TestCleanup are not run for each assertion.
[DataTestMethod] [DataRow("One", "1")] [DataRow("Two", "2")] [DataRow("Three", "3")] [DataRow("Four", "4")] [DataRow("Five", "5")] [DataRow("Six", "6")] [DataRow("Seven", "7")] [DataRow("Eight", "8")] [DataRow("Nine", "9")] [DataRow("Zero", "0")] public void CalculatorKeysShouldDisplay(string key, string expected) { _calculatorSession.FindElementByName(key).Click(); var actual = _calculatorSession.FindElementByAccessibilityId("CalculatorResults") .Text.Replace("Display is", "").Trim(); Assert.AreEqual(expected, actual); }
One note, here – to run successfully these tests, you should install the latest NuGet packages for MSTest.TestFramework and MSTest.TestAdapter. I was using the default ones that came with the Visual Studio 2017 templates and this test wasn’t being detected. As soon as I updates the NuGet packages, everything was fine.
Now that we have our first tests in place, let’s continue to test the calculator’s UI. Instead of testing the standard calculator, we will test the programmer’s calculator. In the project, create a new class and name it ProgrammersCalculatorTests.
Add this code for the initialization of the class and tests:
private static WindowsDriver _calculatorSession; [ClassInitialize] public static void StartCalculator(TestContext context) { DesiredCapabilities appCapabilities = new DesiredCapabilities(); appCapabilities.SetCapability("app", "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"); appCapabilities.SetCapability("deviceName", "WindowsPC"); _calculatorSession = new WindowsDriver(new Uri("http://127.0.0.1:4723"), appCapabilities); _calculatorSession.FindElementByAccessibilityId("NavButton").Click(); _calculatorSession.FindElementByName("Programmer Calculator").Click(); _calculatorSession.FindElementByAccessibilityId("decimalButton").Click(); } [ClassCleanup] public static void CloseCalculator() { _calculatorSession.Dispose(); _calculatorSession = null; } [TestInitialize] public void ResetCalculator() { _calculatorSession.FindElementByAccessibilityId("decimalButton").Click(); _calculatorSession.FindElementByName("Clear").Click(); }
The initialization code is similar to the other test, but instead of setting the type of calculator for every test, we will set the programmer’s calculator at the start of the tests. Then, we’ll clear the calculator and reset to the Decimal entry before each test.
The first test we’ll create is to make sure that the display changes to the correct value when you click the Hex, Octal and Binary buttons:
[DataTestMethod] [DataRow("hexButton","C")] [DataRow("octolButton","14")] [DataRow("binaryButton","1100")] public void Number12ShouldConvertOkToHexOctalAndBinary(string buttonId, string result) { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Two").Click(); _calculatorSession.FindElementByAccessibilityId(buttonId).Click(); var actual = _calculatorSession.FindElementByAccessibilityId("CalculatorResults") .Text.Replace("Display is", "").Trim(); Assert.AreEqual(result, actual); }
We are using the parametrized tests again, passing the ids of the buttons and the expected results for the decimal “12”.
Now, let’s create the tests for the keys that should be enabled/disabled for each mode set:
[TestMethod] public void InHexModeAllButtonsShouldBeEnabled() { var enabledButtons = new[] {"One","Two","Three" ,"Four","Five","Six", "Seven","Eight","Nine","Zero","A", "B", "C", "D", "E", "F" }; _calculatorSession.FindElementByAccessibilityId("hexButton").Click(); foreach (var buttonName in enabledButtons) { Assert.IsTrue(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } } [TestMethod] public void InOctalModeButtonsZeroToSevenShouldBeEnabled() { var enabledButtons = new[] { "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Zero" }; _calculatorSession.FindElementByAccessibilityId("octolButton").Click(); foreach (var buttonName in enabledButtons) { Assert.IsTrue(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } } [TestMethod] public void InBinaryModeAllButtonsExceptZeroAndOneToSevenShouldBeDisabled() { var disabledButtons = new[] {"Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine", "A", "B", "C", "D", "E", "F" }; _calculatorSession.FindElementByAccessibilityId("binaryButton").Click(); foreach (var buttonName in disabledButtons) { Assert.IsFalse(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } }
These tests have several asserts in a single test (the assert is in a foreach loop). Although I don’t like many asserts in a single test, in this case, it makes sense: if only one of these keys is not enabled or disabled, the test should fails. It doesn’t make sense to create one test for every single key. In these tests, I have added a message, so if one fails, I can check which key made the test fail.
The next test are for the operations:
[TestMethod] public void NotZeroShouldBeMinus1() { _calculatorSession.FindElementByName("Zero").Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual("-1", actual); } [TestMethod] public void NotOneShouldBeMinus2() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual("-2", actual); } [TestMethod] public void OneAndZeroShouldBeZero() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("And").Click(); _calculatorSession.FindElementByName("Zero").Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual("0", actual); } [TestMethod] public void OneOrZeroShouldBeOne() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Or").Click(); _calculatorSession.FindElementByName("Zero").Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual("1", actual); } [TestMethod] public void OneXorZeroShouldBeOne() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Exclusive or").Click(); _calculatorSession.FindElementByName("Zero").Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual("1", actual); } [TestMethod] public void OneLshOneShouldBeTwo() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Left shift").Click(); _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual("2", actual); } [TestMethod] public void OneRshOneShouldBeZero() { _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Right shift").Click(); _calculatorSession.FindElementByName("One").Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual("0", actual); }
Looking at these tests, you can see some patterns – there are two kinds of tests – for unary operators (NOT) and for binary operators (AND, OR, XOR, LSH, RSH). We can refactor these tests to create parametrized tests (thank you Microsoft for introducing this )
[DataTestMethod] [DataRow("Zero","-1")] [DataRow("One","-2")] [DataRow("Nine","-10")] public void NotTestsForPositiveNumbersShouldBeOk(string key, string result) { _calculatorSession.FindElementByName(key).Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual,$"Test for key {key}"); } [DataTestMethod] [DataRow("One", "0")] [DataRow("Two", "1")] [DataRow("Nine", "8")] public void NotTestsForNegativeNumbersShouldBeOk(string key, string result) { _calculatorSession.FindElementByName(key).Click(); _calculatorSession.FindElementByName("Positive Negative").Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual, $"Test for negative {key}"); } [DataTestMethod] [DataRow("One","And","Zero","0")] [DataRow("One","And","One","1")] [DataRow("One","And","Three","1")] [DataRow("One", "Or", "Zero", "1")] [DataRow("One", "Or", "One", "1")] [DataRow("One", "Or", "Three", "3")] [DataRow("One", "Exclusive or", "Zero", "1")] [DataRow("One", "Exclusive or", "One", "0")] [DataRow("One", "Exclusive or", "Three", "2")] [DataRow("One", "Left shift", "Zero", "1")] [DataRow("One", "Left shift", "One", "2")] [DataRow("One", "Left shift", "Three", "8")] [DataRow("One", "Right shift", "Zero", "1")] [DataRow("One", "Right shift", "One", "0")] [DataRow("One", "Right shift", "Three", "0")] public void TestsForOperatorsShouldBeOk(string first, string oper, string second, string result) { _calculatorSession.FindElementByName(first).Click(); _calculatorSession.FindElementByName(oper).Click(); _calculatorSession.FindElementByName(second).Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual,$"Test for {first} {oper} {second}"); }
Now he tests are cleaner and we also introduced a lot of new tests for the operators. If we want to add a new test here is just a matter of adding a new data row.
Now, we can go on and test another kind of applications: Win32 applications.
Testing the UI of Win32 applications
We have seen how to test the UI of a UWP application, but that’s not all that we can do with Appium. With it, we can test any kind of desktop application, including Win32 applications.
We will test another calculator, the old Windows calculator, downloaded from http://winaero.com/blog/get-calculator-from-windows-8-and-windows-7-in-windows-10/. After downloading and installing it, you can run it by typing calc.exe on the search bar:
Now we can create our tests for the calculator. Create a new class and name it OldCalculatorTests. Add this code in the class:
private static WindowsDriver _calculatorSession; [TestMethod] public void CalculatorIsNotNull() { Assert.IsNotNull(_calculatorSession); } [DataTestMethod] [DataRow("1", "1")] [DataRow("2", "2")] [DataRow("3", "3")] [DataRow("4", "4")] [DataRow("5", "5")] [DataRow("6", "6")] [DataRow("7", "7")] [DataRow("8", "8")] [DataRow("9", "9")] [DataRow("0", "0")] public void CalculatorKeysShouldDisplay(string key, string expected) { _calculatorSession.FindElementByName(key).Click(); var actual = _calculatorSession.FindElementByName("Result").Text.Trim(); Assert.AreEqual(expected, actual); } [ClassInitialize] public static void StartCalculator(TestContext context) { DesiredCapabilities appCapabilities = new DesiredCapabilities(); appCapabilities.SetCapability("app", "calc.exe"); appCapabilities.SetCapability("deviceName", "WindowsPC"); _calculatorSession = new WindowsDriver(new Uri("http://127.0.0.1:4723"), appCapabilities); _calculatorSession.FindElementByName("Calculator").SendKeys(Keys.Alt+"1"); _calculatorSession.FindElementByName("Clear").Click(); } [ClassCleanup] public static void CloseCalculator() { _calculatorSession.Dispose(); _calculatorSession = null; } [TestInitialize] public void ResetCalculator() { _calculatorSession.FindElementByName("Clear").Click(); }
As you can see, the tests are pretty much the same ones as the UWP calculator. You just have to change the app name to calc.exe (you must add the command line text for activating the program – in this case, the calculator is in the path) and change the control names. Then, the tests are the same. The tests for the Programmer’s calculator are in the ProgrammersOldCalculatorTests:
public class ProgrammersOldCalculatorTests { private static WindowsDriver _calculatorSession; [ClassInitialize] public static void StartCalculator(TestContext context) { DesiredCapabilities appCapabilities = new DesiredCapabilities(); appCapabilities.SetCapability("app", "calc.exe"); appCapabilities.SetCapability("deviceName", "WindowsPC"); _calculatorSession = new WindowsDriver(new Uri("http://127.0.0.1:4723"), appCapabilities); _calculatorSession.FindElementByName("Calculator").SendKeys(Keys.Alt + "3"); _calculatorSession.FindElementByName("Clear").Click(); } [ClassCleanup] public static void CloseCalculator() { _calculatorSession.Dispose(); _calculatorSession = null; } [TestInitialize] public void ResetCalculator() { _calculatorSession.FindElementByName("Decimal").Click(); _calculatorSession.FindElementByName("Clear").Click(); } [DataTestMethod] [DataRow("Hexadecimal", "C")] [DataRow("Octal", "14")] [DataRow("Binary", "1100")] public void Number12ShouldConvertOkToHexOctalAndBinary(string buttonId, string result) { _calculatorSession.FindElementByName("1").Click(); _calculatorSession.FindElementByName("2").Click(); _calculatorSession.FindElementByName(buttonId).Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual); } private static string GetDisplayText() { return _calculatorSession.FindElementByAccessibilityId("Result").Text.Trim(); } [TestMethod] public void InDecimalModeLetterButtonsShouldBeDisabled() { var disabledButtons = new[] { "A", "B", "C", "D", "E", "F" }; foreach (var buttonName in disabledButtons) { Assert.IsFalse(_calculatorSession.FindElementByName(buttonName).Enabled); } } [TestMethod] public void InHexModeAllButtonsShouldBeEnabled() { var enabledButtons = new[] {"1","2","3" ,"4","5","6", "7","8","9","0","A", "B", "C", "D", "E", "F" }; _calculatorSession.FindElementByAccessibilityId("hexButton").Click(); foreach (var buttonName in enabledButtons) { Assert.IsTrue(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } } [TestMethod] public void InOctalModeButtonsZeroToSevenShouldBeEnabled() { var enabledButtons = new[] { "1", "2", "3", "4", "5", "6", "7", "0" }; _calculatorSession.FindElementByAccessibilityId("octolButton").Click(); foreach (var buttonName in enabledButtons) { Assert.IsTrue(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } } [TestMethod] public void InBinaryModeAllButtonsExceptZeroAndOneToSevenShouldBeDisabled() { var disabledButtons = new[] {"2", "3", "4", "5", "6", "7", "8", "9", "A", "B", "C", "D", "E", "F" }; _calculatorSession.FindElementByAccessibilityId("binaryButton").Click(); foreach (var buttonName in disabledButtons) { Assert.IsFalse(_calculatorSession.FindElementByName(buttonName).Enabled, $"Test for {buttonName}"); } } [DataTestMethod] [DataRow("0", "-1")] [DataRow("1", "-2")] [DataRow("9", "-10")] public void NotTestsForPositiveNumbersShouldBeOk(string key, string result) { _calculatorSession.FindElementByName(key).Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual, $"Test for key {key}"); } [DataTestMethod] [DataRow("1", "0")] [DataRow("2", "1")] [DataRow("9", "8")] public void NotTestsForNegativeNumbersShouldBeOk(string key, string result) { _calculatorSession.FindElementByName(key).Click(); _calculatorSession.FindElementByName("Negate").Click(); _calculatorSession.FindElementByName("Not").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual, $"Test for negative {key}"); } [DataTestMethod] [DataRow("1", "And", "0", "0")] [DataRow("1", "And", "1", "1")] [DataRow("1", "And", "3", "1")] [DataRow("1", "Or", "0", "1")] [DataRow("1", "Or", "1", "1")] [DataRow("1", "Or", "3", "3")] [DataRow("1", "Exclusive or", "0", "1")] [DataRow("1", "Exclusive or", "1", "0")] [DataRow("1", "Exclusive or", "3", "2")] [DataRow("1", "Left shift", "0", "1")] [DataRow("1", "Left shift", "1", "2")] [DataRow("1", "Left shift", "3", "8")] [DataRow("1", "Right shift", "0", "1")] [DataRow("1", "Right shift", "1", "0")] [DataRow("1", "Right shift", "3", "0")] public void TestsForOperatorsShouldBeOk(string first, string oper, string second, string result) { _calculatorSession.FindElementByName(first).Click(); _calculatorSession.FindElementByName(oper).Click(); _calculatorSession.FindElementByName(second).Click(); _calculatorSession.FindElementByName("Equals").Click(); var actual = GetDisplayText(); Assert.AreEqual(result, actual, $"Test for {first} {oper} {second}"); } }
As you can see, there is not much difference from the previous tests. Testing a Win32 application is almost the same as testing an UWP app.
Conclusions
With Appium, you can test the UI of your application the same way you do with you unit tests: you can use the same framework and create UI tests similar to unit tests. This is a great bonus, as there is almost no learning curve and, as an added bonus, you can test all your desktop applications the same way, whether they are UWP, .NET or Win32 apps. Nice, no?
All the source code for this article is in https://github.com/bsonnino/CalcTests
This is not about Appium at all. Please change the title.
Why not? WinAppDriver for Appium allows you to test the UI Windows Apps. The title is clear: Test the UI of your Windows Applications with Appium
I could read in this article that it seems you are unit testing a single Fib() method using Data Driven Test.
Hence, would you please add an example of UI Application using Appium and N-Unit (preferred X-Unit)?
I don’t know what happened, it was showing a duplicate of my previous post. It should be ok, now.
Greetings to everyone.
I’d like to know if we can test delphi desktop applications with appium+winappdriver.
I don’t see why not – I can test the calculator, so any app could be tested
It would be so wonderful to find one single text showing all possibilities *after* calculator stuff.
How can I access list elements, how can I test the order within a grid…