Automated UI testing

This post is based on my replay in Dynamics Community forum, therefore if you read it there, you can safely move on.

When I hear “automated testing” in Dynamics AX world, people almost always mean replicating what a human tester does – opening a form, clicking around and so on. It makes some sense, because that’s what end users do as well.

On the other hand, it’s a poor choice for automation, because:

  • It’s extremely fragile. A simple change in user interface (even caused by things like security) can break all your tests. If it uses image recognition, the problem is insanely huge – I see people struggling with tests if they change screen resolution, for example. You want tests fail if your application is broken, not because the resolution has changed.
  • It lacks isolation. If something fails, you know that your have a problem somewhere in a business process, calling several thousand methods. Or maybe you have a problem with data. You simply don’t know. You’ll have to perform expensive debugging before you finally isolate the bug.
    That’s why automated tests should test small isolated units.
  • You must use AX database, therefore you have to maintain test data. Because database is shared for all tests, you can get into all the problems associated with shared test fixture (such as that one test can influence other tests, or you have to expensively cleanup the state before/after each test).
  • It’s the slowest type of test you can imagine.

From my opinion, this is a kind of testing that you should consider when you already have all easier, faster and more reliable tests in place, you have experience with test automation and you want to get rid of the last manually intervention in integration and UI tests. If this is your first attempt for automated testing, it will very likely fail (you won’t achieve it at all or maintenance cost will negate all gains) and people often give up automated testing completely because of this experience. But the problem was that they knew nothing about automatic testing and therefore they made wrong decisions.

I would recommend two things:

  • Use unit tests as the primary mean for verification of implementation. If you don’t test code at the lowest level and get poor implementation of methods, you’ll never build a high-quality product from them. Don’t worry about code coverage too much, but ensure yourself that all complicated logic you write is covered by unit tests.
  • Use Microsoft Test Manager (or a similar tool) to plan and design your manual tests. Designing tests and expected results is the first and mandatory step towards automated UI tests, because scripts (unlike human testers) can’t just figure it out by themselves. It’s surprising how often people want to automated tests they’ve never defined.
    Then record these tests and use fast forward to automate some steps. You’ll see that some actions can’t be replayed, therefore many tests can’t be fully automated in this way. That’s the moment when you can start thinking of converting test recordings to coded UI tests and implement logic needed for full automation. Nevertheless even if you never do that, you’ll already have quite a lot of automation and tools to highly improve productivity of human testers.

6 Comments

  1. Hi Martin,

    I asked the question on the Dynamics Community forum. I am glad you found my question so thought provoking! 🙂

    Since I asked the question in May, I have thought a lot about your answer. Let me just say, I am with you on this one… but:

    • There exists a great need for regression testing for new releases/large customizations. Since most developers are not interested in testing (at least that’s my experience), the need is mostly experience by non-technical consultants, functional testers and management.

    • They (the non-technical consultants, testers and management) want a non-technical solution that they can understand, analyze and market as part of our QA strategy to non-technical prospects.

    • Telling them my unit test ran successfully does not mean anything. They want to know if it is still possible to create orders and invoices with all items and for all customers.

    • This regression testing solution must be maintainable and executable by functional consultants and testers with little technical background, since they are the ones most invested in testing. (This is the kind of task you only ask a developer to do if you want them to start looking for another job. 😉 )

    Since we’ve heard that a partner of ours uses an automated tool for their functional tests very successfully, we are planning a field trip to see how they do it.

    Microsoft Test Manager

    In the meantime and based on your recommendation, I have suggested trying Microsoft Test Manager. So far it’s not been strait forward to connect Test Manager to Dynamics AX. I have setup Visual Studio online, upgraded to full version (the free version does not include testing) and installed the correct version of Visual Studio professional that includes the Test Manager. I still cannot get Test Manager linked to one of our R2 environments. It continuously crashes without giving any indication why. Granted, I have not spent weeks on trying to get it to work. It’s been a busy, understaffed summer with more pressing issues. Hopefully I’ll get around to tinkling with it by the end of this week; I have to since my Visual Studio online full trail only has 20 days left!

    I read your intro to VSO and am looking forward to future posts on the topic. I would also love to hear from anyone who successfully used Microsoft Test Manager with Dynamics AX 2012 (R2 or R3). It has to be possible, but have anyone actually done it?

    Unit Tests
    Lately I have also delved into unit tests, reading the Microsoft material (Inside Dynamics AX and the White Paper: Testing best practices for Microsoft Dynamics AX) as well as more general sources like Code Complete. I have also looked at Jan Kjeldsen’s Suduko Unit tests on Axaptapedia.

    I am very enthusiastic about writing unit tests. Still, I find it hard to wrap my head around where to get started and how to practically implement scripts for the customizations I write. I have never written unit tests, even when I worked in other languages where unit tests are more common. My senior colleges, while being AX gurus, also have little or no experience with unit tests. Unless there is a change in the way the AX development community view unit tests, I don’t think I company is likely to develop unit tests in the near future.

    For now, I’m looking into Test Manger and trying to get it to play nice with Dynamics AX…

    Regards,
    Tina

  2. Tina, I’m curious if you ever got any further in the whole testing regime. I’m looking into this for our company & don’t want to reinvent the wheel if it’s not necessary.

  3. Hi Martin

    Your approach makes a lot of sense, you don’t want to be spending 100 hours to automate something that only saves you 10 hours of manual testing.

    We have an automation tool called Rapise (http://www.inflectra.com/Rapise/Technologies/Dynamics.aspx) and we constantly have to educate our new customers / prospects about what is realistic in terms of automated testing. You should automate the manual, repetitive stuff that is boring for a human and is stable to test automatically.

    Regards
    Adam

  4. Hi Adam or Martin,

    Why didn’t you go for Visual studio coded UI and instead why did you go for Rapise?
    Is it for dynamics AX , if yes, which version?
    We are also planning to automate testing on our Dynamics AX 2012 R3 platform , which tool is best , do we have any analysis on same , some client references for same ?

    Thanks.

    regards,
    Ritesh Jain

    • Hi Ritesh, I did mention coded UI tests (“That’s the moment when you can start thinking of converting test recordings to coded UI tests and implement logic needed for full automation.”).
      Adam was promoting a product of his company; it has nothing to do with me.

Leave a Reply

Your email address will not be published.