• What a test automation framework should be about: Building Quality

    I have been working as a software tester for almost all my professional career, around 8 years. That might seem sweet to more experienced people out there, but I have always worked designing and coding test automation libraries and frameworks for different kinds of web and mobile applications, which gives me a pretty good insight in what to expect of them. In all this time though, I couldn’t stop but feeling that most of these frameworks, although doing a good job on helping me test, were missing something.

    For all the people for which the text above rings a bell and are as lost as I was, here’s the deal: don’t worry, the answer will come naturally to you. The more you test and work with software development, the more enlightened you will become about what’s going on. Now, if you’re in this situation I can safely guess you are writing test automation for yourself or for a test team… Right? So here’s the problem: you are doing it for the wrong people and for the wrong reasons.

    The root of the problem is simple: test automation is not meant to be used by us testers.

    “But Chema what are you saying? It allows us to take the time out of repetitive test sessions for more useful explorations!…”

    True, but sadly engineers often focus on that sole idea and forget what software development is truly about. We quickly get used to have testing as a separated step of the development thinking that the release cycles are under control that way, we even hire test engineers to build test automation suites, but in reality we’re making a snowball and throwing it downhill. We’re systematically, irreparably building technical debt, because there is never going to be enough time for a test team to make sure the application works well. We cannot build quality ourselves so there will always be compromises between time and quality.

    So how can we address this? Well, from my experience there are two approaches for building quality:

    1. Bottom —> Up: quality is implicit in the development process by creating CI pipelines that enforce different levels of testing: Unit, Component, Integration and System testing. Thanks to these pipelines we can ensure the logic of the code is robust, but the actual functionality is not verified until later in the process. However, mature development teams can resort to techniques such as TDD that maximise the testability of the code whilst minimising the need for future refactoring and ensure the functional criteria is met from the very first coding steps. This approach is considered a best practice when building new software (it’s the base of Agile development) and it’s usually handled by both programmers and testers.
    2. Top —> Down: quality is measured from a user perspective, creating tests that verify user scenarios and their acceptance criteria. The idea is that even if the logic is a mess, if the user gets all the functionality he needs without major issues then we fulfil our business needs. BDD techniques are used to build automatic functional system tests early on and have the system covered from major issues after changes are done. Later on some BDD tests are transformed into lower level tests to cover critical parts of the logic where a more white-boxed approach makes more sense. This quality approach is usually executed by testers in already existing applications with high amount of technical debt.

    So how do test automation frameworks go with these? Well you probably know the answer: bottom/up approaches use a combination of Unit Testing (done by programmers) and GUI frameworks (done by testers) to do the trick, whilst top/down approaches do only the latter. The problem is that neither of them are going to work in the long term because the persons dealing with them are different, and worst of all, its our -testers- fault.

    We all know that software development should be all about quality. We’ve heard it everywhere, read it everywhere these days. We know programmers should test all their code from all possible perspectives, and we testers should build our frameworks to help us create automated test cases. We take great pride when we put our suites in CI pipelines, yes, but… There is the problem! We build the frameworks for us. We’re always whining about programmers not testing, but we build the test frameworks for us. Now listen: what would happen if we built your test framework for the developers? Well it should not be that strange… those DevOps guys have been in our development teams for a while now and they’re always building tools for making development and testing easier, right? Now if we focused in creating test frameworks that developers can use, to apply our knowledge of testing into creating lower level frameworks in the programming languages they’re used to work and with the tools they need for their daily work… We could easily combine bottom/up and top/down approaches into a 360° one, where tests are created by different people, with different skills. A test framework that adapts to everyone: BDD tools for PO’s and non-technical testers and TDD tools for developers… Now we’re talking!

    So here’s what you can do: leave the comfort of your chair and your cool Selenium scripts and sit with the programmers to ask the hardest of questions: what do you need to test this properly? Help them help you and become a developer yourself! Because well… Isn’t this, my friends, the purpose of being a tester? To help programmers to build quality? Think and feel free to comment about it!

  • 0
  • 4
  • What a test automation framework should be about: Building Quality

    I have been working as a software tester for almost all my professional career, around 8 years. That might seem sweet to more experienced people out there, but I have always worked designing and coding test automation libraries and frameworks for different kinds of web and mobile applications, which gives me a pretty good insight in what to expect of them. In all this time though, I couldn’t stop but feeling that most of these frameworks, although doing a good job on helping me test, were missing something.

    For all the people for which the text above rings a bell and are as lost as I was, here’s the deal: don’t worry, the answer will come naturally to you. The more you test and work with software development, the more enlightened you will become about what’s going on. Now, if you’re in this situation I can safely guess you are writing test automation for yourself or for a test team… Right? So here’s the problem: you are doing it for the wrong people and for the wrong reasons.

    The root of the problem is simple: test automation is not meant to be used by us testers.

    “But Chema what are you saying? It allows us to take the time out of repetitive test sessions for more useful explorations!…”

    True, but sadly engineers often focus on that sole idea and forget what software development is truly about. We quickly get used to have testing as a separated step of the development thinking that the release cycles are under control that way, we even hire test engineers to build test automation suites, but in reality we’re making a snowball and throwing it downhill. We’re systematically, irreparably building technical debt, because there is never going to be enough time for a test team to make sure the application works well. We cannot build quality ourselves so there will always be compromises between time and quality.

    So how can we address this? Well, from my experience there are two approaches for building quality:

    1. Bottom —> Up: quality is implicit in the development process by creating CI pipelines that enforce different levels of testing: Unit, Component, Integration and System testing. Thanks to these pipelines we can ensure the logic of the code is robust, but the actual functionality is not verified until later in the process. However, mature development teams can resort to techniques such as TDD that maximise the testability of the code whilst minimising the need for future refactoring and ensure the functional criteria is met from the very first coding steps. This approach is considered a best practice when building new software (it’s the base of Agile development) and it’s usually handled by both programmers and testers.
    2. Top —> Down: quality is measured from a user perspective, creating tests that verify user scenarios and their acceptance criteria. The idea is that even if the logic is a mess, if the user gets all the functionality he needs without major issues then we fulfil our business needs. BDD techniques are used to build automatic functional system tests early on and have the system covered from major issues after changes are done. Later on some BDD tests are transformed into lower level tests to cover critical parts of the logic where a more white-boxed approach makes more sense. This quality approach is usually executed by testers in already existing applications with high amount of technical debt.

    So how do test automation frameworks go with these? Well you probably know the answer: bottom/up approaches use a combination of Unit Testing (done by programmers) and GUI frameworks (done by testers) to do the trick, whilst top/down approaches do only the latter. The problem is that neither of them are going to work in the long term because the persons dealing with them are different, and worst of all, its our -testers- fault.

    We all know that software development should be all about quality. We’ve heard it everywhere, read it everywhere these days. We know programmers should test all their code from all possible perspectives, and we testers should build our frameworks to help us create automated test cases. We take great pride when we put our suites in CI pipelines, yes, but… There is the problem! We build the frameworks for us. We’re always whining about programmers not testing, but we build the test frameworks for us. Now listen: what would happen if we built your test framework for the developers? Well it should not be that strange… those DevOps guys have been in our development teams for a while now and they’re always building tools for making development and testing easier, right? Now if we focused in creating test frameworks that developers can use, to apply our knowledge of testing into creating lower level frameworks in the programming languages they’re used to work and with the tools they need for their daily work… We could easily combine bottom/up and top/down approaches into a 360° one, where tests are created by different people, with different skills. A test framework that adapts to everyone: BDD tools for PO’s and non-technical testers and TDD tools for developers… Now we’re talking!

    So here’s what you can do: leave the comfort of your chair and your cool Selenium scripts and sit with the programmers to ask the hardest of questions: what do you need to test this properly? Help them help you and become a developer yourself! Because well… Isn’t this, my friends, the purpose of being a tester? To help programmers to build quality? Think and feel free to comment about it!

  • 0
  • 4

What a test automation framework should be about: Building Quality