The classic dilemma for testers – Robot or pytest

July 26, 2018 3 26
Share
The classic dilemma for testers – Robot or pytest

For any organization, customer’s confidence is of the highest priority and a lot of it is dependent on reliability which is mostly dependent on the quality assured by the product and service you are providing. This has increased the importance of QA, Testing teams and the framework you are using for testing. Choosing the right test framework for a project is a critical decision. As of today, there are many open source as well as proprietary test automation frameworks in the market which you can consider depending on the project’s requirements. Through a series of blogs, with this being the first part, I will attempt to explore the criterion for selecting such frameworks and evaluate some of the best ones.

With Python’s increasing advent in the data science and machine learning domains, it is often confusing even for experienced professionals to pre-decide on their approach to testing for agile projects. With Robot’s keyword-driven approach which makes it user-friendly and pytest being easily maintainable for continuous changes, it is but natural that these frameworks continue to gain the most popularity.

The Robot framework can be thought of as a test automation template which can be used to automate scenarios with a keyword-driven approach. The keyword-driven approach simplifies the task of writing the test cases. The user just needs to import the specific libraries (default as well as custom) for using the inbuilt keywords. This makes the code extremely easy to read and understand as well. It can be used to automate complex scenarios and at the same time offers the flexibility to use languages like Java and Python.

The pytest framework, however, is specifically designed to simplify and enhance the unit testing experience. pytest proves extremely handy for unit testing scenarios wherein the test code block remains almost similar while only the inputs vary. Built-in tags like ‘parameterize’, ‘fixtures’ help in avoiding the repetitive code and make the code more maintainable but only at the cost of reduced readability. Due to their inherent capabilities, Robot proves itself more useful while testing the entire stack whereas pytest is limited only to unit testing. Let’s try and get a deeper perspective of both of them through a set of examples:

Usage:

Being a data-driven test automation framework, Robot needs the user to provide the test data through a config file. A simple example using this approach for logging into an application has been shared below:

*** Settings ***
Test Template         Login with invalid credentials should fail

*** Test Cases ***    USERNAME             PASSWORD
Invalid Username      invalid              ${VALID PASSWORD}
Invalid Password      ${VALID USERNAME}    invalid
Invalid Both          invalid              invalid
Empty Username        ${EMPTY}             ${VALID PASSWORD}
Empty Password        ${VALID USERNAME}    ${EMPTY}
Empty Both            ${EMPTY}             ${EMPTY}

*** Keywords ***
Login with invalid credentials should fail
    [Arguments]    ${username}    ${password}
    Input Username    ${username}
    Input Password    ${password}
    Submit Credentials
    Error Page Should Be Open

In the above example, “Input Username, Input Password, Submit Credentials, Error page Should be Open” are nothing but the keywords which indeed refer to a python library function. The Keywords are composable. This means the user can define new keywords(custom) which use existing keywords. This way, the user can abstract details of testing to something that makes immediate sense. For example, we don’t need to know what exactly the “Push buttons” Credential actually does, unless we want to. Test cases are therefore clear and readable with just the right level of abstraction to convey the intent of the test.

On the other hand, pytest uses fixtures/parameterization to build functions/test cases. The parameterization allows users to define a function that takes multiple arguments. This proves advantageous with each of it constituting a single test case as opposed to Robot framework where the user needs to write a separate test case every time even though just the inputs are different. A simple example of this is:

Example:

@pytest.mark.parametrize("username, password, expected_result", [
(“invalid_username”,”valid_password”, “Login failed”),
(“valid_username",”invalid_password”, “Login failed”),
("valid_username", “”, “Login failed”),
])
def test_eval(username, password, expected_result):
   Login_result = login(username,password)
   assert  Login_result == expected_result

The above example shows how a single test_eval can execute 3 test cases using parameterize tag. Similar to parameterize, we can also pass arguments using fixtures. Fixtures are very handy when you want to artificially separate out the test cases without having to move any data from one file to another by defining the scope of execution.

Example:

@pytest.fixture
def test(username,password):
  a = username
  b = password
  obj = cal(a,b)
  return obj
 
def test_wrong_username(test):
  response = login(test.a,test.b)
  assert response == “Login failed” 

def test_wrong_password(test):
  response = login(test.a,test.b)
  assert response == “Login failed” 

def test_empty_password(test):
   response = login(test.a,test.b)
   assert response == “Login failed” 

The above example shows how a single fixture is shared for fetching the data inputs to perform login operations with a scope of the fixture being just the scenario under execution.

Reporting:

The Robot framework gives extremely presentable test reports while offering the ability to the user to organize the tests with a very simple and robust tagging system. Robot has a test runner that can provide output not only in XML but also in multiple other formats using its listener interface.

robo test report demo

On the other hand, pytest’s built-in reporting mechanism does not provide a full-fledged report and the user may have to get some external plugins to get a more detailed report.

robo test report

Some more fundamental differences between the Robot and pytest automation framework are:

Modular Design:

By using modular fixtures, it is easy to modify or add assertions because test cases are reduced in the pytest functions as compared to Robot framework. By taking advantage of pytest.mark.parametrize decorator, adding an extra scenario is just a matter of adding the additional input tuple.

Custom Asserts:

pytest allows the users to use the standard python assert for verifying expectations and values in Python tests or the users can implement their own assertion using:

pytest_assertrepr_compare hook.

Execution Time:

With pytest, execution time is reduced drastically (35 to 40%) as compared to Robot Framework test-suite.

Plugins:

Another advantage of switching to pytest is the no. of readily available plugins and a provision to design your own plugins. In Robot, you won’t find the plugins readily available. The best example to illustrate this is Cookiecutter.
Simply install Cookiecutter and generate a new pytest plugin project:

$ pip install cookiecutter
$ cookiecutter https://github.com/pytest-dev/cookiecutter-pytest-plugin

… and you are done. The task of Cookiecutter is to simply create a task project from task template.

Test data organization/maintainability:

Robot framework is an efficient way to organize the test cases. It is very handy with the manner in which we can write the test cases. It increases code readability. On the other hand, pytest is an effective way of clubbing repetitive test cases into a single test function and passing the variables using fixtures/parameterization. If we compare code maintainability, pytest is better than Robot, since it eliminates the compulsion to use config file for each suite. All the configurations/parameters are present in the pytest file itself as part of the main code and thus user overhead to maintain clumps of config files is nullified.

Both these frameworks are open source as of today and offer great flexibility for test automation depending on the focal points of the project’s testing effort. So, in case you are still doubtful and undecided on either of them, you may want to wait it out till the next blog in this series is out wherein I will focus on actually working with these frameworks.

Tags
Leave Comment

Your email address will not be published.

  1. Thanks for your interesting post!
    Couple of comments:

    > whereas pytest is limited only to unit testing
    I think pytest can be used for integration/E2E tests as well.
    In Python you can basically do anything that is done in Robot Framework (which is itself built in Python).
    It is just a matter of picking the appropriate Python library (like subprocess to start a SUT on the command line)

    > This proves advantageous with each of it constituting a single test case as opposed to
    > Robot framework where the user needs to write a separate test case every time even
    > though just the inputs are different
    You can use Test Template in Robot Framework to build Data-driven tests so you don’t have to create a full test case for each data set:
    http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#data-driven-style

    • Yes Laurent, we can definitely leverage capabilities of pytest with additional plugins for other testings as well. The aim of the blog was a close comparison of pytest & Robot in their vanilla forms without plugins. Combine plugins like pytest-bdd and we have a much more flexible framework for testing. And for Robot, yes you can build data-driven tests but then you’ll need to write the test-cases in the robot file. However, with pytest, each data input to a test function is automatically treated as a test-case and you don’t need to explicitly write them out.

Stay up to date with the latest and the best from the world of automation!