Testing Frameworks for Python

Something that is untested, is broken.

This is our first article on Software deployment in Python about Testing Frameworks. Hope you enjoy this series and make sure to check out our other articles

Cowboy Coding:

Cowboy coding is software development where programmers have autonomy over the development process. This includes control of the project’s schedule, languages, algorithms, tools, frameworks and coding style.

A cowboy coder can be a lone developer or part of a group of developers working with minimal process or discipline. Usually it occurs when there is little participation by business users, or fanned by management that controls only non-development aspects of the project, such as the broad targets, timelines, scope, and visuals (the “what”, but not the “how”).

So the cowboy approach to coding typically focuses on quick fixes and getting a working product into production as quickly as possible. They may also edit code on a live production server, skipping over the formal processes for quality assurance testing, regression testing and documentation required by continuous integration and other Agile software development methodologies. Instead of producing lean, well-written code, cowboy code often has errors that cause failures upon deployment or make it difficult to maintain over time. Integrating the various components of the code may also be a challenge since with cowboy coding there are no agreed-upon best practices to provide continuity.

Test Driven Development

What is TDD:

TDD can be defined as a programming practice that instructs developers to write new code only if an automated test has failed. This avoids duplication of code. TDD means “Test Driven Development”. The primary goal of TDD is to make the code clearer, simple and bug-free.

Test-Driven Development starts with designing and developing tests for every small functionality of an application. In TDD approach, first, the test is developed which specifies and validates what the code will do.

Test-Driven development is a process of developing and running automated test before actual development of the application. Hence, TDD sometimes also called as Test First Development.

TDD Cycle

  1. Write a test
  2. Make it run.
  3. Change the code to make it right i.e. Refactor.
  4. Repeat process.

Behavioral Driven Development (BDD)

What is BDD

Behavioral Driven Development (BDD) is a software development approach that has evolved from TDD (Test Driven Development). It differs by being written in a shared language, which improves communication between tech and non-tech teams and stakeholders. In both development approaches, tests are written ahead of the code, but in BDD, tests are more user-focused and based on the system’s behavior.


TDD works satisfactorily, as long as the business owner is familiar with the unit test framework being used and their technical skills are strong enough, which is not always the case. In these circumstances, BDD has the advantage because the test cases can be written in a common language used by the stakeholders such as, for example, English. This access to clearer, low-jargon communication is probably the biggest advantage to using BDD, making it possible for collaboration between the technical and non-technical teams to run with improved efficiency.

Characteristics of BDD

As described above, the advantage to BDD test cases being written in a common language is that details of how the application behaves can be easily understood by all. For example, test cases can be written using real-time examples of the actual requirements, to explain the behavior of the system.

The ‘Given-When-Then’ formula BDD example

This is the proposed template (aka Gherkin Language) for writing BDD test cases for a user story, which can be defined as:

Given a certain scenario
When an action takes place
Then this should be the outcome.

A practical example would be:

Given the User has not entered any data on the form
When they click the submit button
Then proper validation messages should be show.

Some Benefits to using BDD

If you plan to implement BDD, here are a few points that will benefit the software team.

  1. You are no longer defining ‘test’, but are defining ‘behavior’.
  2. Better communication between developers, testers and product owners.
  3. Because BDD is explained using simple language, the learning curve will be much shorter.
  4. Being non-technical in nature, it can reach a wider audience.
  5. The behavioral approach defines acceptance criteria prior to development.

Disadvantages of BDD

Even the best development approaches can have pitfalls and BDD is no exception. Some of them are:

  1. To work in BDD, prior experience of TDD is required.
  2. BDD is incompatible with the waterfall approach.
  3. If the requirements are not properly specified, BDD may not be effective.
  4. Testers using BDD need to have sufficient technical skills.

Most Common Testing Frameworks for Python

Robot Framework

Used mostly for development that is acceptance test-driven as well as for acceptance testing.


  • No need a programming language so non-technical person can be involved
  • Keyword-driven-test approach
  • Easy to learn, Test data syntax can be used easily
  • Highly-extensible
  • Parallel tests via a Selenium
  • Open source


  • Skip test by Tags only


A simple example to login with some credentials:

*** Settings ***
Documentation     A test suite with a single test for valid login.
...               This test has a workflow that is created using keywords in
...               the imported resource file.
Resource          resource.txt

*** Test Cases ***
Valid Login
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser


Pytest is a testing framework which allows us to write test codes using python. You can write code to test anything like database , API, even UI if you want. But pytest is mainly being used in industry to write tests for APIs.


  • Easy to start with
  • Open source
  • Many plugins: pytest-bdd, pytest-cov, etc and hypothesis support.
  • Parallel tests
  • Skip tests easily
  • Compact test suites
  • Minimal boilerplate


  • Compatibility issues with other testing frameworks: due to the fact that pytest uses its own routines to write tests


The purpose of test fixtures is to provide a fixed baseline upon which tests can reliably and repeatedly execute.

Test functions can receive fixture objects by naming them as an input argument. For each argument name, a fixture function with that name provides the fixture object. Fixture functions are registered by marking them with @pytest.fixture. Let’s look at a simple self-contained test module containing a fixture and a test function using it:

def smtp_connection():
    import smtplib

    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250

Nose Tests

Nose extends unittest to make testing easier.

Nose has been in maintenance mode for the past several years and will likely cease without a new person/team to take over maintainership. The successor to nose, based on unittest2, is Nose2.


  • Wraps unittest
  • Minimal Boilerplate
  • Many plugins


  • will likely cease


It’s so simple to write tests. Here want to make sure our function multiply works fine:

from unnecessary_math import multiply

def test_numbers_3_4():
     assert multiply(3,4) == 12

def test_strings_a_3():
     assert multiply('a',3) == 'aaa'  


Behavior-driven development (or BDD) is an agile software development technique that encourages collaboration between developers, QA and non-technical or business participants in a software project.

Behave uses tests written in a natural language style, backed up by Python code.


  • Supports the Gherkin language.
  • Flask & Django integration
  • PyCharm professional edition support
  • Environmental functions and fixtures make setup and cleanup easy.


  • A standalone framework
  • Sharing steps between feature files can be a bit of a hassle.


Behave operates on directories containing:

  1. feature files written by your Business Analyst / Sponsor / whoever with your behaviour scenarios in it, and
  2. a “steps” directory with Python step implementations for the scenarios.

A feature file has a natural language format describing a feature or part of a feature with representative examples of expected outcomes. They’re plain-text (encoded in UTF-8) and look something like:

Feature: some feature

  Scenario: Search for an account
    Given I search for a valid account
    Then I will see the account details

Step code implementing the two steps here might look like (using selenium webdriver and some other helpers):

@given('I search for a valid account')
def step_impl(context):
    form = get_element(context.browser, tag='form')
    get_element(form, name="msisdn").send_keys('61415551234')

@then('I will see the account details')
def step_impl(context):
    elements = find_elements(context.browser, id='no-account')
    eq_(elements, [], 'account not found')
    h = get_element(context.browser, id='account-head')
    ok_(h.text.startswith("Account 61415551234"),
        'Heading %r has wrong text' % h.text)

PyTest-BDD Vs Behave

When it comes to behave and pytest, you may think how to integrate them together. But because behave is a standalone framework, it’s impossible to integrate them for now. So the community made a new plugin for pytest to support BDD.


Both of them uses Gherkin langauge.

Unit Tests:

PyTest-BDD is compatible with pytest whereas behave is a standalone framework, so if you want to run unittests without tdd, you have to make use of other framework.

PyCharm Support:

Both pytest-bdd and behave supports pycharm professional edition.


Pytest offers fixtures and because pytest-bdd is a plugin for pytest, it makes use of them to make it easier to manage context between steps using fixtures.

In short:

PyTest-BDD Behave
Language Gherkin Gherkin
Unit Tests Compatible with pytest
PyCharm support Professional Edition Professional Edition
Fixtures Yes, uses them to manage context between steps
What is it? A plugin for pytest A standalone framework

To summarize:

The importance of TDD cannot be overemphasized. TDD means less bugs, higher quality software, and a razor focus, but it can slow development down and the test suites can be tedious to manage and maintain. All in all, it is a recommended approach as the Pros far outweighs the Cons.

BDD makes it easier to write tests and it improves communication between tech and non-tech teams and stakeholders.

Python has many testing frameworks from pytest and nose for TDD to pytest-bdd and behave for BDD.


Leave a Reply

Your email address will not be published. Required fields are marked *