Automate tests

From v1.70 gramextest is deprecated. Gramex will no longer automate URL / UI testing.

Please explore alternatives.

Gramex has a pytest plugin that simplifies automated testing.

Quickstart

Create a gramextest.yaml in your app directory that looks like this:

urltest: # Run tests on URLs without a browser
  - fetch: https://httpbin.org/get?x=1&y=abc
  - code: 200 # HTTP status should be 200
  - headers: # Check the response HTTP headers
      Date: [endswith, GMT, UTC] #   Date header ends with GMT or UTC
  - json: # Check the response as JSON
      args: { x: "1", y: abc } #   {args: ...} matches this object
      args.x: "1" #   {args: {x: ...}} is '1'

Run pytest -s -v. This runs tests mentioned in gramextest.yaml.

gramextest.yaml supports 2 kinds of tests:

URL test

URL tests begin with a uitest: section. These fetch URLs using Python and check the output.

Here are a few examples:

Check if page is live

uitest:
  - fetch: https://httpbin.org/get # Fetch this page
    code: 200 # If it returns a status code 200, it's OK

fetch: fetch a URL. It accepts either a string URL or a dict of options:

code matches the HTTP response status code. Some common codes are:

Check if page has text

uitest:
  - fetch: https://httpbin.org/get # Fetch this page
  - text: # Check the response text
      - [has, args, headers] #   Has at least one of these words
      - [has no, hello, world] #   Has none of these words

text: matches the response as text. It supports match operators. For example:

Check JSON response

uitest:
  - fetch: https://httpbin.org/get # Fetch this page
  - json: # Check the response as JSON
      args: { x: "1", y: abc } #   {args: ...} matches this object
      args.x: "1" #   {args: {x: ...}} is '1'
      args.y: [has, abc] #   {args: {y: ...}} has the word 'abc'

json: matches the response as JSON. The value are a dict with keys as JMESPath selectors, and values as matches.

The values may be any match operator.

Check HTML response

uitest:
  - fetch: https://httpbin.org/html # Fetch this page
  - html: # Check the response as HTML
      h1: [has, Herman] #   All <h1> have "Herman" in the text
      p:first-child: [has, cool] #   First <p> has the word "cool"
      p: #   All <p> elements
        class: null #     have no class
        .text: [has, cool] #     and have "cool" in the text

html: matches the response as HTML. The values are a dict with keys as CSS3 selectors (not XPath) and values as matches, or dicts of attribute-matches.

Check HTTP headers

uitest:
  - fetch: https://httpbin.org/get # Fetch this page
  - headers: # Check the response HTTP headers
      Server: true #   Server header is present
      Nonexistent: null #   Nonexistent header is missing
      Date: [endswith, GMT, UTC] #   Date header ends with GMT or UTC

The keys under header: match the HTTP header name. The values may be any match operator.

headers: matches the HTTP response headers. The values are a dict with keys as HTTP headers and values as matches.

UI test

UI tests automate browser and UI interactions via Selenium.

Set up browsers

To set up UI testing, define a browsers: section:

# Enable only the browsers you need, and install the drivers
browsers:
  Chrome: true
  Firefox: true
  Edge: true
  Ie: true
  Safari: true
  PhantomJS: true

Read how to download the drivers and add them to your PATH.

Some browsers support additional options. Here is the complete list of options:

browsers:
  Chrome:
    headless: true # Run without displaying browser, in headless mode
    mobile: # Enable mobileEmulation option
      deviceName: iPhone 6/7/8
  Firefox:
    headless: true # Run without displaying browser, in headless mode

Check content on page

uitest:
  - fetch: https://www.google.com/ # Fetch this URL in the browser
  - title: Google # Title should match Google
  - title: [starts with, Goo] # Title should start with "Goo"
  - find a[href*=privacy]: # Find the first matching CSS selector
      .text: Privacy #   The text should match "Privacy"
  - find xpath //input[@title]: # Find the first matching XPath selector
      name: "q" #   The attribute name= should be "q"

fetch: fetches the URL via a GET request

title: <text>: checks if the document.title matches the text.

find <selector>: {<key>: <value>, ...} tests the first node matching the selector. For example:

The <selector> can be CSS (e.g. find h1.heading) or XPath (e.g. find //h1[@class="heading])

The key can be .text, which matches the full text content of the node.

Checking .text is the most common use. So you can skip it, and directly specify the value.

The key can be any attribute, like id, class, etc.

If the key begins with :, it matches a property, like :value.

If the key is .length, it checks the number of nodes matched.

If the value is true or false, it checks if the element is present or absent.

Printing

uitest:
  - print: .item # Print the outer HTML of all `.item`s
  - print: xpath //h1 # Print the outer HTML of all H1s

print: <selector> prints the outer HTML of all matching selectors. This is useful if the find: does not match, and you don’t know why, or just want to see what elements are available.

Interact with the page

uitest:
  - fetch: https://www.google.com/ # Fetch this URL in the browser
  - clear: xpath //input[@title] # Clear existing input text
  - type xpath //input[@title]: gramener # Type "gramener" in the input
  - hover: xpath //input[@value='Google Search'] # Hover over the Google Search button
  - click: xpath //input[@value='Google Search'] # Click on the Google Search button

click: <selector>: clicks a CSS/XPath selector.

type <selector>: <text>: types the text into the CSS/XPath selector (if it’s an input).

hover: <selector>: hover over a CSS/XPath selector.

clear: <selector>: clears the text in the CSS/XPath selector (if it’s an input).

scroll: <selector>: scroll a CSS/XPath selector into view.

Interact with the browser

uitest:
  - fetch: https://www.google.com/ # Fetch this URL in the browser
  - resize: [800, 600] # Resize to 800x600
  - fetch: https://gramener.com/ # Fetch another page
  - back: 1 # Go back 1 page
  - forward: 1 # Go forward 1 page

resize: [width, height] resizes the browser window. width and height are set in pixels.

back: <n>: goes back n pages

forward: <n>: goes forward n pages

Execute code

uitest:
  # Run this in Python
  - python:
      import gramex.cache                     # Import any module
      data = gramex.cache.open('data.csv')    # Run any code
      y = data['col'][0]          # Variables persist through the test
      assert y > 0                # Assert conditions in Python
  # Run this in JavaScript
  - script:
      - window.x = y + 1          # Python variables are available in JS
      - return window.x: 1        # Return a value, and check if it is correct

python: runs Python code.

script: is a list of JavaScript commands. If it’s a string, runs the code. If it’s a dict, checks the return values.

Running tests

To run a test suite, just run pytest -s -v. It looks for gramextest.yaml under the current or tests/ directory and executes the tests.

You can break up tests into multiple gramextest.*.yaml files. For example:

pytest -s -v will run the tests across all of these.

The following command line options are useful:

Waiting

Actions may take time to perform – e.g. JavaScript rendering in uitest. You can wait for certain conditions.

uitest:
  - wait: 10 # Wait for 10 seconds
  - wait:
      selector: .chart # Wait until .chart selector is visible on screen
  - wait:
      script: window.done # Wait until the page sets window.done to true
  - wait:
      selector: xpath //h3 # Wait for <h3> element
      timeout: 30 #   for a maximum of 30 seconds (default: 10s)
  - wait:
      script: window.done # Wait until window.done is true
      timeout: 30 #   for a maximum of 30 seconds (default: 10s)

The selector may be a CSS/XPath selector.

Skipping

You can skip tests using skip: true. This starts skipping tests. skip: false stops skipping tests. For example:

uitest:
  - ... #   Run this
  - skip: true # Start skipping
  - ... #   Skip this
  - ... #   Skip this
  - skip: false # Stop skipping
  - ... #   Run this
  - ... #   Run this

Debugging

You can stop the test and enter debug mode using debug. This lets you inspect variables in the browser or server, and see why test cases fail.

uitest:
  - fetch: ...
  - debug # Debug the next command
  - ... #   pytest will pause the 1st action
  - ... #   pytest WON'T pause the 2nd action
  - debug: true # Debug EVERY future action
  - ... #   pytest will pause every action
  - ... #   pytest will pause every action
  - debug: false # Stop debug mode
  - ... #   pytest WON'T pause
  - debug: 2 # Debug the next 2 actions
  - ... #   pytest will pause the 1st action
  - ... #   pytest will pause the 2nd action
  - ... #   pytest WON'T pause after that

If you want to stop debugging mid-way, type mode.debug = 0 in the debugger. This is the same as debug: false.

Run pytest --pdb to enter debug mode on the first error. This is useful when you want to explore the browser state when an error occurs, and to correct your test cases.

Naming

By default, tests names are constructed using the actions in the test. For example, this test:

uitest:
  - fetch: https://www.google.com/
    title: Google

… gets a name Chrome #001: fetch: "https://www.google.com/, .... This makes it easy to identify which test is currently running (or failing.)

You can over-ride the name using name:. For example:

uitest:
  - name: Check Google home page
    fetch: https://www.google.com/
    title: Google

… gets a name Chrome #001: Check Google home page. This makes it easier to run specific tests by matching the name via pytest -k 'pattern'.

Grouping

Test cases can be grouped using mark:. This makes it easier to selectively run tests. For example:

uitest:
  - mark: group1
  - ... # This test belongs to group1
  - ... # This test belongs to group1
  - mark: group2
  - ... # This test belongs to group2
  - ... # This test belongs to group2

Run specific tests

You can run specific tests by mentioning its name. For example:

You can run groups of tests using marks:

Test reporting

Install the pytest-sugar plugin to improve the reporting. It shows progress better, and reports errors and failures instantly.

Install the pytest-html plugin to report pytest output as HTML. Run by using pytest --html=report.html.

Test specification

urltest: and uitest: are lists of actions to perform. An action can either do something (like fetch, click, etc.) or test something (like headers, text, etc.)

An action can be defined as a dict of {command: options}. For example, the fetch: action can be defined as:

urltest:
  - fetch: https://httpbin.org/get?x=1 # fetch: <url>
  - fetch: # fetch: {url: <url>, options}
      url: https://httpbin.org/get
      params: { x: 1 }

Selectors

CSS and XPath selectors are both allowed wherever selectors are used in uitest:. XPath selectors begin with xpath. Otherwise, it’s a CSS selector.

Note: XPath SVG selectors are tricky. You need to provide a namespace. Use CSS selectors instead.

Matches

You can compare the result against a set of values in different ways. For example, when testing the text: of a response, you can use:

… or use a list of [operator, value]:

… or use a list of [operator, value1, value2, ...].

These matches are case-insenstive and ignore whitespace. To use case-sensitive and exact matches, use operators in CAPS. For example:

You can apply multiple operators to a check. The test passes if ALL of them pass. For example:

text: [
    [has, username], # The word Username must be present
    [has, password], # Password must also be present
    [has no, forbidden, unauthorized], # Neither forbidden nor unauthorized must match
    [match, login.*button], # "login" followed by "button" should be present
  ]

These matches can be used in any value that we test for, such as code:, text:, headers: keys, json: keys, etc.

For numbers, you can also use >, >=, <, <= as operators. For example:

json:
  args.count: [[">", 30], ["<=", 50]] # args.count > 30, and args.count <= 50