Overview:

Running ​automated regression on a daily basis as part of the daily build is inevitable nowadays! But it is very painful to maintain hundreds of automated tests & remote parallel test execution results!

In this article, lets see how we can use TestArchiver + PostgreSQL + Grafana to archive all our robot framework test results and create an advanced robot framework test execution dashboard.

In future article we will use the archived test results for our AI driven analysis/predictions.

I already have been using MySQL + Grafana for storing my robot framework test execution results. You can find more details in RF + MySQL + Grafana

Key Benefits:

  • Unified storage for the test run results
  • Archiving your test results to PostgreSQL database
  • Meeting the test data challenge for AI and ML

Metrics Covered:

  • Execution Status – Latest
  • Total Executions – Over Time Period
  • Average Pass and Failure Percentage
  • Top 5 Failed Test Cases
  • All Execution Status Over Time Period
  • Average Execution Time
  • Execution Trend Graph

Grafana & PostgreSQL Integration:

PostgreSQL Setup:

  • Download PostgreSQL – Version > 9.5
  • Use the following link to install PostgreSQL DB
  • Make sure PostgreSQL server is up and running
  • Connect to PostgreSQL database server using PgAdmin
  • Create a new database schema robot_results using PgAdmin

Grafana Setup:

  • Update Grafana data source to point to the PostgreSQL instance
    • Click on Configuration Menu in the Left Panel and Select Datasource
    • Click on Add data source and select PostgreSQL database
  • Enter PostgreSQL server details as below and click on Save & Test. The connection should be successful.

Once integrating PostgreSQL and Grafana is completed, then the next step is to send the robot test results data to PostgreSQL DB and enable the default Grafana dashboard.

Robot Framework & PostgreSQL Integration:

Well at this point, I assume that you have Python 3+, Robot Framework installed already and you are running your automated test cases.

Additionally, if you have few robot test results output.xml files then it would be very helpful for you to follow the below details.

Python Library Setup:

We will be using TestArchiver library in order to parse the robot framework output.xml to PostgreSQL database. Download the entire project using the following link TestArchiver-Git

TestArchiver:

TestArchiver is a tool for archiving your test results to a SQL/PostgreSQL database. This way the future `Robot Framework` related tools and plugins will have a unified storage for the test run results.

I would like to say Thank you for SALabs team for the effort to build such a useful library to support different testing frameworks. Please refer the below link if you would like to learn more.. TestArchiver-Git

Create DB Config File:

Sample configuration details for your reference. Update the values as per your PostgreSQL setup. Please save the file with name test_config.json under TestArchiver-master folder.

Adding Application Name as Metadata:

As each execution is for an application or project, we will add the application/project details with –metadata option when we trigger the robot test execution. Please the below command or include the application in using -M with the existing robot command line option.

Push Test Results to PostgreSQL DB:

Now we are all set to push the report to PostgreSQL DB. Please use the below command to parse the details in output.xml and push it to DB.

Additionally, we have used 2 more parameters to distinguish the test execution while parsing the output.xml. In our pre-defined dashboard we have used both the values as drop down options, in case if you would like to ignore these parameters then the dashboard queries needs to be customized accordingly.

–team TEAM Team name for the test series (Eg: QA_Team)

–series SERIES Name of test series (and Optionally build number – ‘SERIES_NAME#BUILD_NUM’)

Example:

Below is the sample command line code after replacing all the parameters specific to project for reference.

Create View in PostgreSQL DB:

Alright, at this point you would have successfully pushed the robot framework results to PostgreSQL DB. When you check the database you will see the below tables created automatically and parse results will be stored in respective tables.

PostgreSQL DB Tables

Finally, we will create a view in PostgreSQL DB. This is very important step as the default Grafana dashboard rely only on this view. Check out the following link as reference to create views in db – How to create views in PostgreSQL.

View Name: automation_execution_results

Use following query to create the new view automation_execution_results:

Robot Framework Test Results in Grafana Dashboard:

Grafana Plugins:

We will have to install following grafana plugins to incorporate the robot framework dashboard.

  • Click on Configuration Menu in the Left Panel and Select Plugins
  • Click on Find more plugins Grafana.com and Search for below plugins

Select Panel under Plugin Type and Click on Pie Chart | D3 Guage

Please follow the instruction in the below plugin installation page and install the plugins

I have created a decent looking dashboard and published them under grafana dashboards. You can download the pre-build dashboards using the following link Robot Framework Grafana Dashboard

The downloaded zip file should also contain the Grafana dashboards (in JSON format). Import those dashboard files in Grafana (You can find the Import feature under Home as shown below).

Once it successfully imports, It will create dashboards as shown below.

Hope you like this article!! Please provide your rating and helps us to improve 🙂

Robot_Framework_Grafana_PostgreSQL