Skip to content

TestZeus Hercules#
Author Juha-Matti Hietala
Last Updated 10.03.2025

Requirements#

* Tested with Windows 10 * Python 3.11 or newer * OpenAI API key

What is Hercules#

Hercules is a open-source testing agent that turns steps made in Gherkin language in to fully automated tests. It aims to minimize the coding skills needed to perform test automation. Based on the steps given in Gherkin, Hercules executes the test by automatically searching the code of the target website for the elements needed and using them accordingly (clicking links, writing in input fields etc.), and finally giving results in .html and .xml files.

This guide is for installing Hercules in Windows, for other platforms read the official repository linked below.

Windows installation#

  • Download the official repository from their Github as a zip file
  • Extract the zip file to your preferred path
  • Open PowerShell as administrator
  • In PowerShell, Navigate to the helper_scripts folder that you just extracted
  • cd path/to/helper_scripts
  • Once inside the folder, execute the script hercules_windows_setup.ps1 to install and configure Hercules
  • .\hercules_windows_setup.ps1
  • The script will install all required dependencies
  • Once installed, you can run Hercules from PowerShell or Command Prompt. Works in VSCode terminal also, where I've been using it.

To run Hercules, create a folder for your project (test case) and three subfolders; input, output and test_data. 

PROJECT_FOLDER/
├── input/
│   └── test.feature
├── output/
└── test_data/
    └── test_data.txt
Inside input and test_data folders create the test.feature and test_data.txt files as shown above. The test.feature file will contain the test steps with Gherkin language, the test_data.txt will include some decision making information for the Hercules agent. After you have created this folder structure, you can run Hercules with the following command:

testzeus-hercules --project-base PROJECT_FOLDER  --llm-model gpt-4o --llm-model-api-key sk-proj-k.......
  • --project-base Path to the project directory, Hercules will assume that the three subfolders created earlier are inside
  • --llm-model LLM model (Large Language Model, machine learning) the agent will use

    * If using OpenAI premium account, use gpt-4o

    * If using OpenAI free account, use gpt-4o-mini

    * Notice: The free account API has very limited use, I was able to run two tests through and the third one was canceled midway because the resources were used

  • --llm-model-api-key API key for the LLM model, which you can find in your OpenAI account
  • If you happen to have API from a different AI model, check the official readme if it is supported and how to use it

After running the test, the .html and .xml result files will be in the output folder and Hercules will create new subfolders for gherkin files, log files and proof files. The proofs folder will include screenshots and a video of the test, even if you use headless browser, which is the default in Hercules.

Example#

A simple example of a Sign in test case, where we only sign in to PrestaShop, check that after signin in we end up in the account page and the sign in link at the top of the page changes to sign out.

Folder structure as mentioned: 

Sign_in/
├── input/
│   └── test.feature
├── output/
└── test_data/
    └── test_data.txt
The content of test.feature file:

Feature:    Login to Prestashop

    Scenario:   User tries to login to Prestashop with correct credentials

        Given I have a web browser open
        When I navigate to PrestaShop login page
        And I Sign in
        Then I should see the "Your account" page
        And I should see the "Sign out" link at the top of the page
As you can see the feature file only describes the steps a user would make without any detailed information, so for Hercules to know where is PrestaShop login page and credentials to Sign in, they are given in the test_data.txt file.

test_data.txt:

URL: https://fip-AAA-BBB-CCC-DDD.kaj.poutavm.fi/en/login?back=my-account

Email: working@email.fi
Password: WorkingPassword
These could be given in the Gherkin steps inside the feature file, but what I've noticed is that Hercules operates faster if these url, credentials etc. information is kept seperately in the test_data.txt file.

Running the test with command: 

testzeus-hercules --project-base Sign_in --llm-model gpt-4o --llm-model-api-key sk-svcacct-.....YOUR API KEY
After the test is completed the folder structure should look something like this: 
Sign_in/
├── gherkin_files/
├── input/
│   └── test.feature
├── log_files/
├── output/
│   ├── test.feature_result.html
│   └── test.feature_result.xml
├── proofs/
│   └── User_tries_to_login_to_Prestashop_with_correct_credentials/
│       ├── network_logs.json
│       ├── screenshots/
│       └── videos/
└── test_data/
    └── test_data.txt
The result .html file is still less detailed than in Robot Framework, but it shows some basic information, most importantly did the test pass or fail. In the videos folder should be a video of Hercules executing the testcase in a browser, if you need visual clarification.

Part of the result file:

screenshot