Performance testing with RF#
| Author | Juha-Matti Hietala |
| Last Updated | 18.03.2025 |
Requirements#
* Tested with Windows 10 * Python 3.6 minimum for Agents * Python 3.7 minimum for Manager * Python 3.9 minimum for Reporter * tkinter might be needed, if you don't have it installed, do it with their guide
Introduction#
Robot Framework can be used for performance testing. This guide uses a tool called rfswarm for this purpose.
What is rfswarm#
With rfswarm you can use Robot Framework test cases for performance or load testing. It allows you to take an existing Robot Framework test case, with some adjustments to make it suitable for performance testing and run the test case with virtual users (robots), as many as needed for the desired load to be generated.
Installation#
There are 3 components in rfswarm, the Manager for planning and running your test scenario, the Agents that run the tests and the optional Reporter that assist you in reporting the test results. The Manager and Agents are mandatory, the Reporter optional.
The Manager#
The swarm manager is where you monitor and schedule your robots. The manager machine does not have demanding cpu or memory requirements, and you only need one, so it can be your ordinary desktop machine. Robot Framework is not required in this machine. For more information about the Manager read the official documents.
Installing the manager#
Use pip, pip3 or sudo depending on your system. In command prompt:
If you want you can install a desktop icon with the following command. The icon will appear in the start menu.
The Manager needs TCP port 8138 inbound open by default for the agents to connect, so check your firewall rules and make changes if needed.To run rfswarm manager use the command
or click the icon if createdThe Agents#
Agents are what actually execute the tests, so these machines require Robot Framework to be installed. In rfswarm documentation it is mentioned that a mid range PC should be able to support around 50 virtual users for tests using SeleniumLibraby, depending on factors you can read more from the documentation. Start testing with about 10, see how your computer handles it and ramp up if possible. For more information read the official documents.
Installing the Agent#
Use pip, pip3 or sudo depending on your system. In command prompt:
Run Agent in cmd prompt
When you run the agent, you should see something like this$ rfswarm-agent
Robot Framework Swarm: Run Agent
Version 0.6.1
Configuration File: path_to/rfswarm_agent/RFSwarmAgent.ini
RfSwarmAgent.ini
[Agent]
agentname = NAme of the agent machine, example DESKTOP-xxxxx
agentdir = path/to/agent
xmlmode = False
excludelibraries = BuiltIn,String,OperatingSystem,perftest
properties =
swarmmanager = http://localhost:8138/
robotcmd = robot
Reporter#
Reporter is used after a performance test has been completed for a more detailed report.
Installing Reporter#
Use pip, pip3 or sudo depending on your system. In command prompt:
If you want you can install a desktop icon with the following command. The icon will appear in the start menu.
To run the reporter, in cmd prompt or use the icon if created:
Using rfswarm#
Open the manager and run the agent, you should see this:
Manager
Agent running
C:\>rfswarm-agent
Robot Framework Swarm: Run Agent
Version 1.4.0
Configuration File: path\to\rfswarm_agent\RFSwarmAgent.ini
Manager Connected http://localhost:8138/ 2025-03-16 14:46:19 ( 1742129179 )
When you see the Agent is connected, go back to the Plan tab in the Manager. Here you can plan your test scenario.
-
Index
You can run multiple Test Cases at once, this is just the index number for different Test Cases. Add a second row from the green + sign.
-
Robots
This is where you put the number of robots you want to be executing the Test Case you will choose.
-
Delay
This is the delay from when you start the scenario to when this particular Test Case activates. So if you have multiple rows of Test Cases, you can choose when each starts.
-
Ramp Up
This is the duration for how long it takes for all the robots you assigned for this Test Case to be running. For example if you have 10 robots assigned and you put Ramp Up to 30 seconds, it means that every 3 seconds a new robot activates until all 10 are running.
-
Run
How long will the Test Case run after all robots are activated. After the run period there will be a Ramp Down, which is the same time as Ramp Up. So if you have a 1 minute Ramp Up and 2 minute Run, there will be a 1 minute Ramp Down also, which means the total duration for the scenario would be 4 minutes. Time set to Delay will also be added to the total time.
-
Script
The script you will be using, press the icon and search for the right script.
-
Test
Test Case that you want to run from the script you chose. If you have multiple in one script, you can only choose one per row.
When you have your scenario planned, press the blue Play button top-left of the planning tab. Now you can follow the test progress from the Run tab.
Example#
In this example I have assigned 40 robots to run a simple Sign in and out test case.
I have declared a 2 minute Ramp Up and a 2 minute Run time for the robots. This means there will be a 2 minute Ramp Down after Run, so the total runtime will be 6 minutes. The black line in the diagram visualizes this. The script I used looks like this:
Change url and login credentials to your test page
*** Settings ***
Documentation Sign in and out of Prestashop
Library SeleniumLibrary
*** Variables ***
${BASE_URL} https://fip-AAA-BBB-CCC-DDD.kaj.poutavm.fi/en/login?back=my-account
${signin_submit} id=submit-login
${login_email} id=field-email
${login_password} id=field-password
${sign_out} //div[@class='user-info']/a[contains(., 'Sign out')]
*** Keywords ***
Open Browser To Login Page
Open Browser ${BASE_URL} headlesschrome
Title Should Be Login
Type In Username
[Arguments] ${username}
Input Text ${login_email} ${username}
Type In Password
[Arguments] ${password}
Input Text ${login_password} ${password}
Submit Login
Click Element ${signin_submit}
Title Should Be My account
Log Out
Click Element ${sign_out}
Title Should Be Login
*** Test Cases ***
Sign in and out
Open Browser To Login Page
FOR ${index} IN RANGE 30
Type In Username working@email
Type In Password WorkingPassword
Submit Login
Log Out
END
[Teardown]
The FOR ${index} IN RANGE 30 means that Keywords between that and the END will be repeated 30 times before the test continues further. This is so the robots don't have to open new browsers constantly and the test if more effective. With 30 in this scenario you get about 2000 runs for the Test Case with 40 robots. You can put what number you think fits your purpose.
After starting the scenario, in the Run tab you can see the robots ramping up and the test case being executed with time stats for each keyword. In this test we can already notice that the server was getting stressed, as can be seen in the stats after the test was completely run, Max times are several seconds:
If you want, you can manually test how the server is stressed, by browsing the Prestashop page when the test is running.
Now that the test has completed, go to the Plan tab in Manager, press the cogwheel at the top left and see where the Results Location is. After that open the Reporter from cmd prompt or the icon in start menu if installed.
A windows like this should open:
In the Reporter press Results->Open at the top left corner and browse to the just mentioned Results Location folder. Open the created .db file from the scenario folder. Press the Preview tab and you see a more detailed report of the scenario. Example from the report:
Response time means the time the keyword took, as measured by robot framework and then reported back to the Manager by the Agent.
For more details about the Reporter, read the official document.