The roles required of a software deployment team generally revolves around the following,
- Provisioning the correct app environment
- Deploying the correct app version
- Configuring the apps, including any data or state they require
The task of provisioning the app environment typically involves spinning up the required number of virtual machines with specific resource configurations, operating systems, network connectivity, block/file/object storage and external services such as databases, directory services, time servers and so on. Ensuring the app environment is setup correctly is a critical necessity, at the same time the manpower resource required to validate the app environment can become non-trivial especially for a large deployment footprint. An automated system validation setup can alleviate some of the pain, if not most.
Pytest for Infrastructure Testing
Pytest is a full-featured Python testing tool that is usually employed for unit and functional testing during the software development phase. Since it is based on Python scripts, Pytest can also be extended to validate the actual state of your deployed servers as part of a production test automation setup.
Installing Pytest & Support Packages on Ubuntu
- Pre-requisites: Python is probably already installed on your system. Check if you also have pip:
$ which pipor
$ pip –V- Run the following command in your command line to install Pytest
$ pip install -U --user pytestFor Python3 compatibility, run the following too
$ pip3 install -U --user pytest- You’ll also need the Spur.py Python package to spawn shell commands on remote machines over SSH
$ sudo apt-get install python3-spur- If you want to run tests in parallel, you should also install the Pytest-Parallel package
$ pip install -U --user pytest-parallel
$ pip3 install -U --user pytest-parallelValidating Machines are Online
The network ping function can be used as a basic test to check if a cluster of servers is online. The Pytest script to validate a remote machine (e.g. 192.168.56.101) is online is as follows.
import os
def check_ping(hostname):
response = os.system("ping -c 1 " + hostname)
return response
def test_online():
assert(check_ping("192.168.56.101") == 0)
Save this script as “test_local.py” and launch Pytest,
$ python3 -m pytest -v test_local.pyIf the remote target machine is online the expected result is a green pass.
It is trivial to extend the test script to validate the online statuses of multiple machines on the network.
Validating Ping Connectivity on Remote Machines
Usually the test framework is not deployed on the production machines. It is possible to launch Pytest from a dedicated test server (as a Launchpad) to initiate shell commands on remote machines over SSH. The following script checks if an arbitrary website of your choice (www.xyz.com) is reachable from the remote machine at 192.168.56.103.
import spur
def check_remote_ping(hostname):
shell = spur.SshShell(hostname="192.168.56.103", username="test", password="test", missing_host_key=spur.ssh.MissingHostKey.accept)
with shell:
result = shell.run(["ping", "-c 1 -W 2 ", hostname])
if (b', 0% packet loss' in result.output):
return 0 # success
else:
return 1 # fail
def test_remote_online():
assert(check_remote_ping("www.xyz.com") == 0)
If the network connectivity is working correctly on the remote machine the expected result is a pass. If you receive a warning about the crypto function used for the SSH connection, this is likely due to an invalid installation of the cryptography module required by Paramiko, the framework on which Spur is based.
You should be able to fix it by installing the cryptography module,
$ sudo apt-get install build-essential libssl-dev libffi-dev python-dev
followed by,
$ pip install cryptography $ pip3 install cryptography
Running Tests in Parallel
When your number of test cases grows, the test execution time will accumulate as well. The pytest-parallel library allows Pytest scripts to be launched concurrently to achieve a reduction in test execution time.
For instance running pytest in this way allows it to launch the tests with two concurrent threads,
$ python3 -m pytest -v --workers 2
Compare this to running the tests serially,
Auditing Server Configuration, Resource Allocation and More Complex Test Tasks
The Testinfra pytest plugin can be used for scripting more sophisticated tests against the remote server configuration and infra resource allocation.
- Install the Testinfra plugin as follows,
$ pip install -U --user testinfra
$ pip3 install -U --user testinfra- Create a sample testinfra test script (test_infra.py)
def test_passwd_file(host):
passwd = host.file("/etc/passwd")
assert passwd.contains("root")
assert passwd.user == "root"
assert passwd.group == "root"
assert passwd.mode == 0o644
and run it.
$ python3 -m pytest -v test_infra.pyIf the host isn’t specified, the test is run against the local machine. The expected result:
Spawning testinfra test scripts on remote machines requires a little more configuration involving generating an SSH key pair:
- Generate a public-private key pair on the test server
$ cd $HOME/.ssh
$ ssh-keygen -t rsa -f test_server- Copy the public key to the target machine
$ scp test_server.pub test@192.168.56.103:/var/tmp/- Configure the local ssh config file
$ vi $HOME/.ssh/ssh_configEnter the following,
Host test_server
HostName 192.168.56.103
Port 22
User test
IdentityFile ~/.ssh/test_server
Save the SSH config file.
- SSH into the target machine
$ ssh -i $HOME/.ssh/test_server test@192.168.56.103Import the public key on the target machine
$ mkdir $HOME/.ssh
$ mv /var/tmp/test_server.pub $HOME/.ssh/
$ cd $HOME/.ssh/
$ cat test_server.pub >> authorized_keys
$ chmod 600 authorized_keys
$ rm test_server.pubExit ssh and return to the test machine
- On the test machine, launch pytest against the remote machine over ssh as follows,
$ python3 -m pytest -v --connection=ssh --hosts=test@test_server --ssh-config='/home/test/.ssh/ssh_config' test_infra.pyand the result:
Using Fabric
Fabric is an alternative to testinfra. Some tasks are more easily accomplished on one than the other. Install Fabric as such,
$ pip install -U --user fabric
$ pip3 install -U --user fabric
If you encounter an error prompting, “failed building wheel for fabric”, try this:
$ sudo apt-get install build-essential libssl-dev libffi-dev python-devfollowed by,
$ pip install cryptography
$ pip3 install cryptographythereafter try re-installing fabric.
Initiating remote SSH connections using Fabric
Here we can reuse the public-private key pair previously generated in the testinfra section.
connect_kwargs = {"key_filename":['PATH/KEY.pem']}
result = Connection(host="192.168.56.103", user="test", connect_kwargs=connect_kwargs).run(‘uname –s’, hide=True)
e.g. connect_kwargs = {"key_filename":['/home/test/.ssh/test_server']}The following example uses Fabric to audit the amount of free memory and swap space on a remote machine.
from fabric import Connection
# check physical memory
def query_free_memory_percent(remotehost):
connectinfo = {"key_filename":['/home/test/.ssh/test_server']}
shell = Connection(host=remotehost, user='test', connect_kwargs=connectinfo)
result = shell.run('free -m', hide=True)
lines = result.stdout.split('\n')
total_m, used_m, free_m, x, y, z = map(int, lines[1].split()[1:])
mem_free = free_m / total_m * 100
print ('Free memory (percent): %.0f' % (mem_free))
return mem_free
def query_free_swap_percent(remotehost):
connectinfo = {"key_filename":['/home/test/.ssh/test_server']}
shell = Connection(host=remotehost, user='test', connect_kwargs=connectinfo)
result = shell.run('free -m', hide=True)
lines = result.stdout.split('\n')
total_m, used_m, free_m = map(int, lines[2].split()[1:])
swap_free = free_m / total_m * 100
print ('Free swap (percent): %.0f' % (swap_free))
return swap_free
def test_remote_system_memory():
assert(query_free_memory_percent('192.168.56.103') > 50)
def test_remote_swap_space():
assert(query_free_swap_percent("192.168.56.103") > 50)
NOTE: Execute this in pytest using the ‘-s’ (no capture) option, otherwise a thread exception error could be thrown (“reading from stdin while output is captured”).
$ python3 -m pytest -s -v test_memory.pyAudit System Disk Partition Usage
from fabric import Connection
# check disk utilization
def query_disk_free_percent(remotehost):
connectinfo = {"key_filename":['/home/test/.ssh/test_server']}
shell = Connection(host=remotehost, user='test', connect_kwargs=connectinfo)
result = shell.run("df -h / | tail -n1 | awk '{print $5}'", hide=True)
diskfree = result.stdout.strip()
print ("Disk free = " + diskfree)
return int(''.join(c for c in diskfree if c.isdigit()))
def test_remote_disk_free():
TestGroup = ['192.168.56.103']
for svr in TestGroup:
assert(query_disk_free_percent(svr) > 10)
Validate Time Synchronization
from fabric import Connection
# check ntp service
def check_ntp_enabled(remotehost):
connectinfo = {"key_filename":['/home/test/.ssh/test_server']}
shell = Connection(host=remotehost, user='test', connect_kwargs=connectinfo)
is_active_result = shell.run("systemctl is-active systemd-timesyncd.service", hide=True)
is_enabled_result = shell.run("systemctl is-enabled systemd-timesyncd.service", hide=True)
return is_active_result.stdout.strip() == 'active' and is_enabled_result.stdout.strip() == 'enabled'
def check_time_sync(remotehost):
connectinfo = {"key_filename":['/home/test/.ssh/test_server']}
shell = Connection(host=remotehost, user='test', connect_kwargs=connectinfo)
result = shell.run("timedatectl", hide=True)
lines = result.stdout.split('\n')
networkTimeOn = False
ntpSynced = False
for line in lines:
if 'Network time on:' in line:
status = line.split(':')
if status[1].strip() == 'yes':
networkTimeOn = True
elif 'NTP synchronized:' in line:
status = line.split(':')
if status[1].strip() == 'yes':
ntpSynced = True
return networkTimeOn and ntpSynced
def test_time_synchronization():
TestGroup = ['192.168.56.103']
for remotemachine in TestGroup:
assert(check_ntp_enabled(remotemachine) == True)
assert(check_time_sync(remotemachine) == True)
Summary
This article just touches the surface to show how pytest can be used as a test automation framework to perform system validation against a large system infrastructure deployment. All test scripts are in plain text and can therefore be readily version-controlled in Git or Subversion. In fact the pytest scripts can be integrated and launched from CI/CD platforms such as Gitlab, therefore allowing combining configuration management and test automation into a single platform.
Inspired Technologies 