In most of the Python projects I’m working with Pytest is used to test the code, and Coverage is used to check what lines that the tests validate. For this to work, Coverage must take part in the execution of the Python code. While this isn’t a problem for most projects working with NSO poses a challenge since the actual Python code for each NSO package gets executed in a separate Python virtual machine. The goal of this article is to show you how you can overcome this obstacle and gain some insight into your test coverage for your NSO Python packages.
To make things simpler to understand, we’re going to start by looking at how Coverage works outside of NSO. Start by installing the package:
pip install coverage
To test this, we are going to use a simple Python application we call demo.py.
import sys
def show_output(choice):
if choice == "hello":
print("Hello to you too!")
elif choice == "goodbye":
print("I wish you goodbye!")
else:
print("I don't know what to make of that")
if __name__ == "__main__":
show_output(sys.argv[1])
Here we have an application that takes at least one argument and prints one of three things depending on what the parameter is.
ᐅ python demo.py hello
Hello to you too!
ᐅ
Running the same thing through Coverage:
ᐅ coverage run demo.py hello
Hello to you too!
ᐅ
It looks the same only Coverage now creates a .coverage file. We then use Coverage to read and parse that file.
ᐅ coverage report -m
Name Stmts Miss Cover Missing
---------------------------------------
demo.py 9 3 67% 7-10
ᐅ
In this scenario, we don’t have any tests. We only see that 67% of the file got executed. We also see that lines 7 to 10 are missing from the report, and as such, we can’t know if they work. If we were writing real tests for this program, we might want to add tests to cover the missing lines.
It’s also worth pointing out that while we have a code coverage of 67 %, we still haven’t written any tests. All that we know is that that 67 % didn’t cause anything to crash. So, code coverage in itself doesn’t guarantee that much. It can, however, help you to see which parts of your code gets used within the tests. If you have lines or entire functions that might be dead branches and should get removed from your codebase.
I have created two packages in NSO, the packages themselves won’t do anything exciting; their purpose is to have something in place to show how to use Coverage from within NSO. So, instead of having something that would require network access and a specific NED, these are generic actions that anyone can run.
The packages are:
The code for the packages live in the network-lore demos repository in Github: https://github.com/networklore/networklore-demos
The NSO action for the calculator can be triggered like this:
admin@ncs> request calc calc number-a 12 operation multiplication number-b 45
message 12 * 45 =
value 540.0
[ok][2019-11-18 18:54:03]
admin@ncs> request calc calc number-a 66 operation division number-b 6
message 66 / 6 =
value 11.0
[ok][2019-11-18 18:54:32]
admin@ncs>
The relevant Python code for the action looks like this:
class CalcAction(Action):
"""calc action."""
@Action.action
def cb_action(self, uinfo, name, keypath, user_input, output):
"""cb_action."""
first = user_input.number_a
second = user_input.number_b
value = False
if user_input.operation == "addition":
message = f"{first} + {second} ="
value = first + second
elif user_input.operation == "subtraction":
message = f"{first} - {second} ="
value = first - second
elif user_input.operation == "multiplication":
message = f"{first} * {second} ="
value = first * second
elif user_input.operation == "division":
try:
message = f"{first} / {second} ="
value = round(first / second, 2)
except ZeroDivisionError:
message = "You have to pay extra for that operation"
output.message = message
if value is not False:
output.value = value
The greeter package is even simpler. You can run it like this:
admin@ncs> request greeter greet
message Good evening!
[ok][2019-11-18 19:00:37]
admin@ncs>
The code behind the action:
def time_of_day(hour):
"""Return pleasant time of day."""
if hour < 5:
return "night"
elif hour >= 5 and hour < 12:
return "morning"
elif hour == 12:
return "noon"
elif hour > 12 and hour < 18:
return "afternoon"
elif hour >= 18 and hour < 19:
return "dinner time"
elif hour >= 19:
return "evening"
class GreetAction(Action):
"""Greet action."""
@Action.action
def cb_action(self, uinfo, name, keypath, user_input, output):
"""cb_action."""
hour = datetime.datetime.now().hour
period = time_of_day(hour)
if period == "dinner time" or period == "noon":
output.message = "You must be getting hungry!"
else:
output.message = f"Good {period}!"
Both of these packages are obviously quite silly and wouldn’t be terribly useful in a real setup. We only want to use them for demonstration purposes.
As we saw above Coverage needs to take part in the execution of the code it is analyzing. When you create a new package in NSO, a simple test using Lux gets created as a starting point. There’s no option to use Coverage from there. For my part, I prefer to write NSO tests using Pytest and communicate with the server over Netconf. However, even if Coverage can see the Python code that the tests execute, it doesn’t help us with our NSO packages. When we connect to NSO, the server, in turn, handles the code through Python virtual machines.
We can see these VMs from the NSO box.
root@50f85dd023c6:/nso/run# ps -x | grep python
186 ? Ssl 0:00 python -u /opt/ncs/current/src/ncs/pyapi/ncs_pyvm/startup.py -l info -f ./logs/ncs-python-vm -i greeter
203 ? Ssl 0:00 python -u /opt/ncs/current/src/ncs/pyapi/ncs_pyvm/startup.py -l info -f ./logs/ncs-python-vm -i calc
root@50f85dd023c6:/nso/run#
The startup.py file itself gets started by an ncs-start-python-vm shell script that looks like this:
#!/bin/sh
pypath="${NCS_DIR}/src/ncs/pyapi"
# Make sure everyone finds the NCS Python libraries at startup
if [ "x$PYTHONPATH" != "x" ]; then
PYTHONPATH=${pypath}:$PYTHONPATH
else
PYTHONPATH=${pypath}
fi
export PYTHONPATH
main="${pypath}/ncs_pyvm/startup.py"
echo "Starting ${main} $*"
exec python -u ${main} $*
So, this is the place where we need to insert Coverage. The documentation for NSO specifically says that you shouldn’t make modifications to that this file since it can get wiped during an upgrade. The correct way is to make a copy of the file and tell NSO to use our modified copy.
The standard startup script starts Python in unbuffered mode (-u) so we should do the same, this doesn’t seem to be an option with Coverage, but we can also set the environment variable PYTHONUNBUFFERED to x. A modified startup script for Coverage can look like this:
#!/bin/sh
pypath="${NCS_DIR}/src/ncs/pyapi"
# Make sure everyone finds the NCS Python libraries at startup
if [ "x$PYTHONPATH" != "x" ]; then
PYTHONPATH=${pypath}:$PYTHONPATH
else
PYTHONPATH=${pypath}
fi
export PYTHONPATH
main="${pypath}/ncs_pyvm/startup.py"
echo "Starting ${main} $*"
export PYTHONUNBUFFERED=x
exec coverage run --parallel-mode ${main} $*
By default, Coverage writes its findings to a .coverage file to the current directory. However, the above startup script is used to start multiple Python VMs, and it would be unpredictable for them to write to the same file. To avoid that issue, the --parallel-mode setting is used to start Coverage, which creates a separate file for each process.
According to the documentation, we would refer to our start script by adding a section like this to the ncs.conf configuration file.
<python-vm>
<start-command>
/nso/run/bin/start-python-coverage-vm.sh
</start-command>
</python-vm>
However, it seems that changing the configuration in this way causes NSO not to send in any arguments to the shell script, i.e., the $* part which is needed in order to tell the startup script which package to start. An example of what this data might look like: -l info -f ./logs/ncs-python-vm -i calc
A workaround was to instead set the python-vm start-command from NSO, or if this is a throwaway test container it should be fine to modify the original script.
admin@ncs% set python-vm start-command /nso/run/bin/start-python-coverage-vm.sh
[ok][2019-11-19 18:57:21]
[edit]
admin@ncs% commit
Commit complete.
[ok][2019-11-19 18:57:23]
[edit]
admin@ncs%
After the changes are committed, you need to reload the packages, or restart NSO.
Now when we start NSO all of the Python VMs gets executed by Coverage, and when we run our test suite, we can see which part of our code gets hit.
We can verify that we the Python VMs are started correctly using coverage:
root@231b8fe85843:/nso/run# ps -x | grep python
237 ? Ssl 0:00 /usr/bin/python3 /usr/local/bin/coverage run --parallel-mode /opt/ncs/current/src/ncs/pyapi/ncs_pyvm/startup.py -l info -f ./logs/ncs-python-vm -i greeter
238 ? Ssl 0:00 /usr/bin/python3 /usr/local/bin/coverage run --parallel-mode /opt/ncs/current/src/ncs/pyapi/ncs_pyvm/startup.py -l info -f ./logs/ncs-python-vm -i calc
root@231b8fe85843:/nso/run#
At this point, you can run your tests just as you would typically do, be it with Lux, Pytest, or through some other means. Since this article isn’t about any specific testing framework, I’m just going to connect to the CLI and run a few actions. In a real test scenario, the output from the commands or some other condition would get verified, but today, we only care about which part of the code gets hit.
admin@ncs> request greeter greet
message Good evening!
[ok][2019-11-19 19:05:23]
admin@ncs> request calc calc number-a 5 operation multiplication number-b 62
message 5 * 62 =
value 310.0
[ok][2019-11-19 19:05:34]
admin@ncs> request calc calc number-a 1812 operation division number-b 5
message 1812 / 5 =
value 362.4
[ok][2019-11-19 19:05:43]
admin@ncs> exit
The Coverage files are still not created at this time:
root@231b8fe85843:/nso/run# ls -la .cov*
ls: cannot access '.cov*': No such file or directory
root@231b8fe85843:/nso/run#
This is because the Python VMs are still running so Coverage is still waiting for things to happen. The files will only be created when we shutdown NSO:
root@231b8fe85843:/nso/run# ncs --stop
root@231b8fe85843:/nso/run# ls -la .cov*
-rw-r--r-- 1 root root 12762 Nov 19 19:06 .coverage.231b8fe85843.237.722345
-rw-r--r-- 1 root root 12895 Nov 19 19:06 .coverage.231b8fe85843.238.879368
root@231b8fe85843:/nso/run#
At this stage, we have one Coverage file for each of our NSO Python packages. We can merge them all into one file before looking at the result.
root@231b8fe85843:/nso/run# coverage combine
root@231b8fe85843:/nso/run# ls -la .cov*
-rw-r--r-- 1 root root 13119 Nov 19 19:07 .coverage
root@231b8fe85843:/nso/run#
By default, we see Coverage data for all Python packages, including the ones from the NSO Pyapi as well as any third-party package you are using. As all of the code we are interested in lives within the ./state folder, we can filter what to include before generating a report.
root@231b8fe85843:/nso/run# coverage report -m --include=./state/*
Name Stmts Miss Cover Missing
----------------------------------------------------------------------------------------------
state/packages-in-use.cur/1/calc/python/calc/__init__.py 0 0 100%
state/packages-in-use.cur/1/calc/python/calc/calc.py 31 6 81% 16-17, 19-20, 28-29
state/packages-in-use.cur/1/greeter/python/greeter/__init__.py 0 0 100%
state/packages-in-use.cur/1/greeter/python/greeter/greeter.py 29 6 79% 12, 14, 16, 18, 20, 35
----------------------------------------------------------------------------------------------
TOTAL 60 12 80%
root@231b8fe85843:/nso/run#
In this test run, the complete code coverage is 80%. We also see that the code coverage for the calc package is higher than the greeter, and we also see which lines in the Python file we miss in our test suite. I realize that the greeter app is problematic as it is dependant on the time of day. You could, however, create a unit test outside of NSO to test the time_of_day() function, and if desired, you can combine the .coverage file you get from that run with the one above to get a complete picture.
You wouldn’t want to replace the startup command in your production environment. Examine the nso-docker repository for some information regarding how you can create different containers for your development and production environments.
For this test, I was running NSO 4.7.5. The code for the packages and coverage startup script are up on Github at: https://github.com/networklore/networklore-demos
Specifically, the files for this article are in the coverage folder
You should now be able to add Coverage data to your CI pipeline when testing your NSO Python packages. Keep in mind, though, that in the article, we also saw that we had quite a high coverage without validating anything. So, we saw that nothing crashed, but there still might be bugs lurking there, i.e., coverage in itself might not always mean that much.
I hope you found this useful!