Firewalld Testsuite Primer
Over the past two major releases firewalld has seen vast improvements to its testsuite. This post will discuss how to run the testsuite, how to debug a failure, and finally we’ll go through an exercise of adding a new test case. The main target is for current and future contributors to firewalld. However, since firewalld’s testsuite utilizes autotest some of the knowledge gained here may also carry over to other projects.
Running the testsuite
Running the testsuite is very simple. To start build the code just as you would if installing from source.
$ git clone https://github.com/firewalld/firewalld $ cd firewalld $ ./autogen.sh $ ./configure $ make
Then run the testsuite using the
check make target. This needs to be run as
# make check
These tests are non-destructive to the host running them and there is no need to stop the host’s running firewalld instance. They are run inside of network namespaces (containers) which allows numerous benefits; reliability, non-destructive to the host, and they can be run in parallel.
At the time of writing firewalld has 112 test groups - most of which include multiple tests. As such, the testsuite takes a long time to run. The good news is you can speed things up by running them in parallel.
To run test groups in parallel pass
-j4 to autotest via the
# make check TESTSUITEFLAGS="-j4"
Running the testsuite against the host installed firewalld
Firewalld’s testsuite also provides an
installcheck make target. This is
useful for running tests against the hosts installed version of firewalld.
check target runs against the executables built in the source tree.
This is very important for development as it allows you to run a newer
testsuite against an older version of firewalld. As we’ll find later on in this
post, this can be used to leverage test driven development.
To run the testsuite against the installed firewalld
# make installcheck
Debugging a failed test case
Inevitably a test will fail. Whether it’s a bug in firewalld, a permission issue, or a compatibility issue, you’ll have to debug the problem. There are a couple of ways to inspect a failure. The most straight forward way is to view the testsuite log.
To view the testsuite log for failed test number 42
# vi ./src/tests/testsuite.dir/042/testsuite.log
Alternatively, you can enable the verbose flag to cause the testsuite to dump to standard output.
# make check TESTSUITEFLAGS="42 -v"
To enable firewalld’s debug output, you can use the
# make check TESTSUITEFLAGS="42 -v -d"
The test numbers passed to
TESTSUITEFLAGS are very flexible. It will accept
individual numbers or ranges.
For example this is also valid
# make check TESTSUITEFLAGS="42 1-5 110 13-14 17"
Writing a new test case
In this example we’ll follow a test driven development style to fix a real
world firewalld bug. We’ll be using Red Hat bugzilla 1404076 for this
exercise. This bug occurs when a port range is opened using the
--query-port fails to return the expected result if querying a
single port within the range.
Basic test layout
All of the tests follow the same basic layout. Since this is a new test case in
response to an existing bug we’ll add it to the set of regression tests in
At the absolute minimum your new test should look like this
FWD_START_TEST([test description]) ... FWD_CHECK([some command]) ... FWD_END_TEST
The testsuite makes heavy use of m4 macros (
etc.) to simplify test creation. m4 is a macro language which autotest uses to
generate the testsuite script.
A consequence of the macro magic is most tests will be run multiple times. Once for each firewall backend; nftables and iptables. This ensures that both backends are tested equally.
The new test case
We start by creating a new test that reproduces the issue from the bugzilla report.
Here is our new test case
FWD_START_TEST([query single port added with range]) dnl add a set of ports by range, then query a specific port inside that range. FWD_CHECK([-q --add-port=8080-8090/tcp]) FWD_CHECK([-q --query-port=8085/tcp]) FWD_CHECK([-q --query-port=webcache/tcp]) dnl named port FWD_CHECK([-q --query-port=8091/tcp], 1) dnl negative test FWD_CHECK([-q --query-port=8085/udp], 1) dnl negative test dnl same thing, but for permanent configuration. FWD_CHECK([-q --permanent --add-port=8080-8090/tcp]) FWD_CHECK([-q --permanent --query-port=8085/tcp]) FWD_CHECK([-q --permanent --query-port=webcache/tcp]) dnl named port FWD_CHECK([-q --permanent --query-port=8091/tcp], 1) dnl negative test FWD_CHECK([-q --permanent --query-port=8085/udp], 1) dnl negative test FWD_END_TEST
dnl is m4’s way of starting a comment.
This test is fairly exhaustive by including negative tests and named ports. This gives us even more confidence in our code changes.
To attach the new test case to the testsuite we need to append the new test to
$ vi src/tests/regression.at [...] m4_include([regression/rhbz1404076.at])
This new test case can be found upstream on github.
The commit is
3fb707228ced ("tests/regression: add coverage for rhbz 1404076").
Verifying the new test reproduces the issue
Now that the test has been added you can verify it reproduces the issue by
running it against the host’s firewalld using the
installcheck make target.
# make installcheck [...] regression (nftables) [...] 67: query single port added with range FAILED (rhbz1404076.at:5) [...] regression (iptables) [..] 110: query single port added with range FAILED (rhbz1404076.at:5)
Here we see the test case failed as expected for both firewall backends.
Fix the bug
The next step is to make code changes to fix the bug. This is out of scope for
this post, but for those interested the fix can be found on github.
The commit is
2925de324443 ("ports: allow querying a single added by range").
Verify the fix against the in-tree source code
Now that the test case has been written and code changes have been made we can
run the test case again, but this time against the source tree version. This
will verify our code changes fix the bug. Be sure to use the
target this time.
# make check TESTSUITEFLAGS="67 110"
Once your new test case passes you should run the whole testsuite again to verify you didn’t introduce a regression.
# make check
If that was successful, then you’re ready to submit your changes upstream with a pull request! Once your pull request is submitted the testsuite will automatically be run again by travis-ci against multiple versions of python.
This post has shown how easy it is to get started with the firewalld testsuite. We hope it will encourage contributions from others while simultaneously improving firewalld’s quality. Additionally we hope it turns some users into active contributors!
A new release of firewalld, version 0.5.4, is available.
This is a bug fix only release.
- update translations
- fw: if failure occurs during startup set state to FAILED
- fw_direct: avoid log for untracked passthrough queries
- firewall-config: fix some untranslated strings
- Rich Rule Masquerade inverted source-destination in Forward Chain
- don’t forward interface to zone requests to NM for generated interfaces
- firewall-cmd: add –check-config option
- firewall-offline-cmd: add –check-config option
- ipset: check type when parsing ipset definition
- firewall-config: Add ipv6-icmp to the protocol dropdown box
- core: logger: Remove world-readable bit from logfile
- IPv6 rpfilter: explicitly allow neighbor solicitation
Source available here:
A new release of firewalld, version 0.6.1, is available.
This is a bug fix only release.
- Correct source/destination in rich rule masquerade
- Only modify ifcfg files for permanent configuration changes
- Fix a backtrace when calling common_reverse_rule()
- man firewalld.conf: Show nftables is the default FirewallBackend
- firewall-config: fix some untranslated strings that caused a UI bug causing rich rules to not be modify-able
- services/steam-streaming: replace unicode quotation with ASCII apostrophe
- fw_direct: avoid log for untracked passthrough queries
- fixed many issues if iptables is actually iptables-nft
- Use preferred location for AppData files
- ipXtables: fix ICMP block inversion with set-log-denied
- fixes ICMP block inversion with set-log-denied with IndividualCalls=yes
- nftables: fix set-log-denied if target is not ACCEPT
- fw_direct: strip _direct chain suffix if using nftables
Improved interactions with NetworkManager:
- For generated NetworkManager connections, don’t forward request to NetworkManager
- For non-permanent interface to zone assignments don’t involve NetworkManager
- Query NetworkManager for permanent interface assignments
- tests/functions: Remove bashism
- installcheck: pass PYTHON to testsuite, fixes python tests on distro with mixed python versions
- testsuite: avoid multiple inclusion warning
- testsuite: Only enable debug output if testsuite ran with debug flag
Source available here:
As noted in the v0.6.0 release announcement, firewalld recently gained support for using nftables as a firewall backend. This post will highlight why that’s a good thing, how it affects firewalld, and how to start using it. The content here may be interesting to intermediate to advanced users of firewalld or anyone generally interested in firewalling.
nftables in a replacement for all of; iptables, ip6tables, arptables, ebtables, and ipset (henceforth know as “iptables and family”). It brings many advantages, some examples are; built in sets, faster rule updates, and combined ipv4/ipv6 processing. One of the most relevant advantages for firewalld is the ability to maintain all firewall rules through a single interface.
One issue with firewalld’s use of iptables and family is that firewalld assumes complete control of the hosts firewalling. With the nftables backend this is no longer true. Since nftables allows multiple namespaces (tables in nftables vernacular), firewalld will scope all of its rules, sets, and chains to the firewalld table. This will avoid much of the contention with other pieces of software that don’t interact directly with firewalld.
Another nicety of nftables is you can combine logging with verdicts in the same rule. This means you can optimize rules, e.g. log and drop a packet in the same rule. In iptables this would require two rules, both with the same match parameters.
How does firewalld use nftables?
firewalld interacts with nftables directly through the
nft binary. This is
similar to how firewalld currently interacts with iptables and family. In a
future release interaction with nftables will be further improved by using the
newly minted libnftables.
As can be seen in the firewalld structure diagram, nftables fits into firewalld alongside the other firewall backends.
All firewalld’s primitives (services, ports, forward ports, etc.) use nftables by default. In addition some translations occur; ipsets will be translated to native nftables sets, and ICMP types are morphed into nftables equivalents.
What about the direct interface?
Seasoned firewalld users may already be asking themselves, What about the direct interface? No worries, it’s still there and works almost exactly as it did in previous releases. When the nftables backend is enabled direct rules are treated specially and still use iptables and family. This means your existing configurations with custom direct rules will continue to work. However, there is one deliberate behavioral change - direct rules take precedence over all other firewalld rules. For further details see the Behavorial Changes section below.
How do I use the nftables backend?
In firewalld 0.6.0 and later nftables is the default backend - so all you have to do is upgrade. The switch over should be transparent to users. The nftables backend has feature parity with the old iptables backend. That means any issues or missing functionality will be treated as bugs.
If for some reason you need to revert to the old iptables backend, you can
easily do so by setting
iptables, then restart firewalld. However, please realise that future
firewalld development will focus on the nftables backend and not iptables.
This new configuration option is documented in the man pages.
What the future holds
The future of firewalld development will focus on the nftables backend. While the iptables backend is still supported new features won’t necessarily be implemented.
With the nftables backend firewalld has many possible improvements on the horizon:
- output filtering
- DoS protection or rate limiting
- log targets can be combined into the same rule as accept/drop
- avoid creating empty chains or rules
With all that said, the iptables backend is not going away anytime soon. It will be supported for many, many releases. It’s simply just not the primary focus.
Here follows further information that may be useful for those wanting some grittier details.
The nftables backend does bring some behavioral changes. In most scenarios these won’t affect users, but may be relevant for more advanced use cases.
Direct interface precedence
In previous versions of firewalld rules added via the direct interface were still subject to firewalld’s rules that occur early in the ruleset. Most notably is the early acceptance of packets that are part of existing connections. This has led to confusion for users, for example see github issue 44.
With the nftables backend direct rules are deliberately given a higher precedence than all other firewalld rules. Now when a user adds a direct rule to drop traffic existing connections will also be affected.
Firewalld hook priority
nftables allows multiple chains to hook into netfilter at the same point. This is in contrast to iptables which only allowed one and the concept was completely hidden from users. firewalld takes advantage of this feature to add predictability by giving firewalld’s rules slightly lower precedence than iptables and default nftables hook priority values. Without this it would be undefined if firewalld’s chains would execute before or after other iptables/nftables chains.
The main consequence for users is that firewall rules created outside of firewalld (e.g. libvirt, docker, user, etc) will take precedence over firewalld’s rules.
Only flush firewalld’s rules
Since nftables allows namespaces (via tables) firewalld no longer does a complete flush of firewall rules. It will only flush rules in the firewalld table. This avoids scenarios where custom user rules or rules installed by other tools are unexpectedly wiped out when firewalld was restarted or reloaded.
Packet accept/drop precedence
As mentioned above, nftables allows multiple chains to use the same netfilter hook. A consequence of this is that packets that are accepted are still subject to the rules of other chains hooked into the hook type. For firewalld this means packets may be accepted early by custom iptables or nftables rules, but will still be subject to firewalld’s rules. In the drop case processing always stops immediately and no other hooks will process the packet.
A new release of firewalld, version 0.6.0, is available.
This is a large feature release. Some new features warrant a separate blog post. As such, the release is only summarized here.
User facing features:
- nftables backend
This is the new default for all firewalld’s abstractions. The direct interface still supports iptables, ip6tables, and ebtables. It is configurable via
/etc/firewalld/firewalld.conf- valid values are;
- new services
apcupsd, cockpit, distcc, etcd, finger, iSNS, llmnr, matrix, mqtt, nut, plex, rtsp, salt-master, samba-dc, slp, steam, subversion, svdrp, wbem-http, wsman
- updated translations
A lot of development time was spent on improving continuous integration and sanity checks to give confidence in new code changes and contributions from others. This is reflected below in the list of developer focused improvements.
Developer focused improvements:
- lots of code refactoring to abstract firewall backend
- flake8 source code checking
- better debug output on exceptions (tracebacks)
- testsuite runs inside network namespaces
- improved testsuite coverage
New dependencies for nftables backend:
- nftables >= 0.9.0
- linux >= 4.18
While most of the nftables backend will function with earlier versions of nftables and Linux it is not recommended. Many bugs were found and fixed in these packages while firewalld’s nftables backend was being developed. Some examples are; iptables and nftables NAT coexistence, nftables AUDIT support, nftables set ranges with timeouts.
Source available here: