Testsuite Primer

Firewalld Testsuite Primer

Over the past two major releases firewalld has seen vast improvements to its testsuite. This post will discuss how to run the testsuite, how to debug a failure, and finally we’ll go through an exercise of adding a new test case. The main target is for current and future contributors to firewalld. However, since firewalld’s testsuite utilizes autotest some of the knowledge gained here may also carry over to other projects.

Running the testsuite

Running the testsuite is very simple. To start build the code just as you would if installing from source.

$ git clone https://github.com/firewalld/firewalld
$ cd firewalld
$ ./autogen.sh
$ ./configure
$ make

Then run the testsuite using the check make target. This needs to be run as root.

# make check

These tests are non-destructive to the host running them and there is no need to stop the host’s running firewalld instance. They are run inside of network namespaces (containers) which allows numerous benefits; reliability, non-destructive to the host, and they can be run in parallel.

At the time of writing firewalld has 112 test groups - most of which include multiple tests. As such, the testsuite takes a long time to run. The good news is you can speed things up by running them in parallel.

To run test groups in parallel pass -j4 to autotest via the TESTSUITEFLAGS variable.

# make check TESTSUITEFLAGS="-j4"

Running the testsuite against the host installed firewalld

Firewalld’s testsuite also provides an installcheck make target. This is useful for running tests against the hosts installed version of firewalld. The check target runs against the executables built in the source tree. This is very important for development as it allows you to run a newer testsuite against an older version of firewalld. As we’ll find later on in this post, this can be used to leverage test driven development.

To run the testsuite against the installed firewalld

# make installcheck

Debugging a failed test case

Inevitably a test will fail. Whether it’s a bug in firewalld, a permission issue, or a compatibility issue, you’ll have to debug the problem. There are a couple of ways to inspect a failure. The most straight forward way is to view the testsuite log.

To view the testsuite log for failed test number 42

# vi ./src/tests/testsuite.dir/042/testsuite.log

Alternatively, you can enable the verbose flag to cause the testsuite to dump to standard output.

# make check TESTSUITEFLAGS="42 -v"

To enable firewalld’s debug output, you can use the -d flag.

# make check TESTSUITEFLAGS="42 -v -d"

The test numbers passed to TESTSUITEFLAGS are very flexible. It will accept individual numbers or ranges.

For example this is also valid

# make check TESTSUITEFLAGS="42 1-5 110 13-14 17"

Writing a new test case

In this example we’ll follow a test driven development style to fix a real world firewalld bug. We’ll be using Red Hat bugzilla 1404076 for this exercise. This bug occurs when a port range is opened using the --add-port option. --query-port fails to return the expected result if querying a single port within the range.

Basic test layout

All of the tests follow the same basic layout. Since this is a new test case in response to an existing bug we’ll add it to the set of regression tests in src/tests/regression.

At the absolute minimum your new test should look like this

FWD_START_TEST([test description])

...
FWD_CHECK([some command])
...

FWD_END_TEST

The testsuite makes heavy use of m4 macros (FWD_START_TEST, FWD_CHECK, etc.) to simplify test creation. m4 is a macro language which autotest uses to generate the testsuite script.

A consequence of the macro magic is most tests will be run multiple times. Once for each firewall backend; nftables and iptables. This ensures that both backends are tested equally.

The new test case

We start by creating a new test that reproduces the issue from the bugzilla report.

Here is our new test case
src/tests/regression/rhbz1404076.at:

FWD_START_TEST([query single port added with range])

dnl add a set of ports by range, then query a specific port inside that range.
FWD_CHECK([-q --add-port=8080-8090/tcp])
FWD_CHECK([-q --query-port=8085/tcp])
FWD_CHECK([-q --query-port=webcache/tcp]) dnl named port
FWD_CHECK([-q --query-port=8091/tcp], 1) dnl negative test
FWD_CHECK([-q --query-port=8085/udp], 1) dnl negative test

dnl same thing, but for permanent configuration.
FWD_CHECK([-q --permanent --add-port=8080-8090/tcp])
FWD_CHECK([-q --permanent --query-port=8085/tcp])
FWD_CHECK([-q --permanent --query-port=webcache/tcp]) dnl named port
FWD_CHECK([-q --permanent --query-port=8091/tcp], 1) dnl negative test
FWD_CHECK([-q --permanent --query-port=8085/udp], 1) dnl negative test

FWD_END_TEST

Note: dnl is m4’s way of starting a comment.

This test is fairly exhaustive by including negative tests and named ports. This gives us even more confidence in our code changes.

To attach the new test case to the testsuite we need to append the new test to src/tests/regression.at.

$ vi src/tests/regression.at
[...]
m4_include([regression/rhbz1404076.at])

This new test case can be found upstream on github. The commit is 3fb707228ced ("tests/regression: add coverage for rhbz 1404076").

Verifying the new test reproduces the issue

Now that the test has been added you can verify it reproduces the issue by running it against the host’s firewalld using the installcheck make target.

# make installcheck
[...]
regression (nftables)

[...]
 67: query single port added with range              FAILED (rhbz1404076.at:5)
[...]
regression (iptables)

[..]
110: query single port added with range              FAILED (rhbz1404076.at:5)

Here we see the test case failed as expected for both firewall backends.

Fix the bug

The next step is to make code changes to fix the bug. This is out of scope for this post, but for those interested the fix can be found on github. The commit is 2925de324443 ("ports: allow querying a single added by range").

Verify the fix against the in-tree source code

Now that the test case has been written and code changes have been made we can run the test case again, but this time against the source tree version. This will verify our code changes fix the bug. Be sure to use the check make target this time.

# make check TESTSUITEFLAGS="67 110"

Once your new test case passes you should run the whole testsuite again to verify you didn’t introduce a regression.

# make check

If that was successful, then you’re ready to submit your changes upstream with a pull request! Once your pull request is submitted the testsuite will automatically be run again by travis-ci against multiple versions of python.

Conclusions

This post has shown how easy it is to get started with the firewalld testsuite. We hope it will encourage contributions from others while simultaneously improving firewalld’s quality. Additionally we hope it turns some users into active contributors!


firewalld 0.5.4 release

A new release of firewalld, version 0.5.4, is available.

This is a bug fix only release.

  • update translations
  • fw: if failure occurs during startup set state to FAILED
  • fw_direct: avoid log for untracked passthrough queries
  • firewall-config: fix some untranslated strings
  • Rich Rule Masquerade inverted source-destination in Forward Chain
  • don’t forward interface to zone requests to NM for generated interfaces
  • firewall-cmd: add –check-config option
  • firewall-offline-cmd: add –check-config option
  • ipset: check type when parsing ipset definition
  • firewall-config: Add ipv6-icmp to the protocol dropdown box
  • core: logger: Remove world-readable bit from logfile
  • IPv6 rpfilter: explicitly allow neighbor solicitation

Source available here:


firewalld 0.6.1 release

A new release of firewalld, version 0.6.1, is available.

This is a bug fix only release.

  • Correct source/destination in rich rule masquerade
  • Only modify ifcfg files for permanent configuration changes
  • Fix a backtrace when calling common_reverse_rule()
  • man firewalld.conf: Show nftables is the default FirewallBackend
  • firewall-config: fix some untranslated strings that caused a UI bug causing rich rules to not be modify-able
  • services/steam-streaming: replace unicode quotation with ASCII apostrophe
  • fw_direct: avoid log for untracked passthrough queries
  • fixed many issues if iptables is actually iptables-nft
  • Use preferred location for AppData files
  • ipXtables: fix ICMP block inversion with set-log-denied
  • fixes ICMP block inversion with set-log-denied with IndividualCalls=yes
  • nftables: fix set-log-denied if target is not ACCEPT
  • fw_direct: strip _direct chain suffix if using nftables

Improved interactions with NetworkManager:

  • For generated NetworkManager connections, don’t forward request to NetworkManager
  • For non-permanent interface to zone assignments don’t involve NetworkManager
  • Query NetworkManager for permanent interface assignments

testsuite fixes:

  • tests/functions: Remove bashism
  • installcheck: pass PYTHON to testsuite, fixes python tests on distro with mixed python versions
  • testsuite: avoid multiple inclusion warning
  • testsuite: Only enable debug output if testsuite ran with debug flag

Source available here:


nftables backend

Introduction

As noted in the v0.6.0 release announcement, firewalld recently gained support for using nftables as a firewall backend. This post will highlight why that’s a good thing, how it affects firewalld, and how to start using it. The content here may be interesting to intermediate to advanced users of firewalld or anyone generally interested in firewalling.

Why nftables?

nftables in a replacement for all of; iptables, ip6tables, arptables, ebtables, and ipset (henceforth know as “iptables and family”). It brings many advantages, some examples are; built in sets, faster rule updates, and combined ipv4/ipv6 processing. One of the most relevant advantages for firewalld is the ability to maintain all firewall rules through a single interface.

One issue with firewalld’s use of iptables and family is that firewalld assumes complete control of the hosts firewalling. With the nftables backend this is no longer true. Since nftables allows multiple namespaces (tables in nftables vernacular), firewalld will scope all of its rules, sets, and chains to the firewalld table. This will avoid much of the contention with other pieces of software that don’t interact directly with firewalld.

Another nicety of nftables is you can combine logging with verdicts in the same rule. This means you can optimize rules, e.g. log and drop a packet in the same rule. In iptables this would require two rules, both with the same match parameters.

How does firewalld use nftables?

firewalld interacts with nftables directly through the nft binary. This is similar to how firewalld currently interacts with iptables and family. In a future release interaction with nftables will be further improved by using the newly minted libnftables.

As can be seen in the firewalld structure diagram, nftables fits into firewalld alongside the other firewall backends.

firewalld-structure+nftables

All firewalld’s primitives (services, ports, forward ports, etc.) use nftables by default. In addition some translations occur; ipsets will be translated to native nftables sets, and ICMP types are morphed into nftables equivalents.

What about the direct interface?

Seasoned firewalld users may already be asking themselves, What about the direct interface? No worries, it’s still there and works almost exactly as it did in previous releases. When the nftables backend is enabled direct rules are treated specially and still use iptables and family. This means your existing configurations with custom direct rules will continue to work. However, there is one deliberate behavioral change - direct rules take precedence over all other firewalld rules. For further details see the Behavorial Changes section below.

How do I use the nftables backend?

In firewalld 0.6.0 and later nftables is the default backend - so all you have to do is upgrade. The switch over should be transparent to users. The nftables backend has feature parity with the old iptables backend. That means any issues or missing functionality will be treated as bugs.

If for some reason you need to revert to the old iptables backend, you can easily do so by setting FirewallBackend in /etc/firewalld/firewalld.conf to iptables, then restart firewalld. However, please realise that future firewalld development will focus on the nftables backend and not iptables.

This new configuration option is documented in the man pages.

What the future holds

The future of firewalld development will focus on the nftables backend. While the iptables backend is still supported new features won’t necessarily be implemented.

With the nftables backend firewalld has many possible improvements on the horizon:

  • output filtering
  • DoS protection or rate limiting
  • log targets can be combined into the same rule as accept/drop
  • avoid creating empty chains or rules

With all that said, the iptables backend is not going away anytime soon. It will be supported for many, many releases. It’s simply just not the primary focus.

Further information

Here follows further information that may be useful for those wanting some grittier details.

Behavioral Changes

The nftables backend does bring some behavioral changes. In most scenarios these won’t affect users, but may be relevant for more advanced use cases.

Direct interface precedence

In previous versions of firewalld rules added via the direct interface were still subject to firewalld’s rules that occur early in the ruleset. Most notably is the early acceptance of packets that are part of existing connections. This has led to confusion for users, for example see github issue 44.

With the nftables backend direct rules are deliberately given a higher precedence than all other firewalld rules. Now when a user adds a direct rule to drop traffic existing connections will also be affected.

Firewalld hook priority

nftables allows multiple chains to hook into netfilter at the same point. This is in contrast to iptables which only allowed one and the concept was completely hidden from users. firewalld takes advantage of this feature to add predictability by giving firewalld’s rules slightly lower precedence than iptables and default nftables hook priority values. Without this it would be undefined if firewalld’s chains would execute before or after other iptables/nftables chains.

The main consequence for users is that firewall rules created outside of firewalld (e.g. libvirt, docker, user, etc) will take precedence over firewalld’s rules.

Only flush firewalld’s rules

Since nftables allows namespaces (via tables) firewalld no longer does a complete flush of firewall rules. It will only flush rules in the firewalld table. This avoids scenarios where custom user rules or rules installed by other tools are unexpectedly wiped out when firewalld was restarted or reloaded.

Packet accept/drop precedence

As mentioned above, nftables allows multiple chains to use the same netfilter hook. A consequence of this is that packets that are accepted are still subject to the rules of other chains hooked into the hook type. For firewalld this means packets may be accepted early by custom iptables or nftables rules, but will still be subject to firewalld’s rules. In the drop case processing always stops immediately and no other hooks will process the packet.


firewalld 0.6.0 release

A new release of firewalld, version 0.6.0, is available.

This is a large feature release. Some new features warrant a separate blog post. As such, the release is only summarized here.

User facing features:

  • nftables backend
    This is the new default for all firewalld’s abstractions. The direct interface still supports iptables, ip6tables, and ebtables. It is configurable via FirewallBackend in /etc/firewalld/firewalld.conf - valid values are; nftables, iptables.
  • new services
    apcupsd, cockpit, distcc, etcd, finger, iSNS, llmnr, matrix, mqtt, nut, plex, rtsp, salt-master, samba-dc, slp, steam, subversion, svdrp, wbem-http, wsman
  • updated translations

A lot of development time was spent on improving continuous integration and sanity checks to give confidence in new code changes and contributions from others. This is reflected below in the list of developer focused improvements.

Developer focused improvements:

  • lots of code refactoring to abstract firewall backend
  • flake8 source code checking
  • better debug output on exceptions (tracebacks)
  • testsuite runs inside network namespaces
  • improved testsuite coverage

New dependencies for nftables backend:

  • nftables >= 0.9.0
  • linux >= 4.18

While most of the nftables backend will function with earlier versions of nftables and Linux it is not recommended. Many bugs were found and fixed in these packages while firewalld’s nftables backend was being developed. Some examples are; iptables and nftables NAT coexistence, nftables AUDIT support, nftables set ranges with timeouts.


Source available here: