libnftables JSON

Sep 13, 2019 • Eric Garver

A future firewalld feature release (0.8.0) will contain a significant change to the nftables backend. In the past firewalld has always called the nft binary. As of git commit 1582c5dd736a firewalld will instead use the libnftables library. This is the same library that nft uses internally. All the communication with the library is a through a thin python wrapper and uses structured JSON. This means firewalld never calls the nft binary.

Why it’s good

Outside of using structured JSON there are a couple other advantages:

  • faster rule application
    firewalld used to make a call to nft for every rule. With libnftables it’s now passing large blobs of JSON that can contain hundreds of rules. This reduces the numbers of round trips to the kernel and yields a significant performance improvement.
  • no fork()ing
    Each call to nft would be a matter of forking a new process and waiting on the result. With libnftables there is no more forking.
  • atomic transactions
    Internally firewalld builds a set of changes and attempts to apply them all at once. The old nft backend still had to call nft once for every rule so this transaction wasn’t atomic and there were small periods of time between the rules being applied. This time frame is small, but can cause real issues in some scenarios. With libnftables the rules are applied in an “all or nothing” manner. Either they all take effect at once or not at all.

What it looks like

From the user’s perspective nothing has changed. With the exception that firewalld now has a new dependency.

From a developer’s point of view rule generation looks quite different. Instead of building strings that are identical to the nft CLI firewalld needs to generate a python dictionary containing other dictionaries and lists. This will then be converted to JSON before being passed to libnftables.

For example the following is an example rule from before libnftables.

"add rule inet %s filter_%s ct state established,related accept" % (TABLE_NAME, "INPUT"))

And here is the equivalent python dictionary.

{"add": {"rule":  {"family": "inet",
                   "table": TABLE_NAME,
                   "chain": "filter_%s" % "INPUT",
                   "expr": [{"match": {"left": {"ct": {"key": "state"}},
                                       "op": "in",
                                       "right": {"set": ["established", "related"]}}},
                            {"accept": None}]}}}

It’s a definitely more verbose, but the nested dictionaries are easier to work with.

You can learn more about libnftables JSON by reading the man page libnftables-json(5). It covers the basics and the full syntax.

python-nftables

As briefly mentioned above, firewalld uses a thin python wrapper around libnftables. This is shipped with the nftables package and is being referred to as python-nftables by the firewalld developers.

Requirements

Short answer: nftables >= 0.9.3

Long answer:

With this change firewalld has new dependencies; libnftables, and python-nftables. Currently both libnftables and python-nftables are shipped with the standard nftables package. Distribution packagers shouldn’t have to change much. It really amounts to calling the nftables configure script with --with-python --with-json.

libnftables has been around for awhile, but due to bug fixes and other changes the firewalld developers are claiming the libnftables backend only works with nftables >= 0.9.3.

firewalld 0.7.1 release

Jul 22, 2019 • Eric Garver

A new release of firewalld, version 0.7.1, is available.

This is a bug fix only release.

  • fix: firewall-offline-cmd: service: use dict based APIs
  • fix: client: service: use dict based dbus APIs
  • fix: dbus: new dict based APIs for services
  • fix: dbus: add missing APIs for service includes
  • fix: dbus: fix service API break
  • fix: CLI: show service includes with –info-service

Source available here:

firewalld 0.7.0 release

Jun 11, 2019 • Eric Garver

A new release of firewalld, version 0.7.0, is available.

This is a feature release. It also includes all bug fixes since v0.6.0.

New features:

  • Rich Rule Priorities
  • Service Definition Includes
    Service definitions can now include lines like: <include service="https"/> which will include all the ports, etc from the https service.
  • RFC3964 IPv4 filtering
    A new option RFC3964_IPv4 in firewalld.conf is available. It does filtering based on RFC3964 in regards to IPv4 addresses. This functionality was traditionally in network-scripts.
  • FlushAllOnReload
    A new option FlushAllOnReload in firewalld.conf is available. Older releases retained some settings (direct rules, interface to zone assignments) during a --reload. With the introduction of this configuration option that is no longer the case. Old behavior can be restored by setting FlushAllOnReload=no.
  • 15 new service definitions

Statistics since v0.6.0:

  • 266 commits
  • 111 files changed, 4752 insertions(+), 1986 deletions(-)

Source available here:

  • Tarball: firewalld-0.7.0.tar.gz
  • SHA256: da872394ecdc6584fbcefb8044a9a5492f9a3176e864e31b3b082d0b79e5e755
    Note: This release tarball was updated due to missing distfiles. Old hash was bef3e555d99fba51487095e6977aed7cfd582a7ab8505e6d7335e6e833ea42a1.
  • Complete changelog on github: 0.6.0 to 0.7.0

firewalld 0.6.4 release

May 17, 2019 • Eric Garver

A new release of firewalld, version 0.6.4, is available.

This is a bug fix only release.

  • treewide: fix over indentation (flake8 E117)
  • test: travis: add another test matrix for omitting ip6tables
  • chore: travis: split test matrix by keywords
  • chore: tests: add AT_KEYWORDS for firewall-offline-cmd
  • improvement: tests: Use AT_KEYWORDS for backends
  • fix: tests: guard occurrences of IPv6
  • fix: tests/functions: ignore warnings about missing ip6tables
  • test: add macro IF_IPV6_SUPPORTED
  • test: add macro HOST_SUPPORTS_IP6TABLES
  • test: pass IPTABLES make variables down to autotest
  • fix: avoid calling backends that aren’t available
  • fix: tests/regression/rhbz1601610: ignore warning about version mismatch
  • fix: tests/regression/pr323: don’t check for nf_nat_proto_gre
  • fix: do not flush entire ruleset in CHECK_NAT_COEXISTENCE
  • fix: propagate exception if backend fails with IndividualCalls=yes
  • fix: tests nftables: constant set compat between releases
  • fix: document –check-config option
  • fix: tests/nftables: compatibility with numeric output changes
  • test/functions: Strip nft hook and policy from output
  • tests/functions: normalize nft list rule output
  • fix: on reload, set policy before cleanup
  • fix: tests/regression/gh453: guarantee automatic helpers disabled
  • test: add macro CHECK_NFT_CT_HELPER()
  • test: add test to check for nftables helper objects
  • fix: nftables: make helpers work by creating ct helper objects
  • fix: ipXtables: using “mangle” in zone not dependent on “nat”
  • fix: ipXtables: don’t use tables that aren’t available
  • test: add tests for rich rule mark action
  • fix: nftables rich rule mark not marking every packet
  • rich rules: fix Rich_Mark logic
  • test: add coverage for gh #482
  • fix: rich rule forward-port deletion after reload
  • tests/regression: coverage for enabling IP forwarding via forward-ports
  • fw_zone: forward-ports: only enable IP forwarding if toaddr used
  • doc: note that masquerade will enable IP forwarding
  • doc: note that forward-port may enable IP forwarding
  • fix issue #457
  • ipXtables: Avoid inserting rules with index
  • ipXtables: simplify rpfilter rule generation
  • tests/functions: normalize ebtables inversion output
  • tests/firewall-cmd: Coverage for interface wildcarding
  • nftables: Allow interfaces with wildcards
  • tests/firewall-cmd: remove redundant checks for TESTING_FIREWALL_OFFLINE_CMD
  • tests/functions: for list macros skip if testing firewall-offline-cmd
  • tests/functions: m4_strip expected output
  • tests/functions: implement a better m4_strip()
  • tests/regression/rhbz1601610: modify test to satisfy buggy kernel versions
  • ipset: fix set apply if IndividualCalls=yes
  • nftables: fix ipv6 rich rule forward-ports
  • nftables: fix rich rule masquerade
  • fw_zone: fix IPv6 rich rule forward-port without toaddr
  • fw_zone: fix rich rule masquerading
  • firewalld.spec: enforce nftables version requirements
  • firewalld.spec: fix packaging of appdata
  • nftables: fix panic mode not filtering output packets
  • services/steam-streaming: update udp ports
  • zanata: use version stable-0.6
  • update translations
  • rich rules: fix mark action
  • tests/regression/rhbz1571957: exercise log-denied=broadcast
  • ipXtables/nftables: Fix “object has no attribute ‘_log_denied’”
  • config/lockdown-whitelist: Don’t auto add “-Es” to interpreter

Source available here:

Rich Rule Priorities

Dec 18, 2018 • Eric Garver

Recently firewalld gained support for a priority field in the rich rule syntax. It allows fine grained control over rich rules and their execution order. This enables using rich rules in ways not possible before.

Why is it needed?

One issue with current rich rules is that they are organized based on their rule action. Log always occurs before deny. Deny always occurs before allow. This has led to confusion from users as this implicitly reorders rules. It also made it impossible to add a catch-all rich rule to deny traffic.

More information on this can be found in the firewalld.richlanguage man page in the section “Information about logging and actions”.

What does it look like?

The syntax modifications add a new priority field. This can be any number between -32768 and 32767, where lower numbers have higher precedence. This range is large enough to allow automatic rule generation from scripts or other entities.

Example:

# firewall-cmd --add-rich-rule='rule priority=1234 service name="mdns" allow'

Based on the priority rules are organized into different chains.

  • If priority < 0, the rule goes into a chain with the suffix _pre.
  • If priority > 0, the rule goes into a chain with the suffix _post.
  • If priority == 0, the rule goes into a chain ( _log, _deny, _allow ) based on their action. This is the same behavior as rich rules before priority support.

Inside these sub-chains rules are sorted according to their priority value. If they have the same priority value, then it’s undefined in what order they will be executed.

Putting it all together a zone’s set of chains now looks like below:

# nft list chain inet firewalld filter_IN_public
table inet firewalld {
        chain filter_IN_public {
                jump filter_IN_public_pre
                jump filter_IN_public_log
                jump filter_IN_public_deny
                jump filter_IN_public_allow
                jump filter_IN_public_post
                meta l4proto { icmp, ipv6-icmp } accept
        }
}

A couple key points from this layout:

  • _pre can occur before normal log rules.
  • _post execution always occurs after firewalld’s other primitives (services, ports, etc). This makes it a good place for catch-all type rules.
  • _pre and _post chains may contain rich rules with any type of action (accept, deny, log, audit, etc)

Examples (use cases)

Below are some examples, but they don’t even scratch the surface of what’s possible now that rich rules support arbitrary ordering.

Log all traffic not caught by other rules

Using a very low precedence rich rule you can log all traffic that has not yet been denied or accepted. This is useful to flag any unexpected traffic. It can also be a way to implement the zone level equivalent to –log-denied.

# firewall-cmd --add-rich-rule='rule priority=32767 log prefix="UNEXPECTED: " limit value="5/m"'

This results in the following:

# nft list chain inet firewalld filter_IN_public_post
table inet firewalld {
        chain filter_IN_public_post {
                log prefix "UNEXPECTED: " limit rate 5/minute
        }
}

Special policy for subset of traffic

To mimic a policy for only a subset of source addresses you can use a low precedence rule.

# firewall-cmd --add-rich-rule='rule family="ipv4" priority=32767 source address="10.1.1.0/24" reject'

This results in the following:

# nft list chain inet firewalld filter_IN_public_post
table inet firewalld {
		chain filter_IN_public_post {
				ip saddr 10.1.1.0/24 reject
		}
}

Allow a service for a subset of sources

This example allows a service for a subset of sources, then logs and denies it for everyone else.

# firewall-cmd --add-rich-rule='rule family="ipv4" priority=-100 source address="10.1.1.0/24" service name="ssh" accept'
# firewall-cmd --add-rich-rule='rule priority=-99 service name="ssh" log'
# firewall-cmd --add-rich-rule='rule priority=-98 service name="ssh" reject'

This results in the following:

# nft list chain inet firewalld filter_IN_public_pre
table inet firewalld {
        chain filter_IN_public_pre {
                ip saddr 10.1.1.0/24 tcp dport 22 ct state new,untracked accept
                tcp dport 22 ct state new,untracked log
                tcp dport 22 ct state new,untracked reject
        }
}

Compatibility

To maintain compatibility rich rules that have a priority == 0 or an absent priority will behave as they’ve done in the past. They’ll be sorted into the _log, _deny, and _allow chains based on their action.

When will they be available?

Rich rules with priority support will be available in the next minor firewalld release, which will most likely be v0.7.0. However, the feature may be backported to distributions that do that sort of thing.