Version 3.31 of syslog-ng has been released recently. One of its most user-visible features is the parser for Fortigate logs, yet another networking vendor that produces log messages not conforming to syslog specifications. Parsing Fortigate logs builds upon the new no-header flag of syslog-ng combined with the key-value and date parsers. Other features include a new silent message option for the Telegram destination and automatic directory creation for disk-buffer files.

Note: as you could guess from the previous paragraph, Fortigate is not alone. Cisco also has “interesting” log messages, and a bit of extra parsing also helps with PAN-OS, even if their messages conform to syslog specifications.

Before you begin

In this blog, we check out some of the new features of syslog-ng 3.31. If you want to test them, you need to install syslog-ng 3.31 or later (I’m running a Git snapshot version on my laptop). It is available already in FreeBSD ports, but most Linux distributions carry older versions. Check the 3^rd-party binaries page on the syslog-ng website if there are up-to-date 3^rd-party packages for your distribution of choice: https://www.syslog-ng.com/3rd-party-binaries

Once you have syslog-ng 3.31 up and running, you are ready to test some of the new features. Let’s start with some of the smaller features and finish with the big one!

Disk buffer directory

Prior to version 3.31 of syslog-ng, you had to create a directory for disk buffer files manually before starting syslog-ng. If the directory was missing, syslog-ng stopped with an error. In my latest syslog-ng article on opensoure.com, where syslog-ng on a Raspberry Pi machine buffers sensor data until the server becomes available, I also warn you to create the directory.

Starting with version 3.31, if the directory specified in the configuration does not exist, syslog-ng creates it automatically.

Silent Telegram messages

Sending alerts to Telegram from syslog-ng has been possible for years already. But there are some situations when you want to receive all the interesting messages on your mobile without being disturbed each time a new message arrives. This is where the new disable_notification() option to the Telegram destination can come handy. If set to `True`, your mobile receives the new messages without alerting you each time.

“no-header” flag for the syslog parser

Some log messages only keep the PRI field from the syslog specification, but they lack a proper syslog header. Among others, Fortigate logs belong to these log messages. In this case, you can use the “no-header” flag on incoming log messages. This way, syslog-ng parses the PRI field, and puts the rest of the log message into $MSG. If the message is otherwise structured, you can parse it further, extract the date using the date parser, and create name-value pairs from other useful information.

Working with Fortigate logs

Here are some sample Fortigate log messages. These were used to test the new parser, and we will use them as well. You can copy and paste them from here or from /usr/share/syslog-ng/include/scl/fortigate/fortigate.conf as well:

<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683498829 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413214 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1610704739683525562 tz="+0300" srcip=10.9.1.26 srcname="sotina-sv" srcport=61105 srcintf="9 VLAN" srcintfrole="lan" dstip=77.88.55.66 dstname="www.yandex.ru" dstport=443 dstintf="wan1" dstintfrole="wan" sessionid=2364410752 proto=6 action="close" policyid=42 policytype="policy" poluuid="c1e5431a-d082-51e7-53e0-d3a8ab1a3ee2" service="HTTPS" dstcountry="Russian Federation" srccountry="Reserved" trandisp="snat" transip=213.59.243.9 transport=61105 appid=42899 app="Yandex" appcat="General.Interest" apprisk="elevated" applist="Application Control_User" duration=15 sentbyte=7286 rcvdbyte=1490 sentpkt=16 rcvdpkt=8 wanin=1158 wanout=6470 lanin=6470 lanout=6470 utmaction="allow" countapp=1 osname="Windows" srcswversion="7" unauthuser="sotina-sv" unauthusersource="kerberos" mastersrcmac="00:24:21:ac:fb:da" srcmac="00:24:21:ac:fb:da" srcserver=0
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683532607 tz="+0300" srcip=94.143.50.155 srcname="94.143.50.155" srcport=56368 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413216 proto=6 action="deny" policyid=0 policytype="local-in-policy" service="tcp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="tcp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"

As you can see, these lines start as any other log messages start, with a <PRI> field, but otherwise, they lack a proper syslog header. All further data is described as name-value pairs.

When we send this log to a port without Fortigate parsing enabled, the results are far from ideal. The date and time of the message will reflect the current time instead of the time encoded in the log message:

Mar 12 15:14:21 localhost date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"

By using the fortigate-parser(), we can extract the date from the message and also create name-value pairs from the message. This enables easier filtering (alerting), saving only specific fields to save space, or storing specific fields to an SQL or noSQL database, like Elasticsearch.

Luckily, there is no need to configure the fortigate-parser() yourself. Unless you have a high message rate, where even the smallest extra processing counts, using the default-network-drivers() of syslog-ng is easier. It parses Fortigate logs, and many other message types automatically. For this scenario and for other debugging ideas, check the Cisco parser blog at https://www.syslog-ng.com/community/b/blog/posts/parsing-cisco-logs-in-syslog-ng

The following configuration uses the default-network-drivers with a JSON-formatted destination file. This way, you can see that syslog-ng parses the logs and creates name-value pairs from them.

source s_net {
    default-network-drivers();
};
template t_jsonfile {
    template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};
destination d_fromfortigate {
    file("/var/log/fromfortigate" template(t_jsonfile));
};
log {
    source(s_net);
    destination(d_fromfortigate);
};

The source is default-network-drivers() and it listens on TCP port 514. The template uses JSON formatting, includes any syslog fields and removes the leading dots from name-value pairs (by default syslog-ng parsers create names that start with a dot, but it can cause weird problems for example with Elasticsearch) before storing them. Then, apply this template to a destination file. Finally, a log statement connects the source to the destination.

Using netcat, you can send the test messages to the network source on port 514 and check the results:

czplaptop:~ # cat fortigate.txt | netcat -4 -n -N -v 127.0.0.1 514
Connection to 127.0.0.1 514 port [tcp/*] succeeded!
czplaptop:~ # cat /var/log/fromfortigate
{"fortigate":{"vd":"root","tz":"+0300","type":"traffic","trandisp":"noop","time":"12:58:59","subtype":"local","srcport":"45295","srcname":"91.234.154.139","srcip":"91.234.154.139","srcintfrole":"wan","srcintf":"wan1","srccountry":"Russian Federation","sessionid":"2364413215","service":"udp/46730","sentpkt":"0","sentbyte":"0","rcvdbyte":"0","proto":"17","policytype":"local-in-policy","policyid":"0","logid":"0001000014","level":"notice","eventtime":"1610704739683510055","duration":"0","dstport":"46730","dstname":"213.59.243.9","dstip":"213.59.243.9","dstintfrole":"undefined","dstintf":"unknown0","dstcountry":"Russian Federation","devname":"FORTI_111","devid":"FG100D3G12801312","date":"2021-01-15","crscore":"5","crlevel":"low","craction":"262144","appcat":"unscanned","app":"udp/46730","action":"deny"},"app":{"name":"fortigate"},"SOURCE":"s_net","PRIORITY":"notice","MESSAGE":"date=2021-01-15 time=12:58:59 devname=\"FORTI_111\" devid=\"FG100D3G12801312\" logid=\"0001000014\" type=\"traffic\" subtype=\"local\" level=\"notice\" vd=\"root\" eventtime=1610704739683510055 tz=\"+0300\" srcip=91.234.154.139 srcname=\"91.234.154.139\" srcport=45295 srcintf=\"wan1\" srcintfrole=\"wan\" dstip=213.59.243.9 dstname=\"213.59.243.9\" dstport=46730 dstintf=\"unknown0\" dstintfrole=\"undefined\" sessionid=2364413215 proto=17 action=\"deny\" policyid=0 policytype=\"local-in-policy\" service=\"udp/46730\" dstcountry=\"Russian Federation\" srccountry=\"Russian Federation\" trandisp=\"noop\" app=\"udp/46730\" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat=\"unscanned\" crscore=5 craction=262144 crlevel=\"low\"","ISODATE":"2021-01-15T12:58:59+01:00","HOST_FROM":"localhost","HOST":"localhost","FACILITY":"local7","DATE":"Jan 15 12:58:59"}

[…]

You can see all the name-value pairs extracted from the log messages, including a proper date as found in the log message.

What is next?

These were just my highlights from the 3.31 syslog-ng release. For the complete list of changes, check https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.1 where you might find other new features or bugfixes that might convince you to upgrade syslog-ng on your hosts.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.

By default, openSUSE Leap 15.x is using the firewalld firewall implementation (and the firewalld backend is using iptables under the hood).

But since a while, openSUSE also has nftables support available - but neither YaST nor other special tooling is currently configured to directly support it. But we have some machines in our infrastructure, that are neither straight forward desktop machines nor do they idle most of the time. So let’s try out how good we are at trying out and testing new things and use one of our central administrative machines: the VPN gateway, which gives all openSUSE heroes access to the internal world of the openSUSE infrastructure.

This machine is already a bit special:

• The “external” interface holds the connection to the internet
• The “private” interface is inside the openSUSE heroes private network
• We run openVPN with tun devices (one for udp and one for tcp) to allow the openSUSE heroes to connect via a personal certificate + their user credentials
• In addition, we run wireguard to connect the private networks in Provo and Nuremberg (at our Sponsors) together
• And before we forget: our VPN gateway is not only a VPN gateway: it is also used as gateway to the internet for all internal machines, allowing only ‘pre-known traffic’ destinations

All this makes the firewall setup a little bit more complicated.

BTW: naming your interfaces by giving them explicit names like “external” or “private”, like in our example, has a huge benefit, if you play along with services or firewalls. Just have a look in /etc/udev/rules.d/70-persistent-net.rules once your devices are up and rename them according to your needs (you can also use YaST for this). But remember to also check/rename the interfaces in */etc/sysconfig/network/ifcfg-** to use the same name before rebooting your machine. Otherwise your end up in a non-working network setup.

Let’s have a short look at the area we are talking about:

As you hopefully notice, none of the services on the community side is affected. There we have standard (iptables) based firewalls and use proxies to forward user requests to the right server.

On the openSUSE hero side, we exchanged the old SuSEfirewall2 based setup with a new one based on nftables.

There are a couple of reasons that influenced us in switching over to nftables:

• the old SuSEfirewall2 worked, but generated a huge iptables list on our machine in question
• using ipsets or variables with SuSEfirewall2 was doable, but not an easy task
• we ran into some problems with NAT and Masquerading using firewalld as frontend
• Salt is another interesting field:
  • Salt’ing SuSEfirewall2 by deploying some files on a machine is always possible, but not really straight forward
  • there is no Salt module for SuSEfirewall2 (and there will probably never be one)
  • there are Salt modules for firewalld and nftables, both on nearly the same level
• nftables is integrated since a while in the kernel and should replace all the *tables modules long term. So why not jumping directly to it, as we (as admins) do not use GUI tools like YaST or firewalld-gui anyway?

So what are the major advantages?

1. Sets are part of the core functionality. You can have sets of ports, interface names, and address ranges. No more ipset. No more multiport. ip daddr { 1.1.1.1, 1.0.0.1 } tcp dport { dns, https } oifname { "external", "wg_vpn1" } accept; This means you can have very compact firewall sets to cover a lot of cases with a few rules.
2. No more extra rules for logging. Only turn on counter where you need it. counter log prefix "[nftables] forward reject " reject
3. You can cover IPv4 and IPv6 with a single ruleset when using table inet, but you can have per IP protocol tables as well. And sometimes even need them e.g. for postrouting.

Starting from scratch

A very basic /etc/nftables.conf would look something like this

#!/usr/sbin/nft -f

flush ruleset

# This matches IPv4 and IPv6
table inet filter {
    # chain names are up to you.
    # what part of the traffic they cover, 
    # depends on the type line.
	chain input {
		type filter hook input priority 0; policy accept;
	}
	chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

But so far we did not stop or allow any traffic. Well actually we let everything in and out now because all chains have the policy accept.

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain base_checks {
        ## another set, this time for connection tracking states.
        # allow established/related connections
        ct state {established, related} accept;

        # early drop of invalid connections
        ct state invalid drop;
    }

   	chain input {
		type filter hook input priority 0; policy drop;
        
        # allow from loopback
        iif "lo" accept;

        jump base_checks;

        # allow icmp and igmp
        ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, packet-too-big, time-exceeded, parameter-problem, destination-unreachable, packet-too-big, mld-listener-query, mld-listener-report, mld-listener-reduction, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept;
        ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept;
        ip protocol igmp accept;
        
        # for testing reject with logging
        counter log prefix "[nftables] input reject " reject;
	}
    chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

You can activate the configuration with nft --file nftables.conf, but do NOT do this on a remote machine. It is also a good habit to run nft --check --file nftables.conf before actually loading the file to catch syntax errors.

So what did we change?

1. most importantly we changed the policy of the chain to drop and added a reject rule at the end. So nothing gets in right now.
2. We allow all traffic on the localhost interface.
3. The base_checks chain handles all packets related to established connections. This makes sure that incoming packets for outgoing connections get through.
4. We allowed important ICMP/IGMP packets. Again this is using a set and the type names and not some crtyptic numbers. YAY for readability.

Now if someome tries to do a ssh connect to our machine, we will see:

[nftables] input reject IN=enp1s0 OUT= MAC=52:54:00:4c:51:6c:52:54:00:73:a1:57:08:00 SRC=172.16.16.2 DST=172.16.16.30 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22652 DF PROTO=TCP SPT=55574 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0

and nft list ruleset will show us

counter packets 1 bytes 60 log prefix "[nftables] input reject " reject

So we are secure now. Though maybe allowing SSH back in would be nice. You know just in case. We have 2 options now. Option 1 would be to insert the following line before our reject line.

tcp dport 22 accept;

But did we mention already that we have sets and that they are great? Especially great if we need the same list of ports/ip ranges/interface names in multiple places?

We have 2 ways to define sets:

define wanted_tcp_ports {
  22,
}

Yes the trailing comma is ok. And it makes adding elements to the list easier. So we do them all the time. This will change our rule above to

tcp dport $wanted_tcp_ports accept;

If we load the config file and run nft list ruleset, we will see:

tcp dport { 22 } accept

But there is actually a slightly better way to do this:

    set wanted_tcp_ports {
        type inet_service; flags interval;
        elements = {
           ssh
        }
    }

That way our firewall rule becomes:

tcp dport @wanted_tcp_ports accept;

And if we dump our firewall with nft list ruleset afterwards it will still be shown as @wanted_tcp_ports and not have variable replaced with the value. While this is great already, the 2nd syntax actually has one more advantage.

$ nft add element inet filter wanted_tcp_ports \{ 443 \}

Now our wanted_tcp_ports list will allow port 22 and 443. This is of course often more useful if we use it with IP addresses.

    set fail2ban_hosts {
        type ipv4_addr; flags interval;
        elements = {
           192.168.0.0/24
        }       
    }

Let us append some elements to that set too.

$ nft add element inet filter fail2ban_hosts \{ 192.168.254.255, 192.168.253.0/24 \}
$ nft list ruleset

… and we get …

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

Now we could change fail2ban to append elements to the set instead of creating a new rule for each new machine it wants to block. Fewer rules. Faster processing.

But with reloading the firewall we dropped port 443 from the port list again. Oops. Though … if you are happy with the rules. You can just run

$ nft list ruleset > nftables.conf

When you are using all the sets instead of the variables, all your firewall rules will still look nice.

Our complete firewall looks like

table inet filter {
        set wanted_tcp_ports {
                type inet_service
                flags interval
                elements = { 22, 443 }
        }

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

        chain base_checks {
                ct state { established, related } accept
                ct state invalid drop
        }

        chain input {
                type filter hook input priority filter; policy drop;
                iif "lo" accept
                jump base_checks
                ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
                ip protocol icmp icmp type { echo-reply, destination-unreachable, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept
                ip protocol igmp accept
                tcp dport @wanted_tcp_ports accept
                counter packets 12 bytes 828 log prefix "[nftables] input reject " reject
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

For more see the nftables wiki

Growing up, an influential television character for me was Jean Luc Picard from Star Trek the Next Generation.

Picard was a portrayal of a different sort of leader to most. Picard didn’t order people about. He didn’t assume he knew best.  He wasn’t seeking glory. In his words: “we work to better ourselves, and the rest of humanity”. The Enterprise was a vehicle for his crew to better themselves and society. What a brilliant metaphor for organisations to aspire to.

My current job title is “Director of engineering”. I kind of hate it. I don’t want to direct anybody. I represent them as part of my first team. People don’t report to me; I support people. My mission is to clarify so they can decide for themselves, and to help them build skills so they can succeed. 

Director is just a word, but words matter. Language matters.

Picard was an effective leader in part due to the language he used. Here’s a few lessons we can learn from the way he talked.

“Make it So!”

Picard is probably best known for saying “make it so!”. 

This catchphrase says so much about his leadership style. He doesn’t bark orders. He gives the crew the problems to solve. He listens to his crew and supports their ideas. His crew state their intent and he affirms (or not) their decisions. 

I think “Make it so” is even more powerful than the more common “very well” or “do it”, which are merely agreeing with the action being proposed.

“Make it so” is instead an agreement with the outcome being proposed. The crew are still free to adjust their course of action to achieve that outcome. They won’t necessarily have to come back for approval if they have to change their plan to achieve the same outcome. They understand their commander’s intent.  

And of course it’s asking for action “wishing for a thing does not make it so”

“Oh yes?” 

Picard’s most common phrase was not affirming decisions, but “oh yes”. Because he was curious, he would actively listen to his crew. He sought first to understand.  

It’s fitting that he says this more than make it so. Not everything learned requires action. It’s easy to come out of retrospectives with a long list of actions. I’d rather we learned something and took no actions than took action without learning anything. 

“Suggestions?” 

In “Cause and Effect” (The one with Captain Fraiser Crane) there’s an imminent crisis. The Enterprise is on a collision course with another ship and are unable to maneuver. What does Picard do? Resort to command and control? No; despite the urgency he asks for suggestions from the crew. Followed by a “make it so” agreement to act.

Asking for suggestions during a crisis requires enough humility to realise your crew or team is collectively smarter than you are. Picard trusts his team to come up with the best options. 

He is also willing to show vulnerability, even during a crisis. His ego doesn’t get in the way. 

In this episode, the crew did not automatically follow the suggestion of the most senior person in the room. The solution to the crisis is eventually found in the second suggestion, after they tried the first. They succeeded because they had diverged and discovered options first, before converging on a solution.

The crew were aware of the other options open to them, and when the first failed they acted to try another (successful) option. Crucially, they did not wait for their captain to approve it. There wasn’t time to let the captain make the decision, but they were free to try the other options because they’d been told to “make it so” not “do it”.

To achieve the best outcomes as a team, intentionally seek divergent ideas first before converging on a decision. Avoid converging too soon on an idea that sounds promising, when others in the group may have better ideas. If your first choice doesn’t work out you will be able to try the other ideas that came up. 

“Nicely done!”

Picard made sure his crew knew when he thought they had done well. Even when they violated orders! He was not one to blindly follow orders himself and he praised his crew when they violated orders for good reasons.

Data’s violation of orders is met not with a reprimand but a “nicely done!”; when Data questions it Picard responds “The claim ‘I was only following orders’ has been used to justify too many tragedies in our history”

How different might the tech industry be if more people carefully considered whether doing what they’ve been told is the right or wrong thing to do? 

The post Leadership Language Lessons from Star Trek appeared first on Benji's Blog.

Java 16 brings Pattern Matching for instanceof. It’s a feature with exciting possibilities, though quite limited in its initial incarnation. 

Basic Pattern Matching

We can now do things like 

Object o = "hello";
if (o instanceof String s) {
    System.out.println(s.toUpperCase());
}

Note the variable “s”, which is then used without any casting needed.

Deconstructing Optionals

It would be even nicer if we could deconstruct structured types at the same time. I think it’s coming in a future version, but I’m impatient. Let’s see how close we can get with the current tools.

One thing we can do is use a function on the left hand side of this expression.

Let’s consider the example of dealing with unknown values in Java. There’s a fair amount of legacy to deal with. Sometimes you’ll get a value, sometimes you’ll get a null. Sometimes you’ll get an Optional<T> , sometimes you’ll get an Optional<Optional<T>> etc.

How can we make unknowns nicer to deal with?

We could create a little utility function that lets us convert all of these forms of unknown into either a value or not. Then match with an instanceof test.

Object unknown = Optional.of("Hello World");
assertEquals(
       "hello world",
       unwrap(unknown) instanceof String s
               ? s.toLowerCase()
               : "absent"
);

Thanks to instanceof pattern matching we can just use the string directly, without having to resort to passing method references i.e optional.map(String::toLowercase) 

The unwrap utility itself uses pattern matching against Optional to recursively unwrap values from nested optionals. It also converts nulls and Optional.empty() to a non-instantiatable type for the purposes of ensuring they can never match the above pattern.

static Object unwrap(Object o) {
   if (o instanceof Optional> opt) {
       return opt.isPresent() ? unwrap(opt.get()) : None;
   } else if (o != null) {
       return o;
   } else {
       return None;
   }
}
static class None {
   private None() {}
   public static final None None = new None();
}

Here’s several more examples, if you’d like to explore further.

Deconstructing Records

What about more complex structures? Now that we have record types, wouldn’t it be great if we can deconstruct them to work with individual components more easily. I think until more powerful type patterns exist in the language we’ll have to diverge from the instanceof approach. 

I previously showed how we could do this for records we control, by having them implement an interface. What about records we do not control? How can we deconstruct those? 

This is about the closest I can get to what I’d hope would be possible as a first class citizen in the language in future.

record Name(String first, String last) {}
Object name = new Name("Benji", "Weber");
If.instance(name, (String first, String last) -> {
   System.out.println(first.toLowerCase() + last.toLowerCase()); // prints benjiweber
});

It takes a record (Name) and a lambda where the method parameters are of the same types as the component types in the record. It deconstructs the record component parts and passes them to the lambda to use (assuming the record really matches).

We could also use as an expression to return a value, as long as we provide a fallback for the case when the pattern does not match.

Object zoo = new Zoo(new Duck("Quack"), new Dog("Woof"));

String result = withFallback("Fail").
    If.instance(zoo, (Duck duck, Dog dog) ->
       duck.quack() + dog.woof()
    ); // result is QuackWoof

So how does this work? 

If.instance is a static method which takes an Object of unknown type (we hope it will be a Record), and a lambda function that we want to pattern match against the provided object.

How can we use a lambda as a type pattern? We can use the technique from my lambda type references article—have the lambda type be a SerializableLambda which will allow us to use reflection to read the types of each parameter. 

static  void instance(Object o, MethodAwareTriConsumer action) { 

}

So we start with something like the above, a method taking an object and a reflectable lambda function.

Next we can make use of pattern matching again to check if it’s a record.

static  void instance(Object o, MethodAwareTriConsumer action) {
   if (o instanceof Record r) {
	// now we know it's a record
   }
}

Records allow reflection on their component parts. Let’s check whether we have enough component parts to match the pattern.

static  void instance(Object o, MethodAwareTriConsumer action) {
   if (o instanceof Record r) {
       if (r.getClass().getRecordComponents().length < 3) {
           return;
       }
       
	 // at this point we have a record with enough components and can use them.
   }
}

Now we can invoke the passed action itself 

action.tryAccept((T) nthComponent(0, r), (U) nthComponent(1, r), (V) nthComponent(2, r));

Where nthComponent uses reflection to access the relevant component property of the record.

private static Object nthComponent(int n, Record r)  {
   try {
       return r.getClass().getRecordComponents()[n].getAccessor().invoke(r);
   } catch (Exception e) {
       throw new RuntimeException(e);
   }
}

tryAccept is a helper default method I've added in MethodAwareTriConsumer. It checks whether the types of the provided values match the method signature before trying to pass them. Avoiding ClassCastExceptions 

interface MethodAwareTriConsumer extends TriConsumer, ParamTypeAware {
   default void tryAccept(T one, U two, V three) {
       if (acceptsTypes(one, two, three)) {
           accept(one, two, three);
       }
   }
   default boolean acceptsTypes(Object one, Object two, Object three) {
       return paramType(0).isAssignableFrom(one.getClass())
               && paramType(1).isAssignableFrom(two.getClass())
               && paramType(2).isAssignableFrom(three.getClass());
   }
  
   default Class> paramType(int n) {
       int actualParameters = method().getParameters().length; // captured final variables may be prepended
       int expectedParameters = 3;
       return method().getParameters()[(actualParameters - expectedParameters) + n].getType();
   }
}

Then put all this together and we can pattern match against Objects of unknown type, and deconstruct them if they're records matching the provided lambda type-pattern.

record Colour(Integer r, Integer g, Integer b) {}

Object unknown = new Colour(5,6,7); // note the Object type

int result = withFallback(-1).
    If.instance(unknown, (Integer r, Integer g, Integer b) ->
       r + g + b
    );

assertEquals(18, result);

Degrading safely if the pattern does not match

Object unknown = new Name("benji", "weber");

int result = withFallback(-1).
    If.instance(unknown, (Integer r, Integer g, Integer b) ->
       r + g + b
    );

assertEquals(-1, result);

Code for the record deconstruction and several more examples all in this test on github. Hopefully all this will be made redundant by future enhancements to Java's type patterns :)

The post Java 16 Pattern Matching Fun appeared first on Benji's Blog.

Today is March 14^th, or as many geeks refer to it: Pi day. On this occasion, I would like to show you a syslog-ng configuration, which prints a huge π on the screen, and two recent articles that feature syslog-ng on the Raspberry Pi.

Printing π

The following configuration – that I got from my colleagues – prints a huge π sign on screen. Create a new configuration, copy and paste the configuration below and save it on your host running syslog-ng with the name pi.conf:

@version: 3.31
@include "scl.conf"

template-function "pi-art" "

      3.141592653589793238462643383279
    5028841971693993751058209749445923
   07816406286208998628034825342117067
   9821    48086         5132
  823      06647        09384
 46        09550        58223
 17        25359        4081
           2848         1117
           4502         8410
           2701         9385
          21105        55964
          46229        48954
          9303         81964
          4288         10975
         66593         34461
        284756         48233
        78678          31652        71
       2019091         456485       66
      9234603           48610454326648
     2133936            0726024914127
     3724587             00660631558
     817488               152092096

";

block source pi-art(...) {

example-msg-generator(template("$(pi-art)") `__VARARGS__`);

};

log {
  source { pi-art(); };

   destination { file("/dev/stdout"); };
};

What happens in this configuration?

• As usual, the syslog-ng configuration starts with a version number declaration. The current version is 3.31, but I also tested this config with syslog-ng version 3.26.

• We include scl.conf (SCL is the syslog-ng configuration library). It contains many useful configuration snippets. None of them is used here, but we are too much used to including it :-)

• We define a custom template function, called “pi-art” that shows a huge π sign.

• The source block is also named “pi-art” and calls example-msg-generator() to print the π character once each second, using the previously defined template function.

• Finally, the log statement connects the pi-art() source with the standard output (our screen).

How to test it? Start syslog-ng in the foreground with the freshly saved configuration:

syslog-ng -F -f pi.conf

syslog-ng on the Raspberry Pi

Recently, Pi day is not just about physics and mathematics (and eating pies), but also about the Raspberry Pi. Syslog-ng runs on these tiny machines and I contributed two articles to the Pi day article series on https://opensource.com/:

• Containers became widely popular because of Docker on Linux, but there are much earlier implementations, including the jail system on FreeBSD. A container is called a `jail` in the FreeBSD terminology. The jail system was first released in FreeBSD 4.0, way back in 2000, and it has been continuously improved since then. While 20 years ago it was used mostly on large servers, now you can run it on your Raspberry Pi.
  From this article, you can learn the major differences between containers on Linux and FreeBSD, getting started with FreeBSD on the Raspberry Pi, and creating your first two containers using Bastille, a jail management application for FreeBSD: https://opensource.com/article/21/3/bastille-raspberry-pi

• I have lived in 100-plus-year-old brick houses for most of my life. They look nice, they are comfortable, and usually, they are not too expensive. However, humidity is high in winter in my climate, and mold is a recurring problem. A desktop thermometer that displays relative humidity is useful for measuring it, but it does not provide continuous monitoring.
  In comes the Raspberry Pi: it is small, inexpensive, and has many sensor options, including temperature and relative humidity. It can collect data around the clock, do some alerting, and forward data for analysis.
  From this article, you can learn not just about how to forward sensor data from your Raspberry Pi to Elasticsearch using syslog-ng, but also how relative humidity is changing when opening the windows for shorter or longer periods of time: https://opensource.com/article/21/3/sensor-data-raspberry-pi

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @Pczanik.

Dear Tumbleweed users and hackers,

This week, we were finally getting some fixes together for the glibc/i586 issues that plagued us for a while. Unfortunately, also for the x86_64 users, this meant once again a full rebuild of the distribution (which was published just recently with snapshot 0311). But the 4 snapshots published during this week (0305, 0306, 0307, and 0311) also had something for everybody anyway.

The main changes in those 4 snapshots include:

• glibc: Disable x86 ISA level for now
• Linux kernel 5.11.2 & 5.11.4
• gnutls 3.7.0
• openssl 1.1.1j, based on a centralized crypto-policy package
• Libvirt 7.1.0
• KDE Applications 20.12.3
• KDE Plasma 5.21.2
• LibreOffice 7.1.1.2

That actually almost is everything I had listed last week as coming soon. Only GCC 11 as the default compiler remained, and that one is still going to be with us for a few weeks. But things did not stop there, and the stagings are already filled with other things again:

• Linux kernel 5.11.6
• Python 3.9 modules: besides python36-FOO and python38-FOO, we are testing to also shop python39-FOO modules; we already have the interpreter after all. Python 3.8 will remain the default for now.
• UsrMerge is gaining some traction again, thanks to Ludwig for pushing for it
• GCC 11 as default compiler

By default, openSUSE Leap 15.x is using the firewalld firewall implementation (and the firewalld backend is using iptables under the hood).

But since a while, openSUSE also has nftables support available - but neither YaST nor other special tooling is currently configured to directly support it. But we have some machines in our infrastructure, that are neither straight forward desktop machines nor do they idle most of the time. So let's try out how good we are at trying out and testing new things and use one of our central administrative machines: the VPN gateway, which gives all openSUSE heroes access to the internal world of the openSUSE infrastructure.

This machine is already a bit special:

• The "external" interface holds the connection to the internet
• The "private" interface is inside the openSUSE heroes private network
• We run openVPN with tun devices (one for udp and one for tcp) to allow the openSUSE heroes to connect via a personal certificate + their user credentials
• In addition, we run wireguard to connect the private networks in Provo and Nuremberg (at our Sponsors) together
• And before we forget: our VPN gateway is not only a VPN gateway: it is also used as gateway to the internet for all internal machines, allowing only 'pre-known traffic' destinations

All this makes the firewall setup a little bit more complicated.

BTW: naming your interfaces by giving them explicit names like "external" or "private", like in our example, has a huge benefit, if you play along with services or firewalls. Just have a look in /etc/udev/rules.d/70-persistent-net.rules once your devices are up and rename them according to your needs (you can also use YaST for this). But remember to also check/rename the interfaces in /etc/sysconfig/network/ifcfg-* to use the same name before rebooting your machine. Otherwise your end up in a non-working network setup.

Let's have a short look at the area we are talking about:

As you hopefully notice, none of the services on the community side is affected. There we have standard (iptables) based firewalls and use proxies to forward user requests to the right server.

On the openSUSE hero side, we exchanged the old SuSEfirewall2 based setup with a new one based on nftables.

There are a couple of reasons that influenced us in switching over to nftables:

• the old SuSEfirewall2 worked, but generated a huge iptables list on our machine in question
• using ipsets or variables with SuSEfirewall2 was doable, but not an easy task
• we ran into some problems with NAT and Masquerading using firewalld as frontend
• Salt is another interesting field:
  • Salt'ing SuSEfirewall2 by deploying some files on a machine is always possible, but not really straight forward
  • there is no Salt module for SuSEfirewall2 (and there will probably never be one)
  • there are Salt modules for firewalld and nftables, both on nearly the same level
• nftables is integrated since a while in the kernel and should replace all the *tables modules long term. So why not jumping directly to it, as we (as admins) do not use GUI tools like YaST or firewalld-gui anyway?

So what are the major advantages?¶

1. Sets are part of the core functionality. You can have sets of ports, interface names, and address ranges. No more ipset. No more multiport. ip daddr { 1.1.1.1, 1.0.0.1 } tcp dport { dns, https } oifname { "external", "wg_vpn1" } accept; This means you can have very compact firewall sets to cover a lot of cases with a few rules.
2. No more extra rules for logging. Only turn on counter where you need it. counter log prefix "[nftables] forward reject " reject
3. You can cover IPv4 and IPv6 with a single ruleset when using table inet, but you can have per IP protocol tables as well. And sometimes even need them e.g. for postrouting.

Starting from scratch¶

A very basic /etc/nftables.conf would look something like this

#!/usr/sbin/nft -f

flush ruleset

# This matches IPv4 and IPv6
table inet filter {
    # chain names are up to you.
    # what part of the traffic they cover, 
    # depends on the type line.
	chain input {
		type filter hook input priority 0; policy accept;
	}
	chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

But so far we did not stop or allow any traffic. Well actually we let everything in and out now because all chains have the policy accept.

#!/usr/sbin/nft -f

flush ruleset

table inet filter {
    chain base_checks {
        ## another set, this time for connection tracking states.
        # allow established/related connections
        ct state {established, related} accept;

        # early drop of invalid connections
        ct state invalid drop;
    }

   	chain input {
		type filter hook input priority 0; policy drop;
        
        # allow from loopback
        iif "lo" accept;

        jump base_checks;

        # allow icmp and igmp
        ip6 nexthdr icmpv6 icmpv6 type { echo-request, echo-reply, packet-too-big, time-exceeded, parameter-problem, destination-unreachable, packet-too-big, mld-listener-query, mld-listener-report, mld-listener-reduction, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept;
        ip protocol icmp icmp type { echo-request, echo-reply, destination-unreachable, router-solicitation, router-advertisement, time-exceeded, parameter-problem } accept;
        ip protocol igmp accept;
        
        # for testing reject with logging
        counter log prefix "[nftables] input reject " reject;
	}
    chain forward {
		type filter hook forward priority 0; policy accept;
	}
	chain output {
		type filter hook output priority 0; policy accept;
	}
}

You can activate the configuration with nft --file nftables.conf, but do NOT do this on a remote machine. It is also a good habit to run nft --check --file nftables.conf before actually loading the file to catch syntax errors.

So what did we change?

1. most importantly we changed the policy of the chain to drop and added a reject rule at the end. So nothing gets in right now.
2. We allow all traffic on the localhost interface.
3. The base_checks chain handles all packets related to established connections. This makes sure that incoming packets for outgoing connections get through.
4. We allowed important ICMP/IGMP packets. Again this is using a set and the type names and not some crtyptic numbers. YAY for readability.

Now if someome tries to do a ssh connect to our machine, we will see:

[nftables] input reject IN=enp1s0 OUT= MAC=52:54:00:4c:51:6c:52:54:00:73:a1:57:08:00 SRC=172.16.16.2 DST=172.16.16.30 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22652 DF PROTO=TCP SPT=55574 DPT=22 WINDOW=64240 RES=0x00 SYN URGP=0 

and nft list ruleset will show us

counter packets 1 bytes 60 log prefix "[nftables] input reject " reject

So we are secure now. Though maybe allowing SSH back in would be nice. You know just in case.
We have 2 options now. Option 1 would be to insert the following line before our reject line.

tcp dport 22 accept;

But did we mention already that we have sets and that they are great? Especially great if we need the same list of ports/ip ranges/interface names in multiple places?

We have 2 ways to define sets:

define wanted_tcp_ports {
  22,
}

Yes the trailing comma is ok. And it makes adding elements to the list easier. So we do them all the time.
This will change our rule above to

tcp dport $wanted_tcp_ports accept;

If we load the config file and run nft list ruleset, we will see:

tcp dport { 22 } accept

But there is actually a slightly better way to do this:

    set wanted_tcp_ports {
        type inet_service; flags interval;
        elements = {
           ssh
        }
    }

That way our firewall rule becomes:

tcp dport @wanted_tcp_ports accept;

And if we dump our firewall with nft list ruleset afterwards it will still be shown as @wanted_tcp_ports and not have variable replaced with the value.
While this is great already, the 2nd syntax actually has one more advantage.

$ nft add element inet filter wanted_tcp_ports \{ 443 \}

Now our wanted_tcp_ports list will allow port 22 and 443.
This is of course often more useful if we use it with IP addresses.

    set fail2ban_hosts {
        type ipv4_addr; flags interval;
        elements = {
           192.168.0.0/24
        }       
    }

Let us append some elements to that set too.

$ nft add element inet filter fail2ban_hosts \{ 192.168.254.255, 192.168.253.0/24 \}
$ nft list ruleset

... and we get ...

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

Now we could change fail2ban to append elements to the set instead of creating a new rule for each new machine it wants to block. Fewer rules. Faster processing.

But with reloading the firewall we dropped port 443 from the port list again. Oops.
Though ... if you are happy with the rules. You can just run

$ nft list ruleset > nftables.conf

When you are using all the sets instead of the variables, all your firewall rules will still look nice.

Our complete firewall looks like

table inet filter {
        set wanted_tcp_ports {
                type inet_service
                flags interval
                elements = { 22, 443 }
        }

        set fail2ban_hosts {
                type ipv4_addr
                flags interval
                elements = { 192.168.0.0/24, 192.168.253.0/24,
                             192.168.254.255 }
        }

        chain base_checks {
                ct state { established, related } accept
                ct state invalid drop
        }

        chain input {
                type filter hook input priority filter; policy drop;
                iif "lo" accept
                jump base_checks
                ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept
                ip protocol icmp icmp type { echo-reply, destination-unreachable, echo-request, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept
                ip protocol igmp accept
                tcp dport @wanted_tcp_ports accept
                counter packets 12 bytes 828 log prefix "[nftables] input reject " reject
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}

For more see the nftables wiki