Configure fail2ban properly and test it before rollout to prod.

Issue

We need to add fail2ban config to the ssh proxy node to protect it against brute-force password and DDoS attacks when exposed to the internet. Remember users interface with the ssh-proxy for logging into Cheaha. So adding fail2ban config is important.

The ansible role for installing and configuring fail2ban was merged but never deployed in production because the testing was not comprehensive.

It was discovered that we need to add additional configuration to the fail2ban.

Proposed solution

The proposed solution comes in two layers.

1. First, enable the failtoban plugin provided by sshpiper. SSHPiper offers a plugin at the application level - failtoban that acts as a drop-in for fail2ban. It would initially count failures and trigger the failtoban feature when a threshold (maxtries) is reached. The bantime and maxtries are configurable via the plugin.

This would prevent the user from logging in but doesn't prevent the application from being hit by requests. This is where the second layer comes to the rescue.

2. Second, configure fail2ban to ban IP addresses conducting too many failed login attempts via jail.local configuration. Create fail2ban filter for sshpiperd which uses failregex to parse failed login attempt patterns in the log files. It adds firewalld rules in an ad-hoc manner via actions config in /etc/fail2ban/actions.d/ to reject new connections from those IP addresses, for the configured amount of time.

added Backlog label

changed the description

added Sprint 25-02 label and removed Backlog label

assigned to @louistw

changed title from Test fail2ban before rollout the change to Configure fail2ban properly and test it before rollout to prod.

changed the description

assigned to @atlurie and unassigned @louistw

changed the description

marked this issue as related to #211 (closed)

mentioned in merge request !178 (merged)

Introduction

Fail2ban works by parsing the log lines of an application containing failure messages and preventing brute force attacks on the application. It does this by capturing the IPs making too many login attempts and banning them via firewall rules.

Fail2ban comes with configurations for many applications like https, ssh, nginx etc. to prevent bruteforce attacks. There are three parts to configuring fail2ban -

1. jails
2. filters and
3. actions

Understanding Fail2ban Configuration

1. Configuring jail

The Jail configuration defines the rules to implement if an IP makes too many login attempts.

Start configuring the jail by copying the jail.conf to jail.local and editing to it.

Application-specific jail config can be placed in /etc/fail2ban/jail.d/ dir

In the next section, let's look at configuring a custom filter

Before you start developing a custom filter for your application, you need to know how to incrementally test the filter config. Even if you are not developing a filter config from scratch, it is still important to know how to test when you want to edit an already existing config.

2. Testing the Filter Configuration

To test a custom filter against your log file:

Run Fail2Ban Regex Test:

sudo fail2ban-regex [-v] \
[-l HEAVYDEBUG] \
--print-no-missed \
/path/to/logfile /etc/fail2ban/filter.d/custom-filter.conf

NOTE: You must run the above test after every single change while developing the regex for your custom filter. Otherwise, you are bound to fail and be stuck in the perennial loop of frustration. You have been warned ;)

3. Configuring the filters

The custom fail2ban filters are configured per application. The configs are in /etc/fail2ban/filters.d dir.

For sshpiper the filter config would be placed in /etc/fail2ban/filters.d/sshpiperd.local

Here's the config,

# Refer to https://github.com/fail2ban/fail2ban/wiki/Developing-Regex-in-Fail2ban for developing regex using fail2ban
#
[INCLUDES]
before = common.conf

[DEFAULT]
_daemon = sshpiperd
__iso_datetime = "\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:[+-]\d{2}:\d{2}|Z)"
__pref = time=%(__iso_datetime)s level=(?:debug|error)

[Definition]
# Define the prefix regex for the log lines
prefregex = ^<F-MLFID>%(__prefix_line)s%(__pref)s</F-MLFID>\s+<F-CONTENT>.+</F-CONTENT>$

# Failregex to match the specific failure log lines (prefregex is automatically included)
failregex = ^msg="connection from .*failtoban: ip <HOST> too auth many failures"$

ignoreregex =

mode = normal

maxlines = 1

Let's see what each section means and cover some important concepts involved in understanding the filter config.

1. The [INCLUDES] section includes the common.conf file for shared patterns and variables, ensuring consistency across filters. It reuses common regex definitions, such as __prefix_line, for modularity.

2. The [DEFAULT] section defines reusable variables for the filter and the services that need to be monitored.

3. The [DEFINITION] defines the main patterns and behavior for this filter.

   • prefregex defines the regex to match the prefix pattern of the log lines for eg, date, hostname, service, port
     • <F-MLFID>: Encapsulates the prefix metadata i.e. wraps the logline
     • %(__prefix_line)s: A pre-defined variable from common.conf. Pattern Matches Common log prefixes (date, hostname) depending on the log type (file journal syslog).
     • %(__pref)s: A User-defined variable. Service-specific prefix matching timestamp and log level in this case.
     • <F-CONTENT>: Marks the log content to be processed by failregex.

   Note: A variable is defined as %(__VARIABLE)s

   • failregex matches log entries that indicate a failure due to too many authentication attempts from a specific IP.
     • <HOST> Dynamically matches offending IP

4. ignoreregex defines patterns to explicitly exclude certain log lines from being processed.

5. mode = normal defines the mode for the filter. The normal mode processes log lines line by line.

6. maxlines = 1 defines that the log entries are single line and doesn't span across multiple lines.

---

Example Log Line Matching

Log Line:

Jan 22 20:16:19 ssh-proxy sshpiperd[5093]: time="2025-01-22T14:16:19-06:00" level=debug msg="connection from 172.19.0.171:64390 establishing failed reason: rpc error: code = Unknown desc = failtoban: ip 172.19.0.171 too auth many failures"

Pattern matching flow:

__prefixline matches:

Jan 22 20:16:19 ssh-proxy sshpiperd[5093]:

__pref matches:

time="2025-01-22T14:16:19-06:00" level=debug

prefregex matches:

Jan 22 20:16:19 ssh-proxy sshpiperd[5093]: time="2025-01-22T14:16:19-06:00" level=debug

failregex matches:

msg="connection from 172.19.0.171:64390 establishing failed reason: rpc error: code = Unknown desc = failtoban: ip 172.19.0.171 too auth many failures"

<HOST>: Extracts 192.168.1.100 for banning.

---

Recommended reading

1. https://github.com/fail2ban/fail2ban/wiki/Developing-Regex-in-Fail2ban
2. https://github.com/fail2ban/fail2ban/wiki/Proper-fail2ban-configuration

Closing issue because the Merge request is merged

closed