Splunk for Security Monitoring, Part 3: Creating a Real-Time Alert

Welcome back, my aspiring cyber warriors!

In earlier posts here at Hackers-Arise, I introduced you to Splunk, an excellent tool for managing all your machine-generated data.

In this tutorial, I will show you how to use Splunk to generate real-time alerts on just about any machine-data condition that arises on your system or network.

Splunk alerts can be created based upon any search that you can create in the Search window (see Splunk SPL). The alerts will then be triggered when the search results condition are met, much like an IDS creates alerts on signatures (see Snort Rule Writing). These alerts can;

1. Create an entry in Triggered alerts

2. Log an event

3. Output results to a lookup file

4. Send emails

5. Use a Webhook

6. Perform a custom action

Here we will create a real-time alert that emails us whenever a privileged account is used.

Step #1: Create A Search

The first step is to create a search condition. In this case, we have created a search among the Security and System event logs for EventCode=4672 (this event code is triggered whenever anyone or any service logs on with with administrator rights on Windows 7/2008 through Windows 2019 systems).

First, we need to create the search. To find all EventCode=4672, we can create a search such as;

(source="WinEventLog:Security" OR source="WinEventLog:System") | where EventCode=4672

Next, we click on the "Save As" tab in the upper right corner. When we click on "Alert", it open a window like that below. Go ahead and give your Alert a name (I named mine "EventCode 4672 Alert") in the Title space and then add a Description.

Just below the Description, you will see the "Permissions". If you set it to Private, only you can access, edit and view the alerts. If you choose "Shared in App" others can view the alerts through the shared app. Here, I chose my alerts to be private.

Step #2: Scheduled vs. Real-Time

On the next line we can choose either a Scheduled alert or a Real time alert. Here I select a Real-Time alert.

The next line is the "Trigger Conditions". These allow you to capture a larger data set and then apply additional conditions to the results before the alert is triggered. For instance, you may want to see more than one occurrence of the condition within a specified time period before triggering the alert.

These per result conditions include;

1. Per-Result

2. Number of Results

3. Number of Hosts

4. Number of Sources

5. Custom

Here we set the Per-Result condition to 3 results per minute.

Step #3: Set the Action

In the final step, we determine what action we want the alert to take. These actions include;

1. Log the Event

2. Output the results t lookup

3. Output results to Telemetry endpoint

4. Run a script

5. Send email

6. Webhook

Here I want to set the alert to send an email. When you go to save the alert you will be asked for;

1. Who to send the alert to;

2. Priority

3. Subject

4. Message

5. Type of Message (HTML or Plain Text)

Finally, when you save the alert, you will be greeted with a result like that below.

Now, whenever your search condition is triggered more than 3 times in one minute, Splunk will email an alert notifying you.

Summary

Splunk is an excellent and powerful tool for security monitoring. Not only does it gather and index all your machine data, but it has the capability to be used much like an intrusion detection system (IDS), if you know what to look for. While your IDS is looking for signatures of malware, your Splunk instance can be looking for behaviors that indicate suspicious activity and notify you in real-time!