Elasticsearch Machine Learning with Zeek IDS to Detecting Malware Behavior2

Related articles:

• Step-by-step installation of Elasticsearch
• Elasticsearch Enables minimal security
• Installing Kibana step by step
• Step by step installation of Zeek IDS
• C2 server Anti-Virus evasion techniques
• c2 traffic analysis

Elasticsearch Machine Learning and Network traffic anomaly detection

network traffic can be divided into three types:

• Normal traffic — This traffic includes traffic used by installed apps or by users that use apps, email traffic, file sharing, apps updates, etc.
• Beacon — This traffic can be detected when there is a fixed network traffic in time and size, for example, every 3 seconds a request is made to a DNS address. This traffic usually points to a remote Command and Control (C2) server connected to a company network.
• Jitter — Unlike beacon traffic, there is no fixed time for this type of request, but a pattern can still be found between requests

Elasticsearch Machine Learning spike traffic

in the picture below, the network shows normal activity until 7/3/22. On that date, there was a spike of traffic directed towards Russia for a 15-minute period. This spike in traffic towards Russia may not be considered normal behavior and may warrant further investigation to determine if it is indicative of a security concern or not. It is important to note that without more context and information about the network, it’s difficult to say if the spike is normal or not

Elasticsearch Machine Learning (ML) is a feature that allows users to analyze and detect anomalies in their data using machine learning algorithms. It allows for the creation and management of machine learning jobs, including anomaly detection, classification, and regression. These jobs can be applied to data stored in Elasticsearch indices and can be used to identify patterns, trends, and outliers in network traffic data, which can help detect network anomalies, and potential security issues. Additionally, Elasticsearch ML also provides visualization tools to help users understand and interpret the results of the machine learning jobs.

Elasticsearch Machine Learning functions

1. Info_content
2. High_info_content
3. Low_info_content

For example, searching for multiple DNS requests in a short period of time using the high_info_content function will allow us to detect anomalies, and often also beacon traffic on the network

{
"function": "high_info_content",
"field_name": "query",
"over_field_name": "src_ip"
}

In addition, to detect anomalies using Elasticsearch, you can utilize machine learning tasks tailored to the network or operating system. By selecting ‘Security Network’ as shown in the image below, all the relevant tasks for anomaly detection can be used as a baseline for detection rules.

Zeek IDS Anomaly Detection with Elasticsearch Machine Learning

To enable a Zeek-based IDS server to look for anomalies in DNS and send alerts, it’s necessary to install and configure a model called AnomalousDNS. This model uses machine learning algorithms to detect unusual patterns in DNS traffic that may indicate a security incident. Once the model is installed and configured, it can be used to analyze data collected by the Zeek IDS and identify potential anomalies in the network traffic. By using Elasticsearch ML, the results can be stored, indexed and analyzed, providing a powerful tool for detecting and responding to security threats

Preparing for Zeek Package Management with Zkg

Zkg (Zeek Package Manager) is a package manager for Zeek ids that makes it easy to install and manage Zeek scripts and plugins. It allows you to easily find, download, and install packages that have been created by the Zeek community, as well as to manage and update the packages you have installed. Zkg also makes it easy to develop and distribute your own packages.With Zkg you can install the package with a single command, and it also keeps track of the package version, dependencies, and allows you to easily update or remove the package.

Zkg is a powerful tool that helps simplify the management of Zeek scripts and plugins, making it easy to add new functionality and keep your Zeek installation up to date.

To install Zkg on a Zeek management server, you will first need to ensure that the necessary prerequisites are met. These include:

• python3-pip
• GitPython:  https://pypi.org/project/GitPython
• semantic-version:  https://pypi.org/project/semantic-version

*add additional sources to source file /etc/apt/sources.list if pip3 is not available for download via apt

deb http://archive.ubuntu.com/ubuntu bionic main universe
deb http://archive.ubuntu.com/ubuntu bionic-security main universe
deb http://archive.ubuntu.com/ubuntu bionic-updates main universe

installing all software requirements using apt

sudo apt-get install python3-pip python3-git python3-semantic-version

Using AnomalousDNS module for Advanced DNS Traffic Anomaly Detection

AnomalousDNS is a set of Zeek scripts that provide a module for tracking and adjusting abnormal DNS behavior. Tunnel and C&C detection by checking connection duration and connection volume, request and response size, DNS request type and unique queries for each domain.

cd /usr/local/zeek/bin

./zkg install anomalous-dns

you can check what the anomalous-dns script includes in the following path:

cd /usr/local/zeek/share/zeek/site/packages/anomalous-dns/

The anomalous-dns model includes useful scripts for detecting anomalies in DNS requests. It is recommended to install another model called zeek-tls-log-alternative, which allows for more logs of the TLS protocol. When you are ready to publish the anomalous-dns model, you can share it to all Zeek agents by using the following commands:

su zeek

zeekctl status

zeekctl deploy

After the anomalous-dns model is activated, you can look for alerts in the notice file and through Kibana apps

Summary

In this article, we have explained how Elasticsearch Machine Learning and Network traffic anomaly detection can be used to detect and respond to security threats in a computer network. By using Elasticsearch ML, network anomalies and potential security issues can be identified and stored, indexed, and analyzed. Additionally, by utilizing machine learning tasks tailored to the network or operating system, and by installing and configuring a model called AnomalousDNS, a Zeek-based IDS server can be used to detect unusual patterns in DNS traffic that may indicate a security incident. To make the management of Zeek scripts and plugins easier, Zkg (Zeek Package Manager) can be installed on a Zeek management server. To get a complete picture of behavior in the computer network, use Wazuh with EDR detection rules or any other EDR solutions