Implicit Neural Network Rules Are Here to Complement Intrusion Detection Systems
An intrusion detection system (IDS) detects suspicious or unusual activity in the network traffic at an early stage and generates an alert or a log entry. IDSs are commonly classified into two categories: Network-based Intrusion Detection System (NIDS) and Host-based Intrusion Detection System (HIDS). NIDS monitors the network incoming traffic while HIDS collects and analyzes the operating system’s files/logs. There is another classification based on the detection approach, i.e., signature-based and anomaly-based. In an anomaly-based detection system, a range of normal events is developed as a baseline where the real-time events are later compared with the baseline to determine if an alert should be raised. This approach could be leveraged to detect zero-day exploits, however, it generates a high false positive rate. In contrast, a signature-based approach looks for known patterns (signatures) in the captured data and sends an alert to the administrator if the signatures do match. Although this approach can explicitly detect known patterns, it is unable to identify unknown attacks.
Most of the IDSs like Snort, Suricata, OSSEC, and SolarWinds are equipped with event correlation rules to shut down malicious activity instantly and implement mitigation activities. Using finely-tuned event correlation rules, IDSs are trying to decrease the number of false positives to avoid significant disruption to services. In Snort, which is the industry leader in NIDS and still free to use, the rules will detect events such as stealth port scans, buffer overflow attacks, CGI attacks, SMB probes, and OS fingerprinting. The detection method of snort comprises both signature-based and anomaly-based detection. For anomaly detection, for example, TCP protocol anomalies, such as data on SYN packets, data received outside the TCP window could be detected per-target basis.
However, Snort and other rule-based IDSs suffer from some drawbacks, despite all salient features they might offer. Although the rules are usually written using a fairly simple and flexible rule description language, adapting the set of rules for the ever-increasing number of threats and demands of high-speed networks is so challenging. The rule library needs to be updated as new threats evolve and this should be done in no time.
This is where Artificial Intelligence (AI) comes to play to avoid burdensome rule generation task. It is not a new idea to use AI in intrusion detection. Security experts have made some attempts to introduce AI mostly in the context of anomaly-based detection. However, the idea has not been extensively welcomed by the security companies since it will not be easy for AI to detect unknown attack signatures. Due to changing to network structures, usage patterns in large companies, and the prominent number of varied attacks, the AI-based IDS solutions will generate a large number of false positives. False positive alerts occur when benign traffic is mispredicted for a malicious activity by an IDS. So, what if we apply AI as a supplemental technology to complement human intelligence. In the realm of NIDS, AI could be used to generate signatures for novel attacks automatically from real-time traffic that completes the existing hand-made rules.
Neural networks, in particular, could be an excellent source of implicit rules. Neural networks are computational models inspired by biological neurons that are designed to learn numerical patterns in input data like text, image, or voice. A neural network is a collection of artificial neurons aggregated into layers that transmit signals from one neuron into another to classify the input data. The proliferation of the neural networks-based applications justifies the broad range of capabilities they contain. Out of a myriad of characteristics, a neural network incorporates, we can name a straightforward process through which a neural network is trained; compact numerical form in which the information is stored; and a high degree of accuracy acquired in the presence of noise in the input data. However, the success of neural networks is influenced by the lack of inner explanation and the black box way of making decisions. By adding an explanation feature to a neural network, we would enhance the level of reliability, especially in safety-critical systems. In this case, we can integrate the worlds of symbolic AI and connectionist AI in form of the rules.
For an NIDS, to be more specific, neural networks could be trained either on packet captures or netflows to find the inherent patterns. The correlation could be found and extracted as the rules which are well-defined to substitute and complete the signatures of the existing NIDSs. In packet captures, we grab the whole packet header and payload to examine and diagnose network anomalies. In Snort, the rules generated from packet captures comprise two logical sections, namely the rule header and the rule options. The rule header contains the information about action, protocol, source and destination IP addresses and netmasks, and the source and destination ports. The rule option section contains information by which we can identify the parts of the packet needed to be inspected to take the rule action. It also includes alert messages and implements responses and reactions to traffic that complies with a Snort rule. On the other hand, netflow creates a lightweight view of all the network’s traffic like the source/destination IP address, the source/destination ports, the protocol, the time and date of the communication, the flow duration, and the amount of data that was transferred. The figure below shows an example of a Snort rule:
For the deep packet inspection, we need to find patterns in the header and payload section of a packet separately. The packets might have a dependency relationship that resembles the language modeling task. Therefore a sophisticated deep learning model like Recurrent Neural Network (RNN) or Long Short-Term Memory (LSTM) could be employed on the whole packet to learn the representation of the packet header and payloads, individually. Using these deep learning models for handling sequential data, we are able to discover anomalies in the packets from a long-term perspective. In other words, we treat the packet contents as streams of text data and learn the language of attacks in terms of rules extracted from trained RNN/LSTM. For netflow, a Feed-Forward Neural Network (FFNN) or an Autoencoder fits well that can extract conjunctive rules in the form of if/conditions/then, for example,
If source_port=35468 and destination_port=49856 and protocol= 17 and flow_duration=47 then action=’alert’.
Whether we are dealing with raw packet captures or lightweight netflow, we need to combine human knowledge with AI to refine the set of rules for detecting network intrusions in enterprises. Human expertise could be useful for delineating the proper actions that must be taken in case a threat occurs and planning the corresponding remediation techniques. Moreover, getting help from neural networks and deep learning techniques, we can end up with an incremental rule generation process. The rules could be easily updated by retraining the neural network as the new traffic comes and enhancing the extracted rules. This addresses the issue of conventional IDSs of not being adaptable to dynamically increasing volumes of incoming threats.
Extracting rules from a trained neural network dates back to the early 1990s where numerous rule extraction algorithms were proposed to date, however, they are still not operational due to additional effort and computational costs they incur. As there is an ongoing demand for IDSs to come up with an automatic signature/rule generation process, it is expected that the vision of designing an IDS product that includes well-defined rule sets along with a finely-tuned neural network will be accessible in the foreseeable future. In the end, AI can never completely replace human intelligence; instead, it should be a supplemental material to benefit the field of cybersecurity and intrusion detection.
