The search for synergy between employee, SOC analyst, and machine learning technology to combat targeted phishing
Have you ever received a dubious text message or e-mail from an unknown recipient? I bet you have! Nowadays, we are all familiar with those bulk phishing campaigns sent to a large number of recipients. But what if such a malicious message is personally addressed to you? What if the content of the message is well-written and seems authentic to you because it is related to your daily work activities or personal interests? What if the sender seems like a legitimate source, and it makes sense for them to approach you with a great opportunity for you or your company? All you have to do is click on the link! Would you fall for it?
Although you might not realise it, targeted phishing is still one of the biggest cyber security challenges for large companies. According to FireEye1, targeted phishing – also known as spear phishing – was the most popular method for initial access by APTs2 in 2010, and this was still the case in 2020. Our conclusion: phishing is still a very successful attack vector for adversaries. It is a persistent problem and apparently there are no solutions available that completely solve this issue. PCSI proposes a holistic solution in which human and machine work closely together to identify phishing e-mails before any harm is done: the AIWARENESS project.
How do PCSI partners deal with phishing e-mails?
Most companies have implemented a first-line-of-defence control aimed at intercepting and quarantining spam and phishing e-mails, consisting of products mostly provided by security companies. “This first line of defence intercepts a large percentage of phishing attempts, but unfortunately, a 100% solution is not available,” explains Bianca Verhagen, platform engineer at de Volksbank. “All e-mails are scanned, but as a consumer of an e-mail security product, we remain dependent on the vendor. We do not know what logic is used by vendors to intercept e-mails and place them in quarantine, and we do not have the knowledge and expertise to develop a product ourselves.”
At ABN AMRO, a team of data scientists develops machine learning models to improve the bank’s overall security posture. This team has worked on developing a phishing detection model as a second line of defence, in close collaboration with the Security Operations Centre. The biggest challenge for them is reducing the number of false positives (e-mails that are incorrectly classified as malicious) in such models. “Every day, adversaries send newly crafted phishing e-mails, which are often indistinguishable from benign e-mails. It takes a lot of time for us to handle this properly,” says Victor Garçia Cazorla, data scientist at the CISO department of ABN AMRO.
"Apart from a few exceptions, such as DDoS attacks, most cyber crime attacks start with social engineering. We need to make and keep our employees continuously aware of that"
Richard Verbrugge, awareness expert at ABN AMRO, is continuously working on enhancing employee awareness with respect to cyber crime and cyber security. “Apart from a few exceptions, such as DDoS attacks, most cyber crime attacks start with social engineering. We need to continuously make and keep our employees aware of that. We therefore have a continuous learning programme in place, which is updated monthly with the latest trends and developments. Furthermore, we test the knowledge level of our employees and aim to monitor which topics need additional attention.”
How does the PCSI propose to approach this challenge collaboratively?
The PCSI is working on a holistic solution, the AIWARENESS project, in which we aim to optimise the synergy between:
- the employee and their context knowledge about their own inbox and group mailboxes
- the SOC analyst with expert knowledge about phishing and the latest threats
- machine learning-based detection technology that can recognise patterns and detect anomalies in large amounts of data.
Within this project, we aim to provide real-time awareness support to employees, while at the same time improving supervised machine learning-based detection models by means of the incoming judgements (labels) of the employees and SOC analysts.
But how do we propose to do this? A purely technological approach has proven very difficult, since it is next to impossible to develop a perfect detection capability that will spot all incoming malicious e-mails (true positives) and will not trigger on any benign e-mails (false positives) in a continuously changing threat landscape. Even though machine learning-based targeted phishing detection models might not be perfect, they form a solid starting point for making sense of large amounts of e-mail data.
Retraining the model using employee feedback
A reasonably well-performing model can be used to alert employees at the right time, such as when opening a potentially malicious e-mail. Subsequently, the employee’s own judgement of the e-mail can be used as a label for retraining the machine learning (ML) model to enhance its performance. In an ideal world, an employee would be able to judge these e-mails effectively, in which case we would not even require a ML model. However, this is often not the case and therefore we need to:
- empower the employee to better judge these e-mails
- account for the fact that the judgements fed back to the ML model might not always be correct.
Garçia Cazorla continues: “The ‘employee crowd-sourcing’ aspect could be a very interesting addition to existing models and tools. Instead of alerting the SOC analyst and using their judgement to improve the machine learning (with labelled data), we will receive feedback from all employees. If this turns out to be effective, it will significantly lower the workload for SOC analysts.”
Figure 1: Interaction between Classifier (ML model) and the Human oracle (employee or SOC analyst)
In Figure 1, we show how the interaction between the human oracle (employee or SOC analyst) and the ML-based detection model (classifier) is designed in an Explainable Active Learning (XAL) loop.
The Human oracle can either be an employee who receives an e-mail and has to judge whether or not to trust it, or a SOC analyst who is going through reported e-mails. The Classifier is a machine learning model that uses large amounts of incoming data to optimally classify an e-mail as phishing or trustworthy.
The virtuous feedback loop that we are trying to achieve is that the classifier will become better at identifying potentially malicious e-mails, making it better at presenting the human oracle with the most relevant e-mails, which helps the human oracle to focus only on judging those e-mails that need most attention, eventually going full circle because these incoming judgements (labels) will help to improve classifier performance even further.
We can empower the human oracle by presenting them with a clear explanation of why the model has identified an e-mail as malicious, using a technology called Explainable AI (XAI). In return, a human oracle can help improve the classifier most efficiently by labelling those entries that are most informative for the classifier, by means of a technology called Active Learning (AL). The combination of the two is called Explainable Active Learning (XAL).
"If this AIWARENESS solution turns out to be effective, it should significantly lower the workload for SOC analysts"
In the following paragraphs, we will explain the most important technological challenges for a smooth interaction between human oracle and classifier. As we are planning to run a pilot experiment with the AIWARENESS solution, we will narrow the scope in these paragraphs.
Firstly, for an initial pilot implementation we will focus on URL-based phishing detection. We will reuse an existing model that can classify URLs as benign (not phishing) or malicious (phishing). This classifier was partly developed in earlier PCSI research. We are focusing on URL-based detection as it is currently the most frequently used payload in phishing e-mails by adversaries. Phishing e-mails will usually contain many URLs, potentially also benign URLs, so our reasoning is that if an e-mail contains at least one truly malicious URL, we will conclude that it is a malicious e-mail.
Secondly, in the pilot experiment we will focus on the interaction with the employee (and thus not the SOC analyst), as this is one of the most novel aspects of this project and is expected to require most research efforts. Moreover, if we can empower the employee to make better judgements, it will alleviate the SOC analysts’ workloads.
Active learning – efficiently labelling data
One of the biggest challenges for building good models, including machine learning models, to filter targeted phishing is the lack of data. Successful phishing e-mails nowadays are specifically crafted for a single person or a small group of people. This makes them harder to distinguish from regular e-mails for “generic” machine learning models that are already being used to filter more obvious phishing e-mails. The only way to obtain correctly labelled e-mails or URLs for these sophisticated phishing e-mails is to have the employees or the SOC analysts of a company go over each individual e-mail/URL manually and label it as benign or malicious. As you can probably imagine, this is a very cumbersome task.
Active learning is being researched as a promising technique to ease this process. The idea of active learning is to automatically select only the most informative URLs that the model can efficiently learn from. This way, a model needs less data to reach a similar performance and thus relieves the employees and the SOC analysts, who, as a result, need to label fewer data points manually.
Figure 2 Source: Settles, B. (2010). Active learning literature survey. University of Wisconsin. Computer Science Department
Figure 2 illustrates this idea, where the goal is to classify between the green and the red data points. In picture (a), the actual labels of the data points are depicted. In (b), the line represents the way a model distinguishes between green and red if it is given a number of labelled data points that are selected at random. As can be seen, the model is somewhat able to find a border between the green and red areas but is nowhere near perfect. Finally, picture (c) depicts a model which has been trained using the same number of data points, but this time selected using active learning. As can be seen, this model is able to distinguish much better between green and red because it received more informative data points to train on, namely the data points that are close to the border. This process is known as uncertainty sampling and is also used in this project to select only the most informative URLs to be labelled by the employees and the SOC analysts.
Empowering the employee with Explainable AI
An integral part of the philosophy of this project is that a standalone ML model will not be able to solve the problem of targeted phishing on its own. On the other hand, an employee without proper information will also have a hard time identifying targeted phishing e-mails. Therefore, the project aims to find a way to make effective use of the strength of AI in classifying URLs quickly, strengthened by the context knowledge of human employees.
"Employees help improve technology, and subsequently the technology becomes better at protecting and empowering the employee"
Explainable AI (XAI) is a technique that enables us to do this. With XAI, an employee or SOC analyst receives insight into why the model thought a URL was suspicious, and thus the human can use this information to better judge the entire e-mail. For example, XAI might tell the employee to be careful of the URL because it contains unusual characters or because its certificate is invalid. These details might be overlooked by an employee routinely working through their inbox. The most challenging task will be to present information at an understandable level for the employee, and also for a SOC analyst.
“Employees help improve technology, and subsequently the technology becomes better at protecting and empowering the employee. A nice continuous loop which will gradually perform better. In addition, it is important to understand why an employee judges an e-mail to be malicious. With this information, we could iteratively improve the machine learning model,” argues Verbrugge.
Figure 3: two examples of XAI explanations and the corresponding URLs
In Figure 3, two examples of XAI explanations and the corresponding URLs are depicted. On the left, we see a URL that has been classified as malicious because of, among other things, its unusually long path. On the right, we see a URL that has been classified as benign because of, among other things, the large number of certificates historically associated with the URL.
“The employee will be responsible for making a context-related judgement, using information such as whether he or she is expecting an e-mail from a certain recipient because there has been contact about that by telephone” explains Verhagen. “An employee must be – and remain – aware of the fact that a malicious e-mail can still reach one’s inbox, even with a first line of defence in place.”
"An employee must be – and remain – aware of the fact that a malicious e-mail can still reach one’s inbox, even with a first line of defence in place"
Results of first experiments
The first experiments on the use of active learning for combatting targeted phishing have had positive results. In Figure 4, the use of active learning to classify a dataset of phishing e-mails has been compared to regular training.
Figure 4: a comparison between normal training and active learning
Here, we see that the model is indeed able to distinguish between phishing e-mails and benign e-mails more effectively with less data. With 50-80 data points, the model was 10% more effective in this scenario. Furthermore, an experiment has been conducted to investigate the working of active learning in a more realistic scenario where an employee or SOC analyst might not always provide the correct label. The results of this experiment can be found in Figure 5.
Figure 5: results of different employee accuracies
As expected, an employee who is always right (the green line) performs best, while an employee who is right only 50% of the time, basically equal to randomly guessing, does not significantly improve the model. Interestingly, an employee who is right in 80% of cases is still able to improve the model quite rapidly, which is a promising result and a motivation for further practical experiments.
The next step in this project is to actually implement this solution in a real environment, to pilot the solution with users. The goal in the current phase of the project is to measure the effectiveness of these techniques with real employees and real phishing e-mails, and ultimately to see whether it does indeed successfully relieve employees and SOC analysts, while improving models to fight targeted phishing.
“We are very curious to see what employees’ experiences will be with the AIWARENESS solution in a pilot experiment. Moreover, if we eventually use this in production, I would also be curious to see how it impacts the performance of Microsoft Outlook. Up to this point, everything looks promising,” says Verhagen.
Verbrugge adds: “The insights that we will gain from our employees are extremely valuable. It is important to quantitatively prove the value of the solution and draw the right conclusions. This takes time and effort.”
Conclusion
To combat targeted phishing, the PCSI is working on a holistic approach in which we aim to find the synergy between the employee with their context knowledge, the SOC analyst with their security expert knowledge, and machine learning technology making sense of big data. This approach will aid and empower the employee in becoming more resilient, which aligns with de Volksbank’s philosophy: “bankieren met de menselijke maat” (banking on the human scale). Verhagen explains: “Phishing is a complex problem to which many people are exposed on a daily basis. It is very interesting to see how different sources of expertise at the PCSI partners complement each other. Personally, I have no expertise in crafting new AI models, but reasoning on the basis of my own expertise, I can formulate valuable contributions and requirements. That is what makes this project special to me.”
In a broader perspective, AIWARENESS should help reduce the number of phishing incidents: fewer phishing e-mails reaching the inbox of employees will result in fewer security incidents in general, as successful phishing attacks are often the start of other security events such as corporate espionage, theft of credentials, loss of money, or ransomware attacks. Indirectly, AIWARENESS therefore has the potential to further reduce the workload of the SOC. “The adversaries will continue to develop their tactics and techniques, and we need to stay on top of that. We believe this solution will result in fewer phishing incidents. But I also expect the adversaries to move towards social engineering via social media. We will need to continue developing without pause,” concludes Verbrugge.
Share this page