  Malicious websites are responsible for a majority of the cyber-attacks and
scams today. Malicious URLs are delivered to unsuspecting users via email, text
messages, pop-ups or advertisements. Clicking on or crawling such URLs can
result in compromised email accounts, launching of phishing campaigns, download
of malware, spyware and ransomware, as well as severe monetary losses. A
machine learning based ensemble classification approach is proposed to detect
malicious URLs in emails, which can be extended to other methods of delivery of
malicious URLs. The approach uses static lexical features extracted from the
URL string, with the assumption that these features are notably different for
malicious and benign URLs. The use of such static features is safer and faster
since it does not involve crawling the URLs or blacklist lookups which tend to
introduce a significant amount of latency in producing verdicts. The goal of
the classification was to achieve high sensitivity i.e. detect as many
malicious URLs as possible. URL strings tend to be very unstructured and noisy.
Hence, bagging algorithms were found to be a good fit for the task since they
average out multiple learners trained on different parts of the training data,
thus reducing variance. The classification model was tested on five different
testing sets and produced an average False Negative Rate (FNR) of 0.1%, average
accuracy of 92% and average AUC of 0.98. The model is presently being used in
the FireEye Advanced URL Detection Engine (used to detect malicious URLs in
emails), to generate fast real-time verdicts on URLs. The malicious URL
detections from the engine have gone up by 22% since the deployment of the
model into the engine workflow. The results obtained show noteworthy evidence
that a purely lexical approach can be used to detect malicious URLs.

Cryptography and Security