Page 124 - ITU Journal Future and evolving technologies Volume 2 (2021), Issue 4 – AI and machine learning solutions in 5G and future networks

ITU Journal on Future and Evolving Technologies, Volume 2 (2021), Issue 4

Fig. 9 – Feature score and importance ranking (Left: RF, Right: XGBoost)

Fig. 10 – Accuracy and precision with different features

failures Node Down, Interface Down, BGP Injection, Table 3 – Comparison of detection accuracy of different algorithms
and Packet Loss & Packet Delay have maintained a
relatively high accuracy rate, while BGP Hijack are rel‑ No. Method Accuracy Training
Time(s)
atively low. 1 XGBoost 0.9369 0.33
The top 150 important features get the best performance 2 LightGBM 0.9333 2.81
which is 94.17% of average accuracy. Also, it can be seen 3 Random Forest 0.9274 0.80
that there are abrupt reductions on Interface Down and 4 MLP 0.8131 1.52
slight reductions on other failures after the number of fea‑ 5 Decision Tree 0.8095 0.38
tures is reduced to 30. So we can use only 30 features to 6 SVM 0.7905 5.53
achieve a good performance, almost the same as the best
performance, and for the following experimentation, we Table 4 – Test cases for different labels
use these 30 features for training.
TypeNo. Type Name Test Cases
1 Node Down 64
= (1)
+ 3 Interface Down 72
9 BGP Injection 156
11 BGP Hijack 180
= (2)
+ 57 Packet Loss and Delay 368

2 ∗ ∗ Eq ((1)), Eq ((2)), Eq ((3)), and Eq ((4)) assigned with the
− = (3)
+ number of True Positive (TP), False Positive (FP), False
Negative (FN), and True Negative (TN).
+
= (4)
+ + + Accuracy is the most intuitive performance measure and
it is simply a ratio of correctly predicted observation to
4.2 Evaluation the total observations. Precision is the ratio of correctly
predicted positive observations to the total predicted pos‑
To measure ML‑based classi iers’ performance, we use itive observations. High precision relates to the low false
the following evaluation metrics: Precision, Recall, F‑ positive rate. The recall is the ratio of correctly pre‑
measure, and Accuracy. These metrics are calculated by dicted positive observations to all observations in the

108 © International Telecommunication Union, 2021