Optimization Strategies for Large-scale Network Traffic Analysis Using Spark

Manuscript submitted June 2, 2026; accepted July 22, 2026; published August 28, 2026

Abstract—This paper presents a comprehensive big data analytics approach for cybersecurity threat detection using a network flow dataset, UWF-ZeekData22. A scalable solution using Apache Spark for distributed processing is presented. Principal Component Analysis (PCA) was used for dimensionality reduction and three different classifiers were used for threat detection. 2,044,734 network traffic records with 23 features were processed, achieving 99.999% variance retention through PCA dimensionality reduction and 100% classification accuracy. The study demonstrates significant performance optimization through systematic parameter tuning, reducing execution time from 9.024 s to 4.567 s (62.7% improvement) while maintaining perfect classification accuracy. These findings contribute to the field of big data cybersecurity analytics by providing evidence-based optimization strategies for large-scale network traffic analysis.

keywords—big data analytics, cybersecurity, Apache spark, principal component analysis, network traffic analysis, machine learning, random forest, support vector machines, naï ve bayes

Cite: Trust Museta, Sikha S. Bagui, Dustin Mink, Subhash C. Bagui, Aaron Webb, Myles Brown, Venkata SriKrishna Garikapati,"Optimization Strategies for Large-scale Network Traffic Analysis Using Spark," Journal of Advances in Artificial Intelligence, vol. 4, no. 3, pp. 164-189, 2026. doi: 10.18178/JAAI.2026.4.3.164-189

Copyright © 2026 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).