Abstract In the era of big data, it is a problem to be solved for promoting the healthy development of the Internet and the Internet+, protecting the information security of individuals, institutions and countries. Hence, this paper constructs a collaborative detection system of cyber security threats in big data. Firstly, it describes the log collection model of Flume, the data cache of Kafka, and the data process of Esper; then it designs one-to-many log collection, consistent data cache, Complex Event Processing (CEP) data process using event query and event pattern matching; finally, it tests on the datasets and analyzes the results from six aspects. The results demonstrate that the system has good reliability, high efficiency and accurate detection results; moreover, the system has the advantages of low cost and flexible operation.

References

[1] Aniello L., Baldoni R., Chockler G., Laventman G., Lodi G., and Vigfusson Y., “Agilis: An Internet-Scale Distributed Event Processing System for Collaborative Detection of Cyber Attacks,” MIDLAB Technical Report, 2011.

[2] Aniello L., Luna G., Lodi G., and Baldoni R., Collaborative Inter-domain Stealthy Port Scan Detection Using Esper Complex Event Processing, Springer, 2012.

[3] Apache Flume, available at: http://flume.apache.org/, Last Visited 2016.

[4] Apache Kafka, available at: http://kafka.apache.org/, Last Visited, 2016.

[5] Apache Software Foundation, available at: http://hadoop.apache.org/, Last Visited, 2016.

[6] Armbrust M., Bateman D., Xin R., and Zaharia M., “Introduction to Spark 2.0 for Database Researchers,” in Proceedings of 16th International Conference on Management of Data, San Francisco, pp. 2193-2194, 2016.

[7] Baldoni R. and Chockler G., Collaborative Financial Infrastructure Protection, Springer, 2012.

[8] Critical Infrastructure in the Age of Cyber War, http://www.mcafee.com/us/resources/reports/rp- in-crossfire-critical-infrastructure-cyber-war.pdf, Last Visited, 2010.

[9] Cutting D., “Hadoop: Industrial Strength Open Source for Data Intensive Supercomputing,” in Proceedings of 17th Conference on Information and Knowledge Management, Napa Valley, 2008.

[10] DARPA Intrusion Detection Scenario Specific Data Sets, available at: http://www.ll.mit.edu/ideval/data/2000data.html, Last Visited, 2016.

[11] Esper Reference (Version 5.2.0), EsperTech Inc., pp. 253-260, 2015.

[12] EsperTech: Event Series Intelligence, available at: http://www.espertech.com/, Last Visited, 2016.

[13] Global Fraud Report-Annual Edition 2011-2012, Kroll,http://www.krollconsulting.com/fraud- report/2011-12/press-only/, Last Visited, 2011.

[14] Hunt P., Konar M., Junqueira F., and Reed B., “Zookeeper: Wait-free Coordination for Internet- Scale Systems,” Usenix Annual Technical Conference, Berkeley, 2010.

[15] Lamport L., “Fast Paxos,” Distributed Computing, vol. 19, no. 2, pp. 79-103, 2006.

[16] Lodi G., Aniello L., Luna G., and Baldoni R., “An Event-based Platform for Collaborative Threats Detection and Monitoring,” Information Systems, pp. 175-195, 2014.

[17] Luna G., A Collaborative Processing System for Cyber Attacks Detection and Crime Monitoring, Theses, Sapienza University, 2010.

[18] Marz N. and Warren J., Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Manning, 2015.

[19] Sergio C., Andrew P., and Christopher B., “The Diamond Model of Intrusion Analysis,” Technical Report, Center for Cyber Threat Intelligence and Threat Research, 2013.

[20] Singh J., Kaur L., and Gupta S., “A Cross-Layer Based Intrusion Detection Technique for Wireless Networks,” The International Arab Journal of Information Technology, vol. 9, no. 3, pp. 201- 207, 2012.

[21] Staheli D., Mancuso V., Harnasch R., Fulcher C., Chmielinski M., Kearns A., Kelly S., and Vuksani E., “Collaborative Data Analysis and Discovery for Cyber Security,” in Proceedings of 12th Symposium on Usable Privacy and Security, Santa Clara, 2016.

[22] Zuech R., Khoshgoftaar T., and Wald R., “Intrusion Detection and Big Heterogeneous Data: a Survey,” Journal of Big Data, vol. 2, no. 1, 2015. Jiange Zhang received the M.S. degree in computer science and technology from Zhengzhou University, Zhengzhou, China, in 2007, and is currently pursuing the Ph.D. degree at the State Key Laboratory of Mathematical Engineering and Advanced Computing. Her research interests include network security, big data and situation awareness. Yuanbo Guo received his Ph.D. degree in computer science and technology from Xidian University, Xi’an, China. His research interests include network security, network protocol design and analysis, threats detection, and situation awareness. He is currently a full Professor of computer science. Yue Chen received his Ph.D. degree in computer science and technology from Zhengzhou Information Science and Technology Institute, Zhengzhou, China. His research interests include network security, network protocol design and analysis, and advanced computing. He is currently a full Professor of computer science.