High speed network traffic capture and analysis

Abstract

Network packet capture performs essential functions in network management such as attack analysis, network troubleshooting, and performance debugging. As the network bandwidth exceeds 10s of Gbps, the demand for scalable packet capture and retrieval is rapidly increasing. However, existing software-based packet capture systems neither provide high performance nor support flow-level indexing for fast query response. This would either prevent important packets from being stored or make it too slow to retrieve relevant flows. In this dissertation, I present FloSIS, a highly scalable, software-based flow storing and indexing system. FloSIS is characterized as the following three aspects. First, it exercises full parallelism in multiple CPU cores and disks at all stages of packet processing. Second, it constructs two-stage flow-level indexes, which helps minimize expensive disk access for user queries. It also stores the packets in the same flow at a contiguous disk location, which maximizes disk read throughput. Third, I optimize storage usage by flow-level content deduplication at real time. My evaluation shows that FloSIS on a dual octa-core CPU machine with 24 HDDs achieves 30 Gbps of zero-drop performance with real traffic, consuming only 0.25% of the space for indexing. I use FloSIS to analyze real network traffic of a public data center in South Korea, that offers cloud services for business enterprises. I capture the entire traffic of the data center for 20 hours and check packet and flow level communication characteristics. In addition, I confirm the application protocols to figure out the applications running in the data center. Finally, I show two network management cases, anomaly detection and SDN system-level simulation.