Tools for modeling the user-traffic

The significance of the simulation outcome depends on the network load or traffic applied the simulation model. Therefore it is required to use real, user-generated traffic for the simulation. The following list outlines tools for the collection of user-generated traffic, the analysis of thus data and related software i.e. for manipulating the recorded data.

Data Collection (sniffing)

libpcap/tcpdump

Tcpdump is the most used tool for network monitoring and data acquisition. Tcpdump uses libpcap (Packet Capture library), a system-independent interface for user-level packet capture. Libpcap/tcpdump was originally developed by the Network Research Group at the Lawrence Berkeley National Laboratory. Tcpdump provides a standard packet capture interface, a common dump format, basic packet decoding features and can filter packets in various ways.

Platform: Unix (WinPCAP/WinDUMP is the porting to the Windows platform of libpcap/tcpdump)
Business Model: Open Source (BSD License)
Internet: http://www.tcpdump.org
Documentation: http://www.tcpdump.org/tcpdump_man.html

ngrep

Ngrep strives to provide most of GNU grep's common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes TCP, UDP and ICMP across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands bpf filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.

Platform: Unix/Windows
Business Model: Open Source ("BSD-style" License, see file COPYRIGHT in source package for details)
Internet: http://www.packetfactory.net/Projects/ngrep
Documentation: http://www.packetfactory.net/Projects/ngrep/ngrep.8.html

TTT: Tele Traffic Tapper

ttt is yet another descendant of tcpdump but it is capable of real-time, graphical, and remote traffic-monitoring. ttt won't replace tcpdump, rather, it helps you find out what to look into with tcpdump. ttt monitors the network and automatically picks up the main contributors of the traffic within the time window. The graphs are updated every second by default. ttt can replay a trace file at a given speed so that it is possible to replay a 1-hour trace in 1 minute.
Features:
  • Automatic Ranking of Protocols and Hosts
  • Real-Time Monitoring
  • Remote Monitoring with IP-Multicast support
  • Accepts tcpdump output
  • IPv6 Aware (experimental)
  • Portable and easy to customize
Platform: Unix
Business Model: Open Source("BSD-style" License, see file Readme in source package for details)
Internet: http://www.csl.sony.co.jp/person/kjc/kjc/software.html#ttt

Data Analysis

capinfo

Capinfo is a tool for displaying statistics about network traffic from files saved with tcpdump or snoop. For pcap files, capinfo will display the following information:
  • byte order of the capture file
  • libpcap version used for packet capture
  • capture timezone (usually empty)
  • significant figures of timestamps (usually empty)
  • snap length: maximum packet size captured
  • data link type
  • number of packets and bytes
  • number of snaplen truncated packets
  • timestamp of the first and last packets
Platform: Unix
Business Model: Open Source
Internet: http://tcpreplay.sourceforge.net
Documentation: http://tcpreplay.sourceforge.net/capinfo.txt

CoralReef

CoralReef is a comprehensive software suite developed by CAIDA to collect and analyze data from passive Internet traffic monitors, in real time or from trace files. Realtime monitoring support includes system network interfaces (via libpcap), FreeBSD drivers for Apptel POINT (OC12 and OC3 ATM) and FORE ATM (OC3 ATM) cards, and support for Linux drivers for WAND DAG (OC3 and OC12, POS and ATM) cards. The package also includes programming APIs for C and perl, and applications for capture, analysis, and web report generation. This package is maintained by CAIDA developers with the support and collaboration of the Internet measurement community. CoralReef is the evolutionary successor of the Coral package and supersedes it.

Platform: Unix
Business Model: Free for educational, research and non-profit purposes (see file Copyright in source package for details)
Internet: http://www.caida.org/tools/measurement/coralreef/
Documentation: http://www.caida.org/tools/measurement/coralreef/doc/doc/index.html

Ethereal

Ethereal is a GUI network protocol analyzer. It can examine data from a live network or from a capture file on disk. You can interactively browse the capture data, viewing summary and detail information for each packet. Ethereal can assemble all the packets in a TCP conversation and show you the ASCII (or EBCDIC, or hex) data in that conversation. Display filters in Ethereal are very powerful; more fields are filterable in Ethereal than in other protocol analyzers.
Features: (see http://www.ethereal.com/introduction.html for a complete list of features)
  • Data can be captured from a live network connection (via libpcap), or read from a capture file.
  • Ethereal can read capture files from tcpdump (libpcap), NAI's Sniffer™ and Sniffer™ Pro (compressed and uncompressed), Sun snoop and atmsnoop, AIX's iptrace, Microsoft's Network Monitor, Novell's LANalyzer, Cisco Secure IDS iplog etc.
  • Live data can be read from Ethernet, FDDI, PPP, Token-Ring, IEEE 802.11, Classical IP over ATM.
  • 280 protocols can currently be dissected.
  • Output can be saved or printed as plain text or PostScript®.
Platform: Unix/Windows
Business Model: Open Source (GPL)
Internet: http://www.ethereal.com
Documentation: http://www.ethereal.com/docs/user-guide/ or http://www.ethereal.com/ethereal.1.html

Microsoft Network Monitor

Network Monitor is a component of Microsoft® Systems Management Server (SMS) that enables you to detect and troubleshoot problems on LANs, WANs, and serial links running the Microsoft® Remote Access Server (RAS). Network Monitor provides real-time and post-capture modes of network data analysis. In real-time analysis, network traffic is examined by real-time monitors. These monitors test network traffic for a specific set of conditions, and when those conditions are detected, display events, which may prompt end-user action. For example, a monitor can detect conditions that indicate a SYN attack and aid a network administer to respond to the potential attack. In post-capture analysis, network traffic is saved in a proprietary capture file so that the captured data can be analyzed later. In this case, analysis can be in the form of protocol parsers picking out specific network frame types and displaying the frame data in the Network Monitor UI; or analysis can be in the form of experts examining the network data and displaying a report (experts may also manipulate the network data).
Features:
  • Captures network data in real-time or delayed mode.
  • Provides filtering capabilities when capturing data.
  • Uses monitors for real-time analysis and security.
  • Uses experts and parsers for detailed post-capture analysis.
Platform: Windows
Business Model: Commercial
Internet: MSDN Library: About network monitor 2.0
Documentation: MSDN Library: Using network monitor 2.0

Network Associates Sniffer™

The Sniffer product family covers different fields of application (Distributed, Portable and Wireless Environment). Sniffer solutions monitor, troubleshoot, analyze, report on, and proactively manage network performance. They ensure peak performance throughout the enterprise infrastructure, across all LAN, WAN and high-speed topologies, from 10/100 Ethernet to the latest high-speed Asynchronous ATM, Gigabit, and Packet-over-SONET (PoS) backbones.
Features of Sniffer Basic: (see http://www.sniffer.com for a complete list of features)
  • Sniffer Basic is tailored for MIS staff supporting small businesses, remote offices, and departmental networks. It's the ideal tool for front-line IS personnel in their mission to cost-effectively diagnose and solve common network problems.
  • Intuitive, consistent user interface for quick capture and display of data.
  • Real-Time Packet Capture and Real-Time Analysis
  • Traffic Generation
  • Supported Protocols Families: Ethernet, Token Ring, VLAN, Novell NetWare 5, IP (TCP/UDP), PPP, AppleTalk, IBM, Microsoft NT and SMB, SNA, Banyan VINES, XNS, DECnet and more.
Platform: Windows
Business Model: Commercial
Internet: http://www.nai.com, http://www.sniffer.com (product overview: http://www.sniffer.com/products/default.asp)
Documentation: http://www.sniffer.com/products/library/default.asp?A=4

OPNET ACE (Application Characterization Environment)

ACE, the Application Characterization Environment for the OPNET family of products, cracks application performance wide open with precise visualization of application dynamics: at the source, on the server, or anywhere on the network. You can troubleshoot existing application performance problems or plan application deployments. ACE graphically reconstructs application behavior on any network segment into intuitive, in-depth diagrams. ACE diagnoses the factors contributing to delay so that you can make the necessary adjustments to your application and/or your network to optimize performance. ACE's data can be imported to OPNET simulations, giving you unparalleled insights into your application and network performance under varying conditions, configurations, and "what-if" scenarios-today and into the future.
Features: (see http://www.opnet.com for a complete list of features)
  • Capture Application Traces: ACE analyzes application transactions from your testbed or production network by directly capturing packet trace files or by importing traces from popular network analyzers, including from Network Associates' Sniffer products and tcpdump. You can manage multiple capture agents easily. ACE can filter the captured data to focus on important transactions. ACE is unique in its ability to automatically synchronize multiple application traces of the same transaction from different network segments.
  • Visualize Transactions: ACE enables you to visualize application behavior by parsing an application trace file consisting of transactions captured from a test-bed or a production network. ACE then graphically reconstructs application behavior on any network segment into intuitive, powerful diagrams. ACE displays transactions at both the application message level and network packet level to provide a clear view of the important dynamics. ACE graphically displays the dependencies among application messages. This is extremely useful for accurately pinpointing the causes of unnecessary delays.
  • Diagnose Performance Problems: In addition to powerful visualization capabilities, ACE enables several techniques for diagnosing end-to-end performance problems. ACE applies expert knowledge to captured application data for automated troubleshooting. Sources of delay are summarized in convenient diagrams. Thresholds for key application statistics are used to generate informative reports that characterize problems. ACE also provides recommendations that support fine-tuning of specific application and network parameters for achieving desired performance levels. The ACE Decode Module (ADM) significantly increases the value of ACE with the industry's most comprehensive application and protocol decode engine (from Sniffer Technologies). Decode over 400 protocols and applications, including database applications (Oracle, Sybase, MS-SQL Server); HTTP, FTP, Email (SMTP, POP3); Novell (IPX and NetWare 5); Microsoft (SMB, NetBIOS, MSRPC, netBEUI); Sun NFS and RPC; Cisco (routing and VLAN).
  • Validate Solutions: Used with OPNET's IT Guru or SP Guru, ACE provides visibility into the performance of applications in different deployment environments. OPNET Guru solutions offer advanced analytical modeling for quick results, as well as advanced hybrid simulation technology to facilitate more sophisticated analysis of response times in different "what-if" scenarios. QuickRecode allows you to recode poorly written applications "virtually" and evaluate the impact of your changes to application design performance. OPNET's QuickPredict feature for IT Guru and SP Guru provides quick analysis of the impact of key network parameters such as latency, bandwidth, packet loss, congestion, and TCP window sizes on application response times.
Platform: Unix (Agents only)/Windows
Business Model: Commercial
Internet: http://www.opnet.com/products/modules/ace/home.html

tcpdstat

Tcpdstat a program to extract statistical information from tcpdump trace files. Tcpdstat reads a tcpdump file using the pcap library and prints the statistics of a trace. The output includes the number of packets, the average rate and its standard deviation, the number of unique source and destination address pairs, and the breakdown of protocols. Tcpdstat is intended to provide a rough idea of the trace content. The output can be easily converted to a HTTP format. It also provides helpful information to find anomaly in a trace.

Platform: Unix
Business Model: Open Source ("BSD-style" License, see source file tcpdstat/stat.c in source package for details)
Internet: http://tracer.csl.sony.co.jp/mawi/ and http://www.csl.sony.co.jp/~kjc/papers/freenix2000/

tcpflow

Tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis or debugging. A program like "tcpdump" shows a summary of packets seen on the wire, but usually doesn't store the data that's actually being transmitted. In contrast, tcpflow reconstructs the actual data streams and stores each flow in a separate file for later analysis. Tcpflow understands sequence numbers and will correctly reconstruct data streams regardless of retransmissions or out-of-order delivery. However, it currently does not understand IP fragments; flows containing IP fragments will not be recorded properly. Tcpflow is based on the libpcap and therefore supports the same rich filtering expressions that programs like tcpdump support.

Platform: Unix
Business Model: Open Source (GPL)
Internet: http://www.circlemud.org/~jelson/software/tcpflow/
Documentation: http://www.circlemud.org/~jelson/software/tcpflow/tcpflow.1.html

tcpstat

Tcpstat reports certain network interface statistics. Tcpstat gets its information by either monitoring a specific interface, or by reading previously saved tcpdump data from a file. It provides more than 15 different types of statistics including the number of packets passed through the interface, the average size of each packet, the standard deviation of the packet size and the bandwidth in bits per second.

Platform: Unix
Business Model: Open Source (BSD License)
Internet: http://www.frenchfries.net/paul/tcpstat/

tcptrace

Tcptrace is a tool for analysis of TCP dump files. It can take as input the files produced by several popular packet-capture programs, including tcpdump, snoop, etherpeek, HP Net Metrix, and WinDump. Tcptrace can produce several different types of output containing information on each connection seen, such as elapsed time, bytes and segments sent and recieved, retransmissions, round trip times, window advertisements, throughput, and more. It can also reconstruct streams and produce a number of graphs for further analysis.

Platform: Unix/Windows
Business Model: Open Source (GPL)
Internet: http://www.tcptrace.org
Documentation: http://www.tcptrace.org/manual.html

Related Programs

FFT-FGN-C

FFT-FGN-C is a program for synthesizing a type of self-similar process known as fractional Gaussian noise. The program is fast but approximate. Fractional Gaussian noise is only one type of self-similar process. When using this program for synthesizing network traffic, you must keep in mind that it may be that the traffic you seek is better modeled using one of the other processes. The output of the program (perhaps transformed - see the documentation) describes an arrival process, not an interarrival process. That is, the output can be interpreted as a count of network arrivals during a particular time interval; but if your application requires exact arrival times, a separate mechanism is needed to transform the count into individual arrivals (doing this is a research area).

Platform: Unix
Business Model: Open Source ("BSD-style" License, see file Copying in source package for details)
Internet: http://ita.ee.lbl.gov/html/contrib/fft_fgn_c.html
Documentation: http://ita.ee.lbl.gov/html/contrib/fft_fgn_c-readme.txt and Fast Approximation of Self-Similar Network Traffic

Libnet

Libnet is a collection of routines to help with the construction and handling of network packets. It provides a portable framework for low-level network packet shaping, handling and injection. Libnet features portable packet creation interfaces at the IP layer and link layer, as well as a host of supplementary and complementary functionality. Using libnet, quick and simple packet assembly applications can be whipped up with little effort.

Platform: Unix
Business Model: Open Source ("BSD-style" License, see file doc/Copying in source package for details)
Internet: http://www.packetfactory.net/libnet/
Documentation: http://www.packetfactory.net/libnet/manual

Sanitize

Sanitize is a collection of five Bourne shell scripts for reducing tcpdump traces in order to address security and privacy concerns, by renumbering hosts and stripping out packet contents. Each script takes as input a tcpdump trace file and generates to stdout a reduced, ASCII file in fixed-column format. The scripts are:
  • sanitize-tcp - reduce all TCP packets
  • sanitize-syn-fin - reduce TCP SYN/FIN/RST packets
  • sanitize-udp - reduce UDP packets
  • sanitize-encap - reduce encapsulated IP packets (usually MBone)
  • sanitize-other - reduce any other types of packets
The scripts discard all packet contents. The size of the packet data contents are retained only for TCP traffic. For encapsulated IP traffic (usually MBone), and for non-TCP, non-UDP, non-encap-IP traffic, only timestamps are generated. The script for reducing TCP SYN/FIN/RST packets is separate from the one for reducing all TCP packets, so the host renumbering performed by each will be independent.

Platform: Unix
Business Model: Open Source
Internet: http://ita.ee.lbl.gov/html/contrib/sanitize.html
Documentation: http://ita.ee.lbl.gov/html/contrib/sanitize-readme.txt

Sniff

Sniff makes output from the tcpdump program easier to read and parse. It features a coloured console and fully customisable output. Sniff directly accepts tcpdump options (including parsing from packet files)

Platform: Unix
Business Model: Open Source
Internet: http://www.thedumbterminal.co.uk/software/sniff.html

TCP replay

Replays a pcap file on an interface using Libnet.

Platform: Unix
Business Model: Open Source (BSD License)
Internet: http://tcpreplay.sourceforge.net/
Documentation: http://tcpreplay.sourceforge.net/tcpreplay.txt

TCP-Reduce

TCP-Reduce is a collection of Bourne shell scripts for reducing tcpdump traces to one-line summaries of each TCP connection present in the trace. The scripts are:
  • tcp-reduce - takes a tcpdump trace file as an argument and writes a sorted summary to stdout.
  • tcp-conn - an internal awk script that does all the work
  • tcp-summary - an awk script that generates a per-protocol summary of all of the TCP connections produced by tcp-reduce.
The scripts look only at TCP SYN/FIN/RST packets. Connections without SYN packets in the trace (such as those on-going at the beginning of the trace) will not appear in the summary. Garbaged packets (those missing some of their contents) are reported to stderr as bogon's and are discarded. Occasionally the script gets fooled by retransmissions with altered sequence numbers, and reports erroneous huge connection sizes - always check large connections (say 100 MB or more) for plausibility.

Platform: Unix
Business Model: Open Source ("BSD-style" License, see script file tcp-conn in source package for details)
Internet: http://ita.ee.lbl.gov/html/contrib/tcp-reduce.html
Documentation: http://ita.ee.lbl.gov/html/contrib/tcp-reduce-doc.html

tcpdpriv

Tcpdpriv is program for eliminating confidential information (user data and addresses) from packets collected on a network interface (or, from trace files created using the -w argument to tcpdump). Tcpdpriv removes the payload of TCP and UDP, and the entire IP payload for other protocols. It implements several address scrambling methods; the sequential numbering method and its variants, and a hash method with preserving address prefix.

Platform: Unix
Business Model: Open Source ("BSD-style" License, see source file tcpdpriv.c in source package for details)
Internet: http://ita.ee.lbl.gov/html/contrib/tcpdpriv.html
Documentation: http://ita.ee.lbl.gov/html/contrib/tcpdpriv.0.txt

tcpslice

Tcpslice allows the subsetting/merging of captured packet trace files (generated using tcpdump's -w flag) by means of various criterea.

Platform: Unix
Business Model: Open Source
Download: ftp://ftp.ee.lbl.gov/tcpslice.tar.gz

Tracelook

Tracelook is an Tcl/TK program for graphically viewing the contents of trace files created using the -w argument to tcpdump. Tracelook should look at all protocols, but presently only looks at TCP connections. The program is slow and uses system resources prodigiously.

Platform: Unix
Business Model: Open Source ("BSD-style" License, see file tracelook in source package for details)
Internet: http://ita.ee.lbl.gov/html/contrib/tracelook.html
Documentation: http://ita.ee.lbl.gov/html/contrib/tracelook.1.cat.txt

wide-tcpdpriv

The original tcpdpriv lacks several features:
  • It does not support IPv6.
  • It does not preserve TCP options that are essential to analyzing TCP behaviors.
  • It does not preserve other protocols such as ICMP, ARP and DNS.
The MAWI (Measurement and Analysis on the WIDE Internet) working group of the WIDE Project modified the original tcpdpriv to support these features. The default settings are also changed to meet their requirements since the options seem to be too complex and a mistake of option selection could be fatal to user privacy.
Modified defaults: The original default values are the most conservative ones. They are modified to meet the WIDE standard setting and equivalent to -A50 -C4 -M99 -P99 (see wide-tcpdpriv/Readme in source package for explanation).
Added features:
  • IPv6 support
  • ICMPv6, TCP, UDP, IPv4 over IPv6 are recognized.
  • preserve TCP options
  • ARP support
  • ICMP support
  • DLT_ATM_RFC1483 support
Platform: Unix
Business Model: Open Source ("BSD-style" License, see source file wide-tcpdpriv/tcpdpriv.c in source package for details)
Internet: http://tracer.csl.sony.co.jp/mawi/ and http://www.csl.sony.co.jp/~kjc/papers/freenix2000/
Documentation: http://ita.ee.lbl.gov/html/contrib/tcpdpriv.0.txt and http://tracer.csl.sony.co.jp/mawi/guideline.txt