Network switches have been used to forward packets from one node to another node. Such network switch devices include a first network port for receiving packets from a first node, and a second network port for passing the packets to a second node.
In a packet-switching network, the transmission, routing, forwarding, and the like of messages between the terminals in the packet-switching network are broken into one or more packets. Typically, data packets transmitted or routed through the packet switching network comprise three elements: a header, a payload, and a trailer. The header may comprise several identifiers such as source and destination terminal addresses, VLAN tag, packet size, packet protocol, and the like. The payload is the core data for delivery, other than header or trailer, which is being transmitted. The trailer typically identifies the end of the packet and may comprise error checking information (e.g., CRC information). Data packets may conform to a number of packet formats such as IEEE 802.1D or 802.3.
Associated with each terminal in the packet-switching network is a unique terminal address. Each of the packets of a message has a source terminal address, a destination terminal address, and a payload, which contains at least a portion of the message. The source terminal address is the terminal address of the source terminal of the packet. The destination terminal address is the terminal address of the destination terminal of the packet. Further, each of the packets of a message may take different paths to the destination terminal, depending on the availability of communication channels, and may arrive at different times. The complete message is reassembled from the packets of the message at the destination terminal. One skilled in the art commonly refers to the source terminal address and the destination terminal address as the source address and the destination address, respectively.
Applicant of the subject application has determined that packet switch devices (appliances) can be used to forward a copy of packets (either obtained through a SPAN port of a switch or router, or by making a copy of each packet through its built-in tap modules) in the packet-switching network, to network monitoring or security tools for analysis thereby. Such packet switch appliances may have one or more network ports for connection to the packet-switching network and one or more instrument ports connected to one or more network instruments for monitoring packet traffic, such as packet sniffers, intrusion detection systems, application monitors, or forensic recorders.
Sometimes, users may wish to deploy various network instruments for monitoring packet traffic. In order to monitor every packet that goes through a switch, a span port is usually set up such that a copy of every packet is made when they pass through the ports, ingress or egress. Therefore, for a packet that enters in one port of the switch and then egresses out of another port of the same switch, at least two copies of this packet are sent out of the span port. If this packet is a multicast packet, then the switch will send out multiple copies of this packet through multiple ports, and hence the span port will send out even more copies of this packet. In this kind of situation, the copies of the packet coming out of the span port are usually identical.
In other situations, the switch may change the VLAN tag of the packet such that within the copies of this packet, some of them may have different VLAN tags. Also, the packet may go through a router, in which case the destination MAC address or even some fields in the IP header information may have been changed, such as the TTL field, but the payload remains the same.
If copies of packets are made at other network devices and forwarded to the same analysis tool, the analysis tool may be receiving packets with the same payload at slightly different times. The generation of duplicate packets can also occur in redundant network segments depending on the location of tapping points within the segments that are used to tap packets to be forwarded to an analysis tool. That is, depending on where taps are located in a redundant network segment, multiple copies of the same packet or multiple copies of packets with the same payload (i.e., packets that only have different destination and/or source address) may be generated. The presence of such duplicate packets can prevent accurate analysis from occurring, can negatively influence available bandwidth in the network, or can overwhelm a tool that does not have the performance to handle all these packets which carry duplicated information. Therefore, Applicant of the subject application determines that it would be desirable to have a new system and method for removing duplicate packets prior to any analysis or monitoring of the packets.