SPAN-aggregation and packet brokers. Packets deduplication

One may ask: is it real to be so stupid implementing TAPs and brokers that packets are duplicated? Yes, of course, and it doesn't indicate architects' stupidity. E.g. we need the datacenter traffic analysis. So, it is necessary to mirror datacenter uplinks (no matter Internet or corporate) to have an incoming/outgoing traffic visibility, and aggregation/service layer links according to the datacenter network design. Inbound/outbound packet has no duplicates if it is going to some segment connected via dedicated physical lines, no router/firewall on a stick etc.

Let's assume some network part on the picture 1. TAPs mirror traffic to aggregators and then it is sent to information security systems. Users' connections path to servers is going through at least 2 TAPs copying traffic to the aggregator. As a result security sensors receive much more traffic for analysis.

  

Picture 1. Network fragment.

This issue may be addressed using the packet deduplication function shown below. If SPAN-aggregator is able to do it without false positives or negatives then information security systems efficiency may increase a lot.


Picture 2. Packet deduplication explanation.

 So it is packet deduplication feature in brief.