DocumentCode :
787281
Title :
Learn more, sample less: control of volume and variance in network measurement
Author :
Duffield, Nick ; Lund, Carsten ; Thorup, Mikkel
Author_Institution :
AT&T Labs.-Res., Florham Park, NJ, USA
Volume :
51
Issue :
5
fYear :
2005
fDate :
5/1/2005 12:00:00 AM
Firstpage :
1756
Lastpage :
1775
Abstract :
This paper deals with sampling objects from a large stream. Each object possesses a size, and the aim is to be able to estimate the total size of an arbitrary subset of objects whose composition is not known at the time of sampling. This problem is motivated from network measurements in which the objects are flow records exported by routers and the sizes are the number of packet or bytes reported in the record. Subsets of interest could be flows from a certain customer or flows from a worm attack. This paper introduces threshold sampling as a sampling scheme that optimally controls the expected volume of samples and the variance of estimators over any classification of flows. It provides algorithms for dynamic control of sample volumes and evaluates them on flow data gathered from a commercial Internet Protocol (IP) network. The algorithms are simple to implement and robust to variation in network conditions. The work reported here has been applied in the measurement infrastructure of the commercial IP network. To not have employed sampling would have entailed an order of magnitude greater capital expenditure to accommodate the measurement traffic and its processing.
Keywords :
IP networks; sampling methods; transport protocols; IP network; Internet protocol; network measurement; threshold sampling; Fluid flow measurement; Heuristic algorithms; IP networks; Optimal control; Protocols; Robustness; Sampling methods; Size measurement; Telecommunication traffic; Volume measurement; Estimation; Internet measurement; flows; sampling; variance reduction;
fLanguage :
English
Journal_Title :
Information Theory, IEEE Transactions on
Publisher :
ieee
ISSN :
0018-9448
Type :
jour
DOI :
10.1109/TIT.2005.846400
Filename :
1424313
Link To Document :
بازگشت