CCMPerf: a benchmarking tool for CORBA component model implementations

Commercial off-the-shelf (COTS) middleware is now widely used to develop distributed real-time and embedded (DRE) systems. DRE systems are themselves increasingly combined to form "systems of systems" that have diverse quality of service (QoS) requirements. Earlier generations of COTS middleware, such as Object Request Brokers (ORBs) based on the CORBA 2.x standard, do not facilitate the separation of QoS policies from application functionality, which makes it hard to configure and validate complex DRE applications. The new generation of component middleware, such as the CORBA component model (CCM) based on the CORBA 3.0 standard, addresses the limitations of earlier generation middleware by establishing standards for implementing, packaging, assembling, and deploying component implementations. There has been little systematic empirical study of the performance characteristics of component middleware implementations in the context of DRE systems. This paper therefore provides three contributions to the study of CCM for DRE systems. First, we describe the challenges involved in benchmarking different CORBA component model (CCM) implementations. Second, we describe key criteria for comparing different CCM implementations using key black-box and white-box metrics. Third, we describe the design of our CCMPerf benchmarking suite to illustrate test categories that evaluate aspects of CCM implementation to determine their suitability for the DRE domain. We demonstrate CCMPerf by using it to collect metrics from a CCM implementation designed for DRE applications.