QoE evaluation of multimedia transmission over wireless networks

Many simulation tools (e.g., NS-2) can examine the quality of service (QoS) of networks but cannot demonstrate the visual and auditory effects of wireless transmission on multimedia quality. This paper presents a wireless multimedia simulator (WMS), which uses a compact graphical user interface to present the real-time packet delay with the playback of streaming media over a wireless channel based on classic radio channel models and IEEE 802.11 medium access control. By using captured packets and reproduced traces, WMS can demonstrate the visual and auditory effects of fading errors, packet delay and loss. Leveraging the real-time playback function, WMS enables quality of experience (QoE) evaluation of multimedia transmission in a controlled wireless environment. We carried out QoE evaluation with 30 participants for 104 test cases comprising 2 videos and 2 audio clips produced by WMS. The valuable test results enable us to quantify the relationship of QoE in terms of mean opinion score (MOS) with network traffic load and QoS metrics such as bit error probability (BEP). We find the subjective QoE is sensitive to media content although consistent with objective QoS metrics. Statistical difference-of-means tests show the video with slower motion and fewer colours is likely to offer better delay tolerance, and audio is less sensitive to bit errors while video is more resistant to network congestion.