Using statistical network link model for routing in ad hoc networks with multi-agent reinforcement learning

Existing mobile ad-hoc routing protocols are based on a discrete, bimodal model for links between nodes: a link either exists or is broken. This model cannot distinguish transmissions which fail due to interference or congestion from those which fail due to their target being out of transmission range. A statistical network link model is introduced to represent the quality of the link by a statistical measure of link performance. Because of dynamic topologies properties of ad hoc network, each node can´t achieve the global information about other nodes in the whole network. In order to define optimal routes in a network with links of variable quality, ad-hoc routing is modeled as a sequential decision making problem with incomplete information. More precisely, ad hoc routing is mapped into a multi-agent reinforcement learning problem involving a partially observable Markov decision processes (POMDPs). A new routing protocol called SNL-Q is proposed based on a combination of continuous (rather than discrete) model for links and the POMDP model within the ad hoc network. Different scenario-based performance evaluations of the protocol in NS-2 are presented. In comparisons with AODV and DSR, SNL-Q routing exhibits improved performance in congested wireless networks.