Does methodology affect the ability to monitor tobacco control activities? implications for HEDIS and other performance measures

Background It is unclear whether methodological differences in sample size, survey methods, and analysis approach significantly affect the ability to accurately monitor tobacco control activities and to make rate comparisons. Methods Questionnaires were sent to 64,764 members of nine health plans in diverse settings soon after their visit to a primary care clinician. Of these 41,677 completed responses were received. We compared responses received by mail and by telephone follow-up for the percentage of smokers, characteristics of smokers, and their rates of reporting physician cessation counseling. Results Overall, 10.2% were current cigarette smokers, but the proportion was 8.6% for mail responders and 17.2% for phone follow-up responders. Smokers identified by phone follow-up were different from mail responders in most demographic and smoking characteristics and their reports of clinical smoking cessation activities differed for six of nine clinician smoking cessation actions. Calculating advice rates as a proportion of visits produced lower rates with more dispersion among plan rates than doing so without accounting for visit variation. Conclusions Smoking surveys using only mailed questionnaires dramatically undersample smokers, especially in some demographic groups. Comparisons of tobacco counseling among health plans can be improved by ensuring an adequate sample size and response rate and by analyzing by frequency of quit advice.