On the Use of Input/Output Queries for Code Search

Context: Programmers frequently compose keyword queries as they use information search engines to look for source code. This syntactic approach to code search is often imprecise and results in wasted efforts looking through irrelevant results. Semantic code search approaches aim to address this weakness by formulating queries that specify behavior, rather than keywords. A recent approach uses input/output examples as queries that illustrate the behavior of desired code. The technical feasibility of this approach has been illustrated, yet the impact of the change in the query model has not been assessed. Objective: We explore the cost and accuracy of using input/output queries for code search from the perspective of the programmer, considering two programming languages, Yahoo! Pipes and SQL. Method: We perform a controlled user study with 109 participants from two groups, students and Mechanical Turk, to assess the cost and accuracy of using input/output search queries. Results: Our results show that programmers can compose input/output queries in the targeted domains with over 92% average accuracy and in less than two minutes. Conclusion: The use of input/output queries does not seem to limit the early promise of semantic searches that depend on it.