A hand-held multimedia translation and interpretation system for diet management

We propose a system for helping individuals who follow a medical diet maintain this diet while visiting countries where a foreign language is spoken. Our focus is on diets where certain foods must either be restricted (e.g., metabolic diseases), avoided (e.g., food intolerance or allergies), or preferably consumed for medical reasons. However, our framework can be used to manage other diets (e.g., vegan) as well. The system is based on the use of a hand-held multimedia device such as a PDA or mobile telephone to analyze and/or disambiguate the content of foods offered on restaurant menus and interpret them in the context of specific diets. The system also provides the option to communicate diet-related instructions or information to a local person (e.g., a waiter) as well as obtain clarifications through dialogue. All computations are performed within the device and do not require a network connection. Real-time text translation is a challenge. We address this challenge with a light-weight, context-specific machine translation method. This method builds on a modification of existing open source Machine Translation (MT) software to obtain a fast and accurate translation. In particular, we describe a method we call n-gram consolidation that joins words in a language pair and increases the accuracy of the translation. We developed and implemented this system on the iPod Touch for English speakers traveling in Spain. Our tests indicate that our translation method yields the correct translation more often than general purpose translation engines such as Google Translate, and does so almost instantaneously. The memory requirements of the application, including the database of picture, are also well within the limits of the device.