Bayesian Thinking on Your Feet: Embedding Generative Models in Reinforcement Learning for Sequentially Revealed Data

Bayesian Thinking on Your Feet: Embedding Generative Models in Reinforcement Learning for Sequentially Revealed Data

Project funded by the National Science Foundation (IIS-1320538)

Principal Investigator: Jordan Boyd‑Graber, Co‑Principal Investigator: Hal Daumé III, University of Maryland

Overview

The project seeks to create algorithms that can “think on their feet,” meaning they incrementally process incoming data and determine the optimal moment to act, requiring advances in both content (generative) models and decision‑making policies.

Two application domains are explored: synchronous (simultaneous) machine translation, where the system must translate a foreign sentence word‑by‑word as it arrives, and quiz‑bowl style question answering, where questions are revealed piece‑by‑piece.

In simultaneous translation, the content model predicts upcoming words in languages with verb‑final order (e.g., German, Japanese) before they are observed, while the policy learns when to trust these predictions versus waiting for more input to balance accuracy and latency.

For question answering, a specially crafted quiz‑bowl dataset provides progressively easier clues; the content model generates answer hypotheses and the policy decides when the confidence is sufficient to commit to an answer.

Beyond the outreach appeal of quiz bowl, the research addresses core natural‑language‑processing challenges such as classification, discourse modeling, and coreference resolution.