Symptoms or Diseases: Understanding Reliability Issues in Deep Learning and Potential Ways to Fix Them

Despite deep learning's wide range of applications, a satisfactory understanding of its fundamental properties such as robustness, interpretability, generalizability and broadly speaking the scope of its applicability still eludes us. These properties are essential to characterize performance guarantees and to identify and prevent failure modes of deep models. Focusing on model sensitivity against various types of adversarial and natural distributional shifts, a common approach is to treat these issues as separate “diseases” and mitigate them independently. Pushing back on this widely used approach, I will show that in fact explicit tradeoffs exist between adversarial and natural distributional robustness. Instead, I will present some evidence advocating for a new school of thought to consider these issues as “symptoms” of a common disease: current models rely heavily on spurious and noisy features instead of meaningful and core ones in their predictions. This is partially owing to the lack of diverse samples and proper supervision in training. I will then present potential ways to tackle this root cause via developing new learning paradigms based on novel data formulations.