On the implicit bias of SGD in deep learning
Tali's work emphasized the tradeoff between compression and information preservation. In this talk I will explore this theme in the context of deep learning. Artificial neural networks have recently revolutionized the field of machine learning. However, we still do not have sufficient theoretical understanding of how such models can be successfully learned. Two specific questions in this context are: how can neural nets be learned despite the non-convexity of the learning problem, and how can they generalize well despite often having more parameters than training data. I will describe our recent work showing that gradient-descent optimization indeed leads to 'simpler' models, where simplicity is captured by lower weight norm and in some cases clustering of weight vectors. We demonstrate this for several teacher and student architectures, including learning linear teachers with ReLU networks, learning boolean functions and learning convolutional pattern detection architectures.
Blindspot: Hidden Biases of Good People
Mahzarin Banaji and her colleague coined the term “implicit bias” in the mid-1990s to refer to behavior that occurs without conscious awareness. Today, Professor Banaji is Cabot Professor of Social Ethics in the Department of Psychology at Harvard University, a member of the American Academy of Arts and Sciences, the National Academy of Sciences and has received numerous awards for her scientific contributions. The purpose of the seminar, Blindspot: Hidden Biases of Good People, is to reveal the surprising and even perplexing ways in which we make errors in assessing and evaluating others when we recruit and hire, onboard and promote, lead teams, undertake succession planning, and work on behalf of our clients or the public we serve. It is Professor Banaji’s belief that people intend well and that the inconsistency we see, between values and behavior, comes from a lack of awareness. But because implicit bias is pervasive, we must rely on scientific evidence to “outsmart” our minds. If we do so, we will be more likely to reach the life goals we have chosen for ourselves and to serve better the organizations for which we work.