Machine Learning is being widely applied across all aspects of our lives, but it is often hard to explain why a machine learning system arrived at a specific decision. In this workshop, we will motivate the need for developing explainable models, and go over some recent developments in the domain and see them in action.