Language-based interactions promise natural and easy access to explanations of machine learning decisions. As complexity of current artificial intelligence (AI) system increases, transparency and thus explanations of the system behaviour get more important especially in critical applications for domains like law or medicine. Towards this goal, my research focuses on implementing a generic dialogue model for interactive machine learning, that describes user and system behaviour on a semantic level (i.e. dialogue moves) and identifies common user requests that are agnostic to the underlying explainable AI method or domain.