The growing importance of Artificial Intelligence (AI) systems as a part of software services, and the failures of these systems, have emphasized the importance of ethical consideration in engineering AI based software systems. Despite the growing number of studies in AI ethics, comparatively little attention has been paid to how ethical issues could be mitigated in software engineering in practice. We currently know little about how to provide useful tools that can help companies transform high-level ethical guidelines for AI ethics into the actual work flow of the developers. In this paper, we explore the idea of using user stories to transform abstract ethical requirements into tangible outcomes in Agile software development. We tested this idea by studying master's level student projects (15 teams) developing web applications for a real industrial client over the course of five iterations. These projects resulted in 250+ user stories that we analyzed for this study. These teams were divided into two groups: half of the teams worked using a method for AI ethics in software engineering, ECCOLA, while the other half was used as a control group in order to evaluate this method. Both teams were tasked with writing user stories to formulate customer needs into system requirements. Based on the data, we discuss the effectiveness of ECCOLA, and lessons learned on formulating ethical user stories in Agile development.