Experienced teams know when to back up seeing a piling debt, but technical debt in machine learning piles extremely fast. You can create months worth of debt in a matter of one working day and even the most experienced teams can miss a moment when the debt is so huge that it sets them back for half a year, which is often enough to kill a fast-pacing project. You end up with the project where the metrics randomly jump up or down, do not reflect the actual quality, and you are not able to improve them.
Machine learning is known for its difficulties with interpretability, or rather its absence. This is an issue if your users have to work with the numeric output, like in the systems used in sales, trading or marketing. If the user’s interpretation of the ML output is wrong the actual metrics won’t matter and you end up with the bad user experience. The problem is even bigger if you try switching users from an old transparent algorithm to ML. Here I outline the recipes for overcoming the user’s push-back once you start switching your system to ML.