3 key lessons from managing ML products

Work backward, bridge the tech/business gap, adjust priorities at different stages of your project

I am a software engineer with more than two decades of experience coding for games, web applications, embedded systems, and more. Since 2017, I’ve working with and studying machine learning models. I’ve also helped run machine learning projects and have consulted several companies in their ML journey.

The most important lessons I learned were from applying ML in real-world projects. Here are some of top product management lessons I learned from applying ML:

Work backward: In essence, creating ML products and features is no different than other products. Don’t jump into Jupyter notebooks and data analysis before you talk to the key stakeholders. Establish deployment goals (how ML will affect your operations), prediction goals (what exactly the model should predict), and evaluation metrics (metrics that matter and required level of accuracy) before gathering data and exploring models. 

Bridge the tech/business gap in your organization: Business professionals don’t know enough about the intricacies of machine learning, and ML professionals don’t know about the practical needs of businesses. Educate your business team on the basics of ML and create joint teams of data scientists and business analysts to define and measure goals and progress of ML projects. ML projects are more likely to fail when business and data science teams work in silos.

Adjust your priorities at different stages of the project: In the early stages of your ML project, aim for speed. Choose the solution that validates/rejects your hypotheses the fastest, whether it’s an API, a pre-trained model, or even a non-ML solution (always consider non-ML solutions). In the more advanced stages of the project, look for ways to optimize your solution (increase accuracy and speed, reduce costs, increase flexibility).

There is a lot more to share, but these are some of the top experiences that would have made my life a lot easier if I had known them before diving into applied ML. 

I first posted this as a discussion on Reddit and got a lot of feedback and question in the thread and through direct messages. The most important question people asked was how to get started on the ML journey and how to move from basic education to real-world ML projects.

The most important thing to know is that the real world is much messier than the clean datasets you work with in ML labs and courses. Furthermore, you will be working with people whose goals are not necessarily aligned with yours. As a data scientist, you will be optimizing for things such as accuracy and loss. But the business people will have other things in mind, such as revenue, churn, customer satisfaction. You have to find ways to bridge the gap between the ML and business world.

For me, two great resources that helped me become a better AI/ML product manager were:

1- The AI/ML Simulator for Product Managers by GoPractice

2- "The AI Playbook" by Eric Siegel (read my review here)

For people who are more interested in ML roles as opposed to product management and have no experience, I would recommend starting with an introductory book such as "Machine Learning With PyTorch and Scikit-Learn" by Sebastian Raschka. You always need to know the basics if you're ever going to work in a real ML job.

If you want to get some hands-on experience with different ML models, you should definitely try your hand at some Kaggle challenges and look at how other people approach solving problems.

But moving on from the basics, you should look into courses and books that teach you how to design machine learning systems. For example "Designing Machine Learning Systems" by Chip Huyen and "Designing Deep Learning Systems" by Chi Wang are great resources. They provide you with information and experience that you can't find in ML tutorials, such as creating model pipelines, versioning datasets and models, deploying at scale, monitoring, etc.

I also look at company engineering blogs, where they share their experience in deploying machine learning systems. For example, the Netflix Tech Blog and LinkedIn Engineering Blog are two great resources.