Eikku Koponen

Machine learning comes with new types of risk. We need to minimize the risk by addressing how we develop these algorithms and also how we apply these algorithms in the real world. In this article, we'll look at three ways of mitigating the latter – i.e. output risk.

Algorithms have become faster, fancier, and more complex in the past couple of years. Still, they haven't gained as much complexity as the systems around algorithms. In this article, we'll discuss three examples of systems complexity.

Sometimes it is hard to combine the world of experimenting and the more dev-oriented world of data science with robust pipelines and modular work. This example combines Weights and Biases experiments with Valohai's production pipelines.

The data you have, is, if not the most, at least close to the most valuable asset you’ve got when creating AI systems. So in practice, what can you do to embrace more data-centric AI then? We have prepared some simple steps for you to keep in mind and implement.

Observability is the collection of statistics, performance data, and metrics from every part of your ML system. Metadata, if you will. We will dig into how we can easily get started with observability and detect data drift using whylogs while executing your pipeline on Valohai.

When it comes to the production phase, actually providing the model to end-users and integrating it to the (existing) tools, Data Scientist often pass the baton to Software engineers. That handover is often quite rocky. Here are a few tips to how the bridge the gap between data science and engineering.

This article shows an example of a pipeline that uses Hugging Face transformers (DistilBERT) to predict the shark species based on injury descriptions. With Valohai, you can easily tie together typical data science workflows into repeatable pipelines.