Integrating MLFlow

Orthrus now includes an example MLFlow project for integrating orthrus pipelines seamlessly into MLFlow. The user can access this example project from the github repository under the mlflow directory.

MLProject File

The mlflow directory contains an MLProject file which contains entry points to be run in the mlflow workflow. For example, we include a tuning script which tunes an orthrus pipeline and can be executed through the tune entry point. To run an entry point, simply change to the mlflow directory, create an experiment, and then execute the mlflow run command:

(base) $ conda activate orthrus
(orthrus) $ cd $ORTHRUS_PATH/mlflow
(orthrus) $ mlflow experiments create --experiment-name hyperparam-tuning
(orthrus) $ mlflow run . --entry-point tune \
-P dataset-path=$ORTHRUS_PATH/test_data/GSE73072/Data/GSE73072.ds \
-P pipeline-path=$ORTHRUS_PATH/mlflow/pipelines/svm.py \
-P sample-query="Disease=='h1n1'" -P num-samples=100 --no-conda --experiment-id 1

Above we use the tag --no-conda to indicate that we are using our own conda environment. If this tag was omitted then mlflow would install the conda environment specified in the MLProject file.

Hyperparameter Tuning Example

For this example we will tune a support vector machine classifier on validation data and test on test data. Our associated orthrus pipeline is contained in the file pipelines/svm.py. Within the file we define our hyperparameter tuning search space and algorithm, hyperparameter tuning metric, and a function to generate a pipeline based on a fixed hyperparameter configuration. Once these parameters. In particular, we define a logarithmic search space for the C tuning parameter in SVM, use HyperOpt as a tuning algorithm, and limit the number of concurrent tuning trials to 4. For a tuning metric we compute the mean validation balanced accuracy rate returned from a pipeline that has been ran. During the tuning procedure, several pipelines will be built with different configurations and then evaluated for their mean performance across validation splits.

Internally, the tune entry point calls the script workflows/tune.py. This script contains the boilerplate code for tuning a pipeline in general and can be reused for different pipelines outside of our current example. This script also contains code for logging the metrics of the tuning process and finally logging the best configuration of hyperparameters as an artifact. All metrics, parameters, and artifacts can be found in the mlruns directory of the mlflow project folder. Read the MLFlow docs for more information on how to incorporate MLFlow into your workflow.