Integrating MLFlow
==================
Orthrus now includes an example `MLFlow <https://www.mlflow.org/>`_ project for integrating orthrus pipelines seamlessly into MLFlow.
The user can access this example project from the github repository under the `mlflow <https://github.com/ekehoe32/orthrus/tree/main/mlflow>`_ directory.

MLProject File
--------------
The mlflow directory contains an `MLProject <https://www.mlflow.org/docs/latest/projects.html>`_ file which contains entry points to be run
in the mlflow workflow. For example, we include a tuning script which tunes an orthrus
pipeline and can be executed through the **tune** entry point. To run an entry point, simply
change to the mlflow directory, create an experiment, and then execute the mlflow run command:

.. code-block:: bash

    (base) $ conda activate orthrus
    (orthrus) $ cd $ORTHRUS_PATH/mlflow
    (orthrus) $ mlflow experiments create --experiment-name hyperparam-tuning
    (orthrus) $ mlflow run . --entry-point tune \
    -P dataset-path=$ORTHRUS_PATH/test_data/GSE73072/Data/GSE73072.ds \
    -P pipeline-path=$ORTHRUS_PATH/mlflow/pipelines/svm.py \
    -P sample-query="Disease=='h1n1'" -P num-samples=100 --no-conda --experiment-id 1

Above we use the tag ``--no-conda`` to indicate that we are using our own conda environment. If this tag was omitted then
mlflow would install the conda environment specified in the MLProject file.

Hyperparameter Tuning Example
-----------------------------
For this example we will tune a support vector machine classifier
on validation data and test on test data. Our associated orthrus pipeline
is contained in the file `pipelines/svm.py <https://github.com/ekehoe32/orthrus/blob/main/mlflow/pipelines/svm.py>`_. Within
the file we define our hyperparameter tuning search space and algorithm, hyperparameter tuning metric, and a function
to generate a pipeline based on a fixed hyperparameter configuration. Once these parameters. In particular, we define a logarithmic
search space for the ``C`` tuning parameter in SVM, use `HyperOpt <https://docs.ray.io/en/latest/tune/api_docs/suggestion.html#tune-hyperopt>`_ as a tuning algorithm, and limit
the number of concurrent tuning trials to 4. For a tuning metric we compute the mean validation balanced accuracy rate returned from
a pipeline that has been ran. During the tuning procedure, several pipelines will be built with different configurations 
and then evaluated for their mean performance across validation splits. 

Internally, the tune entry point calls the script `workflows/tune.py <https://github.com/ekehoe32/orthrus/blob/main/mlflow/workflows/tune.py>`_.
This script contains the boilerplate code for tuning a pipeline in general and can be reused for different pipelines outside of our
current example. This script also contains code for logging the metrics of the tuning process and finally logging the
best configuration of hyperparameters as an artifact. All metrics, parameters, and artifacts can be found in the mlruns directory
of the mlflow project folder. Read the `MLFlow docs <https://www.mlflow.org/docs/latest/index.html>`_ for more information on how to incorporate MLFlow into your workflow.