Test your deployed inference application

From JupyterLab in Operate First

In order to test the predictions from a deployed model/s, you can use the notebook here.

From your local laptop

NOTE:Make sure you are logged in into the cluster where the model is deployed.

If you want to test the application that you just deployed in the cluster, run this integration test using the behave package.

Behave uses behavior-driven development (BDD), an agile software development technique that encourages and facilitates collaboration between developers, QA engineers and business participants.

1. Create a virtual environment and install your dependencies using thamos(the CLI tool for interacting with Thoth and included as a dependency to this repo) from the root directory of this tutorial. Be sure to select the appropriate overlay, which in this case is test-model.

   python3 -m venv venv/ && . venv/bin/activate && thamos install -r test-model

2. Run the behave command from the root directory of the tutorial.

   DEPLOYED_MODEL_URL=http://elyra-aidevsecops-tutorial-thoth-deployment-examples.apps.zero.massopen.cloud behave -i gather_deployment_metrics.feature

   Below is an example of the output you should expect to see if the behave command ran correctly:

   Scenario: Deployment metrics gathering                    # features/gather_deployment_metrics.feature:2
       Given dataset is available                            # features/steps/model_deployment.py:42
       Given deployment is accessible                        # features/steps/model_deployment.py:56
       When I run test to gather metrics on predict endpoint # features/steps/model_deployment.py:72
       Then I should get model and application metrics       # features/steps/model_deployment.py:128
   
   1 feature passed, 0 failed, 0 skipped
   1 scenario passed, 0 failed, 0 skipped
   4 steps passed, 0 failed, 0 skipped, 0 undefined
   Took 0m3.394s

NOTE:If you use Pytorch+Flask+gunicorn you need to add the flag `-i gatherdeploymentmetricsusingpytorch.feature`

3. You can also check the metrics produced by running the following command:

   cat metrics.json

   {'average_latency': 0.03846242621075362, 'average_error': 0.001}

Next Step:

Monitor your model and application with Prometheus and Grafana

References

• behave
• thamos