Probability To Fail

One of the key performance indicators that we would like to create greater visibility into and for a given moment in time, is to calculate what is the probability that a given test will fail. Using this information, we plan to build a model that uses the probability to fail to predict the time to failure.

In order to provide maxium flexibility for the end-user of this work, instead of creating a number of dataframes to answer each of these specifc questions, we will define a long and narrow data structure (a list of tuples saved as a csv for now) that contains only 7 columns ("timestamp", "tab","grid","test","test_duration", "pass_or_fail", "prob"). This allows visualization tools like Superset (or pandas) to perform the last filter and/or aggregate interested columns to an end user. Which is to say, there may be a lot of repetition within the final data set, but each row should be unique, and it should provide the simplest usability for an end-user.

Linked issue(s) : Issue

Calculation

Calculate Probability to Fail using SVM and Calibrated Classification Model

In order to calculate the probability to fail, some of the approaches we could consider are SVM (support vector machines) and a Calibrated Classification Model. In this section of the notebook, we will compare the SVM and Calibrated Classification model, and choose the model giving better results.

The objective of the support vector machine algorithm is to find a hyperplane in an N-dimensional space(N — the number of features) that distinctly classifies the data points. SVM is a good candidate model to calibrate because it does not natively predict probabilities, meaning the probabilities are often uncalibrated. The probabilities are not normalized, but can be normalized when calling the calibration_curve() function by setting the normalize argument to True.

An SVM model can be created using the scikit learn module.

A classification predictive modeling problem requires predicting or forecasting a label for a given observation. An alternative to predicting the label directly, a model may predict the probability of an observation belonging to each possible class label. Although a model may be able to predict probabilities, the distribution and behavior of the probabilities may not match the expected distribution of observed probabilities in the training data. The distribution of the probabilities can be adjusted to better match the expected distribution observed in the data. This adjustment is referred to as calibration, as in the calibration of the model or the calibration of the distribution of class probabilities.

A classifier can be calibrated using the CalibratedClassifierCV sci-kit library. We will generate the labels/classes required for classification and train the model on a subset of the data set.

Resampling the data

From above, we see that we have an imbalanced dataset i.e the classes are not represented equally, and this can lead to misleading classification accuracy results. To handle this issue of imbalanced datasets, we will resample the data. Resampling is a widely-adopted technique for dealing with imbalanced datasets by changing the dataset into a more balanced one by:

• Undersampling - This simply deletes instances from the over-represented class (majority class) in different ways. The most obvious way is to do delete instances randomly.
• Oversampling - This method adds copies of instances from the under-represented class (minority class) to obtain a balanced dataset. There are multiple ways you can oversample a dataset, like random oversampling.

Let's implement the basic method of undersampling and oversampling, which uses the DataFrame.sample method to get random samples of each class:

Undersampling

Here, we delete instances randomly from the over-represented class i.e. for our dataset the 'Pass (1)' class.

Oversampling

Here, we add instances randomly to the under-represented class i.e. for our dataset the 'Fail (0)' class.

SVM

Now, we will fit the SVM model to calculate the probabilities. Let's run it for both the under and over sampled data and compare.

NOTE : The probabilities are not normalized, but can be normalized when calling the calibration_curve() function by setting the normalize argument to True.

Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction.

The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model.

Reliability Diagram

A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.

The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot.

In the above graph, the solid line represents the calibration of the SVMs predicted probabilities and the dotted line represents a comparison to a perfectly calibrated model. The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot. Here, we see that calibration probabilities are much further away from the ideal prediction.

The position of the points or the curve relative to the diagonal can help to interpret the probabilities; for example:

• Below the diagonal: The model has over-forecast; the probabilities are too large.
• Above the diagonal: The model has under-forecast; the probabilities are too small.

SVM for over sampled data

Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction.

The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model.

We also see that we achieved a higher accuracy of ~81% with over sampling of the data compared to the ~78% accuracy with the under sampled data.

Reliability Diagram

A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.

The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot.

In the above graph, the solid line represents the calibration of the SVMs predicted probabilities and the dotted line represents a comparison to a perfectly calibrated model. The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot. Here, we see that calibration probabilities are much further away from the ideal prediction.

The position of the points or the curve relative to the diagonal can help to interpret the probabilities; for example:

• Below the diagonal: The model has over-forecast; the probabilities are too large.
• Above the diagonal: The model has under-forecast; the probabilities are too small.

Since this graph indicates that the probabilities calculated by SVM are too small, we shall move forward and look in the Calibrated Classification Model.

Calibrated Classification Model

• CalibratedClassifierCV class using 5-fold cross-validation, to calibrate the predicted probabilities.
• Since the over sampled data resulted in a better accuracy, we will consider this dataset for training the Calibrated Classification Model

NOTE : The probabilities are not normalized, but can be normalized when calling the calibration_curve() function by setting the normalize argument to True.

Brier Score is a distance in the probability domain. Which means: the lower the value of this score, the better the prediction.

The above calculated score is a harsh metric since we require for each sample that each label set be correctly predicted. This does not give us a clear indication of a better model so we will use the reliability diagram to find out the better suited model.

Reliability Diagram

A reliability diagram is a line plot of the relative frequency of what was observed (y-axis) versus the predicted probability frequency (x-axis). These plots are commonly referred to as ‘reliability‘ diagrams in forecast literature, although may also be called ‘calibration‘ plots or curves as they summarize how well the forecast probabilities are calibrated.

The better calibrated or more reliable a forecast, the closer the points will appear along the main diagonal from the bottom left to the top right of the plot.

Here, we see that the shape of the calibrated probabilities is different, hugging the diagonal line much better, although still under-forecasting and over-forecasting i.e. the solid line is both above and below the ideal model.

Visually, the plot suggests a better calibrated model.

We now have the probabilities for each test likely to fail stored in the column prob.

Save results to Ceph or locally

• Use the following helper function to save the data frame in a parquet format on the Ceph bucket if we are running in automation, and locally if not.

Conclusion :

In this notebook, we calculated for a given moment in time, what is the probability that a test will fail. Using this information, we plan to build a model that uses the probability to fail value to predict the time to failure, that is how many days, weeks, months, hours, minutes or seconds do you have until the test in question stops working. As next steps, we plan to work on this issue.