TestGrid In-Depth EDA

Note: This notebook can be run and experimented with directly using the "ocp-ci-analysis" image at our jupyterhub instance hosted on the MOC.

In our previous notebook, TestGrid_EDA, we did some straightforward data access and preprocessing work in order to take a look at what data TestGrid exposes, how to access it and convert the test grids themselves into 2d arrays. While performing that initial data exploration we came up with a few more questions around how to look at this data in aggregate that we want to address here.

In this notebook we will try and answer the following questions about the TestGrid data:

• How comparable are the TestGrids?
• How do we analyze the TestGrids in aggregate to learn from their combined behavior?
• How many/ which tests are the most common?
• Are their time series dates comparable?
• Are there sub-groups that should only be compared with one another?
• Is looking at the grid matrices independent of test names a valid approach for issue identification?
• What is the expected behavior of a test over time across multiple jobs.
• How does the entire test platform/specific tests perform on a given day?
• How does a test/ the entire test platform behavior evolve over time.
• Is there sufficient data here for useful ML approaches?
• Can we develop some meaningful alerting/ problem identification based on the results of the above questions?

1. How comparable are the TestGrids?

Lets start by answering the first question on our list, "How comparable are the TestGrids?"

TestGrids are made of a set of tests which either pass or fail over time. They are multidimensional time series where the values can take either 12 (fail), 1 (pass) or 0 (not run). There are a few other status codes used, but we will ignore them for now. We want to know if they are always the same shape, and composed of the same tests. And if not, how do they vary?

1.a) Are there duplicate grids?

The first thing we will do is look at all the summaries and grid names to see if there is any overlap. Are there duplicate grids for different platforms?

It appears as though the answer to our question, "are there duplicate grids?" is no. There a 403 individual grids by name and all grids appear to be unique. However, there seems to be a lot of similarity in naming convention between grids where only the only difference is version number. Let's remove those references and see how many groups there are and how they compare.

1.b) Are there sub-groups?

Great, we can now see that there are 131 grid types if we remove the version. This means that on average, a grid type is repeated about 3 times. Or stays useful for 3 versions.

This might represent a challenge since that is a fair amount of uniqueness when it comes to making statistical claims about the behavior of these grids (since there are only 3 members of each sample).

Now that we have identified these 131 cross-version grid types, lets look at each type and see what percent of tests they share. Can they be compared directly (do they have the same shape and tests)?

1.c) Which tests do the groups all share?

For each grid type there is between 1 and 7 individual grids, and for the vast majority of these types, there is very little overlap in the tests used across them all. There appear to be some exceptions, here that we can look into more closely later on. But as far as addressing the question "how comparable are the grids". I think the answer hear appears to be, not very.

Therefore, analyzing the behavior of our jobs based on comparing grids (if we assume tests as the fundamental unit of these grids) does not appear to be a reasonable approach. We may still be able to do this looking purely at the grid patterns independent of tests names, but this will be explored further below.

2. How comparable are individual tests?

Lets ignore the grids as whole units for a minute and examine the occurrence of specific tests. Which are the most common? How many many unique tests are there? How should we expect them to behave?

2.a) What are the most common tests?

We can see above that there are 177,291 test instances across all 403 grids. Of those tests, there are only 7,798 unique tests which is over a ~20x reduction.

Somewhat contrary to the findings in the earlier section, this tells us that there is a fair amount of overlap of tests among grids. Perhaps our criteria for comparison above was too strict, and we should employ some more fuzzy matching methods.

We can also see that of these ~7k unique tests, 3,137 (about 1/3) occur across at least 10 grids. So it is probably this subset that we could use for comparisons and determining average behavior of a test overtime.

Let's look at one of the most common tests and see what we can say about its overall behavior.

2.b) Single Test Behavior Analysis

We will use "operator.All images are built and tagged into stable" since it looks like it impacts ~80% of our grids.

Great, this particular test has 317 examples, which means of the 403 grids its used in 78%. Let see if we have roughly the same number of samples for each instance of this test.

So we can see from the above output that the we have a very different number of data points for a test depending on which grid it is associated with. This again, creates a problem for making direct comparisons between these tests.

But that's alright, there are a number of ways this issue can be addressed. For now, lets go ahead, and generate some statistics about each time series. We can then define "normal behavior" as the average of these statistics as an initial naive approach to comparing tests in aggregate.

Since these time series are of unequal length, and we want to encode their general behavior to make comparisons, lets convert each of them into a vector of features based on some statistical properties of the time series.

Selected Features

• pass rate
• regression slope
• regression intercept
• longest failure streak ratio
• kurtosis ratio
• largest frequency ratio

There are obviously many ways to convert time series into a fixed length feature vector. But we'll use these as I think they are fairly interesting and will allow us to move forward. But I encourage future contributors to question/update/change these selected features. :)

We've now converted our 317 variable length binary time series data into a tabular data set with 6 features that represent the behavior of our tests over time. The benefit of doing this is that we can now apply some classic machine learning approaches, like dimensionality reduction and clustering to our data.

Lets visualize the instances of this test and see if any obvious structure jumps out.

Visualizing our tests

First we will apply PCA to reduce our 6 dimensional data down to 2 for visualization purposes. Then we will apply DBSCAN to identify any clusters or outliers that might be present in our data.

It looks like this method works to cluster individual test behavior across multiple instances of the same test. In this case we have divide the space into 4 clusters with a couple of outliers that we could investigate further.

For the purposes of this notebook,we have established at least one method for encoding and categorizing the behavior of tests that are all of the same type. We will now move on to another topic.

In the future this could be used to classify different types of observed errors for tests found in TestGrids

2.c) Multiple Test Behavior

Lets just quickly take a look at a larger portion of th entire data set that we have and see if we notice anything from clustering across multiple tests, i.e. are there types of behavior clusters we can identify.

note: Using all 117k tests is rather slow, so I'll limit it only to ~12k from the first 200 grids for the purposes of this notebook

Based on the image above, its looks like we get something close to a 2d uniform distribution of our dataset with a few outliers and smaller clusters when we take a larger portion of our tests at once with this feature set.

This might be noise or it might show that most time series behave in a similar way, or cluster into one of the other 10 failure types.

In either case, we can use these methods outlined above to investigate this approach further in a later notebook.

2.d) Comparing tests in aggregate

Now that we've looked at individual tests, lets derive a new object that describes a single test as the aggregation of the features we've defined above for every instance of the same test.

Alright, this looks like it might be a reasonable representation for a summary of expected behavior for a test. What can this tell us?

For the test "operator.All images are built and tagged into stable" we should expect this test to pass about 91% percent of the time regardless of the platform its run on. We have a small negative slope (avg_coef), which means on average we have more "not run" instances as we move forward in time, but this is such a small slope its practically flat, i.e., this is a reliable test. If it does fail, there is 0% average failure streak, meaning if it does fail, it could fail for up to 0% of the total number of tests runs to date in a row (i.e, this test never fails, seems to only pass or not run). This value basically tells us how often we see small flips vs longer outages in our data. Finally, kurtosis and frequency are two additional measures of "steadiness" of our data, but since they are normalized here their interpretation is less direct, but can be understood in relation to other test vectors.

Now, is it useful to represent tests this way? Maybe. Its not fully clear at this point. My concern here would be that we have over-summarized the different dynamics occurring in these individual test runs. Especially, as we saw about 4 clear clusters for this test above. However, we can validate the usefulness of these baselines by repeating the above for all the tests and seeing if there is stronger inter-test relationship than between different tests. (Basically, will the individual tests land closer to the baseline for their test on average or not if we measure their vector distances?)

Lets find out and produce these vectors for each test, and compare them similarly to how we did above.

Now that we have our data converted from individual time series into one data point for each test with 6 features representing the average behavior of a test across all of its instances. Lets visualize it and get a sense for the distribution. To plot this 6d data in 2d we will again use PCA to reduce the dimensions.

Clearly we can see some differentiation in this representation. Let's use DBSCAN on the 6d data and see if there is still some meaningful grouping.

We can see some clustering going on in the plot above, but also a fairly even distribution between tests.

However, what we want to know is if these "baseline" vectors for each test are actually meaningful. We will make that assessment by determining if individual tests are closer to their "baselines" on average than the datasets overall average.

Great, so in 98% of cases the local average distance of individual tests to their baseline is closer than an overall baseline. This tells us that there is some usefulness in describing the expected behavior of a test based on the mean of its individual features.

What have we accomplished up to now?

So far we have looked at the girds as a whole based on their shared collection of tests, determined that that the grids themselves are all unique and not directly comparable with regard to their set of tests.

However, some tests are repeated across many grids and we showed a couple methods to visualize, encode and analyze them both individually and in aggregate.

3. How does the entire test platform (or a specific test) perform on a given day?

Now that we've looked at encoding and comparing tests from a "horizontal perspective" with regard to the grids and examined the tests independent of time. Lets switch to a "vertical perspective" and determine how the entire platform, specific grids, or particular tests behave at specific days and across time.

3.a) Platform performance for a given day

In order to iterate over and search by the epoch time in our dataset, lets quickly collect all the unique times where a test value was logged.

times should contain every possible time stamp in our dataset.

Now let's iterate through our dataset and collect all the data on our platform that occurred on our query date.

Great, so we can use this method to calculate the pass rate, or any other derived value, for the entire platform for a specific day. As we can see above, there was a 60% pass rate across the entire platform on our query day.

We could also use this approach on a subset of the data to get behavior metrics for specific tests, grids, or custom subsets.

Now that we know how to look at behavior for a single day across the entire platform, we could roll this out across each day to see how the platform evolves over time.

3.b) Platform performance over time

Lets create a list of dates that we are interested in reviewing and plot the platform pass rate.

Great, from the above, we can now select any time period of interest (that we have data for) and examine the behavior of the entire platform over time.

And just to do our due diligence, below we will show that this can be done for individual test as well.

3.c) Aggregate single test performance over time

What have we accomplished?

Above we have explored a couple of methods for visualizing and calculating metrics about the TestGrid environment given a specific time or time range. Moving forward this could be used for overall trend analysis or prediction or root cause analysis for a time period with a known issue.

4. Grid matrices as 2d signals

Is looking at the grid matrices independent of test names a valid approach for issue identification?

I don't see why not. Let's take an example: Looking at the image below we can see that it was annotated manually by a human subject matter expert. Most of these errors seem to lie either purely on the vertical or horizontal axes and can be identified by analyzing either a single test or a single day, which we've already explored above. Let's look for an issue that is signaled by behavior across both days and tests.

4.a) Identify Infrastructure Flakes

What about the infrastructure flake case? That seems to be characterized by diagonal test failures over multiple days. We currently do not have any way of addressing that type of data. Lets see if we can use a simple hand crafted 2d filter to start identifying these types of errors.

What have we accomplished?

The above example has shown one way to automatically determined if there was an infrastructure flake in our TestGrid (assuming they can be identified following this diagonal failure pattern rule). This example identified 2 such flakes between October 1st and 2nd for the "release-openshift-origin-installer-e2e-gcp-compact-4.3'" grid. Moving forward we should follow up with an SME to validate these results and refine this approach.

Conclusion

In this notebook, we have done some additional exploration of the TestGrid data; familiarized ourselves with it further and determined a number of facts about the nature of the grids that will help us move forward developing additional analysis and ML tools for CI. We have looked at the entire TestGrid platform, individual grids and individual tests both from a stationary and time-series perspective. We have also done a couple of small experiments with encoding and clustering to explore the dataset's viability for applying ML approaches.

In this notebook we have tried to address the following questions and come up with the following answers:

• How comparable are the TestGrids?

  • The test grids contain nearly unique sets of tests and therefore are not directly comparable. We will either have to consider the aggregation of shared test or define grids purely by their patterns (independent of specific test)

• How do we analyze the TestGrids in aggregate to learn from their combined behavior?

  • I don't think we can analyze the entire grids directly due to the above stated issue of uniqueness, other than addressing purely the patterns independent of the specific test names. It will likely be better to analyze the data one step lower and learn from the tests, then use the tests the grids are composed of to understand the grids better.

• How many/ which tests do they all have in common?

  • The set of tests in each grid appears to be unique and there is not one test that is included in all grids.
  • However, there are about 3k tests identified above that that occur in 10 grids or more.

• Are their time series dates comparable?

  • We have included the timestamps for the time series, so we know when a test passed/failed and we can compare tests on specific days or time ranges.
  • Each grid contains different dates and lengths of runs, however, so they aren't all equally sampled points.

• Are there sub-groups that should only be compared with one another?

  • We removed version identifiers to compare subgroups, but still saw little overall overlap between sets of tests in grids appeared. But further analysis could still be done in this regard.

• Is looking at the grid matrices independent of test names a valid approach for issue identification?

  • We were able to create a toy example using a 2d convolution and an handcrafted filter to identify diagonal failures in the TestGrids. If the annotated image we have is accurate, then we have shown at least one method to achieve this goal. So I would consider this to indeed be a useful approach.

• What is the expected behavior of a test over time across multiple jobs.

  • We developed some basic encoding and clustering examples to compare individual tests, aggregated tests and tests of the same type. We should build this analysis out further in a later notebook.

• How does the entire test platform/specific tests perform on a given day?

  • We were able to develop a number of methods to perform and display this type of analysis. We should build this analysis out further in a later notebook.

• How does a test or the entire test platform behavior evolve over time.

  • Given the above step, we were able to roll that analysis out across time and plot trends in platform pass rate over September. We should build this analysis out further in a later notebook.

• Is there sufficient data here for useful ML approaches?

  • Yes, especially if we focus on the level of individual time-series for tests. However, there are currently not clear labels, so much of the initial work would either be unsupervised learning or pure statistical analysis.

  • If we look only at the Red Hat grids, which are themselves a subset of the entire ecosystem.

    • There are ~400 grids
    • 61 days of data for each grid
    • 117,000 time series

  • Other platforms have also opened their data on Testgrid.io that we could leverage as well.

• Can we develop some meaningful alerting/ problem identification with the results of the above questions?

  • Yes. Given all of the above, I think there are a number of avenues available for developing altering and classification tools for CI based on this data.