Skip to content

Conversation

@vkarak
Copy link
Contributor

@vkarak vkarak commented Nov 20, 2025

This PR offers the following features:

  1. Multiple statistics can be presented at once using the special stats aggregator. These includes mean, standard deviation, and 0.01, 0.05, 0.50, 0.95, 0.99 quantiles.
  2. The cmp_spec syntax is updated to support multiple statistics at once, e.g., mean,min,max, and also aggregations on values other than the performance, e.g., min(num_tasks),min(pref) etc.
  3. Two new command-line options are added to control the name of the left and right terms in the comparisons. These are the --term-lhs and --term-rhs and are only meaningful with the --performance-compare and --performance-report options.
  4. The default name suffixes for left/right terms in comparisons are now (lhs) and (rhs) respectively.
  5. The selectors of left and right columns in the cmp spec are now _L and _R instead of _A and _B.

Closes #3436.

Implementation details

The analytics layers is now using Polars through its Python bindings.

Managing the column names was the most challenging part of the implementation, as we wanted to keep a straightforward for selecting test attributes from the command line, specifying the left/right variants quickly and the same time produce tables with meaningful headers (e.g., include the aggregation in the column name, especially now, that multiple aggregations are supported at once). This is handled the the internal _QueryMatch class which replaces the very thin former _Match named tuple.

Todos

  • Update documentation
  • Update tutorial examples
  • --table-format=pretty still uses | as a separator for join_uniq.

Notes

If the CI does not pass for Python < 3.9, we should simply merge this PR after #3583 is addressed.

This PR depends on #3596.

@vkarak vkarak added this to the ReFrame 4.10 milestone Nov 20, 2025
@vkarak vkarak self-assigned this Nov 20, 2025
@vkarak vkarak added the reporting Issues related to reporting and processing the test results label Nov 20, 2025
@codecov
Copy link

codecov bot commented Nov 20, 2025

Codecov Report

❌ Patch coverage is 95.39749% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.40%. Comparing base (8858b80) to head (cc60ca8).

Files with missing lines Patch % Lines
reframe/frontend/reporting/utility.py 94.54% 9 Missing ⚠️
reframe/frontend/reporting/__init__.py 96.72% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3585      +/-   ##
===========================================
+ Coverage    91.38%   91.40%   +0.01%     
===========================================
  Files           62       62              
  Lines        13484    13505      +21     
===========================================
+ Hits         12323    12344      +21     
  Misses        1161     1161              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vkarak vkarak marked this pull request as draft November 27, 2025 17:37
@vkarak vkarak force-pushed the feat/resultsdb-stats branch 3 times, most recently from 541d9b9 to f9f4fcc Compare December 14, 2025 17:10
@vkarak vkarak force-pushed the feat/resultsdb-stats branch from f9f4fcc to 79fd962 Compare December 16, 2025 22:54
@vkarak vkarak force-pushed the feat/resultsdb-stats branch from 79fd962 to 6dfab74 Compare December 23, 2025 17:29
@vkarak vkarak force-pushed the feat/resultsdb-stats branch from 6dfab74 to cc60ca8 Compare December 23, 2025 18:39
@vkarak vkarak marked this pull request as ready for review December 23, 2025 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement prio: normal reporting Issues related to reporting and processing the test results

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Report important performance statistics all at once with --list-stored-testcases

1 participant