Skip to content

Conversation

@unexge
Copy link
Contributor

@unexge unexge commented May 13, 2025

This PR adds basic benchmarking setup using Criterion.rs. This would be useful to do performance analyses, and also to measure potential performance optimizations.

Main motivation for this PR was to understand how Regorus handles constants vs. data. @balcanuc realized that constants are re-evaluated each time and that was causing noticeable performance differences on sets with ~100 elements compared to ~3000 elements. This seems to be only happen with constants and not with data:
lines
We're trying to get more data and plan to create an issue to discuss in more details.


The benchmarks can be run with cargo bench. Sample output:

simple equality check with constant
                        time:   [1.9276 µs 1.9343 µs 1.9417 µs]

simple equality check with data
                        time:   [2.2182 µs 2.2262 µs 2.2344 µs]

allow with simple membership/with constant/32
                        time:   [5.6988 µs 5.7082 µs 5.7188 µs]

allow with simple membership/with data/32
                        time:   [2.2931 µs 2.3194 µs 2.3555 µs]

allow with simple membership/with constant/64
                        time:   [9.0032 µs 9.0216 µs 9.0388 µs]

allow with simple membership/with data/64
                        time:   [2.3670 µs 2.3767 µs 2.3864 µs]

allow with simple membership/with constant/128
                        time:   [17.455 µs 17.558 µs 17.666 µs]

allow with simple membership/with data/128
                        time:   [2.4630 µs 2.4665 µs 2.4701 µs]

allow with simple membership/with constant/512
                        time:   [79.154 µs 79.484 µs 79.844 µs]

allow with simple membership/with data/512
                        time:   [3.2336 µs 3.2544 µs 3.2933 µs]

allow with simple membership/with constant/1024
                        time:   [174.71 µs 174.99 µs 175.29 µs]

allow with simple membership/with data/1024
                        time:   [4.2587 µs 4.2663 µs 4.2744 µs]

allow with simple membership/with constant/2048
                        time:   [350.20 µs 355.25 µs 362.29 µs]

allow with simple membership/with data/2048
                        time:   [6.3407 µs 6.3947 µs 6.4682 µs]

Criterion also generates HTML reports and graphs automatically, which can be found at ./target/criterion/report/index.html.

@anakrish
Copy link
Collaborator

@unexge @balcanuc Thanks for the benchmark! Looking forward to your findings and patterns of writing policy.

Regorus doesn't do many possible optimizations yet (e.g. hoist common sub-expressions out of loops, fold-constants). The examples you provide will be a good motivator for prioritizing what to implement first.

unexge added 2 commits May 14, 2025 15:50
Signed-off-by: Burak Varlı <burakvar@amazon.co.uk>
Signed-off-by: Burak Varlı <burakvar@amazon.co.uk>
@unexge unexge force-pushed the push-nznuwntyookn branch from 587d905 to 689dcba Compare May 14, 2025 14:50
@anakrish anakrish merged commit 60ac4a7 into microsoft:main May 14, 2025
33 checks passed
@unexge unexge deleted the push-nznuwntyookn branch May 14, 2025 15:45
@anakrish
Copy link
Collaborator

@unexge I took an initial attempt at implementing constant folding in #410.

The idea is that expressions that don't depend on input or non-idempotent builtin functions will always produce the same value and can thus be pre-evaluated and cached. The prototype only caches rule values.

image

@unexge
Copy link
Contributor Author

unexge commented May 16, 2025

That looks great @anakrish! Thanks for the effort! /cc @balcanuc

anakrish pushed a commit to anakrish/regorus that referenced this pull request May 21, 2025
* Basic benchmarking setup with Criterion
* Fix Clippy warnings

Signed-off-by: Burak Varlı <burakvar@amazon.co.uk>
Signed-off-by: Anand Krishnamoorthi <anakrish@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants