Benchmarks

Performance analysis and comparison with SciPy
NoteReference Hardware

Last synced benchmark assets: 2026-08-24T13:24:43.292589+00:00
System: unknown
CPU: arm, 8 cores
Memory: unknown GB
Platform: macOS-26.3.1-arm64-arm-64bit
Python: 3.12.10, NumPy: 2.3.5, SciPy: 1.16.3, Optyx: 1.3.2

1 Benchmarks

Optyx includes a comprehensive benchmark suite measuring end-to-end performance including variable creation, problem setup, constraint construction, and solving. All benchmarks compare against raw SciPy (which has no build phase).

This page renders directly from synced artifacts in docs/assets/benchmarks/:

  • benchmark_results.json for structured tables and summary values
  • benchmark_metadata.json for machine and dependency metadata
  • benchmark_output.txt for the raw console transcript
  • .png plots copied from the latest benchmark run

1.1 Quick Start

# Run all benchmark tests
uv run pytest benchmarks/ -v

# Generate performance analysis plots and sync docs assets automatically
uv run python benchmarks/run_benchmarks.py
NoteWhat We Measure

All benchmarks measure total time including:

  • Variable creation
  • Problem setup
  • Constraint construction
  • Cold solve (first solve, includes compilation)
  • Warm solve (cached subsequent solves)

1.2 Performance Summary

Problem Type Size Cold Overhead Warm Overhead Notes
LP n=50 1.6x 1.2x Near-parity with SciPy linprog
LP n=500 1.2x 1.3x Near-parity with SciPy linprog
LP n=5000 1.0x 1.0x Scales to large LPs while staying near parity
NLP n=50 2.6x 1.5x Autodiff overhead on a trivially simple objective
NLP n=500 2.3x 1.3x Autodiff overhead on a trivially simple objective
NLP n=5000 1.4x 1.4x Simple quadratic; SciPy converges almost instantly
CQP n=50 4.3x 2.8x O(1) Jacobian compilation for vectorized constraints
CQP n=500 1.6x 1.2x O(1) Jacobian compilation for vectorized constraints
CQP n=5000 1.1x 1.0x Exact Jacobians keep constrained solves near parity
MILP n=50 1.3x 1.2x Near-parity with SciPy milp
MILP n=500 1.2x 1.1x Near-parity with SciPy milp
MILP n=5000 1.1x 1.0x Scales to large binary knapsack problems

Key Insight: Cold solves include one-time compilation. At n=5,000, the latest warm measurements are 1.0x for LP, CQP, and MILP and 1.2x for the simple NLP case. Small timings are noisier and should not be treated as stable speedup claims.

1.3 LP Scaling: VectorVariable vs Loop-Based

LP Scaling Comparison

1.3.1 Loop-Based Variables (n ≤ 500)

n Build Cold Solve Warm Solve SciPy Cold Overhead Warm Overhead
10 0.7ms 11.8ms 0.9ms 0.7ms 17.0x 1.3x
25 1.2ms 6.6ms 1.0ms 1.0ms 8.1x 1.0x
50 1.3ms 24.6ms 1.3ms 1.3ms 20.5x 1.0x
100 5.1ms 84.2ms 2.2ms 2.3ms 39.6x 1.0x
200 38.9ms 425.3ms 5.8ms 5.5ms 84.3x 1.1x
500 264.8ms 2,551.5ms 36.9ms 36.5ms 77.1x 1.0x
WarningLoop-Based Variables Don’t Scale

Loop-based variable construction can create O(n²) expression tree nodes and rapidly increasing compilation time. Use VectorVariable for grouped models.

1.3.2 VectorVariable (n ≤ 5,000)

n Build Cold Solve Warm Solve SciPy Cold Overhead Warm Overhead
10 0.1ms 1.4ms 1.0ms 1.0ms 1.6x 1.0x
25 0.1ms 1.4ms 1.3ms 1.0ms 1.4x 1.2x
50 0.1ms 2.1ms 1.6ms 1.4ms 1.6x 1.2x
100 0.2ms 3.7ms 3.1ms 2.4ms 1.6x 1.3x
200 0.3ms 8.8ms 7.2ms 5.7ms 1.6x 1.3x
500 0.6ms 43.5ms 44.9ms 35.6ms 1.2x 1.3x
1000 1.2ms 167.8ms 154.4ms 149.4ms 1.1x 1.0x
2000 2.5ms 737.5ms 682.9ms 665.3ms 1.1x 1.0x
5000 6.2ms 6,874.7ms 6,506.9ms 6,646.4ms 1.0x 1.0x

In this run, vectorized LP warm overhead ranges from 1.0x to 1.3x and reaches 1.0x at n=5,000. Cold overhead falls from 1.8x at n=50 to 1.1x at n=5,000.

1.4 NLP Scaling: Unconstrained Optimization

NLP Scaling Comparison

Objective: min Σx²ᵢ - Σxᵢ (optimal at x* = 0.5)

1.4.1 VectorVariable with x.dot(x) - x.sum()

n Build Cold Solve Warm Solve SciPy Cold Overhead Warm Overhead
10 0.0ms 0.3ms 0.1ms 0.1ms 5.5x 1.7x
25 0.0ms 0.2ms 0.1ms 0.1ms 3.0x 1.5x
50 0.0ms 0.2ms 0.1ms 0.1ms 2.6x 1.5x
100 0.0ms 0.2ms 0.1ms 0.1ms 2.8x 1.5x
200 0.0ms 0.2ms 0.1ms 0.1ms 2.3x 1.4x
500 0.0ms 0.2ms 0.1ms 0.1ms 2.3x 1.3x
1000 0.0ms 1.5ms 0.3ms 0.1ms 11.6x 2.0x
2000 0.0ms 0.5ms 0.2ms 0.2ms 2.7x 1.3x
5000 0.0ms 0.7ms 0.7ms 0.5ms 1.4x 1.4x
NoteInterpreting NLP Overhead

This benchmark uses a trivially simple quadratic (Σx² - Σx) where SciPy’s L-BFGS-B baseline takes roughly 0.06–0.30ms in this run. At that scale, fixed modeling and dispatch costs dominate the ratio. More complex objectives can benefit from exact generated derivatives, but these results do not quantify that benefit.

1.5 Constrained QP Scaling

CQP Scaling Comparison

Objective: min Σx²ᵢ subject to Σxᵢ ≥ 1, xᵢ ≥ 0

1.5.1 VectorVariable with x.dot(x), x.sum()

n Build Cold Solve Warm Solve SciPy Cold Overhead Warm Overhead
10 0.1ms 0.7ms 0.3ms 0.1ms 7.7x 2.7x
25 0.0ms 0.6ms 0.4ms 0.3ms 2.3x 1.4x
50 0.0ms 0.9ms 0.6ms 0.2ms 4.3x 2.8x
100 0.0ms 1.3ms 1.0ms 0.5ms 3.0x 2.2x
200 0.0ms 2.2ms 1.7ms 1.5ms 1.5x 1.1x
500 0.0ms 11.6ms 8.1ms 7.1ms 1.6x 1.2x
1000 0.0ms 44.8ms 37.6ms 34.7ms 1.3x 1.1x
2000 0.1ms 301.4ms 277.3ms 270.9ms 1.1x 1.0x
5000 0.1ms 4,383.8ms 4,005.4ms 3,844.6ms 1.1x 1.0x

With the structured Jacobian path, CQP warm overhead decreases from 1.2x at n=500 to 1.0x at n=5,000 in this run.

1.6 MILP Scaling: Binary Knapsack

MILP Scaling Comparison

Problem: Single-constraint binary knapsack (sum(x) <= n//2)

1.6.1 VectorVariable (n ≤ 5,000)

n Build Cold Solve Warm Solve SciPy Cold Overhead Warm Overhead
10 0.1ms 0.9ms 0.7ms 0.7ms 1.4x 1.0x
25 0.0ms 1.0ms 0.8ms 0.6ms 1.6x 1.2x
50 0.0ms 1.0ms 1.0ms 0.8ms 1.3x 1.2x
100 0.0ms 1.5ms 1.3ms 1.1ms 1.3x 1.2x
200 0.0ms 2.4ms 2.1ms 1.8ms 1.3x 1.1x
500 0.0ms 6.6ms 6.1ms 5.4ms 1.2x 1.1x
1000 0.0ms 20.0ms 20.3ms 18.4ms 1.1x 1.1x
2000 0.0ms 114.7ms 63.0ms 61.8ms 1.9x 1.0x
5000 0.0ms 414.7ms 385.9ms 385.8ms 1.1x 1.0x

MILP timings are noisy at small sizes, including outliers at n=50 and n=100. From n=500 through n=5,000, warm overhead ranges from 1.0x to 1.1x in this run.

1.7 Overhead Summary by Problem Type

Overhead Breakdown
Problem Type Cold Overhead Warm Overhead
LP (n=50) 1.6x 1.2x
LP (n=5000) 1.0x 1.0x
NLP (n=50) 2.6x 1.5x
NLP (n=5000) 1.4x 1.4x
CQP (n=50) 4.3x 2.8x
CQP (n=5000) 1.1x 1.0x
MILP (n=50) 1.3x 1.2x
MILP (n=5000) 1.1x 1.0x

Pattern: The large n=5,000 LP, CQP, and MILP cases are at 1.0x warm overhead after rounding. The NLP case is 1.2x. Ratios for sub-millisecond and small solver runs vary more and should be read alongside the absolute timings.

1.8 When to Use Optyx

1.8.1 Ideal Use Cases

✅ Parameter sweeps: Solve similar problems with different parameters
✅ Real-time optimization: Repeated solves with cached structure
✅ Prototyping: Clean Python API, no manual gradients
✅ Large LP: VectorVariable reaches 1.0x warm overhead at n=5,000 in this run
✅ Non-convex NLP: Automatic differentiation with exact gradients
✅ Mixed-integer programming: MILP reaches 1.0x warm overhead at n=5,000 in this run

1.8.2 Consider Alternatives For

⚠️ One-shot problems: Cold-solve includes compilation overhead
⚠️ Large dense matrix problems (n>1000): CVXPY’s specialized solvers may scale better
⚠️ Loop-based variables at scale: Use VectorVariable instead

1.9 Running Benchmarks

# All benchmarks
uv run pytest benchmarks/ -v

# By category
uv run pytest benchmarks/validation/ -v
uv run pytest benchmarks/performance/ -v
uv run pytest benchmarks/accuracy/ -v
uv run pytest benchmarks/comparison/ -v

# Generate plots
uv run python benchmarks/run_benchmarks.py

1.10 Latest Console Summary

OVERHEAD SUMMARY BY PROBLEM TYPE
================================================================================
LP n=50: Cold=1.6x, Warm=1.2x
LP n=5000: Cold=1.0x, Warm=1.0x
NLP n=50: Cold=2.6x, Warm=1.5x
NLP n=5000: Cold=1.4x, Warm=1.4x
CQP n=50: Cold=4.3x, Warm=2.8x
CQP n=5000: Cold=1.1x, Warm=1.0x
MILP n=50: Cold=1.3x, Warm=1.2x
MILP n=5000: Cold=1.1x, Warm=1.0x

Saved: benchmarks/results/overhead_breakdown.png

================================================================================