Add multivariate optimization by nyo16 · Pull Request #331 · elixir-nx/scholar

nyo16 · 2026-01-30T23:31:20Z

This PR adds two multivariate optimization algorithms to Scholar, completing the optimization module:

BFGS - Quasi-Newton method with automatic differentiation for smooth, differentiable functions
Nelder-Mead - Derivative-free simplex method for functions where gradients are unavailable or expensive

Both implementations follow the patterns established in PRs #327 (Golden Section) and #328 (Brent):

defn entry point with deftransformp for option validation
Initial point as explicit function argument
Module constants used directly (not wrapped in Nx.tensor)
Each module has its own NimbleOptions schema
JIT/GPU compatible

Changes

File	Description
`lib/scholar/optimize/bfgs.ex`	BFGS implementation with backtracking line search
`lib/scholar/optimize/nelder_mead.ex`	Nelder-Mead simplex implementation
`test/scholar/optimize/bfgs_test.exs`	Tests for BFGS (sphere, Rosenbrock, Booth, Beale functions)
`test/scholar/optimize/nelder_mead_test.exs`	Tests for Nelder-Mead
`notebooks/optimize.livemd`	Updated with multivariate optimization section
`notebooks/efficient_frontier.livemd`	Portfolio optimization example using both algorithms
`compare_*.py`	SciPy validation scripts

Test plan

All existing tests pass (mix test)
New algorithm tests pass (mix test test/scholar/optimize/)
Doctests pass
Code formatted (mix format)
Test livebooks manually in Livebook"

- BFGS: Quasi-Newton method with automatic differentiation - Nelder-Mead: Derivative-free simplex method - Update optimize.livemd with multivariate section - Add efficient_frontier.livemd portfolio optimization example

nyo16 · 2026-01-30T23:41:03Z

And comparing them with python

BFGS Results:
  ┌───────────────────┬──────────────┬────────────────────┬────────┐
  │     Function      │ Expected Min │    SciPy Found     │ Status │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ sphere_2d         │ [0, 0]       │ [~0, ~0]           │ ✓      │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ sphere_3d         │ [0, 0, 0]    │ [~0, ~0, ~0]       │ ✓      │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ rosenbrock        │ [1, 1]       │ [0.99999, 0.99999] │ ✓      │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ booth             │ [1, 3]       │ [1, 3]             │ ✓      │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ beale             │ [3, 0.5]     │ [3, 0.5]           │ ✓      │
  ├───────────────────┼──────────────┼────────────────────┼────────┤
  │ shifted_quadratic │ [2, -3, 1]   │ [2, -3, 1]         │ ✓      │
  └───────────────────┴──────────────┴────────────────────┴────────┘
  Nelder-Mead Results:
  ┌───────────────────┬──────────────┬──────────────┬────────┐
  │     Function      │ Expected Min │ SciPy Found  │ Status │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ sphere_2d         │ [0, 0]       │ [~0, ~0]     │ ✓      │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ sphere_3d         │ [0, 0, 0]    │ [~0, ~0, ~0] │ ✓      │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ rosenbrock        │ [1, 1]       │ [1, 1]       │ ✓      │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ booth             │ [1, 3]       │ [1, 3]       │ ✓      │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ beale             │ [3, 0.5]     │ [3, 0.5]     │ ✓      │
  ├───────────────────┼──────────────┼──────────────┼────────┤
  │ shifted_quadratic │ [2, -3, 1]   │ [2, -3, 1]   │ ✓      │
  └───────────────────┴──────────────┴──────────────┴────────┘

josevalim · 2026-04-11T18:56:16Z

@nyo16 I dropped two tiny comments. Please test the notebooks work as expected and we should be good to go!

- BFGS line search: replace manual 10x unroll with `while ... unroll: true` - Nelder-Mead: bundle candidate points/values into a map instead of passing 11 positional arguments to `nelder_mead_update` - Migrate both files from `defn minimize` + `deftransformp transform_opts` to the `deftransform minimize` options pattern used by golden_section and brent

optimize.livemd: - Drop unreachable f32 tolerances in the Brent/GoldenSection internals blocks and the JIT demo (tol 1.0e-10/1.0e-8 on f32 inputs never met the gradient criterion, so blocks printed "Converged: false") efficient_frontier.livemd: - Express monthly returns as percentages so objective magnitudes match achievable gradient tolerances - Replace soft-penalty objective (which produced leveraged long/short weights with multi-thousand-% annualized returns) with a z^2/sum(z^2) simplex reparameterization that enforces w_i >= 0 and sum(w_i) = 1 by construction - Rewrite the lambda sweep for the new scale; every point on the frontier now converges and returns are monotonic and realistic

nyo16 · 2026-04-12T13:04:46Z

@josevalim i resolved the 2 comments :)

josevalim · 2026-04-12T13:12:12Z

@nyo16 the issue description says: "[ ] Test livebooks manually in Livebook". Did you double check the notebooks too? Thanks!

- Switch Mix.install to pull scholar from the feature branch on GitHub so the notebooks can be opened directly from Livebook without a local path checkout - Add a prominent educational-use disclaimer to the efficient frontier notebook clarifying that the data is fabricated, the model ignores real-world frictions, and the notebook must not be used for actual investment decisions

nyo16 · 2026-04-12T13:17:49Z

@josevalim yes, i left the

{:scholar, github: "nyo16/scholar", branch: "add-multivariate-optimization"},

for this branch to be testable.

nyo16 · 2026-04-12T13:54:39Z

I changed it in the notebooks to use the formal scholar 0.4.0

josevalim · 2026-04-12T14:44:47Z

The notebooks need to use path dependencies, no? Otherwise they won’t work because they require unreleased code?

nyo16 · 2026-04-12T16:00:09Z

The notebooks need to use path dependencies, no? Otherwise they won’t work because they require unreleased code?

Correct, sorry I will revert it :)

BFGS and Nelder-Mead are unreleased, so a Hex version pin can't resolve them. Fall back to {:scholar, path: "."} and leave the intended ~> 0.4.1 pin commented out as a reminder to swap once the release containing these modules ships.

nyo16 · 2026-04-12T23:05:42Z

@josevalim i revert it.

josevalim · 2026-04-13T06:27:57Z

@nyo16 thank you! Btw, if you could send a PR that replaces the use of Scidata by an in memory data set or similar, it would be very welcome! ❤️

josevalim · 2026-04-13T06:28:18Z

💚 💙 💜 💛 ❤️

nyo16 · 2026-04-13T18:40:13Z

@akoutmos you maybe find this interesting :)

Add BFGS and Nelder-Mead multivariate optimization algorithms

f2404ad

- BFGS: Quasi-Newton method with automatic differentiation - Nelder-Mead: Derivative-free simplex method - Update optimize.livemd with multivariate section - Add efficient_frontier.livemd portfolio optimization example

nyo16 force-pushed the add-multivariate-optimization branch from f9d143a to f2404ad Compare January 30, 2026 23:33

josevalim reviewed Apr 11, 2026

View reviewed changes

Comment thread lib/scholar/optimize/bfgs.ex Outdated

josevalim reviewed Apr 11, 2026

View reviewed changes

Comment thread lib/scholar/optimize/nelder_mead.ex Outdated

josevalim approved these changes Apr 11, 2026

View reviewed changes

nyo16 added 2 commits April 12, 2026 08:45

nyo16 added 2 commits April 12, 2026 09:27

Pin notebooks to released scholar ~> 0.2.0

e03cf98

Pin notebooks to scholar ~> 0.4.1

836d29b

Use path dependency in notebooks until release

8c89ca0

BFGS and Nelder-Mead are unreleased, so a Hex version pin can't resolve them. Fall back to {:scholar, path: "."} and leave the intended ~> 0.4.1 pin commented out as a reminder to swap once the release containing these modules ships.

josevalim merged commit 7937730 into elixir-nx:main Apr 13, 2026
0 of 2 checks passed

Conversation

nyo16 commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Test plan

Uh oh!

nyo16 commented Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

josevalim commented Apr 11, 2026

Uh oh!

nyo16 commented Apr 12, 2026

Uh oh!

josevalim commented Apr 12, 2026

Uh oh!

nyo16 commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nyo16 commented Apr 12, 2026

Uh oh!

josevalim commented Apr 12, 2026

Uh oh!

nyo16 commented Apr 12, 2026

Uh oh!

nyo16 commented Apr 12, 2026

Uh oh!

josevalim commented Apr 13, 2026

Uh oh!

Uh oh!

josevalim commented Apr 13, 2026

Uh oh!

nyo16 commented Apr 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nyo16 commented Jan 30, 2026 •

edited

Loading

nyo16 commented Apr 12, 2026 •

edited

Loading