Skip to content

fix(database): drop programs rejected by the novelty check - #528

Open
flyingdoubleG wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
flyingdoubleG:fix/novelty-reject-drop
Open

flyingdoubleG wants to merge 1 commit into
algorithmicsuperintelligence:mainfrom
flyingdoubleG:fix/novelty-reject-drop

Conversation

@flyingdoubleG

Copy link
Copy Markdown

Problem

ProgramDatabase._add() stores the candidate in self.programs before running the novelty check (the check looks the program up by id), but when the check fails it returns program.id without undoing that store:

self.programs[program.id] = program
feature_coords = self._calculate_feature_coords(program)
if not self._is_novel(program.id, island_idx):
    return program.id  # Do not add non-novel program

So a program the judge rejected as NOT_NOVEL is still in the database. It is not on any island, but it

  • can be picked by _sample_random_parent() (and therefore by sample()),
  • shows up in global get_top_programs() / get_best_program(metric=...),
  • counts towards population_size — _enforce_population_limit() is skipped on this path, so the database can grow past the limit,
  • is written to checkpoints by save(),
  • has already widened the min/max feature scaling via _calculate_feature_coords().

Since add() returns the id, callers can't tell it was rejected either: ProcessParallelController stores its artifacts, logs it to the evolution trace as an accepted child and increments the generation counter of current_island (the rejected program has no island metadata) instead of the target island.

This path only became reachable after #483 fixed the NOT_NOVEL parsing; before that the judge never rejected anything.

Minimal repro (population_size=3, one island, identical embeddings, judge always answers NOT_NOVEL):

add returned dup0 ... dup4
islands:  [{'seed'}]
programs: ['dup0', 'dup1', 'dup2', 'dup3', 'dup4', 'seed']

Fix

  • On novelty rejection, remove the provisional entry (restoring any previous entry with the same id) and return None. This is the contract add() already has for population_strategy.admit rejections, and both existing callers handle it (ProcessParallelController raises _CandidateRejected, _migrate_one checks is not None).
  • Compute feature coordinates after the novelty check so rejected programs don't change feature_stats.
  • Docstring / log message updated to mention novelty rejection.

Tests

New tests/test_novelty_rejection.py (mocked embedding client + a judge that always returns NOT_NOVEL): rejected programs are not stored, never sampled or ranked, not checkpointed, and don't touch feature_stats. All 4 fail on main and pass with this change. Existing novelty / migration / population-strategy tests still pass; the rest of the unit suite has the same results as main locally.

_add() stored the candidate in self.programs before the novelty check and
returned its id when the check failed, leaving the rejected program in the
database: it could be sampled as a parent, ranked by get_top_programs(),
counted against population_size and written to checkpoints. Callers could
not tell it was rejected, so the parallel controller stored its artifacts,
traced it and bumped the wrong island's generation counter.

Remove the provisional entry and return None, matching the existing
admission-strategy rejection contract, and compute feature coordinates
after the novelty check so rejected programs do not widen feature scaling.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants