Skip to content

TrainingGuide.md

Latest commit

 

History

History
180 lines (134 loc) · 6.36 KB

TrainingGuide.md

File metadata and controls

180 lines (134 loc) · 6.36 KB

Training Guide

Prerequisites

Before starting the training process, ensure you have:

Training Pipeline Overview

The training pipeline follows this sequence:

graph LR
    A[PGN Data] --> B[FEN Extraction]
    B --> C[Preprocess FEN Data]
    C --> D[Filter Static Positions]
    D --> E[Generate Training Data]
    E --> F[Model Training<br/>Separate Repository]
    F --> G[Export Quantized Weights]
    G --> H[Build Engine]
Loading

Training Process

1. Extract FEN data from PGN

Use pgn-extract tool to extract FEN from PGN games:

pgn-extract -Wfen <pgn-file> > <output-fen-file>

2. Preprocess FEN Data

Run the FEN data preprocessing utility to extract FENs and label each FEN with game result and position count:

cargo run --bin preprocess_fen_data [input_dir] [output_dir] [skip_count] [max_count]

Parameters:

  • input_dir: Path to directory containing FEN data files generated by pgn-extract
  • output_dir: Path where preprocessed FEN data files will be saved (CSV format: fen,result,position_count,total_position_count)
  • skip_count: Number of early moves to skip per game (e.g., 10 to skip opening positions)
  • max_count: Maximum number of positions to include per game (e.g., 200 to limit game length)

Output Format:

  • fen: FEN string representation of the position
  • result: Game result (0.0 = loss, 0.5 = draw, 1.0 = win)
  • position_count: Current position count
  • total_position_count: Total number of positions in the game

Example:

cargo run --bin preprocess_fen_data data/fens/ data/preprocessed/ 10 200

Note: Before running this script, you need to extract FEN positions from your PGN files using the pgn-extract tool:

pgn-extract -Wfen -o data/extracted_fens.txt data/games.pgn

3. Filter Non-static FEN Data

Next, filter the preprocessed FEN data to keep only "static" positions (positions that are not in check and have no immediate tactical complications). This step uses an exchange search to determine if positions are static:

cargo run --bin filter_fen_data [input_dir] [output_dir] [batch_size] [num_threads]

Parameters:

  • input_dir: Path to directory containing preprocessed FEN data files (output from step 1)
  • output_dir: Path where filtered static FEN data files will be saved (CSV format: fen,weighted_result)
  • batch_size: Number of positions to buffer before writing to disk (e.g., 10000)
  • num_threads: Number of parallel threads to use (e.g., 4)

Weighted Result Calculation: The output uses a weighted game result based on position timing in the game:

  • result_weight = position_index / total_position_count (how far into the game)
  • weighted_result = 0.5 + 0.5 * result_weight for wins (1.0)
  • weighted_result = 0.5 - 0.5 * result_weight for losses (0.0)
  • weighted_result = 0.5 for draws (0.5)

This gives positions later in the game more extreme values (closer to 1.0 or 0.0) since the game outcome is more certain, while early positions stay closer to 0.5.

Example:

cargo run --bin filter_fen_data data/preprocessed/ data/static/ 10000 4

4. Generate Training Data

Next, generate the training data from the filtered static FEN files. This step converts FEN positions with weighted game results to the compressed training format:

cargo run --bin gen_training_data [input_dir] [output_dir] [batch_size] [num_threads]

Parameters:

  • input_dir: Path to directory containing filtered static FEN data files (output from step 2)
  • output_dir: Path where training data files will be saved (text file)
  • batch_size: Number of positions to buffer before writing to disk (e.g., 10000)
  • num_threads: Number of parallel threads to use (e.g., 4)

Output Format: The output file uses a compressed run-length encoded text format to minimize storage.

  • Features are encoded as: [number_of_zeros]X[number_of_zeros]X...
  • For example: 12X34X5X...,1.0 means 12 zeros, then a 1, then 34 zeros, then a 1, then 5 zeros, then a 1, with a game result of 1.0
  • The last field is the game result (from the current player's perspective)
  • If the current player is BLACK, the result is inverted (1.0 - result)
  • This text format compresses the sparse 768-dimensional feature vector efficiently while remaining human-readable.

5. Train Model

Train the model using the new Rust trainer from the separate repository:

git clone https://github.com/buildingwheels/ShallowGuessTrainer

Then follow instructions in ShallowGuessTrainer for training and exporting the trained weights.

6. Build Engine

  1. Switch hidden layer size by modifying config/network.cfg
  2. Build the engine with the new quantized weights:
cargo build --release

Utilities

Filter PGN Games

Filters games from a PGN file:

cargo run --bin filter_pgn [options] [input.pgn] [output.pgn] [filters]

Positional Parameters:

  • input.pgn: Path to the input PGN file
  • output.pgn: Path where filtered games will be saved
  • filters: Semicolon-separated tag=value pairs (e.g., "WhiteElo>2500;BlackElo>2500")
    • Supported operators: =, !=, >, <

Options:

  • --filter-if-missing-tag: Filter out games that don't have the specified tag (default: games with missing tags are kept)

Build Validation Dataset

Creates validation datasets by randomly sampling from training data with configurable draw ratio:

cargo run --bin build_validation_dataset [input_file] [output_file] [sample_count] [draw_ratio]

Parameters:

  • input_file: Path to the input training data file
  • output_file: Path where the validation dataset will be saved
  • sample_count: Number of samples to randomly select for validation
  • draw_ratio: Fraction of samples that should be draws (0.0 to 1.0, e.g., 0.3 for 30% draws)

Example:

cargo run --bin build_validation_dataset data/training.txt data/validation.txt 10000 0.3

Blend Training Data

Randomly blends multiple training data files into new mixed files:

cargo run --bin blend_data [input_path] [output_path] [num_files]

Split Large Training Data

Splits large training data files into smaller parts:

cargo run --bin split_large_data [input_file] [lines_per_file]