Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions docs/model_customization/model_customization.rst
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,116 @@ Key Features
with clear precedence. Use ``get_resolved_recipe()`` to inspect the merged configuration
before job submission. See :doc:`finetuning_serverful` and :doc:`finetuning_hyperpod` for examples.

**Job Notifications**
Receive SNS notifications when training jobs complete, fail, or stop. Uses EventBridge
rules to route SageMaker Training Job status changes to your SNS topic.

The SDK creates one rule per unique config (topic + events + prefix). Re-running with
the same config reuses the existing rule. Different configs create separate rules.

.. note::
Supported for SMTJ (serverful and serverless) compute only. HyperPod is not currently supported.

.. code-block:: python

trainer = SFTTrainer(
model="nova-textgeneration-micro",
training_dataset="s3://my-bucket/train.jsonl",
accept_eula=True,
notifications={
"sns_topic_arn": "arn:aws:sns:us-east-1:123456789012:my-topic", # Required
"events": ["Completed", "Failed"], # Optional (default: Completed, Failed, Stopped)
"job_name_prefix": "my-team-sft-", # Optional: filter by job name
"event_bus_arn": "arn:aws:events:us-east-1:123456789012:event-bus/custom-bus" # Optional
},
)
job = trainer.train(wait=False) # Notification sent on completion

# Access the rule ARN
print(trainer.notification_rule_arn)

# List and manage rules
rules = trainer.list_notification_rules()
trainer.delete_notification_rule(rule_arn=trainer.notification_rule_arn)

**Prerequisites:**

- An SNS topic (and subscription) with a resource policy allowing ``events.amazonaws.com``.
To set up a topic and subscription, see `Creating an SNS topic and subscription <https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html>`_.
- IAM permissions: ``events:PutRule``, ``events:PutTargets``, ``events:ListRules``,
``events:RemoveTargets``, ``events:DeleteRule``

**Monitoring: show_metrics()**
Plot training metrics after a job completes. Works across all compute types.

- **Nova models**: Metrics parsed from CloudWatch logs.
- **OSS models**: Metrics pulled from MLflow.

.. code-block:: python

# Plot all available metrics
df = trainer.show_metrics()

# Plot specific metrics
df = trainer.show_metrics(metrics=["training_loss", "lr"])

# Filter by step range
df = trainer.show_metrics(starting_step=10, ending_step=100)

# Filter by time window
from datetime import datetime
df = trainer.show_metrics(
start_time=datetime(2026, 1, 1, 10, 0, 0),
end_time=datetime(2026, 1, 1, 12, 0, 0),
)

**After a kernel restart:**

.. code-block:: python

# Standalone (SMTJ)
from sagemaker.train import plot_training_metrics
plot_training_metrics("my-sft-job")

# Re-attach (HyperPod — needs cluster name for log group resolution)
from sagemaker.train import SFTTrainer
from sagemaker.core.training.configs import HyperPodCompute

trainer = SFTTrainer(
model="nova-textgeneration-micro",
training_dataset="s3://unused",
compute=HyperPodCompute(cluster_name="my-cluster", instance_type="ml.p5.48xlarge")
)
trainer._latest_training_job = "my-hp-job"
df = trainer.show_metrics()

**Monitoring: stream_logs()**
Stream CloudWatch logs in real-time while a job is running.

.. code-block:: python

# Start training non-blocking
job = trainer.train(wait=False)

# Stream logs (blocks until job completes or Ctrl+C)
trainer.stream_logs()

# Custom polling interval (seconds)
trainer.stream_logs(poll=10)

# Stream from a specific start time - providing this will speed up execution.
from datetime import datetime
trainer.stream_logs(start_time=datetime(2026, 1, 1, 15, 0, 0))

.. note::

- **SMTJ**: Streaming auto-stops when the job reaches a terminal state.
- **HyperPod**: Streaming runs until you press Ctrl+C. Logs may take a few minutes
to propagate to CloudWatch on first run.


----


.. toctree::
:maxdepth: 1
Expand Down