Merge branch 'tickets/DM-41049'

Author: taranu

.pre-commit-config.yaml

@@ -6,7 +6,7 @@ repos:
       - id: end-of-file-fixer
       - id: trailing-whitespace
   - repo: https://github.com/psf/black
-    rev: 23.3.0
+    rev: 24.3.0
     hooks:
       - id: black
         # It is recommended to specify the latest version of Python
@@ -15,7 +15,7 @@ repos:
         # https://pre-commit.com/#top_level-default_language_version
         language_version: "python3.11"
   - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
+    rev: 5.13.2
     hooks:
       - id: isort
         name: isort (python)

doc/lsst.analysis.tools/faqs.rst

@@ -6,7 +6,7 @@ FAQs
 What Order are Configs Applied In?
 ----------------------------------
 
-In analysis tools there are a number of ways to set config options which are applied in a certain order. If 
+In analysis tools there are a number of ways to set config options which are applied in a certain order. If
 you get this order wrong you might be suprised by what your action is doing. In order from first applied to
 last applied:
 
@@ -16,8 +16,8 @@ last applied:
 4. The pipeline file configs
 5. The command line --config and --config-file options
 
-Due to this configs that are obs package specific should not be specified in the pipeline file because they 
-overwrite the obs specific configs. For example the bands option, if that is set in the pipeline file it will 
+Due to this configs that are obs package specific should not be specified in the pipeline file because they
+overwrite the obs specific configs. For example the bands option, if that is set in the pipeline file it will
 overwrite the obs package which means that if you try to apply the same pipeline to HSC and DC2 data (for
 example) it will not work for both as each survey has different band coverage.
 
@@ -30,12 +30,12 @@ to be applied in the bands that the plot is being made in then the flag config o
 Where is the line between python and YAML?
 ------------------------------------------
 
-This can be a bit of a blurry line and in the end is up to the individual developer. A rough guideline is that 
+This can be a bit of a blurry line and in the end is up to the individual developer. A rough guideline is that
 if the code is reusable either for variants of the original action or useful for other actions then it belongs
 in its own python class. If it is single use or a specific instance of a class then it should be done through
 the YAML config. An example of this is the photometric repeatability metrics. Here there is a base action
 called PhotometricRepeatability, defined in python, which is then configured for each individual application
-in the pipeline YAML. 
+in the pipeline YAML.
 
 .. code-block:: yaml
 
@@ -46,18 +46,18 @@ in the pipeline YAML.
    atools.modelPhotRepGalSn5to10.process.filterActions.perGroupStdevFiltered.selectors.sn.maximum: 10
 
 The first line calls the action defined in python, after that the specific configuration is set in the
-pipeline YAML. This is a good balance between defining the entire action in YAML and having duplicate actions 
+pipeline YAML. This is a good balance between defining the entire action in YAML and having duplicate actions
 defined in the python with only very mild differences.
 
 Can analysis tools put things in the butler?
 --------------------------------------------
 
 Short answer, yes! Long answer, while the tools themselves do not put things in the butler the tasks in the
 pipelines contained in analysis tools can. As an example lets look at the plots and metrics calculated from
-matching to the reference catalogues. First there is a task that matches the input data to the reference 
+matching to the reference catalogues. First there is a task that matches the input data to the reference
 catalogue. This then saves in the butler the matched output so that it can be read in later by multiple
-atools and also used in a notebook or script if further QA is required. After the matched catalogue is made 
-the plotting/metric/calculation atools read it in, then act on it. The atools themselves do not butler put 
+atools and also used in a notebook or script if further QA is required. After the matched catalogue is made
+the plotting/metric/calculation atools read it in, then act on it. The atools themselves do not butler put
 the metric/plot/calculated data but return them to the task that ran them which then puts them in the right
 place. Most of this is hidden from the user and the interfaces handle putting plots and metrics themselves.
 

pipelines/coaddDiffMatchedQualityExtended.yaml

@@ -1,6 +1,8 @@
 description: |
   Matched difference (measured vs reference) plots/metrics
 parameters:
+  color_diff_min: -250
+  color_diff_max: 250
   mag_x_min: 17
   mag_x_max: 29
   mag_diff_min: -1000
@@ -17,10 +19,16 @@ tasks:
     config:
       connections.outputName: diff_matched_truth_summary_objectTable_tract
 
-      # plots
+      atools.matchedRefCModelColorDiff: MatchedRefCoaddDiffColorTool
+      atools.matchedRefCModelColorDiff.produce.plot.xLims: lims_mag_x
+      atools.matchedRefCModelColorDiff.produce.plot.yLims: lims_color_diff
+
+      atools.matchedRefCModelColorChi: MatchedRefCoaddDiffColorTool
+      atools.matchedRefCModelColorChi.compute_chi: true
+      atools.matchedRefCModelColorChi.produce.plot.xLims: lims_mag_x
+      atools.matchedRefCModelColorChi.produce.plot.yLims: lims_mag_chi
+
       atools.matchedRefCModelMagDiff: MatchedRefCoaddDiffMagTool
-      # Doesn't work; interpreted as float
-      # atools.matchedRefCModelMagDiff.produce.plot.xLims: [parameters.mag_x_min, parameters.mag_x_max]
       atools.matchedRefCModelMagDiff.produce.plot.xLims: lims_mag_x
       atools.matchedRefCModelMagDiff.produce.plot.yLims: lims_mag_diff
 
@@ -29,7 +37,6 @@ tasks:
       atools.matchedRefCModelMagChi.produce.plot.xLims: lims_mag_x
       atools.matchedRefCModelMagChi.produce.plot.yLims: lims_mag_chi
 
-      # TODO: Can this be a one liner?
       atools.matchedRefPositionXDiff: MatchedRefCoaddDiffPositionTool
       atools.matchedRefPositionXDiff.coord_meas: x
       atools.matchedRefPositionXDiff.coord_ref: refcat_x
@@ -55,12 +62,13 @@ tasks:
       atools.matchedRefPositionYChi.produce.plot.xLims: lims_mag_x
       atools.matchedRefPositionYChi.produce.plot.yLims: lims_pos_chi
       atools.matchedRefPositionYChi.compute_chi: true
-
       python: |
         from lsst.analysis.tools.atools.diffMatched import (
+          MatchedRefCoaddDiffColorTool,
           MatchedRefCoaddDiffMagTool,
           MatchedRefCoaddDiffPositionTool,
         )
+        lims_color_diff = (parameters.color_diff_min, parameters.color_diff_max)
         lims_mag_x = (parameters.mag_x_min, parameters.mag_x_max)
         lims_mag_chi = (parameters.mag_chi_min, parameters.mag_chi_max)
         lims_mag_diff = (parameters.mag_diff_min, parameters.mag_diff_max)

pipelines/coaddQualityCore.yaml

@@ -27,6 +27,8 @@ tasks:
       atools.shapeSizeDetRadiusVsCmodelMag.size_type: "determinantRadius"
       atools.shapeSizeDetRadiusVsCmodelMag.mag_x: "cmodel_err"
       atools.shapeSizeDetRadiusVsCmodelMag.size_y: "shape_slot"
+      atools.shapeSizeDetRadiusVsCmodelMag.produce.plot.xLims: (17, 29)
+      atools.shapeSizeDetRadiusVsCmodelMag.produce.plot.yLims: (-4, 3)
       atools.shapeSizeDetRadiusVsCmodelMag.applyContext: CoaddContext
       atools.shapeSizeDetRadiusVsPsfMag: SizeMagnitudePlot
       atools.shapeSizeDetRadiusVsPsfMag.size_type: "determinantRadius"
@@ -37,11 +39,15 @@ tasks:
       atools.cModelBulgeSizeVsCmodelBulgeMag.size_type: "singleColumnSize"
       atools.cModelBulgeSizeVsCmodelBulgeMag.mag_x: "bulge_err"
       atools.cModelBulgeSizeVsCmodelBulgeMag.size_y: "bulge"
+      atools.cModelBulgeSizeVsCmodelBulgeMag.produce.plot.xLims: (17, 29)
+      atools.cModelBulgeSizeVsCmodelBulgeMag.produce.plot.yLims: (-4, 3)
       atools.cModelBulgeSizeVsCmodelBulgeMag.applyContext: CoaddContext
       atools.cModelDiskSizeVsCmodelDiskMag: SizeMagnitudePlot
       atools.cModelDiskSizeVsCmodelDiskMag.size_type: "singleColumnSize"
       atools.cModelDiskSizeVsCmodelDiskMag.mag_x: "disk_err"
       atools.cModelDiskSizeVsCmodelDiskMag.size_y: "disk"
+      atools.cModelDiskSizeVsCmodelDiskMag.produce.plot.xLims: (17, 29)
+      atools.cModelDiskSizeVsCmodelDiskMag.produce.plot.yLims: (-4, 3)
       atools.cModelDiskSizeVsCmodelDiskMag.applyContext: CoaddContext
       python: |
         from lsst.analysis.tools.atools import *

pyproject.toml

@@ -8,4 +8,3 @@ line_length = 110
 
 [tool.lsst_versions]
 write_to = "python/lsst/analysis/tools/version.py"
-

python/lsst/analysis/tools/actions/plot/rhoStatisticsPlot.py

@@ -21,22 +21,18 @@
 
 from __future__ import annotations
 
-from typing import TYPE_CHECKING, Any, Iterable, Mapping
-
 __all__ = ("RhoStatisticsPlot",)
 
+from typing import Any, Iterable, Mapping
+
 import numpy as np
 from lsst.pex.config import ConfigDictField
+from matplotlib.figure import Figure
 
-from ...interfaces import PlotAction, Vector
+from ...interfaces import KeyedData, KeyedDataSchema, PlotAction, Vector
 from .plotUtils import addPlotInfo
 from .xyPlot import XYPlot
 
-if TYPE_CHECKING:
-    from matplotlib.figure import Figure
-
-    from ...interfaces import KeyedData, KeyedDataSchema
-
 
 class RhoStatisticsPlot(PlotAction):
     """Make multiple plots of rho statistics.
@@ -46,10 +42,8 @@ class RhoStatisticsPlot(PlotAction):
     :ref:`here <rho_definitions>`.
     """
 
-    rhoPlots = ConfigDictField(
+    rhoPlots = ConfigDictField[str, XYPlot](
         doc="A configurable dict describing the rho statistics to plot.",
-        keytype=str,
-        itemtype=XYPlot,
         default={},
     )
 
@@ -91,15 +85,17 @@ def getInputSchema(self) -> KeyedDataSchema:
 
     def getOutputNames(self) -> Iterable[str]:
         # Docstring inherited
-        return ("rho3alt", "rho1", "rho2", "rho3", "rho4", "rho5")
+        for key in self.rhoPlots.keys():
+            yield key
 
     def __call__(self, data: KeyedData, **kwargs) -> Mapping[str, Figure]:
         self._validateInput(data)
         return self.makePlot(data, **kwargs)
 
     def _validateInput(self, data: KeyedData) -> None:
-        if not set(("rho3alt", "rho1", "rho2", "rho3", "rho4", "rho5")).issubset(data.keys()):
-            raise ValueError("Input data must contain rho3alt, rho1, rho2, rho3, rho4, and rho5.")
+        required = set(self.rhoPlots.keys())
+        if not required.issubset(data.keys()):
+            raise ValueError(f"Input data must contain {', '.join(self.rhoPlots.keys())}")
 
     def makePlot(
         self, data: KeyedData, plotInfo: Mapping[str, str] | None = None, **kwargs: Any
@@ -141,28 +137,20 @@ def makePlot(
         :ref:`getting started guide<analysis-tools-getting-started>`.
         """
         fig_dict: dict[str, Figure] = {}
-        for rho_name in ("rho1", "rho2", "rho3", "rho4", "rho5"):
+        for rho_name in self.rhoPlots.keys():
             rho: XYPlot = self.rhoPlots[rho_name]
-
             subdata = {
                 "x": data[rho_name].meanr,  # type: ignore
-                "y": data[rho_name].xip,  # type: ignore
-                "yerr": np.sqrt(data[rho_name].varxip),  # type: ignore
                 "xerr": None,
             }
+            if rho_name == "rho3alt":
+                subdata["y"] = data[rho_name].xi  # type: ignore
+                subdata["yerr"] = np.sqrt(data[rho_name].varxi)
+            else:
+                subdata["y"] = data[rho_name].xip  # type: ignore
+                subdata["yerr"] = np.sqrt(data[rho_name].varxip)  # type: ignore
             fig = rho(subdata, **kwargs)
             if plotInfo is not None:
                 fig_dict[rho_name] = addPlotInfo(fig, plotInfo)
 
-        # rho3alt is handled differently because its attributes differ.
-        subdata = {
-            "x": data["rho3alt"].meanr,  # type: ignore
-            "y": data["rho3alt"].xi,  # type: ignore
-            "yerr": np.sqrt(data["rho3alt"].varxi),  # type: ignore
-            "xerr": None,
-        }
-        fig = self.rhoPlots["rho3alt"](subdata, **kwargs)  # type: ignore[misc]
-        if plotInfo is not None:
-            fig_dict["rho3alt"] = addPlotInfo(fig, plotInfo)
-
         return fig_dict