|
55 | 55 | "source": [ |
56 | 56 | "## 1. Introduction\n", |
57 | 57 | "\n", |
58 | | - "This notebook demonstrates how to make image subsets (a smaller piece of a parent LSST image) using the Butler. This method for generating small image subsets is the right tool when small images are desired from a single visit image or deep coadd patch, or, when the analysis workflow is already using the Butler for image analysis. Instead, the Rubin cutout service (see tutorial notebooks 103.4 and 103.5) is a better tool when wanting to generate image cutouts at a range of coordinates or observations times.\n", |
| 58 | + "This notebook demonstrates how to make image subsets (a smaller piece of a parent LSST image) using the Butler. This method for generating small image subsets is the right tool when small images are desired from a single visit image or deep coadd patch, or, when the analysis workflow is already using the Butler for image analysis. Instead, the Rubin cutout service is a better tool when wanting to generate image cutouts at a range of coordinates or observations times. The Rubin cutout service is demonstrated in notebooks in the DP1 103 notebook series.\n", |
59 | 59 | "\n", |
60 | 60 | "Butler-related documentation:\n", |
61 | 61 | "* [pipelines middleware Frequently Asked Questions](https://pipelines.lsst.io/middleware/index.html)\n", |
|
70 | 70 | "id": "72fa86ab-0996-4637-9a99-d94b2084b786", |
71 | 71 | "metadata": {}, |
72 | 72 | "source": [ |
73 | | - "**Related tutorials:** The earlier 100-level Butler tutorials in this series show how to retrieve data with the Butler and how to explore and discover the dataset types and their properties." |
| 73 | + "**Related tutorials:** The earlier 100-level Butler tutorials in this series show how to retrieve data with the Butler and how to explore and discover the dataset types and their properties. The 103 series demonstrate the Rubin cutout service to produce image cutouts." |
74 | 74 | ] |
75 | 75 | }, |
76 | 76 | { |
|
91 | 91 | "outputs": [], |
92 | 92 | "source": [ |
93 | 93 | "import matplotlib.pyplot as plt\n", |
94 | | - "import numpy as np\n", |
95 | 94 | "import math\n", |
96 | 95 | "\n", |
97 | 96 | "from lsst.daf.butler import Butler\n", |
98 | 97 | "\n", |
99 | 98 | "import lsst.sphgeom as sphgeom\n", |
100 | 99 | "import lsst.geom as geom\n", |
101 | | - "from lsst.geom import Box2I, Point2I, Extent2I\n", |
| 100 | + "from lsst.geom import Box2I, Point2I, Extent2I, Point2D\n", |
102 | 101 | "\n", |
103 | | - "import lsst.afw.image as afw_image\n", |
104 | 102 | "from lsst.afw.image import ExposureF, LOCAL, PARENT\n", |
105 | 103 | "import lsst.afw.display as afw_display\n", |
106 | 104 | "\n", |
|
395 | 393 | "source": [ |
396 | 394 | "## 3. Retrieve subset from the butler\n", |
397 | 395 | "\n", |
398 | | - "Several methods exist for extracting subsets from the butler. A few are demonstrated below, including passing the bounding box directly to the butler (Section 3.1), retrieving the full deep_coadd and using the subset method (Section 3.2) and passing the bounding box to the full `deep_coadd` via `ExposureF` (Section 3.3). \n", |
| 396 | + "Several methods exist for extracting subsets from the butler. A few are demonstrated below, including passing the bounding box directly to the butler (Section 3.1), retrieving the full `deep_coadd` and using the subset method (Section 3.2) and passing the bounding box to the full `deep_coadd` via `ExposureF` (Section 3.3). \n", |
399 | 397 | "\n", |
400 | 398 | "### 3.1. Pass bounding box to the Butler\n", |
401 | 399 | "\n", |
402 | | - "The bounding box can be passed directly to the butler to retrieve only a subset of the image using the keyword `parameters`. First, define the `parameters` dictionary to contain the bounding box in a dictionary key called `subset_box`. \n" |
| 400 | + "The bounding box can be passed directly to the butler to retrieve only a subset of the image using the keyword `parameters`. First, define the `parameters` dictionary to contain the bounding box in a dictionary key called `subset_box_sky`. \n" |
403 | 401 | ] |
404 | 402 | }, |
405 | 403 | { |
|
462 | 460 | "source": [ |
463 | 461 | "> **Figure 1:** A subset of a sky region in the ECDFS returned after passing a bounding box to the Butler \n", |
464 | 462 | "\n", |
465 | | - "Some attributes of the subset can be displayed with the `getXY0` and `getDimensions` methods." |
| 463 | + "Some attributes of the subset can be displayed with the `getXY0` and `getDimensions` methods. The first returns the difference in x and y values between the global image pixel origin and subset pixel origin, and the second returns the subset dimensions in pixels." |
466 | 464 | ] |
467 | 465 | }, |
468 | 466 | { |
|
472 | 470 | "metadata": {}, |
473 | 471 | "outputs": [], |
474 | 472 | "source": [ |
475 | | - "print(\"The offset in x,y pixels of the bounding box from the origin of the coadd is \", subset_image.getXY0())\n", |
476 | | - "print(\"The pixel dimensions of the coadd is \", subset_image.getDimensions())\n" |
| 473 | + "print(f\"Subset pixel origin relative to the coadd pixel origin (XY0): {subset_image.getXY0()}\")\n", |
| 474 | + "print(f\"Subset dimensions in pixels: {subset_image.getDimensions()}\")" |
477 | 475 | ] |
478 | 476 | }, |
479 | 477 | { |
|
483 | 481 | "source": [ |
484 | 482 | "### 3.2. Subsets after retrieving coadd\n", |
485 | 483 | "\n", |
486 | | - "It is also possible to make subsets after retrieving the full `deep_coadd` from the butler. This may be preferable if many coadds are needed from the same patch and tract image.Now 3 ways to use the `subset_box` to retrieve the subset. \n", |
| 484 | + "It is also possible to make subsets after retrieving the full `deep_coadd` from the butler. This may be preferable if many coadds are needed from the same patch and tract image. There are three ways to use `subset_box` to retrieve the desired subset.\n", |
487 | 485 | "\n", |
488 | 486 | "First, retrieve the full `deep_coadd` of interest.\n" |
489 | 487 | ] |
|
533 | 531 | "\n", |
534 | 532 | "#### 3.2.2. Subset method\n", |
535 | 533 | "\n", |
536 | | - "Extract the same region of interest but now using the subset method. The use case: When the user already has the full image from the butler in memory, and wants to make one or more subsets from it. \n", |
| 534 | + "Extract the same region of interest but now using the `subset` method. The use case: When the user already has the full image from the butler in memory, and wants to make one or more subsets from it. \n", |
537 | 535 | "\n", |
538 | | - "An optional keyword origin will specify the the coordinate system of the input bounding box (options are `PARENT`, the default. Alternatively, if the input bounding box is defined in `LOCAL` coordinates, the keyword should be reset.) Since `subset_box` was defined based on the `PARENT` coordinate system, set `origin=PARENT`.\n", |
| 536 | + "An optional keyword `origin` will specify the the coordinate system of the input bounding box (options are `PARENT`, the default. Alternatively, if the input bounding box is defined in `LOCAL` coordinates, the keyword should be reset.) Since `subset_box` was defined based on the `PARENT` coordinate system, set `origin=PARENT`.\n", |
539 | 537 | "\n", |
540 | | - "Note that subset is not creating a \"deep copy\" of the image. Any operations that are performed on the pixel array generated by the subset method, `subset_subset`, will also happen to `full_exposure`" |
| 538 | + "Note that `subset` is not creating a \"deep copy\" of the image. Any operations that are performed on the pixel array generated by the `subset` method, `subset_subset`, will also happen to `full_exposure`." |
541 | 539 | ] |
542 | 540 | }, |
543 | 541 | { |
|
561 | 559 | "id": "44b72531-ae48-44ee-bfeb-91c90f14f336", |
562 | 560 | "metadata": {}, |
563 | 561 | "source": [ |
564 | | - "> **Figure 3:** A subset of the same sky region in Figure 1 and 2 but generated using the subset method\n", |
| 562 | + "> **Figure 3:** A subset of the same sky region in Figure 1 and 2 but generated using the `subset` method\n", |
565 | 563 | "\n", |
566 | 564 | "#### 3.2.3. Pass bounding box to ExposureF\n", |
567 | 565 | "\n", |
568 | 566 | "Another method to generate the subset is by explicit constructor instantiation, which means passing the `full_exposure` image and bounding box directly into the `ExposureF` image class.\n", |
569 | 567 | "\n", |
570 | | - "The deep keyword controls the type of image subset that is returned, similar to the python `deep_copy` function. If setting `deep=False`: the subset points to the same block of memory as the parent full_exposure. No new pixel data is duplicated into memory. While this is efficient in that it is quick and uses less memory, because the subset and the parent share the same memory, modifying one modifies the other. Any change to the pixel values in the subset will apply also to the `full_exposure` stored in memory.\n", |
| 568 | + "The `deep` keyword controls the type of image subset that is returned, similar to the python `deep_copy` function. If setting `deep=False`, the subset points to the same block of memory as the parent `full_exposure`. No new pixel data is duplicated into memory. While this is efficient in that it is quick and uses less memory, because the subset and the parent share the same memory, modifying one modifies the other. Any change to the pixel values in the subset will apply also to the `full_exposure` stored in memory.\n", |
571 | 569 | "\n", |
572 | | - "Setting `deep=True` instead is like `deep_copy` in that it allocates new memory and physically duplicates the subset into a new ExposureF object. This allows the user to apply mathematical operations on the subset without altering the the `full_exposure` image also stored in memory. However, it takes longer to execute and uses more memory (though for small or few subsets this is negligible).\n", |
| 570 | + "Setting `deep=True` instead is like `deep_copy` in that it allocates new memory and physically duplicates the subset into a new `ExposureF` object. This allows the user to apply mathematical operations on the subset without altering the the `full_exposure` image also stored in memory. However, it takes longer to execute and uses more memory (though for small or few subsets this is negligible).\n", |
573 | 571 | "\n", |
574 | 572 | "Like the `subset` method, `origin` is also a possible keyword here for specifying the coordinate system of the input bounding box." |
575 | 573 | ] |
|
595 | 593 | "id": "443b7ef4-10ba-4190-b7a3-bf0d35063797", |
596 | 594 | "metadata": {}, |
597 | 595 | "source": [ |
598 | | - "> **Figure 4:** A subset of the same sky region in Figures 1-3 but generated by passing the bounding box to ExposureF \n", |
| 596 | + "> **Figure 4:** A subset of the same sky region in Figures 1-3 but generated by passing the bounding box to `ExposureF` \n", |
599 | 597 | "\n", |
600 | 598 | "## 4. Retrieve multiple subsets from the butler\n", |
601 | 599 | "\n", |
602 | | - "This section demonstrates an efficient way to retrieve multiple subsets from the butler. Instead of first retrieving the deep_coadd image into memory (which is many gigabytes is size), an alternative is to use the `butler.getDeferred()` method. This identifies the images on the remote server and returns a lightweight \"pointer\" or \"handle\" (specifically, a `DeferredDatasetHandle`) instead of the actual data. `getDeferred` is fast and uses nearly no memory, while queuing up a bunch of subsets to retrieve at a later time all at once without transferring the `deep_coadd`(s) themselves. The call `handle.get(parameters=params)` moves the subsets to the local disk. Passing the bbox parameter enables the butler to transfer only the pixels inside the bounding box.\n", |
| 600 | + "This section demonstrates an efficient way to retrieve multiple subsets from the butler. Instead of first retrieving the `deep_coadd` image into memory (which is many gigabytes is size), an alternative is to use the `butler.getDeferred()` method. This identifies the images on the remote server and returns a lightweight \"pointer\" or \"handle\" (specifically, a `DeferredDatasetHandle`) instead of the actual data. `getDeferred` is fast and uses nearly no memory, while queuing up a bunch of subsets to retrieve at a later time all at once without transferring the `deep_coadd`(s) themselves. The call `handle.get(parameters=params)` moves the subsets to the local disk. Passing the `bbox` parameter enables the butler to transfer only the pixels inside the bounding box.\n", |
603 | 601 | "\n", |
604 | 602 | "Since the galaxies may not be centered in the middle of pixels, use the `Point2D` to convert the ra and dec to floating point pixel coordinates, rather than rounded integer types returned by `Point2I`. The example below will use the `Box2I.makeCenteredBox` method, which will take as input the subset center and subset edge size, rather than the bottom left coordinate of the subset. This simplifies the bounding box definition for this example.\n", |
605 | 603 | "\n", |
|
672 | 670 | "id": "af62f43e-1fff-461f-a44c-2d10b589a5a3", |
673 | 671 | "metadata": {}, |
674 | 672 | "source": [ |
675 | | - "The cell below loops through the targets to define the deferred handles. For each target, it first identifies the patch and tract containing the galaxy (since they are not all in the same image) and then constructs a 50x50 pixel bounding box centered at the target. As the loop iterates to the next galaxy, the previous subset object is overwritten so that the memory footprint never exceeds the size of a single subset." |
| 673 | + "The cell below loops through the targets to define the deferred handles. For each target, it first identifies the patch and tract containing the galaxy (since the galaxies are not necessarily all in the same patch and tract) and then constructs a 50x50 pixel bounding box centered at the target. As the loop iterates to the next galaxy, the previous `subset` object is overwritten so that the memory footprint never exceeds the size of a single subset." |
676 | 674 | ] |
677 | 675 | }, |
678 | 676 | { |
|
733 | 731 | "id": "89499498-4f17-4e69-b00e-e848cb1c4271", |
734 | 732 | "metadata": {}, |
735 | 733 | "source": [ |
736 | | - "> **Figure 5:** A grid of subsets for 27 AGN in the ECDFS field generated by the Butler using deferred handles." |
| 734 | + "> **Figure 5:** A grid of subsets for 27 AGNs in the ECDFS field generated by the Butler using deferred handles." |
737 | 735 | ] |
738 | 736 | } |
739 | 737 | ], |
|
0 commit comments