Multiple issues when some MPI processes are left unused

[anand-avinash]

If some MPI processes are left unused (for example, while using attribute based detector blocks), it now breaks the workflow at many places. I am listing here some of them that I found:

Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/mpi_set_n_blocks_test2.py", line 117, in <module>
    sim.prepare_pointings()
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1312, in prepare_pointings
    prepare_pointings(
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/pointings_in_obs.py", line 46, in prepare_pointings
    assert all(m is None for m in cur_obs.mueller_hwp), (
                                  ^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not iterable

Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/mpi_set_n_blocks_test2.py", line 167, in <module>
    binned_maps = sim.make_binned_map(nside)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1726, in make_binned_map
    return make_binned_map(
           ^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 330, in make_binned_map
    nobs_matrix = _build_nobs_matrix(
                  ^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 209, in _build_nobs_matrix
    cur_weights = get_map_making_weights(cur_obs, check=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/common.py", line 118, in get_map_making_weights
    assert np.all(
           ^^^^^^^
AssertionError: Not all the detectors' weights are positive: [0.]

Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/mpi_set_n_blocks_test2.py", line 167, in <module>
    binned_maps = sim.make_binned_map(nside)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1726, in make_binned_map
    return make_binned_map(
           ^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 330, in make_binned_map
    nobs_matrix = _build_nobs_matrix(
                  ^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 224, in _build_nobs_matrix
    pixidx_all, polang_all = _compute_pixel_indices(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/common.py", line 206, in _compute_pixel_indices
    assert len(pol_angle_detectors) == num_of_detectors
           ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object of type 'int' has no len()

Up to this point, patching up was trivial. But it is not trivial for the following (and maybe more) issue:

Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/mpi_set_n_blocks_test2.py", line 167, in <module>
    binned_maps = sim.make_binned_map(nside)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1726, in make_binned_map
    return make_binned_map(
           ^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 330, in make_binned_map
    nobs_matrix = _build_nobs_matrix(
                  ^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/binner.py", line 224, in _build_nobs_matrix
    pixidx_all, polang_all = _compute_pixel_indices(
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/mapmaking/common.py", line 215, in _compute_pixel_indices
    curr_pointings_det, hwp_angle = pointings(idet)
                                    ^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/observations.py", line 859, in get_pointings
    return self.pointing_provider.get_pointings(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/pointings.py", line 118, in get_pointings
    full_quaternions = (self.bore2ecliptic_quats * detector_quat).slerp(
                        ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/scanning.py", line 480, in __mul__
    if (self.start_time is not None) and (other.start_time is not None):
                                          ^^^^^^^^^^^^^^^^
AttributeError: 'numpy.int64' object has no attribute 'start_time'

[ziotom78]

Hi @anand-avinash , thanks for the report! The last error looks really weird, as the type of other should always be RotQuaternion. Unfortunately, it’s hard for me to understand the origin of the problem without having a short script that reproduces the problem. Could you please trim down your script mpi_set_n_blocks_test2.py to the shortest form that still triggers the fourth error and post it here please?

[anand-avinash]

Hi @ziotom78! Here is the minimal script to reproduce the issue:

import numpy as np
import litebird_sim as lbs

comm = lbs.MPI_COMM_WORLD

### Mission params
telescope = "LFT"
channel = "L4-140"
detector_list = [
    "000_005_027_UA_140_B",
    "000_005_027_UA_140_T",
]
mission_time_days = 10
detector_sampling_freq = 1

### Simulation params
imo_version = "vPTEP"
base_path = "mwe"
nside = 16

imo = lbs.Imo()

### Simulation initialization
sim = lbs.Simulation(
    base_path=base_path,
    start_time=0.0,
    duration_s=mission_time_days * 24 * 60 * 60.0,
    random_seed=5455,
    numba_threads=1,
    numba_threading_layer="omp",
    mpi_comm=comm,
    imo=imo,
)


### Instrument definition
sim.set_instrument(
    lbs.InstrumentInfo.from_imo(
        imo,
        f"/releases/{imo_version}/satellite/{telescope}/instrument_info",
    )
)

### Detector list
dets = []
for n_det in detector_list:
    det = lbs.DetectorInfo.from_imo(
        url=f"/releases/{imo_version}/satellite/{telescope}/{channel}/{n_det}/detector_info",
        imo=imo,
    )
    det.sampling_rate_hz = detector_sampling_freq
    dets.append(det)

### Scanning strategy
sim.set_scanning_strategy(
    scanning_strategy=lbs.SpinningScanningStrategy.from_imo(
        imo=imo,
        url=f"/releases/{imo_version}/satellite/scanning_parameters",
    ),
)

### Create observations
sim.create_observations(
    detectors=dets,
    num_of_obs_per_detector=3,
    n_blocks_det=2,
    n_blocks_time=1,
    split_list_over_processes=False,
)

# ## Ideal Half-wave plate
# sim.set_hwp(
#     lbs.IdealHWP(
#         sim.instrument.hwp_rpm * 2 * np.pi / 60,
#     ),
# )


### Compute pointings
sim.prepare_pointings()

### Channel info
ch_info = []
n_ch_info = lbs.FreqChannelInfo.from_imo(
    imo,
    f"/releases/{imo_version}/satellite/{telescope}/{channel}/channel_info",
)
ch_info.append(n_ch_info)


### CMB map
Mbsparams = lbs.MbsParameters(
    make_cmb=True,
    make_fg=False,
    seed_cmb=1,
    gaussian_smooth=True,
    bandpass_int=False,
    nside=nside,
    units="uK_CMB",
    maps_in_ecliptic=False,
    output_string="mbs_cmb_lens",
)

mbs_obj = lbs.Mbs(
    simulation=sim,
    parameters=Mbsparams,
    channel_list=ch_info,
)

if comm.rank == 0:
    input_maps = mbs_obj.run_all()
else:
    input_maps = None

input_maps = lbs.MPI_COMM_WORLD.bcast(input_maps, 0)

### Scanning the sky
lbs.scan_map_in_observations(
    sim.observations, maps=input_maps[0][channel], input_map_in_galactic=True
)

binned_maps = sim.make_binned_map(nside)

I am using two detectors with n_blocks_det=2 and n_blocks_time=1. So if I run the script with 3 MPI processes, one process is left unused. The last error comes up after I apply the patches of #365 to fix the first three.

[anand-avinash]

Also at this point, when I enable ideal hwp, I get the following error:

Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/lbsim_mwe.py", line 80, in <module>
Traceback (most recent call last):
  File "/mnt/Data/Projects/uniroma2/coding/dev_cosmomap2/playground/lbsim_set_n_blocks/lbsim_mwe.py", line 80, in <module>
    sim.prepare_pointings()
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    sim.prepare_pointings()
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 250, in profile_wrapper
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1312, in prepare_pointings
    result = function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/simulations.py", line 1312, in prepare_pointings
    prepare_pointings(
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/pointings_in_obs.py", line 52, in prepare_pointings
    if cur_obs.mueller_hwp[idet] is None:
    prepare_pointings(
  File "/mnt/Data/Projects/uniroma2/coding/dev_litebirdsim/litebird_sim/litebird_sim/pointings_in_obs.py", line 52, in prepare_pointings
    if cur_obs.mueller_hwp[idet] is None:
       ~~~~~~~~~~~~~~~~~~~^^^^^^
TypeError: list indices must be integers or slices, not NoneType
       ~~~~~~~~~~~~~~~~~~~^^^^^^
IndexError: index 1 is out of bounds for axis 0 with size 1

[unfunfunt]

I have also run into one of these when calling precompute_pointings. It works fine with exactly the right number of mpi tasks but when given extra ones it crashes with:

Traceback (most recent call last):
  File "/cluster/home/mathewg/litebird_scripts/noise_only_sims.py", line 105, in <module>
    sim.precompute_pointings(pointings_dtype=np.float64) #Avinash's suggestion - to compute pointings in double prec
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/cluster/home/mathewg/litebird_scripts/litebird_env/lib/python3.11/site-packages/litebird_sim/pointings_in_obs.py", line 65, in precompute_pointings
    precompute_pointings(
  File "/cluster/home/mathewg/litebird_scripts/litebird_env/lib/python3.11/site-packages/litebird_sim/pointings_in_obs.py", line 65, in precompute_pointings
    pointing_matrix, hwp_angle = cur_obs.get_pointings(
                                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/cluster/home/mathewg/litebird_scripts/litebird_env/lib/python3.11/site-packages/litebird_sim/observations.py", line 179, in n_detectors
    list(range(self.n_detectors)),
                      ^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()
    return len(self.det_idx)
                       ^^^^^^^^^^^^^^^^^

Apologies if that traceback is missing something, I had to copy and past each line individually out of a file which had 128 copies of each one

[mreineck]

I can't really help with the problem in question, but concerning the replicated lines in the log: could uniq be useful here?

[ziotom78]

Ok, I am debugging this. The culprit seems to be in the method Observation._set_attributes_from_list_of_dict(), which apparently “spreads” the quantities related with each detector into NumPy arrays. If I add the statement

print(f"#{comm.rank + 1}: {sim.observations[0].mueller_hwp=}")

immediately after the call to sim.create_observations(), I see this:

#1: sim.observations[0].mueller_hwp=[None]
#2: sim.observations[0].mueller_hwp=[None]
#3: sim.observations[0].mueller_hwp=None

Unfortunately Observation._set_attributes_from_list_of_dict() is severely underdocumented, so for the moment I am just trying to rename variables, add comments and type hints, and make the code more straightforward, hoping that this will make evident where the fix needs to be applied.