This is a report on the maximum num_envs each RoboLab benchmark task can run with on a single L40 GPU (48GB VRAM) headlessly. Depending on how much load of the GPU is used for other applications, you may consult this guide as an upper bound for --num-envs on RTX GPUs with 48GB memory.
BagelsOnPlateTaskBananaInBowlTaskBananaOnPlateTaskBananaThenRubiksCubeTaskBlockStackingOrderAgnosticTaskBlockStackingSpecifiedOrderTaskBowlInBinTaskBowlStackingLeftOnRightTaskBowlStackingRightOnLeftTaskButterAboveRaisinTaskClampInRightBinTaskFoodPacking1BoxesTaskFoodPacking1CansTaskLargerObjectRaisinBoxInBinTaskMustardAboveRaisinTaskMustardInLeftBinTaskMustardInRightBinTaskNonHammerToolsInRightBinTaskRedDishesInBinTaskRedItemsInBinTaskRubiksCubeAndBananaTaskRubiksCubeBehindBowlTaskRubiksCubeInFrontOfBowlTaskRubiksCubeLeftOfBowlTaskRubiksCubeOrBananaTaskRubiksCubeRightOfBowlTaskRubiksCubeThenBananaTaskSauceBottlesCrateTaskSmallerObjectButterInBinTaskStack3RubiksCubeTaskToolOrganizationTask
AnimalsInBinTaskBlocksInBinTaskCleanUpToysTaskCubesAndBlocksInBinTaskGreenSpoonsInPotTaskHammersInLeftBinTaskPickUpBluePitcherTaskPickUpGreenObjectTaskPutBowlOnShelfTopTaskPutMugsOnShelfTaskPutTwoMugsOnShelfTaskRecycleCartonTaskRecycleCartonsVerticalCrateTaskRubiksCubeTaskRubiksCubesInBinTaskStackYellowOnRedTaskTakeMugsOffOfShelfTaskToolOrganizationBothTaskUnstackRubiksCubeTaskYellowAndWhiteObjectsInBinTask
BananasInBinOneMoreTaskBananasInBinThreeTotalTaskBananasInCrateTaskBananasOutOfBinTaskBigPumpkinInBinTaskBlackItemsInBinTaskClearOrganicObjectsTaskClutterPlasticTaskClutterPumpkinTaskCookingClearPlateTaskCookingPickPastaToolTaskDishesInBinTaskElectronicsInBinTaskFruitsMovingOrangeOrLimeTaskFruitsMovingTaskFruitsOnionTaskFruitsOnionToPlateTaskGrabABagelTaskGrabAFruitTaskKeyboardOutOfBinTaskMarkerInMugTaskMouseOnKeyboardTaskMoveBananaToBagelPlateTaskPhoneOrRemoteInBinTaskPickDrillTaskPickGlassesTaskPickOrangeObjectTaskReorientAllMugsTaskReorientRedMugTaskReorientWhiteMugsTaskSmallPumpkinInBinTaskSmartphoneInBinTaskSpoonInMugTaskSpoonsInPotTaskStackWhiteMugsTaskTakeMeasuringSpoonOutTaskTakeSpatulaOffShelfTaskToyInBinTaskUtensilsInMugTaskWhiteMugInCenterOfTableTaskWhiteMugsInBinTaskYogurtInBowlTask
AppleAndYogurtInBowlTaskBBQSauceInBinTaskCannedFoodInBinTaskCoffeePotInBinTaskCondimentsInBinTaskFoodPacking2BoxesTaskFoodPacking2CansTaskFoodPacking3BoxesTaskFoodPacking3CansTaskFoodPackingByColorTaskFruitsGreenLimesOnPlateTaskFruitsOnPlate3TaskFruitsOrangesOnPlateTaskJugsOnShelfTaskOneBottleInSquarePailTaskOneBottleOnShelfTaskPinkSpoonInPotTaskPlasticBottlesInSquarePailTaskRecycleCartonsOnBoxTaskReorientJugTaskThrowAwayAppleTaskThrowAwaySnacksTaskToolsPickingAllHammersTaskToolsPickingDrillTaskToolsPickingHammerTaskWoodSpatulaToBowlTask
FruitsOnPlateTask
OOM signatures recognized in the job logs:
out of memory,OOMKilled,cudaErrorOutOfMemory,CUDA error: out of memory(standard)Warp CUDA error: Failed to get driver entry point.*cuDeviceGetUuid(Warp init OOM, opaque)ERROR_OUT_OF_DEVICE_MEMORY,vkAllocateMemory failed,Out of GPU memory allocating resource,Failed to execute RenderGraph,HydraEngine::render failed(Vulkan render-graph silent-hang)omni.physx.tensors.*CUDA error: an illegal memory access(PhysX tensor corruption at high envs)PhysX error: GPU.*Kernel fail to launch(PhysX narrow-phase kernel launch failure)
- Setting
--num-envson a L40 — pick the per-task ceiling. - Sizing a batch of tasks under one workflow — use
min(ceiling)across the batch. - Sizing the same task on a different GPU — these numbers are L40-specific.