Release bin strategy

[netmindz]

I would like to float an idea for a change in strategy to our release bins

Given it looks like the web installer is about to get the ability to detect psram and flash size I think it's an appropriate time to review our release bins

The previous approach seems to have been a mix between throwing in as much as possible, but then also keeping inside the limit imposed by our partition table and the rather generous filesystem partition (that goes massively underused in many cases I suspect)

However when it comes to usermods, other than AR they are not included, even if they are potentially quite broad if their appeal and minimal in flash usage

This means users might miss out on functionality and requires the use of custom builds (and the issues when it comes to upgrade)

What I was thinking is that for users with hardware with more than 4MB and/or PSRAM we include more in the build, otherwise those extra resources are just wasted

If for some reason you want a minimalist build, despite your hardware, these power users can still do a smaller build (or just ota to the 4MB image)

[DedeHai]

The previous approach seems to have been a mix between throwing in as much as possible, but then also keeping inside the limit imposed by our partition table and the rather generous filesystem partition (that goes massively underused in many cases I suspect)

the file system is adequate IMHO: if you run a 2D setup a few animated gifs will eat it up quickly - or have 100+ presets with many segments (not actually that uncommon as users use presets and segments in elaborate schemes). Given the possibilities with the new pixelforge tools - especially paint - I expect (a lot) more users to run out of FS space with 16.0.

adding more default usermods is certainly on the table - Quinled already has the multi-relay UM added. Internal temperature would be another one or the autosave. even in the 4M builds.
as for 8M builds I think they already have additional things compiled in - or was that in MM? But sure, making use of the available flash makes sense, I would not pack in too many if they are not useful for many people.

[softhack007]

as for 8M builds I think they already have additional things compiled in - or was that in MM?

If you think about additional usermods - yes that was in MM. Another way to make use of the extra Flash on 8MB/16MB boards is to build the firmware with "-O2" (optimize for speed). This increases size by a few 100kb; it also creates a nice speedup - up to 30% in some cases.

WLED/platformio_override.sample.ini

Lines 199 to 209 in a5f28d0

	[Speed_Flags]
	build_unflags = -Os ;; to disable standard optimization for small size
	build_flags =
	-O2 ;; optimize for speed
	-free -fipa-pta ;; very useful, too
	;;-fsingle-precision-constant ;; makes all floating point literals "float" (default is "double")
	;;-funsafe-math-optimizations ;; less dangerous than -ffast-math; still allows the compiler to exploit FMA and reciprocals (up to 10% faster on -S3)
	# Important: we need to explicitly switch off some "-O2" optimizations
	-fno-jump-tables -fno-tree-switch-conversion ;; needed - firmware may crash otherwise
	-freorder-blocks -Wwrite-strings -fstrict-volatile-bitfields ;; needed - recommended by espressif

Edit: in MM, -O2 is the default for boards with >= 8MB flash since some time already, so I can confirm "it does work this way" :-)

[softhack007]

The 8MB partitions file could use some optimization - currently its 1.95M program, 4M LittleFs. The 16MB partitions file has 3.2MB for program, which is the max possible without risking slow-down due to larger address space.

I think we can safely modify the 8MB partitioning to have some more "headroom" for program,
like 2.2MB program => 3.5MB LittleFS.

This will be "invisible" to OTA updaters, unless we actually hit the 1.9MB program size limit of the current partitions file.

[willmmiles]

I think it's a good idea to start building up standard module sets for inclusion in each build. I have no objections to expanding the set of modules in standard builds, particularly for larger flash systems. In fact, I'd like to suggest that we split the "usermods" folder into "core_mods" and "usermods" -- no difference in functionality, but "core_mods" are the ones we actively maintain and include in standard builds; while "usermods" are those contributed by others (or put another way: the ones we aren't testing...).

In the long term I'd also like to aim towards moving more of our optional integrations in to modules -- essentially, anywhere we've got #ifndef WLED_DISABLE_X is something that could be a module that could be switched in and out instead...

I would very much like to import the Usermod::name/enable and per-usermod settings page logic from MM as part of this process.

Re partitions: @DedeHai is correct that many new features can use up the "data" partition really fast. Graphics assets are big! I don't have a strong opinion on how to strike a balance between code partitions and data partitions -- whatever you guys think is best. (I agree that for 16MB+ devices, reserving the maximum possible size for code makes sense.) I'm not too too worried about changing our minds there -- a good module system will let us build "minimal" and "expanded" flavours to support old and new partition sizes, and if it comes to it, we can document an upgrade path for repartitioning. (It's technically possible to repartition OTA; we'd want a "total data partition backup" feature to close the loop.)

---

For my obligatory pie-in-the-sky plan, what I really want is runtime dynamic linking, so usermods can be placed on the data partition. This is technically possible.

[softhack007]

Yes the "standard usermods set" idea makes sense. There is one problem that we would need to address: "pin stealing" by usermods (includes premature initialisation of I2C and SPI busses, too).

Even an inactive usermod allocates PINs (pinmanager) during usermod::setup(). If we add some more usermods to the standard build, this will become a problem. User who "just used the standard build" will be surprised when 4 pins are allocated to the rotary usermods, but that usermod is not enabled. This is even true for audioreactive, because "allocate all usermod PINs in setup()" was kind of a dogma for a long time.

I think the problem can be addressed, but it means we need to weed through our favourite "curated default" usermods collection, and make sure that none of them allocates pins when not enabled.

[willmmiles]

I think the problem can be addressed, but it means we need to weed through our favourite "curated default" usermods collection, and make sure that none of them allocates pins when not enabled.

Right - historically usermods could reasonably assume that if they were included at all, that meant they should bring along a full default configuration and assume that they're expected to be used. Core mods can't take that assumption.

One tricky bit is that the meaning of "enabled" is pulling double duty here. There's "enabled" like "whether or not to run any code from this module at all", vs "enabled" like "this module has a complete configuration and is ready to actually do work". Thinking it through, I'm less confident that a standard "enable" flag implemented at the top level makes sense -- "unconfigured" states might be a cleaner design. Eg. the rotary usermod could come up with all the pins set to "no pin", and it wakes up and starts working once the pins are assigned. Since "no pin" is a valid selector, it's going to have to handle that internally anyways.

[softhack007]

"unconfigured" states might be a cleaner design. Eg. the rotary usermod could come up with all the pins set to "no pin",

makes sense - in WLED-MM some usermods are delivered "unconfigured" (=cannot enable before valid PINs are set by users) exactly to work around the "pin stealing" problem in some cases. Some users were actually not so happy about it, because they loved that a UM comes with default PINs - apparently this reduces the headache of "OMG which pins should I chose, there are so many" 😉

[softhack007]

PS: this situation also led to another divergence between MM and upstream WLED. In MM, the I2C device is not doing its Wire.Begin() at startup, while in upstream Wire.Begin() will run when I2C pins are valid, not matter if the unit is needed or not.

In MM we've split the procedure - I2C pins are configured globally, but any "module" that wants to use I2C has to call pinManager::joinWire() to really kick-off the I2C bus. I think its the better solution:

• I2C unit is not started unless its needed
• delayed start means that I2C can often be re-configured by the user without reboot (there is no Wire.End() in older arduino)
• conflicting I2C startup by several usermods can be avoided

Edit: I ran into another dogma at that time - "pin manager is just for book-keeping". So this idea stayed in MM only.

[willmmiles]

We're well in to the resource management discussion now!

I concur on all counts (except maybe the naming scheme!). Modules using shared resources like Wire will need to requisition them via the resource manager, and the actual initialization of those resources must be handled by the resource management framework.

It does feel a little dirty to pack that under pinManager though. Time to start bikeshedding on a new namespace.

[DedeHai]

sounds like you are talking about resource manager again ;) edit: @willmmiles beat me to it
my WIP on new bus driver can allocate:

• GDMA channels
• SPI
• UART
• I2S
  maybe more in the future. need to define a priority or is it first come first serve?

[willmmiles]

maybe more in the future.

RMT channels for sure, we aren't currently leveraging them for IR in, but we could be. LEDCs? Probably we just want to go down the ESP32 peripheral list.

need to define a priority or is it first come first serve?

Excellent question. For a first cut, first-come-first-served and leaving conflict handling to the module is probably the place to start (ie. complain to the user to fix their config). I don't think we'll get much value from trying to do some kind of smarter allocation -- there aren't many cases where there are meaningful direct alternatives. Digital bus drivers are maybe the only one, and even then the alternatives aren't always truly substitutable (RMTs can have unique timing parameters for each channel; parallel I2S/SPI have to agree on a base clock; and bit-banging is always possible but slow).