feasibility & wish: modularized boost assimilation

[magenbluten]

• figure out if it would be feasible provide build2 packages for parts of "modularized" boost.
• prioritize
• dedicate time to wrap all packages

the description of ~boostorg/boost is Super-project for modularized Boost. it could be that a port to build2 is straightforward. "modularization" inside boost happens as git submodules and recursive clone. the "final" bootstrap script just creates symbolic links to re-create the monolithic include structure. from my point of view that is exaclty what build2 would completely automate.

[boris-kolpackov]

One of the notable things missing from this "super-project of submodules" is the dependency information between submodules (packages). Maybe we could use vcpkg's information which I've been told provides split Boost packages. But I've also heard that Boost has circular dependencies.

[magenbluten]

hmm, yeah. i thought so. maybe a google soc project?

so you are suggesting querying the reverse dep tree of vcpkg and then generate the manifests and imports?

[boris-kolpackov]

vcpkg has a CONTROL file for each package (which I was told uses the same format as Debian) that lists package's dependencies. For example: https://github.com/microsoft/vcpkg/blob/master/ports/boost-format/CONTROL

So I think handling this part automatically should be possible (though separating them into interface and implementation dependencies would be tricky).

What I am not certain about is the buildfile side of things. It would be great if we could build every package with generic lib{boost-foo}: {hxx cxx}{**} $libs but something tells me it won't work for all of them.

I think a good place to start would be to create our own boost "super-project" repository, identify a couple of "leaf" libraries (based on the vcpkg information), add them as submodules, and try to package them manually. This should give us some sense of what it will take.

[magenbluten]

sounds reasonable.

another question would be if test cases should be supported as well. or at least a subset to "confirm" everything works as expected.

hmm. looking at boost-core it seems doable. since most of boost is header only anyway couldn't every lib then become an interface lib? for the beginning. or what reasons would speak against it?

[boris-kolpackov]

Yes, experience shows that tests will be a tedious part. I've tried to add support for Boost.Predef some time ago and tests were a pain: boostorg/predef#95 What we normally do, as a starter, is add a smoke test that just links the library and potentially exercises some basic functionality.

Regarding treating all the dependencies as interface dependencies, yes, this could be a way to start. One way we could improve this in the future is by scanning (grep'ing) library's headers for inclusion of dependency's headers. Luckily it should be easy to derive the include prefix from the library name (e.g., libboost-core -> <boost/core/.*>).

[magenbluten]

having one "super-repo" has at least one drawback, you will always fetch all submodules? right? unless the packages are all submitted to cppget. then this is basically a non-issue :)

[boris-kolpackov]

Yes, there is that drawback, but I don't think creating 100+ individual repos is the way to go either. Ideally we would add this support directly to the respective library repositories but I don't think Boost folks will be keen on that, especially without any proof that we will actually see this through.

[magenbluten]

getting this into boost would be nice. but i do not accept that challenge ;) ;)

for boost-context(2) we would also need build2s upcoming assembler or custom-build actions support :)

i have the strong feeling that for 80% of the repos the build2 wrapper repos could be fully generated using sh.

so the current idea would be creating a super repo and git submoduleing the others like this:

super-boost/
|-packages.manifset
|-repositories.manifest
|-boost-core/
  |-upstream/
  |-buildfile
  |-manifest
  |-build/
  |-boost
    |-core/ <- ln -s ../upstream/include/boost/core
    |-buildfile
...

?

[boris-kolpackov]

i have the strong feeling that for 80% of the repos the build2 wrapper repos could be fully generated using sh.

I was actually thinking about this the other day: maybe we should use bdep-new to auto-generate all the build/package support(*) and then apply any necessary customizations (bdep-new even has --pre-hook and --post-hook for that). The reason for that is we will presumably end up with 100+ packages and updating their build/package support manually will be a pain (while things are reasonably stable, we do make tweaks to the projects generated by bdep-new).

If we go this way, then maybe it makes sense to make the submodules top-level directories (but please let's make the resulting packages start the lib prefixes, e.g., libboost-core) and save any necessary scripts/customizations (if needed) into separate directories (say, under etc/). Maybe something along these lines:

super-boost/
|-boost-core/
|-boost-config/
|-etc/
  |-boost-core/
  |-boost-config/

(*) For this to work, we will need support for the "split" (e.g., include/src) project structure in bdep-new, which is on our TODO list.

[magenbluten]

we definitely have to think about maintaining all that stuff. 99% automation 99% of the time would be awesome.

just for completeness sake:

super-boost/
|-packages.manifset
|-repositories.manifest
|-boost-core/ <- git submodule
|-boost-config/ <- git submodule
|-etc/
|-libboost-core/
  |-buildfile
  |-manifest
  |-build/
  |-boost
    |-core/ <- ln -s ../boost-core/include/boost/core
    |-buildfile
|-libboost-config/

right?

the split include/src setup sounds interesting. could at least help wrapping other projects as well.

[boris-kolpackov]

I am not even sure we need separate directories for the packages (libboost-* in your listing). If everything in there is auto-generated and not committed, we could just output it straight into the submodule directory (one problem with this approach is that it will dirty the submodule directory for git).

[magenbluten]

i think i am not a fan of "dumping" stuff into submodules. because if you clone without --recursive there will be a submodule directory "precharged" with stuff. possible making submodules {init|update} not working with the "right&magic" flags.

[boris-kolpackov]

Most boost libraries have now been packaged and are available on cppget.org. For details, see: https://build2.org/blog/build2-boost.xhtml

[boris-kolpackov]

Since boost has been packaged and available on cppget.org for a while now, I am going to close this issue.