pkg_tar supports pkg_filegroup framework

[nacl]

[aiuto]

I'm ready to begin this. Overall plan:

• pkg_{tar|zip|deb|rpm|} will process srcs (not deps) for any of the PkgFile types. We use the PkgFile providors in preference to DefaultInfo
• a common method will build srcs into a single map of output_path => [src_path, mode, owner, group, is_dir, is_link]
  • duplicate output_path is a fail
• we sort the map by output_path
• write map to an output file
• build integration tests for pkg_* rule composition against the generated maps.
• rpm & deb packagers might use this structure to build a .spec or control file.
• modify build_zip, build_tar, ... to take that particular format as input

Things to note about srcs that have PkgFile providers. Our goal is to deprecate the special purpose attributes of pkg_tar that handled modes and owners differently than other rules.

• package_dir will be applied
• modes, owner and group names will be used as default values when the provider does not contain an attribute value
• strip_prefix will not be applied.
• the maps of paths to owners and modes will not be applied

Special case for tar deps. There is an existing behavior for pkg_tar where deps which are tar files are expanded into the output. For the time being, those will be spliced into the output tar after all srcs are processed. In the next implementation phase, I will add a pkg_expand() rule that takes a tar/zip as srcs and allows re-prefixing and attribute assignment. I am deferring design for that until later. For now, we can focus on the requirements

• it must be possible to mix the files from expanded tar with the other srcs so that they are all lexically sorted
• that may cause a second read of the tar file, but must not cause writing the entire content to disk
• there must (eventually) be a fast path option that does not sort outputs together, and causes no more than a single read of each expanded tar file.
  A potential time saver might be for pkg_tar/pkg_zip to produce a secondary output that is an index of all the emitted files with modes and owners. A consuming rule could use that index rather than re-reading the content.

[aiuto]

I'm building out a test case here: https://github.com/aiuto/rules_pkg/tree/rich/examples/rich_structure
I want to have a comprehensive set of things people actually do there, even if it seems overly complex for a tiny project.

[nacl]

This looks similar to something I was thinking about -- especially WRT the file interfaces.

Directories can be thought of in the same way as pkg_expand -- namely that they are "pre-expanded" archives. A complexity that comes to mind in all this is that the manifests need to be combined at some point in package builds.

[nacl]

Made #355 for pkg_expand.

[aiuto]

Speaking of the word "manifests". Can we agree that we call the file we pass to build_*.py that lists the source paths and the destinations & modes they get as the "manifest". I would rather nail that down now, than debate it in a PR review.

[nacl]

Speaking of the word "manifests". Can we agree that we call the file we pass to build_*.py that lists the source paths and the destinations & modes they get as the "manifest". I would rather nail that down now, than debate it in a PR review.

I'm fine with that, but would "manifest" also be fine to use in something like #238? Perhaps they could be distinguished with an adjective?

[aiuto]

I think it might work for both uses.

…

On Mon, May 17, 2021 at 5:33 PM Andrew Psaltis ***@***.***> wrote: Speaking of the word "manifests". Can we agree that we call the file we pass to build_*.py that lists the source paths and the destinations & modes they get as the "manifest". I would rather nail that down now, than debate it in a PR review. I'm fine with that, but would "manifest" also be fine to use in something like #238 <#238>? Perhaps they could be distinguished with an adjective? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#310 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHHB7REDYSGWUVMLGPLTOGDQZANCNFSM4YPVPWZA> .

[aiuto]

This has been working for a while now #367

[nacl]

BTW, we do not have any unit tests for the interaction between the pkg_filegroup framework and pkg_tar.