Support force_version of gems to override dependency conflicts

[lukaso]

What was the end-user or developer problem that led to this PR?

As described in the RFC: rubygems/rfcs#13, gems are often abandoned with the authors failing to update dependencies. This allows a user to force a dependency to a specific version so that they are not forced to fork and maintain a gem, which can be a substantial overhead.

To quote the RFC:

"This feature has been requested many times since 2011.

Project authors want the ability to specify newer versions of gems in their Gemfile even if the version is being constrained by another gem.

There several causes for this issue:

The version dependency specified in the gemspec is unnecessarily strict: s.add_dependency 'json', '= 1.7.7'
The gem is slow to update or is no-longer maintained to the point that version constraints do not keep up with the version updates of their dependencies: s.add_dependency 'json', '>= 1.7.7', '<= 2'
The common approach to this problem is to fork the problematic gem, relax the dependencies specified in the gemspec, and reference the fork in the Gemfile. This is a very heavyweight and time intensive solution to a problem that could be easily solved by a well-designed solution to override dependencies from within the Gemfile itself.

Obviously, overriding version constraints of gems would be AT YOUR OWN RISK, however, a properly defined DSL and output messages should mitigate and address these issues.

Most of the time, this kind of override should NOT be a permanent solution, and as such, SHOULD be accompanied with some type of message that is additionally displayed to indicate why the override has occurred."

What is your fix for the problem, implemented in this PR?

This code is a port of a proof of concept done by @segiddins in rubygems/bundler#5670, which has been adjusted to work on latest master.

Here is sample output:

Here is the Gemfile:

source 'https://rubygems.org'

gem "sfmc-fuelsdk-ruby", "1.3.0"
gem "savon", "2.12.1", force_version: true

This means I no longer have to maintain a forked version of sfmc-fuelsdk-ruby.

The run:

$ bundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Using public_suffix 4.0.6
Using builder 3.2.4
Using mini_portile2 2.4.0
Using bundler 2.3.0.dev
Using rack 2.2.3
Using socksify 1.7.1
Using json 1.8.6
Using jwt 1.5.6
Using nori 2.6.0
Using addressable 2.7.0
Using gyoku 1.3.1
Using nokogiri 1.10.10
Using httpi 2.4.5
Using akami 1.3.1
Using wasabi 3.6.1
Using savon 2.12.1 [version forced]
Using sfmc-fuelsdk-ruby 1.3.0
Bundle complete! 2 Gemfile dependencies, 17 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

Here is the Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.7.0)
      public_suffix (>= 2.0.2, < 5.0)
    akami (1.3.1)
      gyoku (>= 0.4.0)
      nokogiri
    builder (3.2.4)
    gyoku (1.3.1)
      builder (>= 2.1.2)
    httpi (2.4.5)
      rack
      socksify
    json (1.8.6)
    jwt (1.5.6)
    mini_portile2 (2.4.0)
    nokogiri (1.10.10)
      mini_portile2 (~> 2.4.0)
    nori (2.6.0)
    public_suffix (4.0.6)
    rack (2.2.3)
    savon (2.12.1)
      akami (~> 1.2)
      builder (>= 2.1.2)
      gyoku (~> 1.2)
      httpi (~> 2.3)
      nokogiri (>= 1.8.1)
      nori (~> 2.4)
      wasabi (~> 3.4)
    sfmc-fuelsdk-ruby (1.3.0)
      json (~> 1.8, >= 1.8.1)
      jwt (~> 1.0, >= 1.0.0)
      savon (= 2.2.0)
    socksify (1.7.1)
    wasabi (3.6.1)
      addressable
      httpi (~> 2.0)
      nokogiri (>= 1.4.2)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  savon (= 2.12.1)
  sfmc-fuelsdk-ruby (= 1.3.0)

BUNDLED WITH
   2.3.0.dev

Make sure the following tasks are checked

• [x ] Describe the problem / feature
• [x ] Write tests for features and bug fixes
• [x ] Write code to solve the problem
• [x ] Make sure you follow the current code style and write meaningful commit messages without tags

[welcome]

Thanks for opening a pull request and helping make RubyGems and Bundler better! Someone from the RubyGems team will take a look at your pull request shortly and leave any feedback. Please make sure that your pull request has tests for any changes or added functionality.

We use GitHub Actions to test and make sure your change works functionally and uses acceptable conventions, you can review the current progress of GitHub Actions in the PR status window below.

If you have any questions or concerns that you wish to ask, feel free to leave a comment in this PR or join our #rubygems or #bundler channel on Slack.

For more information about contributing to the RubyGems project feel free to review our CONTRIBUTING guide

[lukaso]

I will look to add a message in the bundle install command that highlights that the gem has been force overridden. Done in a4e9c66

[lukaso]

1. All changes to vendored Molinillo are removed (I've not had to merge anything to upstream).
2. Unrelated changes have been moved to separate PRs
3. Complexity is vastly reduced (at the cost of an instance variable @forced_requirements in Resolver).

I feel like this is a much improved version of the PR (thanks for the invaluable feedback @simi).

[deivid-rodriguez]

Maybe passing the exact version as the :force_version value would be a slightly better API? It would make it more clear that it's just a temporary hack and it wouldn't require you to modify your original requirement. Like:

gem "minitest", "~> 5.14", force_version: "5.14.2"

[deivid-rodriguez]

I think, if I understand this new approach, you'd need to handle required_ruby_version and required_rubygems_version specially. Probably by changing them to be unconstrained? Otherwise even if all the requirements of the forced version are prioritized, they would still not be met in the case of required_ruby_version and required_rubygems_version. Right?

This makes me wonder whether @simi's suggestion of providing a way of "monkeypatching requirements" of a gem would be better:

 gem "minitest", "~> 5.14" do |s|
  s.required_ruby_version = ">= 2.5"
end

It would also make it possible to use the feature multiple times in a Gemfile which I believe it could currently cause issues with this approach if they have conflicting requirements?

[lukaso]

I don't like the monkey patch syntax because it requires bringing all the parents into the equation in the force_version case and I don't think adds enough extra information to make it worth it. Also, I'm a bit wracking my brain in how to implement.

Here is an implementation of two additional flags to gem, :override_ruby_version and `:override_rubygems_version: lukaso#7 which I can fold into this PR.

The usage example is:

gem 'rutabaga', git: 'git@github.com:simplybusiness/rutabaga.git', branch: 'ruby-24-or-less', override_ruby_version: true, override_rubygems_version: true

There is no easy or clear place to patch into the loading of gemspecs that I could find, so I had to place to overrides in a number of strategic places.

[deivid-rodriguez]

I'm not sure about the implementation either, I can see how it could get super tricky, but if we decided to do this, I was expecting to ship a single feature to accomplish this. Either force_version: <exact_version> as in: "ignore what you have to ignore during resolution to make sure you give me this specific version", or a more finer grained approach of being able to monkeypatch the gemspec constraints used for resolution.

I'll think about it more.

[lukaso]

Maybe passing the exact version as the :force_version value would be a slightly better API? It would make it more clear that it's just a temporary hack and it wouldn't require you to modify your original requirement. Like:

gem "minitest", "~> 5.14", force_version: "5.14.2"

I don't think this helps because in general as there wouldn't be an original requirement in your Gemfile since the dependency you are trying to override would be in a gemspec within one of the dependencies.

Example:

Gemfile:

gem "sfmc-fuelsdk-ruby", "~> 1.3.0"
gem "jwt", "~> 2.2", force_version: true # added here to override gemspec in sfmc-fuelsdk-ruby

sfmc-fuelsdk-ruby.gemspec:

  spec.add_dependency "jwt", "~>1.0",">= 1.0.0"

Or do you have another situation in mind?

[deivid-rodriguez]

@lukaso I believe my proposal is also better when forcing indirect dependencies? It still makes it more clear that it's a temporary hack and forces you to update the Gemfile every time the dependency you're overriding releases a new version if you want to get it, which seems good since you might no longer need to override it.

In any case, your current implementation doesn't work with the example you gave, right? Doesn't it raise a "Cannot use force_version for inexact version requirement" error?

[br3nt]

I’ve got reservations about requiring an exact version.

Imagine if a gem author used force_version And an exact gem version to get around a constraint imposed by one of their other dependencies. It may take a lot of time to remove the problematic dependency, so a quick solution is temporarily required. Once this gem is published, anyone who updates to the latest version now has that dependency locked to the exact version the gem specified. Yikes!

A better approach would be to use the existing constraints system. Allow gem authors to specify a looser constraint as an override. This would allow the dependency to continue to develop and the end users of the gem will be able to get the newer versions of the dependency independently of updates to this gem. Perfect!

Just imagine if a security patch couldn’t be accessed because a dependency forced an exact version. 😥

[hsbt]

I introduced same feature at #9517 and #9530 for next stable version.