Skip to content

feat: improvements in package caching#1274

Closed
a-gave wants to merge 2 commits intoopenwrt:mainfrom
a-gave:feat/improve_package_caching
Closed

feat: improvements in package caching#1274
a-gave wants to merge 2 commits intoopenwrt:mainfrom
a-gave:feat/improve_package_caching

Conversation

@a-gave
Copy link
Copy Markdown
Contributor

@a-gave a-gave commented Feb 25, 2025

This adds some squid directives to consider fresh longer kernel modules of already compiled releases, to prevent revalidation against the origin server which consumes time.

An automatic mechanism to refresh the below squid directive should be included (e.g., 24h after the first appearance on https://downloads.openwrt.org/.versions.json)

refresh_pattern ^http://downloads.openwrt.org/releases/.*(23.05.[6-99]|24.10.[1-99]|25.*|SNAPSHOT)/targets/.* 0 100% 0 refresh-ims

This adds also a fallback to no_proxy (https) in case STDERR contains “package index are corrupt” to rule out cache problems and to allow packages to be considered fresh longer and thus not revalidated against the origin server

@a-gave a-gave force-pushed the feat/improve_package_caching branch from 505be82 to 84ffb51 Compare February 25, 2025 16:31
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 25, 2025

Codecov Report

❌ Patch coverage is 52.38095% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.29%. Comparing base (5e65dec) to head (dc53a56).
⚠️ Report is 395 commits behind head on main.

Files with missing lines Patch % Lines
asu/build.py 25.00% 6 Missing ⚠️
asu/util.py 69.23% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1274      +/-   ##
==========================================
+ Coverage   80.75%   90.29%   +9.53%     
==========================================
  Files          15       13       -2     
  Lines         977     1205     +228     
==========================================
+ Hits          789     1088     +299     
+ Misses        188      117      -71     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@a-gave a-gave force-pushed the feat/improve_package_caching branch from 1ab8034 to 2a44ded Compare February 26, 2025 09:55
@a-gave a-gave changed the title draft: feat: improvements in package caching feat: improvements in package caching Feb 26, 2025
@a-gave a-gave marked this pull request as draft February 26, 2025 10:43
@a-gave a-gave force-pushed the feat/improve_package_caching branch from 2a44ded to 3670a3d Compare February 26, 2025 23:26
@a-gave a-gave force-pushed the feat/improve_package_caching branch from 3670a3d to dc53a56 Compare February 26, 2025 23:28
@aparcar
Copy link
Copy Markdown
Member

aparcar commented Feb 27, 2025

I recon you're really limited on bandwidth? Or what is the motivation to avoid the extra checks?

@a-gave
Copy link
Copy Markdown
Contributor Author

a-gave commented Feb 27, 2025

Yes, this is purely to save bandwidth and to give a crutch to somewhat dated processors.

A build without selecting additional packages, just changing defaults

With a recent processor:
1- 43 seconds (refreshing everything)
2- 39 seconds (refreshing only packages, not kmods)
3- 22 seconds (without any refresh and with all TCP_MEM_HIT from ram)

The same with a 10years old processor:
1- 48s
2- 42s
3- 31s

Also the regex below should cause less fallback to https, if building with feeds from a mirror
(that resynchronizes i.e. every 12 or 24 hours)

refresh_pattern ^http://.*openwrt.*/.* 30 100% 30 override-expire

Close it if not necessary, maybe it could be useful as a reference for someone else, but maybe not.

@a-gave
Copy link
Copy Markdown
Contributor Author

a-gave commented Apr 17, 2026

Obsoleted by 331c976 (squid replaced by nginx-cache)

@a-gave a-gave closed this Apr 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants