fix: harden scrape inputs against SSRF and unsafe URL targets

[chidii]

Closes #6

Reusable URL safety policy applied at the service boundary. 16 tests covering allowed URLs and malicious URL table. Blocks private ranges, cloud metadata, credential URLs, non-HTTP protocols. DNS validation, redirect limits, timeout, response size cap. Safe error messages without internal details.

[vercel]

@chidii is attempting to deploy a commit to the emrekayat's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@chidii Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[emrekayat]

Thanks for the contribution. I cannot merge this PR as-is.

Issue #6 is already closed on main, and this PR only adds a standalone urlSafety helper plus tests. It does not wire the safety policy into the scrape service boundary, so current/future scrape providers would not actually inherit the protection required by the issue.

The missing acceptance point is the important one here: apply the policy at the service boundary before any scrape provider can perform network access. Since the issue is already resolved on main, this PR is now redundant unless it is rebased and changed into a focused improvement on top of the existing implementation.

[emrekayat]

Thanks for the contribution. I reviewed this against #6 and cannot merge it as-is.

The main blocker is that the new urlSafety.ts helper is not wired into the scrape execution path or provider boundary, so current/future scrape providers do not actually inherit the policy at runtime. The acceptance criteria require the policy to be applied before network access, including DNS validation, redirect revalidation, timeout/size/content-type enforcement, and safe client-facing errors.

Also, #6 has already been completed on main, which now has the integrated scrape URL safety policy and docs. If you want to continue from here, please rebase on main and make this a focused improvement to the existing scrape-url-safety implementation rather than a parallel unused helper.