fix: added seo sitemap for product pages#3151
Merged
zamanafzal merged 9 commits intomainfrom Apr 6, 2026
Merged
Conversation
OpenAPI Changes1 changes: 0 error, 0 warning, 1 info Unexpected changes? Ensure your branch is up-to-date with |
Contributor
ChristopherChudzicki
left a comment
ChristopherChudzicki
left a comment
There was a problem hiding this comment.
@zamanafzal This generally looks good to me. Left one very minor comment. Can do full review when out of draft.
| description="Get a paginated list of learning resources with summary fields", | ||
| description=( | ||
| "Get a paginated list of learning resources with summary fields " | ||
| "(id, last_modified, url). Includes ``url`` for sitemap and similar " |
Contributor
There was a problem hiding this comment.
I'm not sure there's much value in listing the fields here. The openAPI spec shows the response fields already.
Contributor
|
@zamanafzal re the failing test... Seems flaky, but i suspect it's also removed (we changed that feature, i think) on main, if you want to rebase. |
zamanafzal
force-pushed
the
zafzal/10755-seo-learn-product-pages
branch
2 times, most recently
from
38eeca2 to
c9a479b
Compare
for more information, see https://pre-commit.ci
zamanafzal
force-pushed
the
zafzal/10755-seo-learn-product-pages
branch
from
c9a479b to
b249d41
Compare
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Implements SEO improvements for Learn product pages by exposing product-page URLs via the learning resources summary API and generating a dedicated products sitemap for MITxOnline courses/programs.
Changes:
- Added
urlto the LearningResource summary API (serializer, view, OpenAPI spec, and generated client types). - Updated MITxOnline ETL to store Learn product-page URLs (courses, programs, and “program-as-course” URLs).
- Added
/sitemaps/products/sitemap/<id>.xmlgeneration + tests, and included it in the sitemap index; removed explicitnoindex, nofollowfrom product page metadata.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| openapi/specs/v1.yaml | Documents new url field in LearningResourceSummary schema. |
| learning_resources/views.py | Adds url to summary() queryset values output. |
| learning_resources/views_test.py | Updates summary endpoint expectations and adds coverage for stored URLs. |
| learning_resources/serializers.py | Includes url in LearningResourceSummarySerializer fields. |
| learning_resources/etl/mitxonline.py | Writes Learn product-page URLs into LearningResource.url for MITxOnline resources. |
| learning_resources/etl/mitxonline_test.py | Updates ETL transform expectations to reflect new URL behavior. |
| frontends/main/src/app/sitemaps/sitemap-index.xml/route.ts | Adds products sitemap entries to the sitemap index. |
| frontends/main/src/app/sitemaps/products/sitemap.ts | New dynamic products sitemap backed by summary API filtering for MITxOnline courses/programs. |
| frontends/main/src/app/sitemaps/products/sitemap.test.ts | Tests sitemap index params and page generation/filtering behavior. |
| frontends/main/src/app/(products)/programs/[readable_id]/page.tsx | Removes explicit robots: noindex, nofollow from program pages. |
| frontends/main/src/app/(products)/courses/p/[readable_id]/page.tsx | Removes explicit robots: noindex, nofollow from program-as-course pages. |
| frontends/main/src/app/(products)/courses/[readable_id]/page.tsx | Removes explicit robots: noindex, nofollow from course pages. |
| frontends/api/src/test-utils/factories/learningResources.ts | Adds url to learning resource summary factory output. |
| frontends/api/src/generated/v1/api.ts | Regenerates client types to include LearningResourceSummary.url. |
mbertrand
approved these changes
Apr 6, 2026
Member
mbertrand
left a comment
mbertrand
left a comment
There was a problem hiding this comment.
The ETL changes look good to me, 👍
ChristopherChudzicki
approved these changes
Apr 6, 2026
Contributor
ChristopherChudzicki
left a comment
ChristopherChudzicki
left a comment
There was a problem hiding this comment.
looks good to me. 👍
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
https://github.com/mitodl/hq/issues/10755
Description (What does it do?)
This PR implements SEO improvements for Learn product pages:
noindex, nofollowfrom course and program product pages to allow search engine indexingproducts/sitemap.tsthat fetches MITxOnline courses and programs from the/api/v1/learning_resources/summary/endpointurlfield to Learn product page URLs (e.g.,/courses/{readable_id},/programs/{readable_id},/courses/p/{readable_id}for programs withdisplay_mode="course")urlin the summary API response for sitemap generationScreenshots (if appropriate):
How can this be tested?
Prerequisites
A running MITxOnline instance with:
display_mode="course"(program-as-course)Steps
Run the MITxOnline ETL to populate URLs:
Verify the summary API includes URLs:
Visit: http://api.open.odl.local:8065/api/v1/learning_resources/summary/?platform=mitxonline&resource_type=course&resource_type=program&limit=1000
Confirm that:
urlfield{APP_BASE_URL}/courses/{readable_id}{APP_BASE_URL}/programs/{readable_id}{APP_BASE_URL}/courses/p/{readable_id}Verify the sitemap index includes products sitemap:
Visit: http://open.odl.local:8062/sitemaps/sitemap-index.xml
Confirm that it includes entries for
/sitemaps/products/sitemap/{id}.xmlVerify the products sitemap contains correct URLs:
Visit: http://open.odl.local:8062/sitemaps/products/sitemap/0.xml
Confirm that:
Verify noindex removal from product pages:
Visit any product page and inspect the HTML
<head>:/courses/{readable_id}/programs/{readable_id}/courses/p/{readable_id}Confirm that
<meta name="robots" content="noindex, nofollow">is NOT present.Additional Context