CVE-2021-41125

HTTP authentication credential leak to target websites in scrapy

Published Oct 6, 2021 CVSS 3.1 · 5.7 Medium EPSS 1.3% KEV · Not listed
Description
Scrapy is a high-level web crawling and scraping framework for Python. If you use `HttpAuthMiddleware` (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the `w3lib.http.basic_auth_header` function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using `HttpAuthMiddleware`.

Remediation

No solution or workaround provided in the CVE record

Tracking

Sign in to view the affected projects.

Advisories

Source ID Title
Debian DLA Debian DLA DLA-2950-1 python-scrapy security update
EUVD EUVD EUVD-2021-0239 Scrapy is a high-level web crawling and scraping framework for Python. If you use `HttpAuthMiddleware` (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the `w3lib.http.basic_auth_header` function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using `HttpAuthMiddleware`.
Github GHSA Github GHSA GHSA-jwqp-28gf-p498 Scrapy HTTP authentication credentials potentially leaked to target websites
Ubuntu USN Ubuntu USN USN-7476-1 Scrapy vulnerabilities
History
Type Values Removed Values Added
Metrics epss

{'score': 0.00493}

epss

{'score': 0.00507}

Weaknesses

Sources

Status
PUBLISHED
Assigner
GitHub_M
Published
Updated
Aug 4, 2024, 2:59 AM
Reserved
Sep 15, 2021, 12:00 AM
Status
Modified
Published
Oct 6, 2021, 6:15 PM
Modified
Jun 17, 2026, 4:07 AM