Skip to content

Commit e1c8879

Browse files
proxymeshproxymesh
authored
Merge pull request #9 from proxymesh/docs/readme-improvements
Improve README with all modules and getting started guide
2 parents 3e1a120 + 7745197 commit e1c8879

1 file changed

Lines changed: 139 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 139 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,155 @@
11
# Python Proxy Headers
22

3-
The `python-proxy-headers` package provides support for handling custom proxy headers when making HTTPS requests in various Python modules.
3+
[![Documentation Status](https://readthedocs.org/projects/python-proxy-headers/badge/?version=latest)](https://python-proxy-headers.readthedocs.io/en/latest/?badge=latest)
4+
[![PyPI version](https://badge.fury.io/py/python-proxy-headers.svg)](https://badge.fury.io/py/python-proxy-headers)
45

5-
We currently provide extensions to the following packages:
6+
Extensions for Python HTTP libraries to support **sending and receiving custom proxy headers** during HTTPS CONNECT tunneling.
67

7-
* [urllib3](https://python-proxy-headers.readthedocs.io/en/latest/urllib3.html) - a user-friendly HTTP client library for Python
8-
* [requests](https://python-proxy-headers.readthedocs.io/en/latest/requests.html) - a simple, yet elegant, HTTP library
9-
* [aiohttp](https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html) - asynchronous HTTP client/server framework for asyncio and Python
10-
* [httpx](https://python-proxy-headers.readthedocs.io/en/latest/httpx.html) - a next generation HTTP client for Python
8+
## The Problem
119

12-
## Purpose
10+
When making HTTPS requests through a proxy, the connection is established via a CONNECT tunnel. During this process:
1311

14-
None of these modules provide good support for parsing custom response headers from proxy servers. And some of them make it hard to send custom headers to proxy servers. So we at [ProxyMesh](https://proxymesh.com) made these extension modules to support our customers that use Python and want to use custom headers to control our proxy behavior. But these modules can work for handling custom headers with any proxy.
12+
1. **Sending headers to the proxy** - Most Python HTTP libraries don't provide an easy way to send custom headers (like `X-ProxyMesh-Country`) to the proxy server during the CONNECT handshake.
1513

16-
*If you are looking for [Scrapy](https://scrapy.org/) support, please see our [scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers) project.*
14+
2. **Receiving headers from the proxy** - The proxy's response headers from the CONNECT request are typically discarded, making it impossible to read custom headers (like `X-ProxyMesh-IP`) that the proxy sends back.
15+
16+
This library solves both problems for popular Python HTTP libraries.
17+
18+
## Supported Libraries
19+
20+
| Library | Module | Use Case |
21+
|---------|--------|----------|
22+
| [urllib3](https://python-proxy-headers.readthedocs.io/en/latest/urllib3.html) | `urllib3_proxy_manager` | Low-level HTTP client |
23+
| [requests](https://python-proxy-headers.readthedocs.io/en/latest/requests.html) | `requests_adapter` | Simple HTTP requests |
24+
| [aiohttp](https://python-proxy-headers.readthedocs.io/en/latest/aiohttp.html) | `aiohttp_proxy` | Async HTTP client |
25+
| [httpx](https://python-proxy-headers.readthedocs.io/en/latest/httpx.html) | `httpx_proxy` | Modern HTTP client |
26+
| [pycurl](https://python-proxy-headers.readthedocs.io/en/latest/pycurl.html) | `pycurl_proxy` | libcurl bindings |
27+
| [cloudscraper](https://python-proxy-headers.readthedocs.io/en/latest/cloudscraper.html) | `cloudscraper_proxy` | Cloudflare bypass |
28+
| [autoscraper](https://python-proxy-headers.readthedocs.io/en/latest/autoscraper.html) | `autoscraper_proxy` | Automatic web scraping |
1729

1830
## Installation
1931

20-
To use these extension modules, you must first do the following:
32+
```bash
33+
pip install python-proxy-headers
34+
```
35+
36+
Then install the HTTP library you want to use (e.g., `pip install requests`).
37+
38+
> **Note:** This package has no dependencies by default - install only what you need.
39+
40+
## Quick Start
41+
42+
### requests
43+
44+
```python
45+
from python_proxy_headers.requests_adapter import ProxySession
46+
47+
with ProxySession(proxy_headers={'X-ProxyMesh-Country': 'US'}) as session:
48+
session.proxies = {'https': 'http://user:pass@proxy.example.com:8080'}
49+
response = session.get('https://httpbin.org/ip')
50+
51+
# Proxy headers are merged into response.headers
52+
print(response.headers.get('X-ProxyMesh-IP'))
53+
```
54+
55+
### httpx
56+
57+
```python
58+
from python_proxy_headers.httpx_proxy import get
59+
60+
response = get(
61+
'https://httpbin.org/ip',
62+
proxy='http://user:pass@proxy.example.com:8080'
63+
)
64+
65+
# Proxy CONNECT response headers are merged into response.headers
66+
print(response.headers.get('X-ProxyMesh-IP'))
67+
```
68+
69+
### aiohttp
70+
71+
```python
72+
import asyncio
73+
from python_proxy_headers.aiohttp_proxy import ProxyClientSession
74+
75+
async def main():
76+
async with ProxyClientSession() as session:
77+
async with session.get(
78+
'https://httpbin.org/ip',
79+
proxy='http://user:pass@proxy.example.com:8080'
80+
) as response:
81+
# Proxy headers merged into response.headers
82+
print(response.headers.get('X-ProxyMesh-IP'))
83+
84+
asyncio.run(main())
85+
```
86+
87+
### pycurl (low-level)
88+
89+
```python
90+
import pycurl
91+
from python_proxy_headers.pycurl_proxy import set_proxy_headers, HeaderCapture
2192

22-
1. `pip install python-proxy-headers`
23-
2. Install the appropriate package based on the Python library you want to use.
93+
c = pycurl.Curl()
94+
c.setopt(pycurl.URL, 'https://httpbin.org/ip')
95+
c.setopt(pycurl.PROXY, 'http://proxy.example.com:8080')
2496

25-
This package does not have any dependencies because we don't know which library you want to use.
97+
# Add these two lines to any existing pycurl code
98+
set_proxy_headers(c, {'X-ProxyMesh-Country': 'US'})
99+
capture = HeaderCapture(c)
100+
101+
c.perform()
102+
103+
print(capture.proxy_headers) # Headers from proxy CONNECT response
104+
c.close()
105+
```
106+
107+
### cloudscraper
108+
109+
```python
110+
from python_proxy_headers.cloudscraper_proxy import create_scraper
111+
112+
# Drop-in replacement for cloudscraper.create_scraper()
113+
scraper = create_scraper(proxy_headers={'X-ProxyMesh-Country': 'US'})
114+
scraper.proxies = {'https': 'http://proxy.example.com:8080'}
115+
116+
response = scraper.get('https://example.com')
117+
# All CloudScraper features (Cloudflare bypass) preserved
118+
```
119+
120+
## Testing
121+
122+
A test harness is included to verify proxy header functionality:
123+
124+
```bash
125+
# Set your proxy
126+
export PROXY_URL='http://user:pass@proxy.example.com:8080'
127+
128+
# Test all modules
129+
python test_proxy_headers.py
130+
131+
# Test specific modules
132+
python test_proxy_headers.py requests httpx
133+
134+
# Verbose output (show header values)
135+
python test_proxy_headers.py -v
136+
```
26137

27138
## Documentation
28139

29-
For detailed documentation, examples, and usage instructions, please see the [full documentation](https://python-proxy-headers.readthedocs.io/en/latest/).
140+
For detailed documentation, API reference, and more examples:
141+
142+
- **Full Documentation:** [python-proxy-headers.readthedocs.io](https://python-proxy-headers.readthedocs.io/en/latest/)
143+
- **Example Code:** [proxy-examples for Python](https://github.com/proxymesh/proxy-examples/tree/main/python)
144+
145+
## Related Projects
146+
147+
- **[scrapy-proxy-headers](https://github.com/proxymesh/scrapy-proxy-headers)** - Proxy header support for Scrapy
148+
149+
## About
150+
151+
Created by [ProxyMesh](https://proxymesh.com) to help our customers use custom headers to control proxy behavior. Works with any proxy that supports custom headers.
152+
153+
## License
30154

31-
You can also find more example code in our [proxy-examples for python](https://github.com/proxymesh/proxy-examples/tree/main/python).
155+
MIT License

0 commit comments

Comments
 (0)