You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+5-2Lines changed: 5 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,10 @@ Surprisingly, the only thing that tells a server the application triggered the r
21
21
22
22
## The source code
23
23
24
-
The project code in this repository is crawling two different public proxy websites http://proxyfor.eu/geo.php and http://free-proxy-list.net.
24
+
The project code in this repository is crawling two different public proxy websites:
25
+
*http://proxyfor.eu/geo.php
26
+
*http://free-proxy-list.net
27
+
25
28
After collecting the proxy data and filtering the slowest ones it is randomly selecting one of them to query the target url.
26
29
The request timeout is configured at 30 seconds and if the proxy fails to return a response it is deleted from the application proxy list.
27
-
I have to mention that for each request a different agent header is used. This headers are strong in the **/data/user_agents.txt** file which contains around 900 different agents.
30
+
I have to mention that for each request a different agent header is used. The different headers are stored in the **/data/user_agents.txt** file which contains around 900 different agents.
0 commit comments