Skip to content

Commit 9fdf00c

Browse files
committed
Update README.md
1 parent 3695dad commit 9fdf00c

1 file changed

Lines changed: 5 additions & 2 deletions

File tree

README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,10 @@ Surprisingly, the only thing that tells a server the application triggered the r
2121

2222
## The source code
2323

24-
The project code in this repository is crawling two different public proxy websites http://proxyfor.eu/geo.php and http://free-proxy-list.net.
24+
The project code in this repository is crawling two different public proxy websites:
25+
* http://proxyfor.eu/geo.php
26+
* http://free-proxy-list.net
27+
2528
After collecting the proxy data and filtering the slowest ones it is randomly selecting one of them to query the target url.
2629
The request timeout is configured at 30 seconds and if the proxy fails to return a response it is deleted from the application proxy list.
27-
I have to mention that for each request a different agent header is used. This headers are strong in the **/data/user_agents.txt** file which contains around 900 different agents.
30+
I have to mention that for each request a different agent header is used. The different headers are stored in the **/data/user_agents.txt** file which contains around 900 different agents.

0 commit comments

Comments
 (0)