You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[](https://shields.io/)
5
+
[](https://shields.io/)<!---->
This repository holds data for the Proxy_Bypass vulnerability research tool with the `user-agents.json` file generated from the `User-Agent-Parser.py` script within this repository. Additionally, the `ua-stats.py` script is used to draw various statistics out of the `user-agents.json` file.
13
+
This repository holds data of all the useragents in the `user_agents.json` file, which can be used directly with any tool that can parse json format.
13
14
14
-
## Overview
15
+
download the repo to update the user-agent data by running `User-Agent-Dict.py` and edit the file per the fields you would like to add or remove.
16
+
17
+
Major intention for creating this dictionary of useragents was to feed it to Proxy_Bypass vulnerability research tool(under development in private repo) to use the `user-agents.json` file as input for fuzzing proxies.
15
18
16
-
The User Agent Data Scraper is designed to:
19
+
As a PoC have provided the `ua-stats.py` script which draw various statistics out of the `user-agents.json` file as input
20
+
21
+
## Overview
17
22
18
23
🎯 **Primary:**
19
-
-Scrape user agent data from [useragentstring.com](https://www.useragentstring.com/pages/All/)
20
-
-Create a dictionary of useragents to be used by the proxy-bypass vulnerability research tool
21
-
- Organize the data into dictionaries.
24
+
-The User Agent Dict python script is designed to scrape from [useragentstring.com](https://www.useragentstring.com/pages/All/)
25
+
-To use as Fuzzing library of user-agents used for vulnerability research tools
26
+
- Organize the data into dictionaries, for faster access (mimicing hashmaps)
22
27
23
28
🚀 **Secondary:**
24
29
- Identify user agent groups based on specified conditions.
30
+
- High used vs low used user-agents to choose for fuzzing
25
31
- Display statistics about general and mobile user agents.
26
32
- Provide options for data visualization using pie charts, word clouds, and more.
27
33
28
-
## Features
29
-
30
-
🌟 **Scrapes User Agent Data:**
31
-
Scrapes user agent data from a URL and stores it in dictionaries.
32
-
33
-
🔍 **Filters and Organizes:**
34
-
Filters and organizes user agents based on conditions.
35
-
36
-
📊 **Provides Statistics:**
37
-
Provides statistics about general and mobile user agents.
38
-
39
-
📈 **Data Visualization:**
40
-
Offers options for data visualization, including:
41
-
- Pie charts for user agent groups.
42
-
- Word clouds for user agent group names.
43
-
44
-
💾 **Save Charts and Data:**
45
-
Allows users to save generated charts and data to a local directory.
46
-
47
-
🎈 **Easy-to-Use Interface:**
48
-
An easy-to-use command-line interface.
49
-
50
34
## How to Use
51
35
52
-
[](video_link)
53
-
54
36
1.**Installation:**
55
37
Clone this repository to your local machine.
56
38
@@ -65,40 +47,113 @@ An easy-to-use command-line interface.
65
47
Open a terminal and navigate to the project directory. Run the script using the following command:
66
48
67
49
```bash
68
-
python User-Agent-Dict.py
50
+
python3 User-Agent-Dict.py
69
51
```
70
52
71
-
Follows the Prompts: The script will prompt you to interactively choose from various options, such as viewing pie charts, generating word clouds, and more.
53
+
## Sample Output
54
+
55
+
***If you would like to print on screen***
56
+
```
57
+
> python3 User-Agent-Dict.py
58
+
Do you want to print the data on the screen? (yes/no): yes
<spanstyle="color: yellow;">[!]</span> No new user-agents found.
92
+
</code>
93
+
</pre>
94
+
## Statistics
95
+
96
+
As a PoC, have added a basic statistics deriving script `ua-stats.py` that uses the `user_agents.json` as input file.
72
97
73
-
Data Visualization: The script generates various types of charts and visualizations to analyze user agent data.
98
+
`ua-stats.py` script will prompt you to interactively choose from various options, such as viewing pie charts and generating word clouds from the `user_agents.json` data.
74
99
75
-
Saving Charts: Choose to save generated charts and data to a local directory.
100
+
## How to Use `ua-stats.py`
101
+
102
+
1.**Run the Script:**
103
+
Open a terminal and navigate to the project directory. Run the script using the following command:
104
+
105
+
```bash
106
+
python3 ua-stats.py
107
+
```
76
108
77
109
## Sample Output
78
110
79
-
Video Tutorial
80
-
Watch the full tutorial on YouTube
111
+
```
112
+
> python3 ua-stats.py
113
+
Select an option:
114
+
1. Pie chart for Mobile User Agents (Count < 10)
115
+
2. Pie chart for Mobile User Agents (10 <= Count < 500)
116
+
3. Pie chart for General User Agents (10 <= Count < 50)
117
+
4. Pie chart for General User Agents (50 <= Count < 500)
118
+
5. Pie chart for General User Agents (Count >= 500)
119
+
6. Word Cloud for Mobile User Agent Group Names
120
+
7. Word Cloud for General User Agent Group Names
121
+
8. Exit
122
+
Enter your choice (1/2/3/4/5/6/7/8):
123
+
```
81
124
82
-
## Statistics
125
+
This is only a PoC to use of using the json file data, there could be more analysis you could think of with this data. 😀
126
+
127
+
**Few Samples Below**
83
128
84
129
**Mobile**
85
130
86
-
Highest Mobile User Agents
87
-
131
+
<palign="center">
132
+
<strong>Highest Mobile User Agents</strong><br>
133
+
<imgsrc="Charts/Highest%20Mobile%20User-agents.png"alt="Highest Mobile User Agents">
134
+
</p>
88
135
89
-
Mobile User Agents < 500
90
-
136
+
<palign="center">
137
+
<strong>Mobile User Agents < 500</strong><br>
138
+
<imgsrc="Charts/Mobile%20User-agents%20less%20than%20500.png"alt="Mobile User Agents < 500">
139
+
</p>
91
140
92
-
***General***
141
+
**General**
93
142
94
-
Highest General User Agents
95
-
143
+
<palign="center">
144
+
<strong>Highest General User Agents</strong><br>
145
+
<imgsrc="Charts/Highest%20General%20User-agents.png"alt="Highest General User Agents">
146
+
</p>
96
147
97
-
General User Agents > 500
98
-
148
+
<palign="center">
149
+
<strong>General User Agents > 500</strong><br>
150
+
<imgsrc="Charts/General%20User-agents%20greater%20than%20500.png"alt="General User Agents > 500">
151
+
</p>
99
152
100
-
General User Agents < 500
101
-
153
+
<palign="center">
154
+
<strong>General User Agents < 500</strong><br>
155
+
<imgsrc="Charts/General%20User-agents%20less%20than%20500.png"alt="General User Agents < 500">
156
+
</p>
102
157
103
158
## Note
104
159
The script may require an internet connection to retrieve data from the specified URL.
0 commit comments