Merge pull request #6 from kirankotari/code-refactor

Code refactor

Author: kirankotari

CHANGELOG.md

@@ -1,3 +1,10 @@
+# New in 2.0
+
+- Added global/local feature
+- Added proxy feature
+- Added test cases
+- Updating README
+
 # New in 1.2
 
 - Code refactoring

README.md

@@ -11,55 +11,139 @@
 ![GitHub pull requests](https://img.shields.io/github/issues-pr/kirankotari/weblinks)
 
 - [Introduction](#introduction)
+- [Pre-requisites](#pre-requisites)
 - [Installation and Downloads](#installation-and-downloads)
-- [Usage](#usage)
+- [Commands](#commands)
+- [Docs](#docs)
+- [Bug Tracker and Support](#bug-tracker-and-support)
 - [License and Copyright](#license-and-copyright)
 - [Author and Thanks](#author-and-thanks)
 
+
 ## Introduction
 
 Weblinks, It get all links from a given website and we can apply filters on top of it to get desired results, when you are good you can start downloading them with `-d` flag.
 
-TODO: many changes, add-ons need to update in full.!
-
-In the library we are supporting plain webpages, authentication based webpages.
-
-## Pre-req
+In the library we are supporting plain webpages, authentication based webpages, proxy, authentication on proxy, etc. 
 
-A system need to have curl and python
-
-## Docs
+We also support storing the config in either local/global configuration, for best practise we suggest not to store your password, you get a prompt on runtime wher you can provide it.
 
-How to use weblinks?
+## Pre-requisites
 
-```python
+A system need to support **curl commands** and **python3**
 
-```
 ## Installation and Downloads
 
-`pip install weblinks`
+```shell
+pip install weblinks
+```
 
-## Usage
+## Commands
 
+```shell
+weblinks --help
 ```
-usage: weblinks [-h] -w WEB [-u USERNAME] [-p PASSWORD] [-e EXT] [-d] [-v]
-                   substring
-
-positional arguments:
-  substring             the sub-string in the links
+```
+usage: weblinks [-h] [-w WEB] [-s SUBSTRING] [-e EXT] [-d] [-u USERNAME]
+                [-p PASSWORD] [-g] [-l] [-v] [--proxy PROXY]
+                [--proxy-username PROXY_USERNAME]
+                [--proxy-password PROXY_PASSWORD] [--version]
 
 optional arguments:
   -h, --help            show this help message and exit
   -w WEB, --web WEB     the website
+  -s SUBSTRING, --substring SUBSTRING
+                        the sub-string in the links
+  -e EXT, --ext EXT     file extention
+  -d, --download        download links
   -u USERNAME, --username USERNAME
                         web login username
   -p PASSWORD, --password PASSWORD
                         web login password
-  -e EXT, --ext EXT     file extention
-  -d, --download        download links
+  -g, --global          global configuration
+  -l, --local           local configuration
   -v, --verbosity
+  --proxy PROXY         proxy address
+  --proxy-username PROXY_USERNAME
+                        proxy username
+  --proxy-password PROXY_PASSWORD
+                        proxy password
+  --version             weblinks version
+```
+
+## Docs
+
+**Weblinks usage**
+
+To see current lib. version
+```shell
+weblinks --version
+# weblinks version: 2.0
+```
+
+To see python file from given url
+```shell
+weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python
+# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found
+# Python-3.8.13.tar.xz
+# Python-3.8.13.tar.xz.asc
+# Python-3.8.13.tgz
+# Python-3.8.13.tgz.asc
+```
+
+Still wanted to filter add file extention
+```shell
+weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz
+# INFO     | 2022-07-23 16:23:33,603 | run     :117  | links found
+# Python-3.8.13.tgz
+```
+
+Start download, listed links are good
+```shell
+weblinks --web https://www.python.org/ftp/python/3.8.13/ --substring Python --ext .tgz -d
+# INFO     | 2022-07-23 16:25:34,807 | run     :117  | links found
+# Python-3.8.13.tgz
+# INFO     | 2022-07-23 16:25:34,807 | run     :124  | start download: Python-3.8.13.tgz
+# INFO     | 2022-07-23 16:25:34,807 | utils   :58   | downloading: Python-3.8.13.tgz
+# INFO     | 2022-07-23 16:25:36,849 | run     :126  | completed: Python-3.8.13.tgz
 ```
 
+For authentication
+```shell
+weblinks --web <url> --substring <sub> --username <kirankotari> --password <xxxxxx> --ext .tgz
+# Note: don't add --password, it will ask dynamically
+```
+
+For verbose add -v
+```shell
+weblinks --web <url> --substring <sub> --username <kirankotari> --ext .tgz -v
+```
+
+To store config add --local or --global respectively
+```shell
+weblinks --local --web <url> --username <kirankotari> --ext .tgz
+```
+
+For proxy
+```shell
+weblinks --proxy <ip>:<port> --web <url> --substring <sub>
+```
+
+For proxy with authentication
+```shell
+weblinks --proxy <ip>:<port> --web <url> --substring <sub> --proxy-username <proxy user> --proxy-password <proxy password>
+```
+
+Adding web authentication on top of proxy auth.
+```shell
+weblinks --proxy <ip>:<port> --web <url> --substring <sub> --proxy-username <proxy user> --proxy-password <proxy password> --username <user>
+```
+
+## Bug Tracker and Support
+
+- Please report any suggestions, bug reports, or annoyances with weblinks through the [Github bug tracker](https://github.com/kirankotari/weblinks/issues). If you're having problems with general python issues, consider searching for a solution on [Stack Overflow](https://stackoverflow.com/search?q=).
+- If you can't find a solution for your problem or need more help, you can [ask a question](https://stackoverflow.com/questions/ask).
+
 ## License and Copyright
 
 - weblinks is licensed [Apache 2.0](https://opensource.org/licenses/Apache-2.0) 2022

setup.py

@@ -2,8 +2,7 @@
 from os import path
 from io import open
 
-version = "1.1.5"
-# version = "2.0" # TODO: planning to release a major version
+version = "2.0"
 
 # setup file path
 here = path.abspath(path.dirname(__file__))

tests/test_weblinks.py

@@ -55,7 +55,7 @@ def test_weblinks_version(self):
         self.applyPatch
         run.main()
         out, err = self._capfd.readouterr()
-        assert out.strip() == 'weblinks version: 1.2'
+        assert out.strip() == 'weblinks version: 2.0'
         del self.args['version']
 
 

tests/test_weblinks_proxy.py

@@ -35,7 +35,6 @@ def get_inputs():
     return args
 
 
-# TODO: need to add proxy test cases
 @pytest.mark.run
 class TestWeblinksVersion(unittest.TestCase):
 

weblinks/run.py

@@ -2,7 +2,9 @@
     and filter them using substring, a file extentions
 
 usage: weblinks [-h] [-w WEB] [-s SUBSTRING] [-e EXT] [-d] [-u USERNAME]
-               [-p PASSWORD] [-g] [-l] [-v] [--version]
+                [-p PASSWORD] [-g] [-l] [-v] [--proxy PROXY]
+                [--proxy-username PROXY_USERNAME]
+                [--proxy-password PROXY_PASSWORD] [--version]
 
 optional arguments:
   -h, --help            show this help message and exit
@@ -18,6 +20,11 @@
   -g, --global          global configuration
   -l, --local           local configuration
   -v, --verbosity
+  --proxy PROXY         proxy address
+  --proxy-username PROXY_USERNAME
+                        proxy username
+  --proxy-password PROXY_PASSWORD
+                        proxy password
   --version             weblinks version
 """
 
@@ -32,7 +39,7 @@
 from .config import Configuration
 
 
-version = "1.2"
+version = "2.0"
 
 
 def parser() -> argparse: