first shot at adding web drivers for hard urls! (#77)

* first shot at adding web drivers for hard urls!
* tweak to dockerfile and failed tests
* update test
* update fake user agent to be fixed fork
* catch any general error with browser and try driver
updating test cases with harder urls (e.g., that return 400)
exception fallback should try driver too
* increase timeout for twitter status

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Author: vsoch

.dockerignore

@@ -0,0 +1,2 @@
+geckodriver
+chromedriver

.github/workflows/main.yml

@@ -1,31 +1,34 @@
-name: docker-deploy
+name: Build and Deploy containers
 
 on:
+  # Always test on pull request
+  pull_request: []
+
+  # Deploy on merge to main
   push:
-    branches:    
-      - master    
+    branches:
+      - main
 
 jobs:
-  testing-docker:
+  deploy-test-containers:
     runs-on: ubuntu-latest
+    name: Build Container
     steps:
-      - uses: actions/checkout@v3
-      - name: Build container image
-        run: |
-          docker build -t quay.io/urlstechie/urlchecker .
-          DOCKER_TAG=$(docker run quay.io/urlstechie/urlchecker --version)
-          printf "Docker Tag is ${DOCKER_TAG}\n"
-          echo "DOCKER_TAG=${DOCKER_TAG}" >> $GITHUB_ENV
-      - name: Docker login
-        env:
-          docker_user: ${{ secrets.DOCKER_USERNAME }}
-          docker_pass: ${{ secrets.DOCKER_PASSWORD }}
+      - name: Checkout
+        uses: actions/checkout@v3     
+
+      - name: Build
         run: |
-          docker login -u="${docker_user}" -p="${docker_pass}" quay.io
-      - name: Push containers
+           docker build -t ghcr.io/urlstechie/urlchecker .
+           DOCKER_TAG=$(docker run ghcr.io/urlstechie/urlchecker --version)
+           printf "Docker Tag is ${DOCKER_TAG}\n"
+           echo "DOCKER_TAG=${DOCKER_TAG}" >> $GITHUB_ENV
+      - name: Login and Deploy Test Container
+        if: (github.event_name != 'pull_request')
         run: |
+          docker images
           printf "Docker Tag is ${DOCKER_TAG}\n"
-          docker tag quay.io/urlstechie/urlchecker:latest "quay.io/urlstechie/urlchecker:${DOCKER_TAG}"
-          docker push quay.io/urlstechie/urlchecker:latest
-          docker push "quay.io/urlstechie/urlchecker:${DOCKER_TAG}"
-
+          echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ secrets.GHCR_USERNAME }} --password-stdin
+          docker tag ghcr.io/urlstechie/urlchecker:latest "ghcr.io/urlstechie/urlchecker:${DOCKER_TAG}"
+          docker push ghcr.io/urlstechie/urlchecker:latest
+          docker push "ghcr.io/urlstechie/urlchecker:${DOCKER_TAG}"

.github/workflows/test.yml

@@ -41,7 +41,6 @@ jobs:
           pip install types-requests
           mypy urlchecker
 
-
   testing:
     needs: type_checking
     runs-on: ubuntu-latest
@@ -50,11 +49,18 @@ jobs:
       - name: Setup testing environment
         run: conda create --quiet --name testing pytest
 
+      - name: Download ChromeDriver
+        run: |
+            wget https://chromedriver.storage.googleapis.com/103.0.5060.134/chromedriver_linux64.zip
+            unzip chromedriver_linux64.zip
+            rm chromedriver_linux64.zip
+
       - name: Test
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate testing
-          pip install .
+          pip install git+https://github.com/danger89/fake-useragent.git
+          pip install .[all]
           pip install -r tests/test-requirements.txt
           pytest -vs -x --cov=./urlchecker tests/test_*.py
 

.gitignore

@@ -3,6 +3,10 @@ __pycache__/
 *.py[cod]
 *$py.class
 
+# Web drivers
+chromedriver
+geckodriver
+
 # C extensions
 *.so
 

CHANGELOG.md

@@ -12,6 +12,7 @@ and **Merged pull requests**. Critical items to know are:
 Referenced versions in headers are tagged on Github, in parentheses are for pypi.
 
 ## [vxx](https://github.com/urlstechie/urlschecker-python/tree/master) (master)
+ - adding support for web driver for harder URLs (0.0.31)
  - use ANSI escape sequences for colors, fake-useragent for agents (0.0.30)
  - adding type hints to code, more tests and logging bug fix (0.0.29)
  - decrease verbosity when filename is None (0.0.28)

Dockerfile

@@ -1,23 +1,36 @@
 FROM bitnami/minideb:buster
-# docker build -t urlschecker .
+# docker build -t ghcr.io/urlstechie/urlchecker .
 WORKDIR /code
 ENV PATH /opt/conda/bin:${PATH}
 ENV LANG C.UTF-8
 ENV SHELL /bin/bash
 RUN apt-get update && \
-    /bin/bash -c "install_packages wget bzip2 ca-certificates git && \
+    /bin/bash -c "install_packages wget bzip2 ca-certificates git unzip gnupg2 && \
+    install_packages libglib2.0-dev libnss3 libfontconfig1 libgconf-2-4 && \
+    install_packages libxcb-randr0-dev libxcb-xtest0-dev libxcb-xinerama0-dev libxcb-shape0-dev libxcb-xkb-dev && \
     wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
     bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda && \
     rm Miniconda3-latest-Linux-x86_64.sh && \
     conda create --name urlchecker && \
     conda clean --all -y"
+
+# Google chrome binary
+RUN /bin/bash -c 'wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \ 
+    && echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list && \
+    apt-get update && apt-get -y install google-chrome-stable'
+
 COPY . /code
 RUN /bin/bash -c "source activate urlchecker && \
     which python && \
     which pip && \
     pip install --upgrade certifi && \
-    pip install ."
+    pip install git+https://github.com/danger89/fake-useragent.git && \
+    pip install .[all]"
+# Download chrome driver for selenium
+RUN /bin/bash -c "wget https://chromedriver.storage.googleapis.com/103.0.5060.134/chromedriver_linux64.zip && \
+    unzip chromedriver_linux64.zip && \
+    rm chromedriver_linux64.zip"
 RUN echo "source activate urlchecker" > ~/.bashrc
-ENV PATH /opt/conda/envs/urlchecker/bin:${PATH}
+ENV PATH /code:/opt/conda/envs/urlchecker/bin:${PATH}
 ENTRYPOINT ["urlchecker"]
 CMD ["check", "--help"]

README.md

@@ -20,7 +20,14 @@ A detailed documentation of the code is available under [urlchecker-python.readt
 
 ### Install
 
-You can install the urlchecker from [pypi](https://pypi.org/project/urlchecker):
+You can install the urlchecker from [pypi](https://pypi.org/project/urlchecker).
+Before you do, it's recommended to install fake-useragent from:
+
+```bash
+pip install git+https://github.com/danger89/fake-useragent.git
+```
+
+And then urlchecker:
 
 ```bash
 $ pip install urlchecker
@@ -557,6 +564,26 @@ In the "client" folder, for example, the commands that are exposed for the clien
 Functions for Github are be provided in `main/github.py`. This organization should
 be fairly straight forward to always find what you are looking for.
 
+### Drivers
+
+To test more difficult urls, we use a web driver, and you can choose between:
+
+ - [Chrome Driver](https://chromedriver.chromium.org/downloads)
+ - [Gecko Driver](https://github.com/mozilla/geckodriver/releases) (firefox)
+ 
+both to be used with selenium. This driver is optional, but will come by default with our action. To install
+it, you can download the driver at either of the links above and ensure you install selenium:
+
+```bash
+$ pip install urlchecker[selenium]
+```
+and either:
+
+1. Add it directly to your path
+2. Export the directory where it lives as `URLCHECKER_DRIVERS_PATH`
+3. Put it in the root of the urlchecker clone (it will be looked for here)
+
+
 ## Support
 
 If you need help, or want to suggest a project for the organization,

setup.py

@@ -21,12 +21,11 @@ def get_lookup():
 
 # Read in requirements
 def get_reqs(lookup=None, key="INSTALL_REQUIRES"):
-    """get requirements, mean reading in requirements and versions from
-    the lookup obtained with get_lookup"""
-
-    if lookup == None:
-        lookup = get_lookup()
-
+    """
+    Get requirements, mean reading in requirements and versions from
+    the lookup obtained with get_lookup
+    """
+    lookup = lookup or get_lookup()
     install_requires = []
     for module in lookup[key]:
         module_name = module[0]
@@ -67,6 +66,8 @@ def get_reqs(lookup=None, key="INSTALL_REQUIRES"):
 
     INSTALL_REQUIRES = get_reqs(lookup)
     TESTS_REQUIRES = get_reqs(lookup, "TESTS_REQUIRES")
+    INSTALL_REQUIRES_ALL = get_reqs(lookup, "INSTALL_REQUIRES_ALL")
+    SELENIUM_REQUIRES = get_reqs(lookup, "SELENIUM_REQUIRES")
 
     setup(
         name=NAME,
@@ -87,7 +88,10 @@ def get_reqs(lookup=None, key="INSTALL_REQUIRES"):
         setup_requires=["pytest-runner"],
         install_requires=INSTALL_REQUIRES,
         tests_require=TESTS_REQUIRES,
-        extras_require={},
+        extras_require={
+            "all": INSTALL_REQUIRES_ALL,
+            "selenium": SELENIUM_REQUIRES,
+        },
         classifiers=[
             "Intended Audience :: Developers",
             "License :: OSI Approved :: MIT License",

tests/test_core_check.py

@@ -37,6 +37,40 @@ def test_check_files(file_paths, print_all, exclude_urls, exclude_patterns):
     )
 
 
+@pytest.mark.parametrize("file_paths", [["tests/test_files/hard_urls.md"]])
+def test_difficult_urls(file_paths):
+    """
+    test difficult urls that likely require selenium.
+    """
+    checker = UrlChecker()
+    results = checker.run(file_paths, timeout=20)
+
+    # This should be the only failing (503)
+    assert (
+        "https://thisurldoesnotexist-pancakes.whatever"
+        in results["failed"]
+    )
+    working = [
+        "https://www.hpcwire.com/2019/01/17/pfizer-hpc-engineer-aims-to-automate-software-stack-testing/",
+        "https://www.sciencedirect.com/science/article/pii/S0013468608005045",
+        "https://doi.org/10.1063/5.0023771",
+        "https://www.linux.org/",
+        "https://drupal.org/",
+        "https://codepen.io/rootwork/",
+        "http://groundwire.org/blog/groundwire-engagement-pyramid/",
+        "https://twig.symfony.com/doc/",
+        "https://groups.drupal.org/node/298298",
+        "https://portland2013.drupal.org/program/sprints.html",
+        "https://twitter.com/wharman",
+        "https://www.progressiveexchange.org",
+        "https://twitter.com/jooy8/status/322734500226412544",
+        "https://www.drupal.org/node/1982024",
+        "https://groups.drupal.org/node/278968",
+    ]
+    for url in working:
+        assert url in results["passed"]
+
+
 @pytest.mark.parametrize("local_folder_path", ["./tests/test_files"])
 @pytest.mark.parametrize("config_fname", ["./tests/_local_test_config.conf"])
 def test_locally(local_folder_path, config_fname):

tests/test_core_fileproc.py

@@ -92,10 +92,12 @@ def test_get_file_paths(base_path, file_types):
         [
             "tests/test_files/sample_test_file.md",
             "tests/test_files/sample_test_file.py",
+            "tests/test_files/hard_urls.md",
         ],
         [
             "tests/test_files/sample_test_file.py",
             "tests/test_files/sample_test_file.md",
+            "tests/test_files/hard_urls.md",
         ],
     ]
     # assert