diff --git a/curl/README.md b/curl/README.md index e83d3a2..94cc509 100644 --- a/curl/README.md +++ b/curl/README.md @@ -35,7 +35,7 @@ For more details on how to use `curl` with ECH, see the [official documentation] To test that your custom `curl` build is working correctly, run it against the DEfO test server: ```sh -./workspace/output/bin/curl" --ech=true --doh-url https://1.1.1.1/dns-query 'https://test.defo.ie/echstat.php?format=json' | jq +./workspace/bin/curl --ech=true --doh-url https://1.1.1.1/dns-query 'https://test.defo.ie/echstat.php?format=json' | jq ``` Example output: diff --git a/dnsreport/README.md b/dnsreport/README.md index 4bb34b4..a0d00a3 100644 --- a/dnsreport/README.md +++ b/dnsreport/README.md @@ -1,34 +1,30 @@ # DNS Report Generation -This document outlines the steps to generate a report on DNS query latency and HTTPS RR feature usage. The process involves collecting data, running analysis scripts, and generating a final report. +This document outlines the steps to generate a report on DNS query latency and HTTPS RR feature usage. The process involves collecting data, running analysis in an interactive Jupyter notebook, and visualizing the findings. -The final report is `report/report.md` and can be converted to a PDF. +The notebook (`dnsreport/report.ipynb`) is pre-populated with cached analysis results so that you can view the final report immediately, but you can also re-run the analysis at any time using your own collected data. ## Step 1: Setup -The analysis scripts are written in Python. +The analysis is packaged within an interactive Jupyter notebook that contains pre-populated results and can be dynamically re-run. -1. **Create a Python virtual environment:** - From the `ech-test` directory, run: - ```sh - python3 -m venv ./workspace/.venv - ``` +We recommend using [uv](https://docs.astral.sh/uv/) for virtual environment and dependency management (you can install it by following the [official installation guide](https://docs.astral.sh/uv/getting-started/installation/)). If you prefer standard Python tools (like `python3 -m venv`), you can use them instead. -2. **Activate the virtual environment:** +1. **Create and activate the virtual environment:** + From the `ech-research` directory, run: ```sh - source ./workspace/.venv/bin/activate + uv venv + source .venv/bin/activate ``` - You will need to do this every time you work on the report in a new terminal session. -3. **Install dependencies:** - From the `ech-test` directory, run: +2. **Install dependencies:** ```sh - pip install -r ./dnsreport/tools/requirements.txt + uv pip install -r requirements.txt ``` ## Step 2: Collect DNS Data -From the `ech-test` folder, run the data collection tool. The following command will query the top 10,000 domains 5 times each, which is a good sample for the report. +From the `ech-research` folder, run the data collection tool. The following command will query the top 10,000 domains 5 times each, which is a good sample for the report. ```sh go run ./dnsreport -topN 10000 -numQueries 5 @@ -60,78 +56,19 @@ The tool generates a CSV file (`workspace/results-top-n.csv`) with the fol * `answers`: The resource records in the answer section (excluding CNAMEs), formatted as a JSON array. * `additionals`: The resource records in the additional section, formatted as a JSON array. +--- -## Step 3: Analyze DNS Query Latency - -The goal of this step is to determine the impact of waiting for the HTTPS RR before proceeding with TCP or TLS connections. The analysis is broken down into: - -* **Duration Distribution:** We create a cumulative distribution of latencies over all queries, broken down by query type (A/AAAA/HTTPS), to visualize the overall performance. -* **Impact of Caching:** To consider the effects of caching, we group queries by domain and query type, and analyze the minimum and median durations. -* **Slowest Queries:** We identify all domains for which the median HTTPS query is > 50ms slower than the A query and put them in a table for detailed inspection. +## Step 3: Analyze and Visualize via Jupyter Notebook -### Generating the Latency Analysis +All sorting, filtering, graphing, and table compilation are carried out interactively inside the `dnsreport/report.ipynb` notebook. -The analysis scripts are in `dnsreport/tools`. The generated plots and tables will be placed in `dnsreport/report`. +You can open this notebook natively in **VS Code** (choose the `.venv` kernel in the top-right corner) or in a web browser using the standard command: -First, sort the data: ```sh -./workspace/.venv/bin/python3 dnsreport/tools/sort_csv_by_rank.py ./workspace/results-top10000-n5.csv +cd dnsreport +jupyter notebook report.ipynb ``` -This creates `./workspace/results-top10000-n5-sorted.csv`. - -Now, run the analysis scripts. - -1. **Generate Latency Plots:** - ```sh - ./workspace/.venv/bin/python3 dnsreport/tools/generate_charts.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report - ``` - **Outputs:** This generates the plots in the `dnsreport/report` directory. - * `duration_by_type_quantile_plot.png`: Overall latency distribution. - * `min_duration_quantile_plot.png`: Best-case (cached) latency distribution. - * `median_duration_quantile_plot.png`: Typical latency distribution. - -2. **Generate Slow Queries Table:** - ```sh - ./workspace/.venv/bin/python3 dnsreport/tools/generate_slow_queries_table.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report/slow_https_queries.md - ``` - **Output:** This creates `slow_https_queries.md` in the `dnsreport/report` directory. - -## Step 4: Analyze HTTPS RR Feature Usage - -The goal of this step is to determine what features of the HTTPS RR are being used in production to inform the priority of implementing support for them. The features include the Alias Mode, the various `alpn` values and the various SVCB parameters (e.g., `ipv4hint`, `ipv6hint`, `ech`). - -### Generating the Feature Analysis - -1. **Generate Feature Usage Plots:** - ```sh - ./workspace/.venv/bin/python3 dnsreport/tools/generate_charts.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report - ./workspace/.venv/bin/python3 dnsreport/tools/unique_domain_analysis.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report - ``` - **Outputs:** This generates the plots in the `dnsreport/report` directory. - * `param_usage.png`: Usage frequency of all HTTPS RR parameters. - * `param_usage_unique_domains.png`: Parameter usage counted only once per domain. - -2. **Generate Feature Usage Table:** - ```sh - ./workspace/.venv/bin/python3 dnsreport/tools/unique_domain_analysis.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report/feature_usage_table.md - ``` - **Output:** This creates `feature_usage_table.md` in the `dnsreport/report` directory. - -## Step 5: Analyze Broken Domains - -The goal of this step is to identify domains where HTTPS queries consistently time out (duration > 2s) and investigate the reasons behind these failures by analyzing RCODEs, errors, and querying authoritative nameservers. - -### Generating the Broken Domains Analysis - -1. **Run the analysis script:** - ```sh - ./workspace/.venv/bin/python3 dnsreport/tools/analyze_broken_domains.py ./workspace/results-top10000-n5-sorted.csv ./dnsreport/report/broken_domains_report.md - ``` - **Output:** This generates `broken_domains_report.md` in the `dnsreport/report` directory. The report includes the `dig` commands used for the analysis, which can be easily copied and pasted to reproduce the results. - -## Step 6: Assemble the Report -1. **Fill in the report:** - Open `dnsreport/report/report.md`. The generated charts are already linked. You can copy the contents of `dnsreport/report/slow_https_queries.md`, `dnsreport/report/feature_usage_table.md`, and `dnsreport/report/broken_domains_report.md` to replace the example tables in the report. Finally, write a conclusion based on the findings. +The notebook is pre-populated with default analysis results for immediate viewing, but you can also rerun the cells at any time to process your newly collected data files. -Remember to `cd ../..` to return to the `ech-test` directory when you are done. +All generated plots, tables, and statistics appear directly inline. Once you have finished executing the analysis cells, you can finalize your conclusions within the notebook's markdown sections or export the entire document as a self-contained HTML/PDF report. diff --git a/dnsreport/report.ipynb b/dnsreport/report.ipynb index b5ca874..74e282c 100644 --- a/dnsreport/report.ipynb +++ b/dnsreport/report.ipynb @@ -70,10 +70,12 @@ }, "outputs": [], "source": [ - "# Uncoment and run these to collect the data and move into the report folder.\n", + "# Uncomment and run these to collect the data and move into the report folder.\n", + "# Note: Shell commands are prefixed with '!' so they can execute in a Python Jupyter kernel.\n", + "# Alternatively, you can run these commands directly in your separate terminal shell (without the '!').\n", "\n", - "#go run . -workspace ../workspace -topN {topN} -numQueries {numQueries} 2> log.txt\n", - "#cp ../workspace/results-top{topN}-n{numQueries}.csv ../workspace/domains-top{topN}.csv ." + "#!go run . -workspace ../workspace -topN {topN} -numQueries {numQueries} 2> log.txt\n", + "#!cp ../workspace/results-top{topN}-n{numQueries}.csv ../workspace/domains-top{topN}.csv ." ] }, { @@ -1357,7 +1359,7 @@ "runs = sorted(https_df['run'].unique())\n", "run_columns = []\n", "for r in runs:\n", - " run_columns.extend([f\"category_{r}\", f\"duration_{r}_ms\"])\n", + " run_columns.extend([f\"category_{r}\", f\"duration_ms_{r}\"])\n", "\n", "# 7. Apply the column order, sort by rank, and make the index start at 1\n", "final_df = consistently_failed[['domain', 'rank', 'first_ns'] + run_columns].sort_values('rank').reset_index(drop=True)\n", diff --git a/greasereport/README.md b/greasereport/README.md index 806a6d2..33eb76c 100644 --- a/greasereport/README.md +++ b/greasereport/README.md @@ -27,7 +27,7 @@ This will: * `-trancoID `: The ID of the Tranco list to use. Defaults to `7NZ4X`. * `-topN `: The number of top domains to analyze. Defaults to 100. * `-parallelism `: Maximum number of parallel requests. Defaults to 10. -* `-curl `: Path to the ECH-enabled curl binary. Defaults to `./workspace/output/bin/curl`. +* `-curl `: Path to the ECH-enabled curl binary. Defaults to `./workspace/bin/curl`. * `-maxTime `: Maximum time per curl request. Defaults to `10s`. ### Output Format @@ -49,8 +49,7 @@ The tool generates a CSV file (`workspace/grease-results-top.csv`) with the f ## Report -After running the `greasereport` tool, a `report` subdirectory is created within the `greasereport` directory. This directory contains: +The repository contains pre-compiled analysis results in the `greasereport/report/` directory: * `report.md`: A summary of the ECH GREASE connectivity analysis. -* `analyze.py`: The Python script used for the analysis. -* `grease-results-top.csv`: The raw data from the test run. +* `grease-results-top10000.csv`: The raw data from a 10,000-domain test run. diff --git a/ispreport/README.md b/ispreport/README.md index fae9d74..91ce758 100644 --- a/ispreport/README.md +++ b/ispreport/README.md @@ -39,7 +39,9 @@ The countries file should be a CSV file containing country names and their 2-let "Virgin Islands, U.S.",VI ``` -You can download a complete list of country codes from [here](https://raw.githubusercontent.com/datasets/country-list/master/data.csv). +A default country list is provided at `ispreport/report/countries.csv`. You can copy this file to your workspace directory (`workspace/countries.csv`) before running the tool, or specify its path using the `--countries` parameter. + +You can download a complete list of country codes from [here](https://raw.githubusercontent.com/datasets/country-list/master/data.csv) as well. ## Running @@ -78,7 +80,7 @@ This will: * `-parallelism `: Maximum number of parallel requests. Defaults to `16`. * `-verbose`: Enable verbose logging. * `-maxTime `: Maximum time per curl request. Defaults to `30s`. -* `-curl `: Path to the ECH-enabled curl binary. Defaults to `./workspace/output/bin/curl`. +* `-curl `: Path to the ECH-enabled curl binary. Defaults to `./workspace/bin/curl`. * `-ipCheckURL `: URL used to discover the real external IP of the proxy. Defaults to `https://ipv4.icanhazip.com/`. * `-asnDB `: Optional path to a MaxMind or DB-IP `.mmdb` database file for independent ASN verification. @@ -115,7 +117,9 @@ The CSV file contains the following columns: ## Generating the Final Report -After running the data collection tool, you can generate a visual report using the provided Jupyter notebook. +The visual report notebook (`ispreport/report/report.ipynb`) is pre-populated with cached multi-country analysis results and plots for domains like `www.google.com` and `cloudflare-ech.com`. You can view and explore the final report immediately without setting up any SOAX proxy accounts or running any data collection. + +If you collect new proxy data, you can easily re-run the notebook to update the visualizations. ### 1. Organize the Data @@ -137,27 +141,25 @@ The notebook expects data to be organized in subdirectories within `ispreport/re Running the notebook requires Python 3 and several data analysis libraries. -```bash -# From the project root: -# 1. Create the virtual environment if it doesn't exist -python3 -m venv workspace/.venv +We recommend using [uv](https://docs.astral.sh/uv/) for virtual environment and dependency management (you can install it by following the [official installation guide](https://docs.astral.sh/uv/getting-started/installation/)). If you prefer standard Python tools (like `python3 -m venv`), you can use them instead. -# 2. Activate the virtual environment -source workspace/.venv/bin/activate +```bash +# From the project root, create and activate the environment: +uv venv +source .venv/bin/activate -# 3. Install required dependencies -pip install pandas numpy matplotlib seaborn ipywidgets jupyter +# Install required dependencies: +uv pip install -r requirements.txt ``` ### 3. Run the Notebook -1. Navigate to the report directory and start Jupyter: - ```bash - cd ispreport/report - # If you didn't activate the venv yet, run: source ../../workspace/.venv/bin/activate - jupyter notebook report.ipynb - ``` +You can open this notebook natively in **VS Code** (choose the `.venv` kernel in the top-right corner) or in a web browser using the standard command: -2. In the first code cell of the notebook, update the `DOMAIN` variable to match the name of the subdirectory you created (e.g., `DOMAIN = "www_google_com"`). +```bash +cd ispreport/report +jupyter notebook report.ipynb +``` -3. Run all cells in the notebook to generate the analysis and visualizations. +1. In the first code cell of the notebook, update the `DOMAIN` variable to match the name of the subdirectory you created (e.g., `DOMAIN = "www_google_com"`). +2. Run the cells in the notebook to generate the analysis and visualizations. diff --git a/ispreport/main.go b/ispreport/main.go index 7a509d0..eeadc46 100644 --- a/ispreport/main.go +++ b/ispreport/main.go @@ -231,7 +231,7 @@ func main() { // Determine curl binary path curlPath := *curlPathFlag if curlPath == "" { - curlPath = filepath.Join(workspaceDir, "output", "bin", "curl") + curlPath = filepath.Join(workspaceDir, "bin", "curl") } // Load ASN database if provided diff --git a/ispreport/report/report.ipynb b/ispreport/report/report.ipynb index f4ff3a1..800e12d 100644 --- a/ispreport/report/report.ipynb +++ b/ispreport/report/report.ipynb @@ -10,8 +10,8 @@ "\n", "> **How to (re-)run this notebook:**\n", "> \n", - "> 1. Ensure you have Python installed and activate your virtual environment (e.g., `source ../../workspace/.venv/bin/activate`).\n", - "> 2. Install the required libraries if you haven't already: `pip install pandas numpy matplotlib seaborn ipywidgets jupyter`.\n", + "> 1. Ensure you have Python installed and activate your virtual environment (e.g., `source ../../.venv/bin/activate`).\n", + "> 2. Install the required libraries if you haven't already: `uv pip install -r ../../requirements.txt` (or `pip install -r ../../requirements.txt`).\n", "> 3. Open this file in Jupyter Notebook (`jupyter notebook report.ipynb`) or an IDE (like VS Code), and execute the cells sequentially.\n" ] }, diff --git a/requirements.txt b/requirements.txt index 80d919c..969d1df 100644 --- a/requirements.txt +++ b/requirements.txt @@ -3,3 +3,5 @@ numpy matplotlib seaborn ipykernel +ipywidgets +jupyter