Finalised episode 2. Challenges need testing once tags are active. Updated episode 3 with example code. 2/3 use cases now complete.

Author: sjdkay

_episodes/02-rucio_usage.md

@@ -192,12 +192,12 @@ The following tags are available as of March 2026:
   - E.g. v25.06.2 -> June 2025 software container, version 2
 - **requester\_pwg**
   - Defines the physics working group (PWG) that the simulated data relates to, options are:
-  - excl\_diff\_tagging
-  - inclusive
-  - jets\_hf
-  - semi\_inclusive
-  - ew\_bsm
-  - other
+    - edt (exclusive, diffractive and tagging)
+    - inclusive
+    - jets\_hf
+    - semi\_inclusive
+    - ew\_bsm
+    - other
   - **Can be one or more**
 - **q2\_min**
   - Minumum Q2 value (GeV^2) in the simulation file, entered as a number.
@@ -213,11 +213,14 @@ The following tags are available as of March 2026:
   - True/false depending upon whether sample includes any background mixing
 - **ion\_species**
   - Ion species in the simulation, defaults to `p`, proton, if not specified
-  - Typed as formatted in files, e.g. `Au197` for gold, `He3` for helium 3 etc.
+    - Typed as formatted in files, e.g. `Au197` for gold, `He3` for helium 3 etc.
+    - `Cu63`, `H2`, `Ru96` and `p` are some other options 
 - **generator**
   - MC event generator used to generate the simulated data
   - E.g. Pythia8, Herwig etc
-  
+  - Entered as all lower case
+    - E.g. `dempgen` *not* `DEMPgen`
+
 As noted on some items in this list, some tags are optional and may not be applied to all datasets. However, the following tags are **required** for all datasets:
 
 - software\_release
@@ -228,18 +231,28 @@ As noted on some items in this list, some tags are optional and may not be appli
 - ion\_species
 - generator
 
+Note that as mentioned for the generator, tags are entered in lower case, **with the exception of ion species**.
+
 We can use these tags to filter through the available datasets and identify those of interest to us. For example:
 
 ```bash
 rucio did list --filter 'TAG==*' 'scope:*'
 ```
 
-So, as an example, we could all DIDs using the latest software release (v26.03.0) via:
+So, as an example, we could list all DIDs with electron beam energies of 10 GeV via:
 
 ```bash
-rucio did list --filter 'software_release==26.03.0*' 'epic:*'
+rucio did list --filter 'electron_beam_energy==10' 'epic:*'
 ```
 
+We can also combine tags and filter on several at once, e.g:
+
+```bash
+rucio did list --filter 'electron_beam_energy==10, ion_beam_energy==250' 'epic:*'
+```
+
+which will return only datasets with 10x250 collisions (10 GeV electron on 250 GeV ions using the standard ePIC conventions). We can keep adding filters in this manner as we like to really narrow down the DIDs we return with our query.
+
 > ## `Exercise:`
 > Using tags, find the DIDs of the **latest**:
 > - DEMP events in the Q2 range of 3 to 10 for 10 GeV electrons on 250 GeV protons

_episodes/03-use_cases.md

@@ -29,35 +29,45 @@ They may also want to only test a small subset of data to test and develop their
 To find files that meet their requirements they could utilise the following tags:
 
 - software\_release
-- physics\_process
+- requester\_pwg
 - electron\_beam\_energy
 - ion\_beam\_energy
+- ion\_species
 
 We can use these tags to filter through the DIDs and find datasets of interest:
 
 ```bash
-Example command
+rucio did list --filter 'software_release==XXX, requester_pwg==YYY, electron_beam_energy==ZZ, ion_beam_energy==iii, ion_species==jjj' 'epic:*'
 ```
 
+Where we can substitute in our chosen values for each in place of `XXX`, `YYY`, `ZZ`, `iii` and `jjj`.
+
 > ## `Beam Energies:` 
 > Whilst we can enter any number for the `electron_beam_energy` and `ion_beam_energy` values, there are only certain combinations actually in use.
 > `electron_beam_energy` is typically 5, 10 or 18 GeV
-> `ion_beam_energy` is typically 41, 100, 130, 250 or 275 for protons.
+> `ion_beam_energy` is typically 41, 100, 130, 250 or 275 GeV for protons.
 > For other ion species, 110 and 166 may also be used.
 {: .callout}
 
-
 Once we have identified a specific dataset of interest, we can look at the files within it using:
 
 ```bash
-Example command
+rucio did content list scope:name
 ```
 
-as we saw in the last episode. We could download this file locally using
+and we can get locations of the files within the dataset via:
 
 ```bash
-Example command
+rucio replica list file --protocols root --pfns --rses isopenaccess scope:name_of_file
+rucio replica list file --protocols root --pfns --rses isopenaccess scope:name_of_Dataset
 ```
+as we saw in the last episode. We can get just the location of a specific file OR the location of all files within the dataest, depending upon which we specify. We could download this file locally using
+
+```bash
+xrdcp FILEPATH ./
+```
+
+where `FILEPATH` is the path to one specific file from the output of one of the rucio commands above.
 
 > ## `Exercise:`
 > Using the suggested tags, find the **latest** available datasets for:
@@ -85,46 +95,81 @@ To find files that meet their requirements they could utilise the following tags
 
 They may also want to use the `q2\_min` ad `q2\_max` tags, along with the `ion\_species` tags to narrow down to an even more specific subset of files. They may also want to analyse files with or without background enabled.
 
-As they want to process a large number of files, **it is unlikely (and not recommended) that they download a large number of files to process them locally**. Instead, they may want to stream their files directly in their analysis script. They could do this via
+As they want to process a large number of files, **it is unlikely (and not recommended) that they download a large number of files to process them locally**. Instead, they may want to stream their files directly in their analysis script. They could do this via:
 
 ```c++
-root based streaming example
-Full working script
+auto f = TFile::Open("FILEPATH");
+auto tree = f->Get<TTree>("events");
 ```
 
-or if they're using python -
+or if they're using python:
 
 ```python
-Python based streaming example
-Full working script
+import uproot
+import XRootD
+file_path = "FILEPATH"
+root_file = uproot.open(file_path)
 ```
 
-As they may wish to process a full dataset, they might want to feed their script a full list of files to stream and run. They could print the full list of files in a dataset via -
+As they may wish to process a full dataset, they might want to feed their script a full list of files to stream and run. They could print the full list of files in a dataset:
 
 ```bash
-Example command to pipe dataset list to a file
+rucio replica list file --protocols root --pfns --rses isopenaccess scope:name_of_Dataset > FileList
 ```
 
-> ## `Note:` 
-> We have limited this to only pipe 5 files in the dataset to our list.
-> Remove the `fragment` part of the command to instead print all lines.
-> Alternatively, edit this to be the number of lines that you want.
-{: .callout}
-
-This could then be processed in the script via -
+This could then be processed in the script:
 
 ```c++
-root based streaming example
-Full working script
+void FileListProcess(){
+  string line;
+  ifstream fstream ("FileList");
+  int FileCount = 0;
+  //TChain *AnalysisChain = new TChain("events"); // We could define a chain to process our files too and add them as we scan over our list
+  while(getline(fstream, line)){
+    if (FileCount > 5) continue; // Stop loop after 5 files, comment out to read full file
+    // Check file exists
+    TString tmpFile{line};
+    auto RootFile = TFile::Open(tmpFile);
+    if(!RootFile){ // Check file exists
+      cout << "File not found:"<<tmpFile << endl;
+      continue;
+    }
+    cout << "Found file - " << line << endl;
+    //AnalysisChain->Add(tmpFile) // Add to our chain if we want
+    FileCount++;
+  }
+}
 ```
 
-or if they're using python -
+or if they're using python:
 
 ```python
-Python based streaming example
-Full working script
+import ROOT
+import uproot
+import XRootD
+import awkward as ak
+
+Files=[]
+
+with open('FileList', 'r') as file:
+    lines_list = file.readlines()
+    for line in lines_list[:5]: # Read only lines 0:5 - remove [:5] to read all or change 5 to N where N is the number of lines you want
+        file_path = line.rstrip() # rstrip to remove trailing white space/new lines
+        try:
+            with uproot.open(file_path) as file:
+                Files.append(file_path) # Add file path to array
+                print("Found file - ", file_path, "and appended to list for processing.")
+        except Exception as e:
+            print(f"Could not open file: {e}")
+        
+# Use the uproot iterate method to process our list of files - See https://uproot.readthedocs.io/en/stable/uproot.behaviors.TBranch.iterate.html
+#for chunk in uproot.iterate({f: "events" for f in Files}, expressions=["MCParticles.PDG"]): # Open files in array f and process events tree with branches specified
+    # Process each chunk - Do something
+    # print(ak.type(chunk))
 ```
 
+Note that we have restricted these examples to only print out the first five files in the list we created. We can comment out the lines noted to process the full list (or adjust the cutoff value in the condition to process a different number).
+
 > ## `Exercise:`
 > Using the suggested tags, find the **latest** available dataset for:
 > - Deeply Virtual Compton Scattering (DVCS) events from the EpIC event generator for 10 GeV electrons colliding with 130 GeV protons *without* background included
@@ -133,14 +178,20 @@ Full working script
 > 3. Stream **five** of the files in this dataset in a script, check the total number of events contained in all five files.
 {: .challenge}
 
-## Detector Designer/Optimiser
+## Detector Designer/Optimiser, Algorithm/Reconstruction Development
+
+Discussion of use case based upon SIM data - To be added soon.
+
+## Conclusion and Comments
+
+That wraps up our introduction to using Rucio and some example use cases and scenarios.
 
-Discussion of use case based upon SIM data
+New tags may be added in the future. We're welcome to take on board any suggestions or changes as we roll out Rucio and it becomes more widely used. Get in touch via:
 
-## Algorithm/Reconstruction Development
+`stephen.kay@york.ac.uk` 
 
-Discussion of use case based upon SIM data and tags - merge with previous?
+or on Mattermost with suggestions, comments and feedback.
 
-## General Comments
+Remember to consider whether you need full datasets before downloading them and keep an eye on whether your files have multiple open access copies when making file lists.
 
-Some general comments and info. Pointers, things to avoid or recommendations etc.
+Also, if you find any nice tricks or develop short scripts (maybe one which makes a file list for the latest version of a dataset based upon inputs?) then feel free to share them too!