Skip to content

Commit dd9ba9c

Browse files
Release v0.9.3: Add PBS environment variable support and update documentation
1 parent cb31472 commit dd9ba9c

176 files changed

Lines changed: 1215 additions & 10 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

Cargo.lock

Lines changed: 10 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "fbqueue"
3-
version = "0.9.2"
3+
version = "0.9.3"
44
edition = "2021"
55

66
[dependencies]

MANUAL.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -181,11 +181,27 @@ FBQueue automatically maps embedded script directives to internal job parameters
181181
| **Dependencies** | `#$ -hold_jid 123` | **`depend`** |
182182
| **Walltime** | `#$ -l h_rt=01:30:00` | **`walltime`** |
183183

184-
## 9. Roadmap
184+
## 9. Job Environment Variables (PBS Compatibility)
185+
186+
FBQueue automatically injects environment variables into the job execution context to ensure compatibility with scripts designed for traditional HPC schedulers like PBS.
187+
188+
| Variable | Description |
189+
| :--- | :--- |
190+
| **`$PBS_JOBID`** | The unique job ID assigned by FBQueue (e.g., `147`). |
191+
| **`$PBS_JOBNAME`** | The name of the job (defaults to the script path if not specified). |
192+
| **`$PBS_O_WORKDIR`** | The absolute path to the directory where the job was submitted. |
193+
| **`$PBS_QUEUE`** | The name of the queue where the job is executing. |
194+
| **`$PBS_NODEFILE`** | Path to a temporary file containing the execution hostname. Deleted after job completion. |
195+
| **`$PBS_ENVIRONMENT`** | Set to `PBS_BATCH` to indicate the job is running in a batch environment. |
196+
| **`$PBS_O_HOST`** | The hostname of the machine where the job was submitted. |
197+
| **`$PBS_O_LOGNAME`** | The username of the user who submitted the job. |
198+
199+
## 10. Roadmap
185200

186201
* [x] **Background Archiving**: Automatic `tar.gz` compression of old job records during idle periods.
187-
* * **Flexible Directory Discovery**: Support for recursive parent directory search for `.fbqueue` configuration.
188-
* * **Detailed Analytics**: Advanced statistics and status summaries via `fbqueue stat`.
202+
* [x] **Full PBS Compatibility**: Support for standard environment variables and script directives.
203+
* [x] **Windows Native Execution**: Support for `.ps1` and `.bat` scripts with automatic path handling.
204+
* [ ] **Detailed Analytics**: Advanced statistics and status summaries via `fbqueue stat`.
189205

190206
---
191207
### Author

PBS_COMPATIBILITY.md

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,22 @@ Job id Name User Time Use S Queue
8383

8484
---
8585

86-
## 4. Why use FBQueue for PBS Workflows?
86+
## 4. Standard PBS Environment Variables
87+
88+
FBQueue injects standard PBS environment variables into your job's execution context. This allows your scripts to remain portable across different clusters.
89+
90+
| Variable | Description |
91+
| :--- | :--- |
92+
| **`$PBS_JOBID`** | The unique job ID assigned by FBQueue. |
93+
| **`$PBS_O_WORKDIR`** | The directory from which the job was submitted. |
94+
| **`$PBS_NODEFILE`** | Path to a file containing the hostname of the execution node. |
95+
| **`$PBS_O_HOST`** | The hostname of the submission machine. |
96+
| **`$PBS_O_LOGNAME`** | The username of the user who submitted the job. |
97+
| **`$PBS_ENVIRONMENT`** | Set to `PBS_BATCH` to indicate a batch job environment. |
98+
99+
---
100+
101+
## 5. Why use FBQueue for PBS Workflows?
87102

88103
- **Personal Sandbox**: Run your PBS scripts on your local workstation or a shared server where you don't have administrative rights to a full PBS cluster.
89104
- **Portability**: Move your research scripts between a massive supercomputer and your local laptop without changing a single line of the `#PBS` directives.

README.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Download the pre-built static binary and place it in your `$PATH`:
3131

3232
```bash
3333
# 1. Download
34-
wget https://github.com/ForblazeProject/fbqueue/releases/download/v0.9.2/fbqueue-linux-x64.tar.gz
34+
wget https://github.com/ForblazeProject/fbqueue/releases/download/v0.9.3/fbqueue-linux-x64.tar.gz
3535

3636
# 2. Extract and move
3737
tar -xzvf fbqueue-linux-x64.tar.gz
@@ -81,6 +81,11 @@ fbqueue sub ./calc.sh
8181

8282
## 📜 Change Log
8383

84+
### v0.9.3
85+
- **Full PBS Environment Emulation**: Jobs now automatically define standard PBS variables like `$PBS_JOBID`, `$PBS_O_WORKDIR`, `$PBS_O_HOST`, and `$PBS_O_LOGNAME`.
86+
- **Nodefile Support**: Dynamically generates and cleans up `$PBS_NODEFILE` for compatibility with scripts expecting a list of execution nodes.
87+
- **Improved Metadata Accuracy**: Enhanced hostname and user detection across both Linux and Windows for more reliable job logging.
88+
8489
### v0.9.2
8590
- **Fixed Process Leak**: Improved job cancellation logic using process groups (`setsid`) on Unix and tree termination (`/T`) on Windows. Subprocesses spawned by job scripts are now correctly terminated.
8691
- **Robust Job Filtering**: `qstat <jobID>` (and `stat <jobID>`) now works reliably even if the job has already finished and moved to history. It also correctly handles the `.master` suffix.

src/daemon.rs

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,8 @@ pub fn run_daemon() {
6969
for entry in entries.filter_map(|e| e.ok()) {
7070
let id = entry.file_name().to_str().unwrap().trim_end_matches(".job").to_string();
7171
if let Some(pos) = running_jobs.iter().position(|(rid, _, _, _, _, _)| rid == &id) {
72-
let (_, mut child, _, _, _, _) = running_jobs.remove(pos);
72+
let (rid, mut child, _, _, _, _) = running_jobs.remove(pos);
73+
let _ = fs::remove_file(fbq_dir.join("run").join(format!("nodefile.{}", rid)));
7374
kill_process_tree(&mut child);
7475
if let Ok(mut f) = fs::OpenOptions::new().append(true).open(fbq_dir.join("queue/running").join(entry.file_name())) {
7576
let _ = writeln!(f, "status: CANCELLED");
@@ -87,6 +88,7 @@ pub fn run_daemon() {
8788
if now - *start_at > wt {
8889
kill_process_tree(child);
8990
let (job_id, _, _, _, _, _) = running_jobs.remove(i);
91+
let _ = fs::remove_file(fbq_dir.join("run").join(format!("nodefile.{}", job_id)));
9092
let fname = format!("{}.job", job_id);
9193
let job_path = fbq_dir.join("queue/running").join(&fname);
9294
if let Ok(mut f) = fs::OpenOptions::new().append(true).open(&job_path) {
@@ -103,6 +105,8 @@ pub fn run_daemon() {
103105
while i < running_jobs.len() {
104106
if let Ok(Some(status)) = running_jobs[i].1.try_wait() {
105107
let (id, _, _, _, _, _) = running_jobs.remove(i);
108+
let nodefile_path = fbq_dir.join("run").join(format!("nodefile.{}", id));
109+
let _ = fs::remove_file(nodefile_path);
106110
let fname = format!("{}.job", id);
107111
let rpath = fbq_dir.join("queue/running").join(&fname);
108112
let exit_code = status.code().unwrap_or(-1);
@@ -202,7 +206,17 @@ pub fn run_daemon() {
202206
process::Command::new(&j.cmd)
203207
};
204208

205-
child_cmd.args(&j.args).current_dir(&j.cwd).envs(j.envs).stdout(out_f).stderr(err_f);
209+
let nodefile_path = fbq_dir.join("run").join(format!("nodefile.{}", j.id));
210+
let hostname = process::Command::new("hostname").output().map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_else(|_| "localhost".to_string());
211+
let _ = fs::write(&nodefile_path, format!("{}\n", hostname));
212+
213+
child_cmd.args(&j.args).current_dir(&j.cwd).envs(j.envs)
214+
.env("PBS_JOBID", &j.id)
215+
.env("PBS_JOBNAME", &j.name)
216+
.env("PBS_QUEUE", &j.queue)
217+
.env("PBS_NODEFILE", nodefile_path.display().to_string())
218+
.env("PBS_ENVIRONMENT", "PBS_BATCH")
219+
.stdout(out_f).stderr(err_f);
206220

207221
if let Ok(child) = child_cmd.spawn() {
208222
running_jobs.push((j.id, child, j.cost, j.queue, now, j.walltime));

src/handlers.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,7 @@ pub fn handle_daemon(args: &[String]) {
104104
if let Ok(pid_str) = fs::read_to_string(&lock_file) {
105105
if let Ok(pid) = pid_str.trim().parse::<u32>() {
106106
#[cfg(unix)] { process::Command::new("kill").arg(pid.to_string()).status().ok(); }
107-
#[cfg(windows)] { process::Command::new("taskkill").arg("/PID").arg(pid.to_string()).arg("/F").status().ok(); }
107+
#[cfg(windows)] { process::Command::new("taskkill").arg("/PID").arg(pid.to_string()).arg("/F").arg("/T").status().ok(); }
108108
}
109109
}
110110
let _ = fs::remove_file(lock_file);

src/job.rs

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
use std::env;
22
use std::fs;
3+
use std::process;
34
use std::io::{self, BufRead};
45
use std::path::{Path, PathBuf};
56
use crate::utils;
@@ -162,6 +163,7 @@ pub fn submit_job(cmd_tmpl: &str, args_tmpl: &[String], cwd: &Path, val: Option<
162163
let job_file_path = fbq_dir.join("queue/new").join(format!("{}.job", job_id));
163164

164165
let current_user = env::var("USER").or_else(|_| env::var("USERNAME")).unwrap_or_else(|_| "unknown".to_string());
166+
let hostname = process::Command::new("hostname").output().map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()).unwrap_or_else(|_| "localhost".to_string());
165167

166168
let mut content = format!("id: {}\nname: {}\ncmd: {}\ncost: {}\nuser: {}\nqueue: {}\n", job_id, final_name, cmd, final_cost, current_user, final_queue);
167169
if let Some(o) = final_out { content.push_str(&format!("stdout: {}\n", o)); }
@@ -172,6 +174,12 @@ pub fn submit_job(cmd_tmpl: &str, args_tmpl: &[String], cwd: &Path, val: Option<
172174

173175
for arg in job_args { content.push_str(&format!("arg: {}\n", arg)); }
174176
content.push_str(&format!("cwd: {}\n", cwd.display()));
177+
178+
// PBS compatibility environment variables (Submission time)
179+
content.push_str(&format!("env: PBS_O_WORKDIR={}\n", cwd.display()));
180+
content.push_str(&format!("env: PBS_O_LOGNAME={}\n", current_user));
181+
content.push_str(&format!("env: PBS_O_HOST={}\n", hostname));
182+
175183
for (key, val) in env::vars() { content.push_str(&format!("env: {}={}\n", key, val)); }
176184

177185
fs::write(&job_file_path, content).expect("Failed to write job file");

tests/tmp/case_1/config

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
capacity: 8
2+
default_queue: batch
3+
queue: batch
4+
priority: 10

tests/tmp/case_1/queue/done/1.job

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
id: 1
2+
name: echo
3+
cmd: echo
4+
cost: 1
5+
user: wsl
6+
queue: batch
7+
arg: Hello FBQueue
8+
cwd: /home/wsl/dev/queue/fbqueue/tests/tmp/work_1
9+
env: SHELL=/bin/bash
10+
env: WSL2_GUI_APPS_ENABLED=1
11+
env: TERM_PROGRAM_VERSION=3.2a
12+
env: WSL_DISTRO_NAME=Ubuntu
13+
env: TMUX=/tmp/tmux-1000/default,1066,0
14+
env: FBQUEUE_DIR=/home/wsl/dev/queue/fbqueue/tests/tmp/case_1
15+
env: NAME=Lenovo2
16+
env: PWD=/home/wsl/dev/queue/fbqueue/tests/tmp/work_1
17+
env: LOGNAME=wsl
18+
env: JIRA_API_TOKEN=ATATT3xFfGF0CBC73AekZT98GiCQlo0ERk36caSu8WrwQtM7vDeEFoVj9FaD5noJaAbjBwBmhxxw4SbeRKLMniw-zXhz9aGIu9d0yWiLn1OYk3h622z3LNDZDks8evpKaYHJ60aQoJp4p7KRu3KEebqqY16jVrqc-5aa8jFHc9QgBRBkqzMM6rE=C3808572
19+
env: MOTD_SHOWN=update-motd
20+
env: HOME=/home/wsl
21+
env: MSI2LMP_LIBRARY=/usr/share/lammps/frc_files
22+
env: LANG=C.UTF-8
23+
env: WSL_INTEROP=/run/WSL/405_interop
24+
env: LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
25+
env: WAYLAND_DISPLAY=wayland-0
26+
env: LAMMPS_POTENTIALS=/usr/share/lammps/potentials
27+
env: LESSCLOSE=/usr/bin/lesspipe %s %s
28+
env: TERM=xterm-256color
29+
env: LESSOPEN=| /usr/bin/lesspipe %s
30+
env: USER=wsl
31+
env: GIT_PAGER=cat
32+
env: TMUX_PANE=%4
33+
env: DISPLAY=:0
34+
env: SHLVL=4
35+
env: PAGER=cat
36+
env: XDG_RUNTIME_DIR=/run/user/1000/
37+
env: GEMINI_CLI=1
38+
env: WSLENV=
39+
env: GEMINI_CLI_NO_RELAUNCH=true
40+
env: XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
41+
env: PATH=/home/wsl/.local/bin:/home/wsl/.local/qe/bin:/home/wsl/.local/bin:/home/wsl/.local/bin:/home/wsl/.cargo/bin:/home/wsl/.local/qe/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Microsoft MPI/Bin/:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/Go/bin:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/CMake/bin:/mnt/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit/:/mnt/c/Program Files/nodejs/:/mnt/c/Program Files/Git/cmd:/mnt/c/Users/sat_i/.cargo/bin:/mnt/c/Python311:/mnt/c/Python311/Scripts:/mnt/c/Users/sat_i/AppData/Local/Microsoft/WindowsApps:/mnt/c/Program Files (x86)/Vim/vim90:/mnt/c/BURAI1.3.2_Windows/exec.WIN/qe:/mnt/c/Program Files/mopac-22.1.0-win/bin:/mnt/c/Program Files/MOPAC/bin:/mnt/c/Users/sat_i/AppData/Roaming/Python/Python311/Scripts:/mnt/c/Users/sat_i/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/sat_i/go/bin:/mnt/c/Qt/6.9.2/msvc2022_64/bin:/mnt/c/Program Files/VTK/bin:/mnt/c/Program Files/LAMMPS 64-bit 22Jul2025 with GUI/bin:/mnt/c/Users/sat_i/AppData/Roaming/npm:/mnt/c/Program Files/jira_1.7.0_windows_x86_64/bin:/snap/bin:/home/wsl/installer/jira_1.7.0_linux_x86_64/bin:/usr/local/go/bin:/home/wsl/installer/jira_1.7.0_linux_x86_64/bin:/usr/local/go/bin
42+
env: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
43+
env: HOSTTYPE=x86_64
44+
env: PULSE_SERVER=unix:/mnt/wslg/PulseServer
45+
env: OLDPWD=/home/wsl/dev/queue/fbqueue
46+
env: TERM_PROGRAM=tmux
47+
env: _=./qsub
48+
start_at: 1772025603
49+
end_at: 1772025604
50+
exit_code: 0
51+
status: DONE

0 commit comments

Comments
 (0)