Skip to content

Commit 8b59a5b

Browse files
committed
feat: implement standardized data pooling and external metadata resolution
1 parent 72f5477 commit 8b59a5b

14 files changed

Lines changed: 501 additions & 50 deletions

File tree

.logs/start/api-server.log

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
INFO: Started server process [1802573]
1+
INFO: Started server process [277692]
22
INFO: Waiting for application startup.
33
INFO: Application startup complete.
44
INFO: Uvicorn running on http://0.0.0.0:4242 (Press CTRL+C to quit)
55
================================================================================
66
EHTOOL ROUTER MODULE LOADED - VERSION: DEBUG v2
77
================================================================================
8-
INFO: 127.0.0.1:52216 - "GET /health HTTP/1.1" 200 OK
9-
INFO: 127.0.0.1:39000 - "GET /api/pm/data HTTP/1.1" 200 OK
10-
INFO: 127.0.0.1:38992 - "GET /files?parent=root HTTP/1.1" 200 OK
11-
INFO: 127.0.0.1:39000 - "GET /api/pm/data HTTP/1.1" 200 OK
12-
INFO: 127.0.0.1:55914 - "OPTIONS /api/pm/login HTTP/1.1" 200 OK
13-
INFO: 127.0.0.1:55930 - "POST /api/pm/login HTTP/1.1" 200 OK
8+
INFO: 127.0.0.1:36132 - "GET /health HTTP/1.1" 200 OK
9+
INFO: 127.0.0.1:36392 - "GET /api/pm/data HTTP/1.1" 200 OK
10+
INFO: 127.0.0.1:36394 - "GET /api/pm/data HTTP/1.1" 200 OK
11+
INFO: 127.0.0.1:36386 - "GET /files?parent=root HTTP/1.1" 200 OK
12+
INFO: 127.0.0.1:53930 - "GET /api/pm/volumes?page=1&page_size=50 HTTP/1.1" 200 OK
13+
INFO: 127.0.0.1:53930 - "GET /api/pm/volumes?page=1&page_size=50 HTTP/1.1" 200 OK
1414
INFO: Shutting down
1515
INFO: Waiting for application shutdown.
1616
INFO: Application shutdown complete.
17-
INFO: Finished server process [1802573]
17+
INFO: Finished server process [277692]

.logs/start/data-server.log

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
127.0.0.1 - - [26/Mar/2026 14:48:47] "GET / HTTP/1.1" 200 -
1+
127.0.0.1 - - [10/Apr/2026 14:25:52] "GET / HTTP/1.1" 200 -

.logs/start/pytc-server.log

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
INFO: Started server process [1802650]
1+
INFO: Started server process [277796]
22
INFO: Waiting for application startup.
33
INFO: Application startup complete.
44
INFO: Uvicorn running on http://0.0.0.0:4243 (Press CTRL+C to quit)
@@ -14,8 +14,8 @@ Working directory: /home/sam/Workshop/pytc-client
1414
SERVER_PYTC: Starting Uvicorn server on port 4243...
1515
================================================================================
1616

17-
INFO: 127.0.0.1:50152 - "GET /hello HTTP/1.1" 200 OK
17+
INFO: 127.0.0.1:43042 - "GET /hello HTTP/1.1" 200 OK
1818
INFO: Shutting down
1919
INFO: Waiting for application shutdown.
2020
INFO: Application shutdown complete.
21-
INFO: Finished server process [1802650]
21+
INFO: Finished server process [277796]

.logs/start/react-build.log

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -55,16 +55,15 @@ src/views/project-manager/QuotaManagement.js
5555
Line 98:5: 'activeWorker' is assigned a value but never used no-unused-vars
5656

5757
src/views/project-manager/VolumeTracker.js
58-
Line 16:3: 'Popconfirm' is defined but never used no-unused-vars
59-
Line 127:6: React Hook useMemo has an unnecessary dependency: 'isAdmin'. Either exclude it or remove the dependency array react-hooks/exhaustive-deps
60-
Line 184:15: 'cfg' is assigned a value but never used no-unused-vars
58+
Line 126:6: React Hook useMemo has an unnecessary dependency: 'isAdmin'. Either exclude it or remove the dependency array react-hooks/exhaustive-deps
59+
Line 183:15: 'cfg' is assigned a value but never used no-unused-vars
6160

6261
Search for the keywords to learn more about each warning.
6362
To ignore, add // eslint-disable-next-line to the line before.
6463

6564
File sizes after gzip:
6665

67-
553.16 kB build/static/js/main.be80df26.js
66+
553.67 kB build/static/js/main.450b07d9.js
6867
465 B build/static/css/main.40821645.css
6968

7069
The bundle size is significantly larger than recommended.

.logs/start/react-dev.log

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,17 @@
33
> react-scripts start
44

55
[baseline-browser-mapping] The data in this module is over two months old. To ensure accurate Baseline data, please update: `npm i baseline-browser-mapping@latest -D`
6-
(node:1802959) [DEP_WEBPACK_DEV_SERVER_ON_AFTER_SETUP_MIDDLEWARE] DeprecationWarning: 'onAfterSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
6+
(node:278495) [DEP_WEBPACK_DEV_SERVER_ON_AFTER_SETUP_MIDDLEWARE] DeprecationWarning: 'onAfterSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
77
(Use `node --trace-deprecation ...` to show where the warning was created)
8-
(node:1802959) [DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning: 'onBeforeSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
8+
(node:278495) [DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning: 'onBeforeSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
99
Starting the development server...
1010

1111
Compiled successfully!
1212

1313
You can now view PyTC Client in the browser.
1414

1515
Local: http://localhost:3000
16-
On Your Network: http://10.101.188.107:3000
16+
On Your Network: http://10.0.0.38:3000
1717

1818
Note that the development build is not optimized.
1919
To create a production build, use npm run build.

README_DEV.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# PyTC Client — Developer Guide (External Data)
2+
3+
This guide explains how to configure the PyTC client to work with externalized metadata and data storage.
4+
5+
## Environment Variables
6+
7+
To use external data, set the following environment variables before starting the backend:
8+
9+
- `PROJECT_METADATA_JSON`: Absolute path to the `project_manager_data.json` file.
10+
- `DATA_ROOT_EM`: Root directory containing the H5 volume files.
11+
12+
### Example Setup (Bash)
13+
14+
```bash
15+
export PROJECT_METADATA_JSON="/home/sam/Workshop/Pytc-data/im_64nm/server_api/data_store/project_manager_data.json"
16+
export DATA_ROOT_EM="/home/sam/Workshop/Pytc-data/im_64nm/im_64nm/"
17+
18+
./scripts/start.sh
19+
```
20+
21+
## Migration Utility (Kanchan's Logic)
22+
23+
To assign volumes in a specific subdirectory to a user (e.g., "kanchan"), use the provided migration script:
24+
25+
```bash
26+
python3 scripts/migrate_metadata.py /path/to/project_manager_data.json --subdir "1440/"
27+
```
28+
29+
This will find all volumes where `rel_path` contains "1440/" and set their assignee to "kanchan".
30+
31+
## Standardized Data Access
32+
33+
The frontend now utilizes a standardized `DataReaderService` (`client/src/services/data_reader.js`).
34+
UI components should only reference volumes by their `taskId` (the volume ID).
35+
36+
### Linking Metadata Dynamically
37+
38+
You can also link to an external metadata file at runtime via the backend API:
39+
40+
```bash
41+
curl -X POST http://localhost:4242/api/pm/data/link \
42+
-H "Content-Type: application/json" \
43+
-d '{"path": "/home/sam/Workshop/Pytc-data/im_64nm/project_manager_data_new.json"}'
44+
```

client/src/contexts/ProjectManagerContext.js

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import React, {
99
import { message, Spin } from "antd";
1010
import { getPMData, savePMData, resetPMData } from "../api";
1111
import { apiClient } from "../api";
12+
import { dataReader } from "../services/data_reader";
1213

1314
// ── Context ───────────────────────────────────────────────────────────────────
1415

@@ -148,11 +149,30 @@ export function ProjectManagerProvider({ children }) {
148149
}
149150
}, []);
150151

152+
const ingestData = useCallback(async () => {
153+
setLoading(true);
154+
try {
155+
const res = await apiClient.post("/api/pm/data/ingest");
156+
if (res.data.ok) {
157+
setPmState(res.data.data);
158+
message.success(
159+
`Ingested ${res.data.data.volumes?.length || 0} volumes.`,
160+
);
161+
}
162+
} catch (err) {
163+
console.error("[PM] Ingestion failed:", err);
164+
message.error(
165+
err.response?.data?.detail || "Failed to ingest data from storage.",
166+
);
167+
} finally {
168+
setLoading(false);
169+
}
170+
}, []);
171+
151172
// ── Volume helpers ───────────────────────────────────────────────────────
152173
const getVolumes = useCallback(async (params = {}) => {
153174
try {
154-
const res = await apiClient.get("/api/pm/volumes", { params });
155-
return res.data;
175+
return await dataReader.getPooledVolumes(params);
156176
} catch (err) {
157177
message.error("Failed to load volumes from server.");
158178
throw err;
@@ -161,15 +181,13 @@ export function ProjectManagerProvider({ children }) {
161181

162182
const updateVolumeStatus = useCallback(async (volumeId, status) => {
163183
try {
164-
const res = await apiClient.patch(`/api/pm/volumes/${volumeId}`, {
165-
status,
166-
});
167-
if (res.data?.global_progress) {
184+
const res = await dataReader.updateStatus(volumeId, status);
185+
if (res?.global_progress) {
168186
setPmState((prev) =>
169-
prev ? { ...prev, global_progress: res.data.global_progress } : prev,
187+
prev ? { ...prev, global_progress: res.global_progress } : prev,
170188
);
171189
}
172-
return res.data;
190+
return res;
173191
} catch (err) {
174192
message.error("Failed to update volume status.");
175193
throw err;
@@ -277,6 +295,7 @@ export function ProjectManagerProvider({ children }) {
277295

278296
// Actions
279297
resetData,
298+
ingestData,
280299
getVolumes,
281300
updateVolumeStatus,
282301
}}

client/src/services/data_reader.js

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
/**
2+
* Data Reader Service
3+
*
4+
* Provides a standardized interface for accessing project volumes
5+
* by taskId, abstracting away file paths and backend specifics.
6+
*/
7+
8+
import axios from 'axios';
9+
10+
// Backend configuration
11+
const API_BASE = process.env.REACT_APP_API_URL || 'http://localhost:4242/api/pm';
12+
13+
class DataReaderService {
14+
/**
15+
* Fetch all pooled volumes (with pagination handled internally if needed)
16+
* The frontend "pools" the list from the active metadata JSON.
17+
*/
18+
async getPooledVolumes(params = {}) {
19+
try {
20+
const response = await axios.get(`${API_BASE}/volumes`, { params });
21+
return response.data; // { total, page, items, ... }
22+
} catch (error) {
23+
console.error('Error fetching pooled volumes:', error);
24+
throw error;
25+
}
26+
}
27+
28+
/**
29+
* Get metadata for a specific volume by taskId
30+
*/
31+
async getVolumeByTaskId(taskId) {
32+
try {
33+
// Since the backend uses id as taskId (e.g. vol_001_em.h5)
34+
// we can just use the volumes endpoint with filtering if supported,
35+
// or we could add a dedicated GET /volumes/{id} if needed.
36+
// For now, we'll assume the frontend already has the list or we fetch one.
37+
const response = await axios.get(`${API_BASE}/volumes`, { params: { id: taskId, page_size: 1 } });
38+
return response.data.items[0] || null;
39+
} catch (error) {
40+
console.error(`Error fetching volume ${taskId}:`, error);
41+
throw error;
42+
}
43+
}
44+
45+
/**
46+
* Update volume status by taskId
47+
*/
48+
async updateStatus(taskId, status) {
49+
try {
50+
const response = await axios.patch(`${API_BASE}/volumes/${taskId}`, { status });
51+
return response.data;
52+
} catch (error) {
53+
console.error(`Error updating volume ${taskId}:`, error);
54+
throw error;
55+
}
56+
}
57+
58+
/**
59+
* Bulk link to an external metadata JSON
60+
*/
61+
async linkExternalMetadata(path) {
62+
try {
63+
const response = await axios.post(`${API_BASE}/data/link`, { path });
64+
return response.data;
65+
} catch (error) {
66+
console.error('Error linking external metadata:', error);
67+
throw error;
68+
}
69+
}
70+
}
71+
72+
export const dataReader = new DataReaderService();
73+
export default dataReader;

client/src/views/project-manager/ProjectManagerLogin.js

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ export default function ProjectManagerLogin() {
107107

108108
<Divider plain>
109109
<Text type="secondary" style={{ fontSize: 12 }}>
110-
Credentials Template
110+
Login Credentials
111111
</Text>
112112
</Divider>
113113

@@ -121,10 +121,10 @@ export default function ProjectManagerLogin() {
121121
>
122122
<Space direction="vertical" size={2}>
123123
<Text type="secondary">
124-
Admin: <Text code>admin / admin123</Text>
124+
Admin Access: <Text code>admin / admin123</Text>
125125
</Text>
126126
<Text type="secondary">
127-
Worker: <Text code>alex / alex123</Text>
127+
Annotator Access: <Text code>alex / alex123</Text>
128128
</Text>
129129
</Space>
130130
</div>

client/src/views/project-manager/VolumeTracker.js

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,10 @@ function VolumeTracker() {
5959
isWorker,
6060
activeWorker,
6161
globalProgress,
62+
getVolumes,
6263
updateVolumeStatus,
64+
ingestData,
65+
loading: pmLoading,
6366
} = useProjectManager();
6467

6568
const [volumes, setVolumes] = useState([]);
@@ -86,20 +89,16 @@ function VolumeTracker() {
8689
const params = { page, page_size: pageSize };
8790
if (workerFilter) params.assignee = workerFilter;
8891
if (statusFilter) params.status = statusFilter;
89-
const res = await fetch(
90-
`http://localhost:4242/api/pm/volumes?${new URLSearchParams(params)}`,
91-
{ credentials: "include" },
92-
);
93-
if (!res.ok) throw new Error(`HTTP ${res.status}`);
94-
const data = await res.json();
92+
93+
const data = await getVolumes(params);
9594
setVolumes(data.items ?? []);
9695
setTotal(data.total ?? 0);
9796
} catch (err) {
9897
console.error("[VolumeTracker] fetch failed:", err);
9998
} finally {
10099
setLoadingVols(false);
101100
}
102-
}, [page, pageSize, workerFilter, statusFilter]);
101+
}, [page, pageSize, workerFilter, statusFilter, getVolumes]);
103102

104103
useEffect(() => {
105104
fetchVolumes();
@@ -323,6 +322,29 @@ function VolumeTracker() {
323322
}
324323
extra={
325324
<Space>
325+
{/* Sync button — admin only */}
326+
{isAdmin && (
327+
<Popconfirm
328+
title="Refresh volume list?"
329+
description="This will scan the physical storage for new .h5 files."
330+
onConfirm={async () => {
331+
await ingestData();
332+
fetchVolumes();
333+
}}
334+
okText="Scan"
335+
cancelText="Cancel"
336+
>
337+
<Button
338+
size="small"
339+
type="primary"
340+
icon={<SyncOutlined />}
341+
loading={pmLoading}
342+
>
343+
Sync with Storage
344+
</Button>
345+
</Popconfirm>
346+
)}
347+
326348
{/* Assignee filter — admin only */}
327349
{isAdmin && (
328350
<Select

0 commit comments

Comments
 (0)