|
|
@@ -1,6 +1,6 @@
|
|
|
# CLAUDE.md — frontend
|
|
|
|
|
|
-Angular 20 SPA. Multi-tenant dashboard with a lazy-loaded `src.palm.vision` sub-app for MPOB-standard palm oil fruit bunch (FFB) ripeness detection. Three inference engines: browser ONNX WASM, NestJS remote (WebSocket), and n8n edge passthrough.
|
|
|
+Angular 21 SPA. Multi-tenant dashboard with a lazy-loaded `src.palm.vision` sub-app for MPOB-standard palm oil fruit bunch (FFB) ripeness detection. Three inference engines: browser ONNX WASM, browser TFLite WASM, and NestJS remote (WebSocket).
|
|
|
|
|
|
## Commands
|
|
|
|
|
|
@@ -20,9 +20,9 @@ npm run clean # Clear Angular + npm caches
|
|
|
|
|
|
### Application Shell (`src/app/`)
|
|
|
|
|
|
-- **`app.config.ts`** — `ApplicationConfig`: bootstraps NGXS root store (`withNgxsStoragePlugin`, `withNgxsReduxDevtoolsPlugin`), Angular Router (hash location), HTTP client, service worker.
|
|
|
+- **`app.config.ts`** — `ApplicationConfig`: bootstraps NGXS root store (`withNgxsStoragePlugin({keys:'*'})`, `withNgxsReduxDevtoolsPlugin`), Angular Router (hash location, `onSameUrlNavigation:'reload'`), HTTP client, service worker (registered immediately).
|
|
|
- **`app.routes.ts`** — Root routes: `dashboard`, `auth`, `leave`, `tender`, `src.palm.vision` (all lazy-loaded except dashboard).
|
|
|
-- **`app.component.ts`** — Root component: session timeout, PWA install prompt, theme management.
|
|
|
+- **`app.component.ts`** — Root component: 30-minute session timeout, PWA install prompt, theme management (light/dark + blue/pink), multi-language (en_US, ms_MY, zh_Hans), maintenance mode alert.
|
|
|
|
|
|
### PalmVision Sub-App (`src/src.palm.vision/`)
|
|
|
|
|
|
@@ -39,10 +39,11 @@ Lazy-loaded at `/src.palm.vision`. Provides its own NGXS `VisionState` via `prov
|
|
|
#### Services (`src/src.palm.vision/services/`)
|
|
|
|
|
|
**`InferenceService`** — Local WASM inference:
|
|
|
-- `analyze(file, batchId?)` → `Observable<InferenceFrame>` — preprocesses image → posts tensor to Web Worker → emits result
|
|
|
-- `preprocessImage(dataUrl)` — resizes to 640×640, converts RGBA→CHW, normalizes `[0.0, 1.0]`
|
|
|
-- `results$: Subject<InferenceFrame>` — all completed frames
|
|
|
-- `queueDepth$: BehaviorSubject<number>` — pending worker frames
|
|
|
+- `analyze(file, batchId?, mode)` → `Observable<InferenceFrame>` — preprocesses image → posts tensor to Web Worker → emits result
|
|
|
+- `preprocessImage(dataUrl)` — resizes to 640×640 on canvas, converts RGBA→CHW, normalizes `[0.0, 1.0]` → `Float32Array [1, 3, 640, 640]`
|
|
|
+- `results$: Subject<InferenceFrame>` — hot stream of all completed frames
|
|
|
+- `queueDepth$: BehaviorSubject<number>` — pending worker frame count
|
|
|
+- `pendingMap: Map<string, fn>` — in-memory frameId → observer resolver
|
|
|
|
|
|
**`RemoteInferenceService`** — WebSocket bridge to NestJS backend:
|
|
|
- `analyze(file, sourceLabel?, batchId?)` → `Observable<InferenceFrame>` — sends `PalmVision:analyze` via FIS envelope
|
|
|
@@ -52,6 +53,8 @@ Lazy-loaded at `/src.palm.vision`. Provides its own NGXS `VisionState` via `prov
|
|
|
- `getImage(archiveId)` → `Observable<{ archiveId, image_data }>` — `PalmHistory:GetImage`
|
|
|
- `saveExternalResult(payload)` → `Observable<any>` — `PalmHistory:SaveExternalResult`
|
|
|
|
|
|
+All calls are wrapped in the FIS envelope and routed via `DpService.stream()`.
|
|
|
+
|
|
|
Connection config loaded from `src/src.palm.vision/config/config.json`:
|
|
|
```json
|
|
|
{ "connection": { "uacp": "http://localhost:3000", "uacp_ws": "ws://localhost:3000/socket.io", "uacpEmulation": "on" } }
|
|
|
@@ -61,7 +64,14 @@ Connection config loaded from `src/src.palm.vision/config/config.json`:
|
|
|
|
|
|
**`VisionStateModel`:**
|
|
|
```typescript
|
|
|
-{ items: any[]; loading: boolean; expandedBatchIds: string[]; currentInference: InferenceFrame | null }
|
|
|
+{
|
|
|
+ items: any[]; // History records from server
|
|
|
+ loading: boolean;
|
|
|
+ expandedBatchIds: string[]; // Expanded accordion groups
|
|
|
+ currentInference: InferenceFrame | null;
|
|
|
+ batchFrames: InferenceFrame[]; // Current batch results
|
|
|
+ selectedFrameIndex: number | null;
|
|
|
+}
|
|
|
```
|
|
|
|
|
|
**Selectors:** `VisionState.items`, `VisionState.loading`, `VisionState.expandedBatchIds`, `VisionState.currentInference`
|
|
|
@@ -70,32 +80,40 @@ Connection config loaded from `src/src.palm.vision/config/config.json`:
|
|
|
|
|
|
| Action | Payload | Effect |
|
|
|
|---|---|---|
|
|
|
-| `SubmitBatchAnalysis` | `{ files: File[]; mode: 'local'\|'remote'\|'n8n' }` | Run batch inference |
|
|
|
+| `SubmitBatchAnalysis` | `{ files: File[]; mode: 'local-onnx' \| 'local-tflite' \| 'remote' }` | Run batch inference; local results persisted via `saveExternalResult` before vault display |
|
|
|
| `ToggleBatchGroup` | `{ batchId: string }` | Expand/collapse history accordion |
|
|
|
| `LoadGroupImages` | `{ batchId: string }` | Lazy-load archived images for a batch |
|
|
|
| `LoadHistory` | — | Fetch last 50 records from SQLite |
|
|
|
| `DeleteHistoryRecord` | `{ id: string }` | Delete record + disk image |
|
|
|
| `ClearAllHistory` | — | Wipe all records |
|
|
|
|
|
|
+**Key `submitBatchAnalysis` behaviour:** generates a shared `batchId = UUID`, fans out per-file streams via `merge()`, collects via `toArray()`, then dispatches `LoadHistory`. In `local-onnx`/`local-tflite` mode each frame is first persisted to the backend via `saveExternalResult()` (with `timeout(5000)` + `catchError` fallback) before being committed to the batch.
|
|
|
+
|
|
|
#### Components
|
|
|
|
|
|
**`AnalyzerComponent`** — Three-mode inference UI:
|
|
|
-- Engine selector: `local` (WASM), `remote` (NestJS), `n8n` (edge)
|
|
|
-- Input: drag-and-drop file zone or live webcam (`getUserMedia`, environment-facing)
|
|
|
-- Canvas overlay via `renderPredictionsWithBoxes(frame)` — draws bounding boxes + confidence labels
|
|
|
+- Engine selector: `local-onnx` (ONNX WASM), `local-tflite` (TFLite WASM), `remote` (NestJS server)
|
|
|
+- Input: drag-and-drop file zone or live webcam (`getUserMedia`, environment-facing, 640×640)
|
|
|
+- Batch carousel: prev/next navigation across `batchFrames`
|
|
|
+- Canvas overlay via `renderPredictionsWithBoxes(frame)` — uses `norm_box` normalized coords preferentially, falls back to `box` pixel coords
|
|
|
- MPOB color palette: `Ripe #4caf50`, `Unripe #ff9800`, `Underripe #ffeb3b`, `Overripe #9c27b0`, `Abnormal #f44336`, `Empty_Bunch #607d8b`
|
|
|
|
|
|
**`HistoryComponent`** — Batch vault:
|
|
|
- Groups `items` by `batch_id` into `BatchGroup[]` via `combineLatest([items$, expandedBatchIds$])`
|
|
|
- `@ViewChildren('thumbCanvas')` — canvas thumbnails rendered via `renderThumbnailWithBoxes()` using `norm_box` coordinates
|
|
|
- `paintAllVisibleCanvases()` with `setTimeout(0)` deferral; `data-archive-id` attribute maps canvas elements to data items
|
|
|
-- Mode badge variants: `local` (WASM), `remote` (Server), `n8n`
|
|
|
+- Mode badge variants: `ONNX`, `TFLite`, `Server`
|
|
|
|
|
|
**`ChatbotComponent`** — Thin wrapper around `angularlib/chat/chat.component` with title "Industrial Intelligence Portal".
|
|
|
|
|
|
#### Web Worker (`src/src.palm.vision/workers/inference.worker.ts`)
|
|
|
|
|
|
-Receives `{ frameId, batchId, imageDataUrl, tensor, processingStart }`, runs ONNX inference, returns `InferenceFrame`. Keeps inference off the main thread for WASM local mode.
|
|
|
+Receives `{ frameId, batchId, imageDataUrl, tensor: ArrayBuffer (transferred), processingStart, mode }`.
|
|
|
+
|
|
|
+- **ONNX path (`local-onnx`):** Loads ONNX session once from `/assets/models/onnx/best.onnx`; runs via `onnxruntime-web`; output shape `[1, N, 6]` (`x, y, x, y, conf, classIdx`); filters at confidence ≥ 0.25.
|
|
|
+- **TFLite path (`local-tflite`):** Dynamically imports `/assets/tflite-wasm/tflite_web_api_client.js`; initializes `TFLiteWebModelRunner` for `/assets/models/tflite/best_float16.tflite`; converts CHW→HWC before upload; model includes internal NMS; output `[1, 300, 6]`; filters at confidence ≥ 0.20; maps `[ymin, xmin, ymax, xmax]` → `[nx1, ny1, nx2, ny2]`.
|
|
|
+
|
|
|
+Returns `InferenceFrame`-shaped `postMessage` correlated by `frameId`.
|
|
|
|
|
|
### Key Interfaces
|
|
|
|
|
|
@@ -105,7 +123,7 @@ interface DetectionResult {
|
|
|
class: string; // MPOB class name
|
|
|
confidence: number; // [0, 1]
|
|
|
is_health_alert: boolean; // true if Abnormal | Empty_Bunch
|
|
|
- box: [x1, y1, x2, y2]; // pixel coords
|
|
|
+ box: [x1, y1, x2, y2]; // absolute pixel coords
|
|
|
norm_box?: [nx1, ny1, nx2, ny2]; // normalized [0,1] coords
|
|
|
}
|
|
|
|
|
|
@@ -126,21 +144,21 @@ interface InferenceFrame {
|
|
|
|
|
|
| Alias | Path | Contents |
|
|
|
|---|---|---|
|
|
|
-| `angularlib/*` | `src/dependencies/angularlib/` | UI components, auth, forms, chat |
|
|
|
-| `dp-ui/*` | `src/dependencies/dp-ui/` | Custom Material extensions, `NgxSocketService` |
|
|
|
-| `fis/*` | `src/dependencies/fis/` | Domain modules (leave, tender, approval) |
|
|
|
+| `angularlib/*` | `src/dependencies/angularlib/` | UI components (forms, dialogs, chat), auth, `UIState`, `ChatState` |
|
|
|
+| `dp-ui/*` | `src/dependencies/dp-ui/` | `DpService`, `NgxSocketService`, `DPState`, FIS message models |
|
|
|
+| `fis/*` | `src/dependencies/fis/` | Domain modules (leave, tender, approval), `MetadataState` |
|
|
|
| `fis-commons/*` | `src/dependencies/fis-commons/` | CDN-hosted shared utilities |
|
|
|
|
|
|
### Models & WASM Assets
|
|
|
|
|
|
-- `src/assets/models/onnx/best.onnx` — YOLOv8 ONNX model for browser WASM inference
|
|
|
-- `src/assets/models/tflite/best_float16.tflite` / `best_float32.tflite` — TFLite variants
|
|
|
-- `src/assets/wasm/` — ONNX Runtime JS WASM bundles
|
|
|
-- `src/assets/tflite-wasm/` — TensorFlow Lite WASM runtime
|
|
|
+- `src/assets/models/onnx/best.onnx` — YOLOv8 ONNX model for browser ONNX WASM inference
|
|
|
+- `src/assets/models/tflite/best_float16.tflite` — TFLite model (loaded by worker in `local-tflite` mode)
|
|
|
+- `src/assets/wasm/` — ONNX Runtime Web JS + WASM bundles
|
|
|
+- `src/assets/tflite-wasm/` — TensorFlow Lite WASM runtime (`tflite_web_api_client.js` + WASM)
|
|
|
|
|
|
### Multi-Build Strategy
|
|
|
|
|
|
-`angular.json` defines four production configurations that swap assets and menu via `fileReplacements`:
|
|
|
+`angular.json` defines four production configurations that swap menu and assets via `fileReplacements`:
|
|
|
|
|
|
| Config | Output | Swaps |
|
|
|
|---|---|---|
|
|
|
@@ -153,8 +171,8 @@ Development build uses a mock socket service (`dp.service.t.ts`) for offline dev
|
|
|
|
|
|
### PWA
|
|
|
|
|
|
-Service worker registered immediately (`registerImmediately`) via `ngsw-config.json`. PWA install prompt handled in `AppComponent`. Three language packs: `en_US`, `ms_MY`, `zh_Hans`.
|
|
|
+Service worker registered immediately (`registerImmediately`) via `ngsw-config.json`. PWA install prompt handled in `AppComponent` (mobile platforms only). Three language packs: `en_US`, `ms_MY`, `zh_Hans`.
|
|
|
|
|
|
### CORS / Connection
|
|
|
|
|
|
-Frontend connects to NestJS backend at `http://localhost:3000` (configurable via `config.json`). Backend CORS whitelist is hardcoded in `server-desktop/src/main.ts` — add new device IPs there.
|
|
|
+Frontend connects to NestJS backend at `https://localhost:3000` (configurable via `src/config/config.json`). Backend CORS whitelist is hardcoded in `server-desktop/src/main.ts` and `server-android/src/main.ts` — add new device IPs there.
|