|
|
@@ -1,6 +1,6 @@
|
|
|
# CLAUDE.md — frontend
|
|
|
|
|
|
-Angular 21 SPA. Multi-tenant dashboard with a lazy-loaded `src.palm.vision` sub-app for MPOB-standard palm oil fruit bunch (FFB) ripeness detection. Three inference engines: browser ONNX WASM, browser TFLite WASM, and NestJS remote (WebSocket).
|
|
|
+Angular 21 SPA. Multi-tenant dashboard with a lazy-loaded **PalmVision** sub-app for MPOB-standard palm oil fruit bunch (FFB) ripeness detection. Three inference engines: browser ONNX WASM, browser TFLite WASM, and NestJS remote (WebSocket).
|
|
|
|
|
|
## Commands
|
|
|
|
|
|
@@ -21,76 +21,40 @@ npm run clean # Clear Angular + npm caches
|
|
|
### Application Shell (`src/app/`)
|
|
|
|
|
|
- **`app.config.ts`** — `ApplicationConfig`: bootstraps NGXS root store (`withNgxsStoragePlugin({keys:'*'})`, `withNgxsReduxDevtoolsPlugin`), Angular Router (hash location, `onSameUrlNavigation:'reload'`), HTTP client, service worker (registered immediately).
|
|
|
-- **`app.routes.ts`** — Root routes: `dashboard`, `auth`, `leave`, `tender`, `src.palm.vision` (all lazy-loaded except dashboard).
|
|
|
+- **`app.routes.ts`** — Root routes: `dashboard`, `auth`, `leave`, `tender`, `src.palm.vision` (all lazy-loaded). PalmVision loads from `angularlib/palm-vision/palm-vision.module`.
|
|
|
- **`app.component.ts`** — Root component: 30-minute session timeout, PWA install prompt, theme management (light/dark + blue/pink), multi-language (en_US, ms_MY, zh_Hans), maintenance mode alert.
|
|
|
|
|
|
-### PalmVision Sub-App (`src/src.palm.vision/`)
|
|
|
+### Connection Config (`src/config/config.json`)
|
|
|
|
|
|
-Lazy-loaded at `/src.palm.vision`. Provides its own NGXS `VisionState` via `provideStates([VisionState])` in the feature module.
|
|
|
+Backend URL and connection settings live here:
|
|
|
|
|
|
-**Sub-routes:**
|
|
|
-
|
|
|
-| Path | Component | Purpose |
|
|
|
-|---|---|---|
|
|
|
-| `analyzer` (default) | `AnalyzerComponent` | Inference UI — file/camera input, bounding box canvas |
|
|
|
-| `vault` | `HistoryComponent` | Batch history vault with canvas thumbnails |
|
|
|
-| `chat` | `ChatbotComponent` | Industrial Intelligence Portal (n8n chat proxy) |
|
|
|
-
|
|
|
-#### Services (`src/src.palm.vision/services/`)
|
|
|
-
|
|
|
-**`InferenceService`** — Local WASM inference:
|
|
|
-- `analyze(file, batchId?, mode)` → `Observable<InferenceFrame>` — preprocesses image → posts tensor to Web Worker → emits result
|
|
|
-- `preprocessImage(dataUrl)` — resizes to 640×640 on canvas, converts RGBA→CHW, normalizes `[0.0, 1.0]` → `Float32Array [1, 3, 640, 640]`
|
|
|
-- `results$: Subject<InferenceFrame>` — hot stream of all completed frames
|
|
|
-- `queueDepth$: BehaviorSubject<number>` — pending worker frame count
|
|
|
-- `pendingMap: Map<string, fn>` — in-memory frameId → observer resolver
|
|
|
-
|
|
|
-**`RemoteInferenceService`** — WebSocket bridge to NestJS backend:
|
|
|
-- `analyze(file, sourceLabel?, batchId?)` → `Observable<InferenceFrame>` — sends `PalmVision:analyze` via FIS envelope
|
|
|
-- `getHistory()` → `Observable<any[]>` — `History:getAll`
|
|
|
-- `getBatchDetails(batchId)` → `Observable<any>` — `History:getBatchDetails` (aggregated stats)
|
|
|
-- `deleteRecord(archiveId)` → `Observable<{ deleted: boolean }>` — `History:delete`
|
|
|
-- `clearHistory()` → `Observable<{ deleted: number }>` — `History:clearAll`
|
|
|
-- `getImage(archiveId)` → `Observable<{ archiveId, image_data }>` — `PalmHistory:GetImage`
|
|
|
-- `saveExternalResult(payload)` → `Observable<any>` — `PalmHistory:SaveExternalResult`
|
|
|
-
|
|
|
-All calls are wrapped in the FIS envelope and routed via `DpService.stream()`.
|
|
|
-
|
|
|
-Connection config loaded from `src/src.palm.vision/config/config.json`:
|
|
|
```json
|
|
|
-{ "connection": { "uacp": "https://localhost:3000", "uacp_ws": "wss://localhost:3000/socket.io", "uacpEmulation": "on" } }
|
|
|
-```
|
|
|
-
|
|
|
-#### State (`src/src.palm.vision/store/`)
|
|
|
-
|
|
|
-**`VisionStateModel`:**
|
|
|
-```typescript
|
|
|
{
|
|
|
- items: any[]; // History records from server
|
|
|
- loading: boolean;
|
|
|
- expandedBatchIds: string[]; // Expanded accordion groups
|
|
|
- currentInference: InferenceFrame | null;
|
|
|
- batchFrames: InferenceFrame[]; // Current batch results
|
|
|
- selectedFrameIndex: number | null;
|
|
|
+ "connection": {
|
|
|
+ "uacp": "https://localhost:3000",
|
|
|
+ "uacp_ws": "wss://localhost:3000",
|
|
|
+ "uacpEmulation": "on"
|
|
|
+ }
|
|
|
}
|
|
|
```
|
|
|
|
|
|
-**Selectors:** `VisionState.items`, `VisionState.loading`, `VisionState.expandedBatchIds`, `VisionState.currentInference`
|
|
|
+- `uacp` — REST/HTTP base URL for the NestJS backend
|
|
|
+- `uacp_ws` — WebSocket URL for Socket.io
|
|
|
+- `uacpEmulation` — `"on"` enables local buffering mode (dev); `"off"` = direct publish (prod)
|
|
|
|
|
|
-**Actions (`vision.actions.ts`):**
|
|
|
+Loaded at runtime via `HttpClient.get('/config/config.json')` by `RemoteInferenceService` and `DpService`.
|
|
|
|
|
|
-| Action | Payload | Effect |
|
|
|
-|---|---|---|
|
|
|
-| `SubmitBatchAnalysis` | `{ files: File[]; mode: 'local-onnx' \| 'local-tflite' \| 'remote' }` | Run batch inference; local results persisted via `saveExternalResult` before vault display |
|
|
|
-| `ToggleBatchGroup` | `{ batchId: string }` | Expand/collapse history accordion |
|
|
|
-| `LoadGroupImages` | `{ batchId: string }` | Lazy-load archived images for a batch |
|
|
|
-| `LoadHistory` | — | Fetch last 50 records from SQLite |
|
|
|
-| `DeleteHistoryRecord` | `{ id: string }` | Delete record + disk image |
|
|
|
-| `ClearAllHistory` | — | Wipe all records |
|
|
|
+### PalmVision — UI Components (`src/dependencies/angularlib/palm-vision/`)
|
|
|
|
|
|
-**Key `submitBatchAnalysis` behaviour:** generates a shared `batchId = UUID`, fans out per-file streams via `merge()`, collects via `toArray()`, then dispatches `LoadHistory`. In `local-onnx`/`local-tflite` mode each frame is first persisted to the backend via `saveExternalResult()` (with `timeout(5000)` + `catchError` fallback) before being committed to the batch.
|
|
|
+Lazy-loaded at `/src.palm.vision` via `PalmVisionModule`.
|
|
|
|
|
|
-#### Components
|
|
|
+**Sub-routes:**
|
|
|
+
|
|
|
+| Path | Component | Purpose |
|
|
|
+|---|---|---|
|
|
|
+| `analyzer` (default) | `AnalyzerComponent` | Inference UI — file/camera input, bounding box canvas |
|
|
|
+| `vault` | `HistoryComponent` | Batch history vault with canvas thumbnails |
|
|
|
+| `chat` | `ChatbotComponent` | Industrial Intelligence Portal (n8n chat proxy) |
|
|
|
|
|
|
**`AnalyzerComponent`** — Three-mode inference UI:
|
|
|
- Engine selector: `local-onnx` (ONNX WASM), `local-tflite` (TFLite WASM), `remote` (NestJS server)
|
|
|
@@ -100,37 +64,49 @@ Connection config loaded from `src/src.palm.vision/config/config.json`:
|
|
|
- MPOB color palette: `Ripe #4caf50`, `Unripe #ff9800`, `Underripe #ffeb3b`, `Overripe #9c27b0`, `Abnormal #f44336`, `Empty_Bunch #607d8b`
|
|
|
|
|
|
**`HistoryComponent`** — Batch vault:
|
|
|
-- Groups `items` by `batch_id` into `BatchGroup[]` via `combineLatest([items$, expandedBatchIds$])`
|
|
|
-- `@ViewChildren('thumbCanvas')` — canvas thumbnails rendered via `renderThumbnailWithBoxes()` using `norm_box` coordinates
|
|
|
-- `paintAllVisibleCanvases()` with `setTimeout(0)` deferral; `data-archive-id` attribute maps canvas elements to data items
|
|
|
-- Mode badge variants: `ONNX`, `TFLite`, `Server`
|
|
|
+- Groups records by `batch_id` into `BatchGroup[]`
|
|
|
+- Canvas thumbnails rendered via `renderThumbnailWithBoxes()` using `norm_box` coordinates
|
|
|
|
|
|
-**`BatchReportComponent`** — Batch detail viewer (`history/batch-report/`):
|
|
|
-- Fetches batch details via `getBatchDetails(batchId)` on open
|
|
|
-- Displays scalar metrics: `frameCount`, `totalDetections`, `avgInferenceMs`, `avgProcessingMs`, `batchStart`, `batchEnd`
|
|
|
-- Shows `classTally` breakdown with MPOB color-coded percentage bars
|
|
|
-- Frame gallery grid — clicking a frame opens `FrameInspectorDialogComponent`
|
|
|
+**`BatchReportComponent`** (`history/batch-report/`) — Batch detail viewer:
|
|
|
+- Scalar metrics: `frameCount`, `totalDetections`, `avgInferenceMs`, `avgProcessingMs`, `batchStart`, `batchEnd`
|
|
|
+- `classTally` breakdown with MPOB color-coded percentage bars
|
|
|
+- Frame gallery grid — clicking opens `FrameInspectorDialogComponent`
|
|
|
|
|
|
-**`FrameInspectorDialogComponent`** — Inline standalone dialog (`history/batch-report/frame-inspector-dialog.component.ts`):
|
|
|
-- Opened via `MatDialog.open()` with `maxWidth: '90vw'` and no forced width — centers to content on desktop
|
|
|
-- Layout: column flex — canvas centered on top, detection chips (`flex-wrap`) below
|
|
|
-- Canvas capped at 720px natural width; CSS `max-width: 100%` handles narrower viewports
|
|
|
-- Bounding boxes drawn using `norm_box` coords preferentially, falls back to `box` pixel coords scaled by the paint scale factor
|
|
|
-- Footer shows `inference_ms` and `processing_ms` timing
|
|
|
+**`FrameInspectorDialogComponent`** (`history/batch-report/frame-inspector-dialog.component.ts`) — Inline dialog:
|
|
|
+- Canvas capped at 720px; bounding boxes drawn using `norm_box` preferentially, falls back to `box`
|
|
|
|
|
|
-**`ChatbotComponent`** — Thin wrapper around `angularlib/chat/chat.component`:
|
|
|
-- Signals for messages, loading state, and `sessionId`
|
|
|
-- Sends via `Chat:send` FIS envelope; clears session via `Chat:clear`
|
|
|
-- Session UUID tracked server-side in `client.data.sessionId`
|
|
|
+**`ChatbotComponent`** (`palm-vision/chatbot/chatbot.component.ts`) — Industrial Intelligence Portal chat UI, sends via `Chat:send`/`Chat:clear` FIS envelope over the socket (see `DpService`/`NgxSocketService` in `dp-ui/`). Not related to the legacy `angularlib/chat/chat.component` (HTTP-based, used elsewhere in the app).
|
|
|
+- Backend (`server-desktop`) normalizes the n8n agent's reply into `{ text, has_visuals, visual_data }` before emitting it (see `server-desktop/CLAUDE.md`).
|
|
|
+- `chatbot/models/intelligence.model.ts` — `ChatResponse`/`ChatVisualData` mirror that schema; `buildIntelligenceMessage()` turns a response into a `TextMessage` or `DataMessage`; `toChartConfiguration()` maps `visual_data` to a Chart.js `ChartConfiguration`.
|
|
|
+- Chart rendering reuses the in-house `<chart>` component (`angularlib/chart/ChartComponent`/`ChartModule`, wraps `chart.js/auto`) — no separate charting library. `visual_data.type` is restricted by the n8n system prompt to `bar`, `line`, `pie`, `doughnut`, `radar`, `polarArea` (the types matching the flat `labels[]`/`datasets[{label,data:number[]}]` shape supported by this schema); `bubble`/`scatter` are not supported since they need `{x,y}` point objects.
|
|
|
|
|
|
-#### Web Worker (`src/src.palm.vision/workers/inference.worker.ts`)
|
|
|
+### PalmVision — Logic (`src/dependencies/fis-vision/`)
|
|
|
|
|
|
-Receives `{ frameId, batchId, imageDataUrl, tensor: ArrayBuffer (transferred), processingStart, mode }`.
|
|
|
+Self-contained library: services, NGXS state, and Web Worker for inference.
|
|
|
|
|
|
-- **ONNX path (`local-onnx`):** Loads ONNX session once from `/assets/models/onnx/best.onnx`; runs via `onnxruntime-web`; output shape `[1, N, 6]` (`x1, y1, x2, y2, conf, classIdx`); filters at confidence ≥ 0.25.
|
|
|
-- **TFLite path (`local-tflite`):** Dynamically imports `/assets/tflite-wasm/tflite_web_api_client.js`; initializes `TFLiteWebModelRunner` for `/assets/models/tflite/best_float16.tflite`; converts CHW→HWC before upload; model includes internal NMS; output `[1, 300, 6]`; filters at confidence ≥ 0.20; maps `[ymin, xmin, ymax, xmax]` → `[nx1, ny1, nx2, ny2]`.
|
|
|
+**`InferenceService`** — Local WASM inference:
|
|
|
+- `analyze(file, batchId?, mode)` → `Observable<InferenceFrame>`
|
|
|
+- Preprocesses image → posts tensor to Web Worker → emits result
|
|
|
+- `preprocessImage(dataUrl)` — resizes to 640×640, converts RGBA→CHW, normalizes `[0.0, 1.0]` → `Float32Array [1, 3, 640, 640]`
|
|
|
+
|
|
|
+**`RemoteInferenceService`** — WebSocket bridge to NestJS backend:
|
|
|
+- Loads `src/config/config.json` for `uacp` / `uacp_ws`
|
|
|
+- `analyze(file, sourceLabel?, batchId?)` → `Observable<InferenceFrame>` via `PalmVision:analyze`
|
|
|
+- `getHistory()`, `getBatchDetails(batchId)`, `deleteRecord(id)`, `clearHistory()`, `getImage(archiveId)`, `saveExternalResult(payload)`
|
|
|
+
|
|
|
+**`VisionState`** (NGXS) — Key actions:
|
|
|
|
|
|
-Returns `InferenceFrame`-shaped `postMessage` correlated by `frameId`.
|
|
|
+| Action | Effect |
|
|
|
+|---|---|
|
|
|
+| `SubmitBatchAnalysis` | Fan-out per-file inference; local results persisted via `saveExternalResult` before vault display |
|
|
|
+| `LoadHistory` | Fetch last 50 records from SQLite |
|
|
|
+| `ToggleBatchGroup` | Expand/collapse history accordion |
|
|
|
+| `LoadGroupImages` | Lazy-load archived images for a batch |
|
|
|
+| `DeleteHistoryRecord` / `ClearAllHistory` | Delete one or all records |
|
|
|
+
|
|
|
+**Web Worker** (`workers/inference.worker.ts`):
|
|
|
+- **ONNX path:** Loads `/assets/models/onnx/best.onnx`; output shape `[1, N, 6]`; confidence threshold ≥ 0.25
|
|
|
+- **TFLite path:** Loads `/assets/models/tflite/best_float16.tflite`; output `[1, 300, 6]`; confidence ≥ 0.20
|
|
|
|
|
|
### Key Interfaces
|
|
|
|
|
|
@@ -157,21 +133,23 @@ interface InferenceFrame {
|
|
|
}
|
|
|
```
|
|
|
|
|
|
-### Shared In-Source Dependencies (`src/dependencies/`)
|
|
|
+### In-Source Dependencies (`src/dependencies/`)
|
|
|
|
|
|
-| Alias | Path | Contents |
|
|
|
-|---|---|---|
|
|
|
-| `angularlib/*` | `src/dependencies/angularlib/` | UI components (forms, dialogs, chat), auth, `UIState`, `ChatState` |
|
|
|
-| `dp-ui/*` | `src/dependencies/dp-ui/` | `DpService`, `NgxSocketService`, `DPState`, FIS message models |
|
|
|
-| `fis/*` | `src/dependencies/fis/` | Domain modules (leave, tender, approval), `MetadataState` |
|
|
|
-| `fis-commons/*` | `src/dependencies/fis-commons/` | CDN-hosted shared utilities |
|
|
|
+| Folder | Purpose |
|
|
|
+|---|---|
|
|
|
+| `angularlib/` | Shared UI components (forms, dialogs, chat, palm-vision UI) |
|
|
|
+| `dp-ui/` | `DpService`, `NgxSocketService`, `DPState`, FIS message models |
|
|
|
+| `fis/` | Domain modules (leave, tender, approval), `MetadataState` |
|
|
|
+| `fis-vision/` | PalmVision inference services, NGXS state, Web Worker |
|
|
|
+
|
|
|
+`fis-commons` is CDN-hosted — no local checkout needed.
|
|
|
|
|
|
### Models & WASM Assets
|
|
|
|
|
|
-- `src/assets/models/onnx/best.onnx` — YOLOv8 ONNX model for browser ONNX WASM inference
|
|
|
-- `src/assets/models/tflite/best_float16.tflite` — TFLite model (loaded by worker in `local-tflite` mode)
|
|
|
+- `src/assets/models/onnx/best.onnx` — YOLOv8 ONNX model
|
|
|
+- `src/assets/models/tflite/best_float16.tflite` — TFLite model
|
|
|
- `src/assets/wasm/` — ONNX Runtime Web JS + WASM bundles
|
|
|
-- `src/assets/tflite-wasm/` — TensorFlow Lite WASM runtime (`tflite_web_api_client.js` + WASM)
|
|
|
+- `src/assets/tflite-wasm/` — TensorFlow Lite WASM runtime
|
|
|
|
|
|
### Multi-Build Strategy
|
|
|
|
|
|
@@ -184,12 +162,10 @@ interface InferenceFrame {
|
|
|
| `quotation-prod` | `dist/quotation` | `menu.ts → menu.quotation.ts`, quotation assets |
|
|
|
| `maf-quot-prod` | `dist/maf/quotation` | MAF quotation assets |
|
|
|
|
|
|
-Development build uses a mock socket service (`dp.service.t.ts`) for offline development.
|
|
|
-
|
|
|
### PWA
|
|
|
|
|
|
-Service worker registered immediately (`registerImmediately`) via `ngsw-config.json`. PWA install prompt handled in `AppComponent` (mobile platforms only). Three language packs: `en_US`, `ms_MY`, `zh_Hans`.
|
|
|
+Service worker registered immediately (`registerImmediately`) via `ngsw-config.json`. PWA install prompt handled in `AppComponent` (mobile only). Three language packs: `en_US`, `ms_MY`, `zh_Hans`.
|
|
|
|
|
|
### CORS / Connection
|
|
|
|
|
|
-Frontend connects to NestJS backend at `https://localhost:3000` (configurable via `src/src.palm.vision/config/config.json`). Backend CORS whitelist is hardcoded in `server-desktop/src/main.ts` — add new device IPs there.
|
|
|
+Frontend connects to NestJS backend at `https://localhost:3000` (change in `src/config/config.json`). Backend CORS whitelist is hardcoded in `server-desktop/src/main.ts` — add new device IPs there.
|
