|
@@ -4,27 +4,24 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|
|
|
|
|
|
|
## What This Is
|
|
## What This Is
|
|
|
|
|
|
|
|
-NestJS backend for the PalmOilAI system. Provides server-side YOLOv8 ONNX inference, SQLite history persistence (with disk image archiving), WebSocket gateways for real-time vision streaming and n8n chat proxying, and a process surveillance monitor for n8n and Ollama.
|
|
|
|
|
|
|
+NestJS backend optimized for ARM/Android (Termux) deployment. Provides server-side YOLOv8 ONNX inference via WASM (single-threaded, no native bindings), SQLite history persistence (sql.js + disk image archiving), a unified FIS-protocol WebSocket gateway, n8n chat proxying, and a process surveillance monitor.
|
|
|
|
|
|
|
|
-Detection classes: `Ripe`, `Unripe`, `Underripe`, `Overripe`, `Abnormal`, `Empty_Bunch` (MPOB standard).
|
|
|
|
|
|
|
+Detection classes: `Ripe`, `Unripe`, `Underripe`, `Overripe`, `Abnormal`, `Empty_Bunch` (MPOB standard, indices 0–5).
|
|
|
|
|
|
|
|
## Commands
|
|
## Commands
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|
|
|
npm install
|
|
npm install
|
|
|
-npm run start:dev # Watch mode dev server (port 3000)
|
|
|
|
|
|
|
+npm run start:dev # Watch mode dev server (HTTPS, port 3000)
|
|
|
npm run start # Single-run dev server
|
|
npm run start # Single-run dev server
|
|
|
npm run start:prod # Production (from dist/)
|
|
npm run start:prod # Production (from dist/)
|
|
|
npm run build # Compile → dist/
|
|
npm run build # Compile → dist/
|
|
|
npm run lint # ESLint with auto-fix
|
|
npm run lint # ESLint with auto-fix
|
|
|
npm run test # Jest unit tests
|
|
npm run test # Jest unit tests
|
|
|
-npm run test:watch # Jest watch mode
|
|
|
|
|
-npm run test:cov # Coverage report
|
|
|
|
|
npm run test:e2e # End-to-end tests
|
|
npm run test:e2e # End-to-end tests
|
|
|
-jest --testPathPattern=palm-oil # Run a single test file
|
|
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
-Set `PORT` env var to override port 3000. Set `N8N_WEBHOOK_URL` in `.env` for chat proxy and agent readiness probing.
|
|
|
|
|
|
|
+Set `PORT` env var to override port 3000. Set `N8N_WEBHOOK_URL` in `.env` for chat proxy and n8n process discovery.
|
|
|
|
|
|
|
|
## Architecture
|
|
## Architecture
|
|
|
|
|
|
|
@@ -32,39 +29,59 @@ Set `PORT` env var to override port 3000. Set `N8N_WEBHOOK_URL` in `.env` for ch
|
|
|
|
|
|
|
|
**`PalmOilModule`** (`src/palm-oil/`)
|
|
**`PalmOilModule`** (`src/palm-oil/`)
|
|
|
|
|
|
|
|
-**`ScannerProvider`** — ONNX inference engine loaded on `OnModuleInit`:
|
|
|
|
|
-- `preprocess(buffer)` — `sharp` resize to 640×640, strip alpha, convert HWC→CHW, normalize to `[0.0, 1.0]`, output `[1, 3, 640, 640]` tensor
|
|
|
|
|
-- `inference(tensor)` — runs `onnxruntime-node` session
|
|
|
|
|
-- `postprocess(tensor, origW, origH, threshold=0.25)` — output is `[1, N, 6]` (`x1, y1, x2, y2, confidence, class_index`):
|
|
|
|
|
- - Captures first 5 raw rows as `raw_tensor_sample` for technical evidence
|
|
|
|
|
- - Filters by confidence threshold
|
|
|
|
|
|
|
+**`OnnxWasmProvider`** (`src/palm-oil/providers/onnx-wasm.provider.ts`) — WASM-based inference engine, `IScannerProvider` implementation:
|
|
|
|
|
+- Loaded on `OnModuleInit`; `ort.env.wasm.numThreads = 1` for ARM/Termux safety
|
|
|
|
|
+- Model loaded from `best.onnx` in the project root
|
|
|
|
|
+- Execution provider: `['wasm']`
|
|
|
|
|
+- `preprocess(buffer)` — Jimp decode → resize 640×640 → RGBA→CHW strip → normalize `[0.0, 1.0]` → `[1, 3, 640, 640]` Float32Array
|
|
|
|
|
+- `inference(tensor)` — runs ONNX session with `{ images: ortTensor }` input, extracts first output key
|
|
|
|
|
+- `postprocess(tensor, origW, origH, threshold=0.25)` — output shape `[1, N, 6]` (`x1, y1, x2, y2, confidence, class_index`):
|
|
|
|
|
+ - Captures first 5 rows as `raw_tensor_sample`
|
|
|
|
|
+ - Filters by confidence threshold (default 0.25)
|
|
|
- Scales coords to original image dimensions
|
|
- Scales coords to original image dimensions
|
|
|
- Maps `class_index` → `MPOB_CLASSES`
|
|
- Maps `class_index` → `MPOB_CLASSES`
|
|
|
- Sets `is_health_alert` for `Abnormal` | `Empty_Bunch`
|
|
- Sets `is_health_alert` for `Abnormal` | `Empty_Bunch`
|
|
|
|
|
|
|
|
-**`PalmOilService`** — Orchestrates inference + persistence. `analyzeImage(imageBuffer, filename, batchId?)` pipeline:
|
|
|
|
|
-1. Preprocess via `ScannerProvider`
|
|
|
|
|
-2. Inference with timing (`inference_ms`, `processing_ms`)
|
|
|
|
|
-3. Postprocess → detections + `raw_tensor_sample`
|
|
|
|
|
-4. Save image buffer to `archive/` directory on disk
|
|
|
|
|
-5. Persist to SQLite (`HistoryEntity`) — **fully awaited before returning** (guarantees DB write before socket emit)
|
|
|
|
|
-6. Returns `AnalysisResponse` with Base64 image, `archive_id`, full `technical_evidence` block
|
|
|
|
|
|
|
+**`PalmOilService`** — Orchestrates inference + persistence. `analyzeImage(imageBuffer, filename?, batchId?)` pipeline:
|
|
|
|
|
+1. Measure original image dimensions via Jimp
|
|
|
|
|
+2. Preprocess → inference → postprocess via `ScannerProvider`
|
|
|
|
|
+3. Capture timing (`inference_ms`, `processing_ms`)
|
|
|
|
|
+4. Write image buffer to `archive/` directory on disk
|
|
|
|
|
+5. Persist to SQLite (`HistoryEntity`) — **fully awaited before returning** (ordering guarantee: DB write completes before socket emit)
|
|
|
|
|
+6. Return `AnalysisResponse` with Base64 image, `archive_id`, full `technical_evidence` block
|
|
|
|
|
|
|
|
-Additional methods: `getHistory()` (last 50 records), `getRecordByArchiveId()`, `deleteRecord()` (removes DB row + disk image), `clearAllHistory()` (removes all records + disk images).
|
|
|
|
|
|
|
+Additional methods: `getHistory()` (last 50 records), `getRecordByArchiveId()`, `deleteRecord()` (removes DB row + disk image), `clearAllHistory()`.
|
|
|
|
|
|
|
|
-**`VisionGateway`** — WebSocket gateway on `/vision` namespace:
|
|
|
|
|
-- `vision:analyze` handler: receives `VisionStreamPayload` (`frame: string, sourceLabel?, batchId?`), strips data-URI prefix, decodes Base64 → Buffer, calls `PalmOilService.analyzeImage()`, emits `vision:result` or `vision:error`
|
|
|
|
|
-- `chat:send` handler: receives `ChatPayload`, POSTs to `N8N_WEBHOOK_URL` server-to-server (bypasses browser CORS), unwraps array response to first element, emits `chat:result` or `chat:error`
|
|
|
|
|
-- Hard rule: accepts raw, uncompressed Base64 strings only — no binary frames, no WebRTC
|
|
|
|
|
|
|
+**`VisionGateway`** (`src/palm-oil/vision.gateway.ts`) — Unified FIS-protocol WebSocket gateway (root namespace, no prefix):
|
|
|
|
|
|
|
|
-**`HistoryEntity`** — TypeORM entity fields:
|
|
|
|
|
|
|
+**Inbound**: single `request` event carrying `FisAppMessage` envelope `{ serviceId, operation, requestId, payload }`.
|
|
|
|
|
+**Outbound**: `response` event with `FisAppResponse` packets (two-packet streaming pattern).
|
|
|
|
|
+
|
|
|
|
|
+Route table (`${serviceId}:${operation}`):
|
|
|
|
|
+
|
|
|
|
|
+| Route | Payload | Behavior |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| `PalmVision:analyze` | `{ frame: string, sourceLabel?, batchId? }` | Base64 decode → Buffer → `PalmOilService.analyzeImage()` → emit two-packet response with `technical_evidence` |
|
|
|
|
|
+| `History:getAll` | — | Query SQLite (last 50), emit array response |
|
|
|
|
|
+| `History:delete` | `{ archiveId: string }` | Delete record + disk image, emit `{ deleted: boolean }` |
|
|
|
|
|
+| `History:clearAll` | — | Wipe all records + images, emit `{ deleted: number }` |
|
|
|
|
|
+| `Chat:send` | `{ message: string }` | POST to `N8N_WEBHOOK_URL` server-to-server, unwrap array to first element, emit response |
|
|
|
|
|
+| `Chat:clear` | — | Generate new session UUID, emit `{ status: 'success' }` |
|
|
|
|
|
+| `PalmHistory:GetImage` | `{ archiveId: string }` | Load image from disk, encode to Base64 data URL, emit response |
|
|
|
|
|
+
|
|
|
|
|
+Client lifecycle:
|
|
|
|
|
+- On connect: push latest surveillance snapshot (if available)
|
|
|
|
|
+- Store session UUID in `client.data.sessionId` (used for Chat payloads)
|
|
|
|
|
+- Every 500ms tick: broadcasts `response` packet with `SystemMetrics` to all connected clients (via `SurveillanceService` callback registered in `onModuleInit`)
|
|
|
|
|
+
|
|
|
|
|
+**`HistoryEntity`** (`src/palm-oil/entities/history.entity.ts`) — TypeORM entity fields:
|
|
|
- `id` (PK auto), `archive_id` (unique), `batch_id` (nullable, indexed), `filename`
|
|
- `id` (PK auto), `archive_id` (unique), `batch_id` (nullable, indexed), `filename`
|
|
|
- `total_count`, `industrial_summary` (simple-json), `detections` (simple-json array)
|
|
- `total_count`, `industrial_summary` (simple-json), `detections` (simple-json array)
|
|
|
- `inference_ms`, `processing_ms`, `image_path` (disk reference)
|
|
- `inference_ms`, `processing_ms`, `image_path` (disk reference)
|
|
|
- `created_at` (auto timestamp)
|
|
- `created_at` (auto timestamp)
|
|
|
|
|
|
|
|
**`mpob-standards.ts`** (`src/palm-oil/constants/`) — Source of truth:
|
|
**`mpob-standards.ts`** (`src/palm-oil/constants/`) — Source of truth:
|
|
|
-- `MPOB_CLASSES`: index 0–5 → class name strings
|
|
|
|
|
|
|
+- `MPOB_CLASSES`: index → class name (`0: Empty_Bunch`, `1: Underripe`, `2: Abnormal`, `3: Ripe`, `4: Unripe`, `5: Overripe`)
|
|
|
- `GRADE_COLORS`: class name → hex color
|
|
- `GRADE_COLORS`: class name → hex color
|
|
|
- `HEALTH_ALERT_CLASSES`: `['Abnormal', 'Empty_Bunch']`
|
|
- `HEALTH_ALERT_CLASSES`: `['Abnormal', 'Empty_Bunch']`
|
|
|
|
|
|
|
@@ -72,32 +89,88 @@ Additional methods: `getHistory()` (last 50 records), `getRecordByArchiveId()`,
|
|
|
|
|
|
|
|
**`SurveillanceModule`** (`src/surveillance/`)
|
|
**`SurveillanceModule`** (`src/surveillance/`)
|
|
|
|
|
|
|
|
-**`SurveillanceService`** — Boots on `OnModuleInit`, runs two background loops:
|
|
|
|
|
-- **500ms poll loop**: discovers PIDs for n8n (Node.js process containing "n8n" in cmd) and Ollama (`ollama_llama_server` or `ollama`), then calls `pidusage` for CPU/memory metrics
|
|
|
|
|
-- **Port-level heartbeat**: n8n PID only included if port 5678 actively accepts TCP connections (800ms timeout); evicts PID if port goes silent or process dies
|
|
|
|
|
-- Exposes: `getLatestMetrics()`, callback registration for tick events
|
|
|
|
|
|
|
+**`SurveillanceService`** — 500ms process monitor via `systeminformation`:
|
|
|
|
|
+- `OnModuleInit`: starts polling interval; `OnModuleDestroy`: clears it
|
|
|
|
|
+- Per tick: queries CPU load, memory, process list via `systeminformation`
|
|
|
|
|
+- Discovers processes by identity:
|
|
|
|
|
+ - **NestJS**: match by `process.pid`
|
|
|
|
|
+ - **n8n**: command substring match (only if `N8N_WEBHOOK_URL` contains `localhost`)
|
|
|
|
|
+ - **Ollama**: command substrings `['ollama', 'ollama_llama_server']`
|
|
|
|
|
+- Classifies each as `ACTIVE` (CPU > 1.0%), `IDLE` (CPU ≤ 1.0%), or `OFFLINE` (not found)
|
|
|
|
|
+- Stores `SystemMetrics` snapshot; triggers registered callback for `VisionGateway` broadcast
|
|
|
|
|
+- Public API: `getLatestMetrics()`, `registerMetricsCallback(cb)`
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+interface ServiceStatus { service: string; pid: number | null; status: 'ACTIVE'|'IDLE'|'OFFLINE'; cpu: number; memory: number; }
|
|
|
|
|
+interface SystemMetrics { cpuLoad: number; memUsed: number; memTotal: number; uptime: number; services: ServiceStatus[]; timestamp: Date; }
|
|
|
|
|
+```
|
|
|
|
|
|
|
|
-**`SurveillanceGateway`** — WebSocket gateway on `/monitor` namespace:
|
|
|
|
|
-- Wires `SurveillanceService` callbacks: every 500ms tick broadcasts `monitor:data` to all clients
|
|
|
|
|
-- On client connect: immediately pushes latest snapshot (no blank load screen)
|
|
|
|
|
-- `monitor:subscribe` handler: acknowledges and re-sends current snapshot
|
|
|
|
|
|
|
+Note: No separate `SurveillanceGateway`. Surveillance broadcasts are handled inside `VisionGateway` via the registered callback.
|
|
|
|
|
|
|
|
### Database
|
|
### Database
|
|
|
-SQLite file `palm_history.db` in the project root. Managed by TypeORM with `synchronize: true` (auto-creates/migrates tables — dev only). No external DB setup required.
|
|
|
|
|
|
|
+
|
|
|
|
|
+SQLite file `palm_history.db` in project root. Managed by TypeORM with driver `sqljs` (`sql.js`) and `autoSave: true`. No external DB setup required. `synchronize: true` auto-creates/migrates tables on startup (dev only).
|
|
|
|
|
+
|
|
|
|
|
+### Key Interfaces (`src/palm-oil/interfaces/`)
|
|
|
|
|
+
|
|
|
|
|
+```typescript
|
|
|
|
|
+interface DetectionResult {
|
|
|
|
|
+ bunch_id: number; // 1-based sequential count
|
|
|
|
|
+ class: string; // MPOB class name
|
|
|
|
|
+ confidence: number; // [0, 1]
|
|
|
|
|
+ is_health_alert: boolean; // true if Abnormal | Empty_Bunch
|
|
|
|
|
+ box: [x1, y1, x2, y2]; // pixel coords in original image dims
|
|
|
|
|
+}
|
|
|
|
|
+
|
|
|
|
|
+interface AnalysisResponse {
|
|
|
|
|
+ detections: DetectionResult[];
|
|
|
|
|
+ archive_id: string;
|
|
|
|
|
+ image_base64: string; // Base64-encoded image
|
|
|
|
|
+ technical_evidence: {
|
|
|
|
|
+ engine: 'NestJS-ONNX';
|
|
|
|
|
+ archive_id: string;
|
|
|
|
|
+ total_count: number;
|
|
|
|
|
+ threshold: number; // 0.25
|
|
|
|
|
+ industrial_summary: Record<string, number>;
|
|
|
|
|
+ raw_tensor_sample: number[][]; // first 5 ONNX output rows
|
|
|
|
|
+ };
|
|
|
|
|
+ inference_ms: number;
|
|
|
|
|
+ processing_ms: number;
|
|
|
|
|
+}
|
|
|
|
|
+```
|
|
|
|
|
|
|
|
### Required Files
|
|
### Required Files
|
|
|
-- `best.onnx` — YOLOv8 ONNX model, must be placed in the **project root** (not `src/`). Loaded by `ScannerProvider` at startup.
|
|
|
|
|
-- `archive/` directory — created automatically by `PalmOilService` for disk image storage.
|
|
|
|
|
|
|
+
|
|
|
|
|
+- `best.onnx` — YOLOv8 ONNX model in **project root** (not `src/`). Loaded at startup.
|
|
|
|
|
+- `cert/127.0.0.1+1-key.pem` and `cert/127.0.0.1+1.pem` — TLS certificates. HTTPS is always on (no HTTP fallback).
|
|
|
|
|
+- `archive/` — Auto-created by `PalmOilService` on first write.
|
|
|
|
|
+- `tflite/` — TFLite model files (present but not loaded by current server code).
|
|
|
|
|
|
|
|
### Configuration (`.env`)
|
|
### Configuration (`.env`)
|
|
|
|
|
+
|
|
|
```
|
|
```
|
|
|
-N8N_WEBHOOK_URL=<n8n webhook URL>
|
|
|
|
|
-PORT=3000 # optional
|
|
|
|
|
|
|
+N8N_WEBHOOK_URL=<n8n webhook URL> # Optional; required for Chat:send and n8n process discovery
|
|
|
|
|
+PORT=3000 # Optional
|
|
|
|
|
+HOST=0.0.0.0 # Default bind address
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
-### WebSocket Event Contracts
|
|
|
|
|
-| Namespace | Client → Server | Payload | Server → Client |
|
|
|
|
|
-|---|---|---|---|
|
|
|
|
|
-| `/vision` | `vision:analyze` | `{ frame: string, sourceLabel?: string, batchId?: string }` | `vision:result`, `vision:error` |
|
|
|
|
|
-| `/vision` | `chat:send` | `{ message: string }` | `chat:result`, `chat:error` |
|
|
|
|
|
-| `/monitor` | `monitor:subscribe` | — | `monitor:data` (MonitorPayload[]) |
|
|
|
|
|
|
|
+### CORS Whitelist
|
|
|
|
|
+
|
|
|
|
|
+Origins hardcoded in `main.ts`:
|
|
|
|
|
+- `https://192.168.100.100:4200`
|
|
|
|
|
+- `https://192.168.100.79:4200`
|
|
|
|
|
+- `https://localhost:4200`
|
|
|
|
|
+
|
|
|
|
|
+Update `main.ts` to add new origins (e.g., new Android device IPs).
|
|
|
|
|
+
|
|
|
|
|
+### Key Differences from `nestjs/` (server-desktop)
|
|
|
|
|
+
|
|
|
|
|
+| Concern | `nestjs/` | `server-android/` |
|
|
|
|
|
+|---|---|---|
|
|
|
|
|
+| ONNX runtime | `onnxruntime-node` (native) | `onnxruntime-web` (WASM, `numThreads=1`) |
|
|
|
|
|
+| Image processing | `sharp` | `Jimp` |
|
|
|
|
|
+| SQLite driver | `sqlite3` | `sql.js` (TypeORM `sqljs`) |
|
|
|
|
|
+| WebSocket namespaces | `/vision` + `/monitor` (separate) | Root namespace (unified) |
|
|
|
|
|
+| WebSocket protocol | Flat events (`vision:analyze`) | FIS envelope (`request`/`response`) |
|
|
|
|
|
+| Surveillance | Separate `SurveillanceGateway` on `/monitor` | Inline broadcast in `VisionGateway` |
|
|
|
|
|
+| Process metrics | `pidusage` | `systeminformation` |
|