PalmOilRipenessDetection-Nest/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What This Is

NestJS backend optimized for ARM/Android (Termux) deployment. Provides server-side YOLOv8 ONNX inference via WASM (single-threaded, no native bindings), SQLite history persistence (sql.js + disk image archiving), a unified FIS-protocol WebSocket gateway, n8n chat proxying, and a process surveillance monitor.

Detection classes: Ripe, Unripe, Underripe, Overripe, Abnormal, Empty_Bunch (MPOB standard, indices 0–5).

Commands

npm install
npm run start:dev               # Watch mode dev server (HTTPS, port 3000)
npm run start                   # Single-run dev server
npm run start:prod              # Production (from dist/)
npm run build                   # Compile → dist/
npm run lint                    # ESLint with auto-fix
npm run test                    # Jest unit tests
npm run test:e2e                # End-to-end tests

Set PORT env var to override port 3000. Set N8N_WEBHOOK_URL in .env for chat proxy and n8n process discovery.

Architecture

Modules

PalmOilModule (src/palm-oil/)

OnnxWasmProvider (src/palm-oil/providers/onnx-wasm.provider.ts) — WASM-based inference engine, IScannerProvider implementation:

• Loaded on OnModuleInit; ort.env.wasm.numThreads = 1 for ARM/Termux safety
• Model loaded from best.onnx in the project root
• Execution provider: ['wasm']
• preprocess(buffer) — Jimp decode → resize 640×640 → RGBA→CHW strip → normalize [0.0, 1.0] → [1, 3, 640, 640] Float32Array
• inference(tensor) — runs ONNX session with { images: ortTensor } input, extracts first output key
• postprocess(tensor, origW, origH, threshold=0.25) — output shape [1, N, 6] (x1, y1, x2, y2, confidence, class_index):
  • Captures first 5 rows as raw_tensor_sample
  • Filters by confidence threshold (default 0.25)
  • Scales coords to original image dimensions
  • Maps class_index → MPOB_CLASSES
  • Sets is_health_alert for Abnormal | Empty_Bunch

PalmOilService — Orchestrates inference + persistence. analyzeImage(imageBuffer, filename?, batchId?) pipeline:

1. Measure original image dimensions via Jimp
2. Preprocess → inference → postprocess via ScannerProvider
3. Capture timing (inference_ms, processing_ms)
4. Write image buffer to archive/ directory on disk
5. Persist to SQLite (HistoryEntity) — fully awaited before returning (ordering guarantee: DB write completes before socket emit)
6. Return AnalysisResponse with Base64 image, archive_id, full technical_evidence block

Additional methods: getHistory() (last 50 records), getRecordByArchiveId(), deleteRecord() (removes DB row + disk image), clearAllHistory().

VisionGateway (src/palm-oil/vision.gateway.ts) — Unified FIS-protocol WebSocket gateway (root namespace, no prefix):

Inbound: single request event carrying FisAppMessage envelope { serviceId, operation, requestId, payload }.
Outbound: response event with FisAppResponse packets (two-packet streaming pattern).

Route table (${serviceId}:${operation}):

Route	Payload	Behavior
PalmVision:analyze	{ frame: string, sourceLabel?, batchId? }	Base64 decode → Buffer → PalmOilService.analyzeImage() → emit two-packet response with technical_evidence
History:getAll	—	Query SQLite (last 50), emit array response
History:delete	{ archiveId: string }	Delete record + disk image, emit { deleted: boolean }
History:clearAll	—	Wipe all records + images, emit { deleted: number }
Chat:send	{ message: string }	POST to N8N_WEBHOOK_URL server-to-server, unwrap array to first element, emit response
Chat:clear	—	Generate new session UUID, emit { status: 'success' }
PalmHistory:GetImage	{ archiveId: string }	Load image from disk, encode to Base64 data URL, emit response
Surveillance:SubscribeTelemetry	—	Join telemetry-room; immediately push latest snapshot; reply { status: 'subscribed' }
Surveillance:UnsubscribeTelemetry	—	Leave telemetry-room; reply { status: 'unsubscribed' }

Client lifecycle:

• On connect: push latest surveillance snapshot (if available); store session UUID in client.data.sessionId
• On disconnect: leave telemetry-room
• Every 500ms tick: broadcasts response packet with SystemMetrics to telemetry-room subscribers only (via SurveillanceService callback registered in onModuleInit)

HistoryEntity (src/palm-oil/entities/history.entity.ts) — TypeORM entity fields:

• id (PK auto), archive_id (unique), batch_id (nullable, indexed), filename
• total_count, industrial_summary (simple-json), detections (simple-json array)
• inference_ms, processing_ms, image_path (disk reference)
• created_at (auto timestamp)

mpob-standards.ts (src/palm-oil/constants/) — Source of truth:

• MPOB_CLASSES: index → class name (0: Empty_Bunch, 1: Underripe, 2: Abnormal, 3: Ripe, 4: Unripe, 5: Overripe)
• GRADE_COLORS: class name → hex color
• HEALTH_ALERT_CLASSES: ['Abnormal', 'Empty_Bunch']

---

SurveillanceModule (src/surveillance/)

SurveillanceService — 500ms process monitor via systeminformation:

• OnModuleInit: starts polling interval; OnModuleDestroy: clears it
• Per tick: queries CPU load, memory, process list via systeminformation
• Discovers processes by identity:
  • NestJS: match by process.pid
  • n8n: command substring match (only if N8N_WEBHOOK_URL contains localhost)
  • Ollama: command substrings ['ollama', 'ollama_llama_server']
• Classifies each as ACTIVE (CPU > 1.0%), IDLE (CPU ≤ 1.0%), or OFFLINE (not found)
• Stores SystemMetrics snapshot; triggers registered callback for VisionGateway broadcast
• Public API: getLatestMetrics(), registerMetricsCallback(cb)

interface ServiceStatus { service: string; pid: number | null; status: 'ACTIVE'|'IDLE'|'OFFLINE'; cpu: number; memory: number; }
interface SystemMetrics { cpuLoad: number; memUsed: number; memTotal: number; uptime: number; services: ServiceStatus[]; timestamp: Date; }

Note: No separate SurveillanceGateway. Surveillance broadcasts are handled inside VisionGateway via the registered callback.

Database

SQLite file palm_history.db in project root. Managed by TypeORM with driver sqljs (sql.js) and autoSave: true. No external DB setup required. synchronize: true auto-creates/migrates tables on startup (dev only).

Key Interfaces (src/palm-oil/interfaces/)

interface DetectionResult {
  bunch_id: number;           // 1-based sequential count
  class: string;              // MPOB class name
  confidence: number;         // [0, 1]
  is_health_alert: boolean;   // true if Abnormal | Empty_Bunch
  box: [x1, y1, x2, y2];     // pixel coords in original image dims
}

interface AnalysisResponse {
  detections: DetectionResult[];
  archive_id: string;
  image_base64: string;       // Base64-encoded image
  technical_evidence: {
    engine: 'NestJS-ONNX';
    archive_id: string;
    total_count: number;
    threshold: number;        // 0.25
    industrial_summary: Record<string, number>;
    raw_tensor_sample: number[][];  // first 5 ONNX output rows
  };
  inference_ms: number;
  processing_ms: number;
}

Required Files

• best.onnx — YOLOv8 ONNX model in project root (not src/). Loaded at startup.
• cert/127.0.0.1+1-key.pem and cert/127.0.0.1+1.pem — TLS certificates. HTTPS is always on (no HTTP fallback).
• archive/ — Auto-created by PalmOilService on first write.
• tflite/ — TFLite model files (present but not loaded by current server code).

Configuration (.env)

N8N_WEBHOOK_URL=<n8n webhook URL>   # Optional; required for Chat:send and n8n process discovery
PORT=3000                           # Optional
HOST=0.0.0.0                        # Default bind address

CORS Whitelist

Origins hardcoded in main.ts:

• https://192.168.100.100:4200
• https://192.168.100.79:4200
• https://localhost:4200

Update main.ts to add new origins (e.g., new Android device IPs).

Key Differences from server-desktop

Concern	server-desktop	server-android
ONNX runtime	onnxruntime-node native (+ WASM fallback via INFERENCE_BACKEND)	onnxruntime-web WASM only (numThreads=1)
SQLite driver	sqlite3 (native C++ bindings)	sql.js (TypeORM sqljs)
Provider selection	Env var INFERENCE_BACKEND (default: onnx-native)	Hardcoded OnnxWasmProvider
postprocessShared	Defined in onnx-native.provider, imported by WASM provider	Defined inline in onnx-wasm.provider
WebSocket surveillance	Separate SurveillanceGateway broadcasts monitor:data	Inline broadcast in VisionGateway to telemetry-room