AI Image Processing Requirements

Overview

This document specifies the technical requirements for implementing Background AI Image Processing in the Plings object creation system. The feature integrates AI-powered image analysis into the existing non-blocking image upload workflow to provide intelligent suggestions for object metadata, classification, and duplicate detection.

Integration with Existing System

Current Implementation Status

✅ Foundation Ready:

  • Image upload system with background processing
  • Non-blocking user experience design
  • State management for upload progress tracking
  • Direct Supabase Storage integration
  • CreateObjectModal with extensible architecture

🔄 Extensions Required:

  • AI analysis state management
  • Suggestion UI components
  • Backend AI service integration
  • Progressive enhancement framework

Technical Architecture

Frontend Component Extensions

1. Enhanced Image Upload State

Extend existing ImageUploadState:

interface AIEnhancedImageUploadState extends ImageUploadState {
  localImages: (LocalImage & {
    // AI Processing Extensions
    aiAnalysisStatus: 'pending' | 'analyzing' | 'completed' | 'failed' | 'disabled';
    aiResults?: AIAnalysisResult;
    aiConfidence?: number;
    aiProcessingTime?: number;
    aiError?: string;
  })[];
  
  // Global AI Settings
  aiEnabled: boolean;
  aiPreferences: AIUserPreferences;
}

interface AIAnalysisResult {
  classification: ObjectClassification;
  brandDetection: BrandDetection;
  visualProperties: VisualProperties;
  similarObjects: SimilarObject[];
  nameSuggestions: NameSuggestion[];
  confidenceScore: number;
  processingTime: number;
}

interface AIUserPreferences {
  autoAcceptClassification: boolean;
  autoAcceptNameSuggestions: boolean;
  showSimilarObjectWarnings: boolean;
  duplicateDetectionThreshold: number; // 0.0 - 1.0
  preferredAIMode: 'suggestions' | 'auto' | 'disabled';
}

2. AI Suggestion Components

Create new React components:

src/components/objects/creation/ai/
├── AISuggestionProvider.tsx       # Context provider for AI state
├── AIAnalysisIndicator.tsx        # Processing progress indicator
├── AIClassificationSuggestion.tsx # Object type suggestions
├── AINameSuggestions.tsx          # Smart name generation
├── AIPropertySuggestions.tsx      # Visual property extraction
├── AISimilarObjectsWarning.tsx    # Duplicate detection alert
├── AIConfidenceDisplay.tsx        # Show confidence scores
├── AISettingsPanel.tsx            # User preference controls
└── hooks/
    ├── useAIAnalysis.tsx          # AI processing hook
    ├── useAISuggestions.tsx       # Suggestion management
    └── useAIPreferences.tsx       # User preference management

3. CreateObjectModal Integration

Enhanced modal with AI components:

const CreateObjectModal = () => {
  const { aiEnabled, aiResults, aiPreferences } = useAIAnalysis();
  
  return (
    <Modal>
      {/* Existing image upload section */}
      <ImageUploadSection onImageUpload={handleImageUpload} />
      
      {/* AI Analysis Section - NEW */}
      {aiEnabled && (
        <AIAnalysisSection>
          <AIAnalysisIndicator />
          {aiResults && (
            <>
              <AIClassificationSuggestion />
              <AINameSuggestions />
              <AIPropertySuggestions />
              <AISimilarObjectsWarning />
            </>
          )}
        </AIAnalysisSection>
      )}
      
      {/* Enhanced existing sections with AI integration */}
      <ObjectDetailsForm 
        aiSuggestions={aiResults?.nameSuggestions}
        aiProperties={aiResults?.visualProperties}
      />
      
      {/* Existing organization and location sections */}
    </Modal>
  );
};

Backend AI Service Architecture

1. AI Analysis Service

New backend service for image processing:

class AIAnalysisService:
    def __init__(self):
        self.vision_client = OpenAIVisionClient()
        self.similarity_engine = ObjectSimilarityEngine()
        self.brand_detector = BrandDetectionService()
        
    async def analyze_object_image(
        self, 
        image_url: str, 
        user_id: str,
        organization_id: str
    ) -> AIAnalysisResult:
        """
        Comprehensive AI analysis of uploaded object image
        """
        tasks = await asyncio.gather(
            self._classify_object(image_url),
            self._detect_brand(image_url),
            self._extract_properties(image_url),
            self._find_similar_objects(image_url, organization_id),
            self._generate_name_suggestions(image_url, user_id),
            return_exceptions=True
        )
        
        return self._combine_results(tasks)

2. GraphQL Schema Extensions

New operations for AI processing:

# AI Analysis Types
type AIAnalysisResult {
  id: ID!
  status: AIAnalysisStatus!
  classification: ObjectClassification
  brandDetection: BrandDetection
  visualProperties: VisualProperties
  similarObjects: [SimilarObject!]!
  nameSuggestions: [NameSuggestion!]!
  confidenceScore: Float!
  processingTimeMs: Int!
  createdAt: String!
}

type ObjectClassification {
  category: String!
  type: String!
  subtype: String
  confidence: Float!
  alternativeTypes: [String!]!
}

type BrandDetection {
  brand: String
  model: String
  detectedText: [String!]!
  confidence: Float!
  logoDetected: Boolean!
}

type VisualProperties {
  colors: [String!]!
  materials: [String!]!
  estimatedSize: String
  condition: String
  style: String
  confidence: Float!
}

type SimilarObject {
  objectId: ID!
  name: String!
  similarity: Float!
  matchReason: String!
  thumbnail: String
  organization: Organization!
}

type NameSuggestion {
  suggestion: String!
  reasoning: String!
  confidence: Float!
  includesUserPattern: Boolean!
  includesBrandInfo: Boolean!
}

enum AIAnalysisStatus {
  PENDING
  ANALYZING
  COMPLETED
  FAILED
  CANCELLED
}

# Mutations
mutation analyzeObjectImage($input: AnalyzeImageInput!) {
  analyzeObjectImage(input: $input) {
    analysisId: ID!
    estimatedCompletionTime: Int!
    status: AIAnalysisStatus!
  }
}

mutation updateAIPreferences($input: AIPreferencesInput!) {
  updateAIPreferences(input: $input) {
    preferences: AIUserPreferences!
  }
}

# Queries
query getAIAnalysis($analysisId: ID!) {
  aiAnalysis(id: $analysisId) {
    ...AIAnalysisResult
  }
}

query getAIPreferences {
  myAIPreferences {
    ...AIUserPreferences
  }
}

# Subscriptions
subscription aiAnalysisProgress($analysisId: ID!) {
  aiAnalysisUpdate(analysisId: $analysisId) {
    status: AIAnalysisStatus!
    progress: Float!
    partialResults: AIAnalysisResult
    estimatedTimeRemaining: Int
  }
}

AI Service Integrations

1. Image Classification Service

OpenAI Vision API Integration:

interface VisionAPIConfig {
  provider: 'openai' | 'google' | 'aws';
  apiKey: string;
  model: string;
  maxTokens: number;
  temperature: number;
}

const classificationPrompt = `
Analyze this image of a physical object and provide:
1. Category (furniture, electronics, tools, etc.)
2. Specific type (lamp, screwdriver, laptop, etc.)
3. Subtype if applicable (desk lamp, Phillips head screwdriver, etc.)
4. Confidence score (0.0-1.0)
5. Alternative classifications

Return as JSON format.
`;

2. Brand Detection Service

OCR + Logo Recognition:

interface BrandDetectionConfig {
  ocrProvider: 'tesseract' | 'google' | 'aws';
  logoRecognitionProvider: 'google' | 'clarifai';
  confidenceThreshold: number;
  supportedBrands: string[];
}

Vector Similarity Engine:

class ObjectSimilarityEngine:
    def __init__(self):
        self.embedding_model = ImageEmbeddingModel()
        self.vector_db = VectorDatabase()
        
    async def find_similar_objects(
        self, 
        image_url: str, 
        organization_id: str,
        threshold: float = 0.75
    ) -> List[SimilarObject]:
        # Generate image embedding
        embedding = await self.embedding_model.embed_image(image_url)
        
        # Search vector database
        similar = await self.vector_db.similarity_search(
            embedding, 
            organization_filter=organization_id,
            threshold=threshold
        )
        
        return similar

User Interface Specifications

1. AI Analysis Progress Indicator

Component: AIAnalysisIndicator.tsx

const AIAnalysisIndicator = ({ analysis }: { analysis: AIAnalysis }) => {
  return (
    <div className="ai-analysis-indicator">
      <div className="analysis-status">
        <AiIcon className="animate-spin" />
        <span>Analyzing image...</span>
        <ProgressBar value={analysis.progress} />
      </div>
      
      <div className="analysis-steps">
        <Step completed={analysis.classificationComplete}>
          Object classification
        </Step>
        <Step completed={analysis.brandDetectionComplete}>
          Brand detection
        </Step>
        <Step completed={analysis.similaritySearchComplete}>
          Finding similar objects
        </Step>
      </div>
      
      <div className="estimated-time">
        ~{analysis.estimatedTimeRemaining}s remaining
      </div>
    </div>
  );
};

2. Classification Suggestion Component

Component: AIClassificationSuggestion.tsx

const AIClassificationSuggestion = ({ 
  classification, 
  onAccept, 
  onReject 
}: ClassificationSuggestionProps) => {
  return (
    <Card className="ai-suggestion classification">
      <CardHeader>
        <div className="suggestion-header">
          <BrainIcon />
          <span>AI Classification</span>
          <ConfidenceBadge confidence={classification.confidence} />
        </div>
      </CardHeader>
      
      <CardContent>
        <div className="classification-result">
          <h3>{classification.type}</h3>
          <p className="category">Category: {classification.category}</p>
          {classification.subtype && (
            <p className="subtype">Subtype: {classification.subtype}</p>
          )}
        </div>
        
        {classification.alternativeTypes.length > 0 && (
          <div className="alternatives">
            <span>Or maybe:</span>
            {classification.alternativeTypes.map(type => (
              <Button 
                key={type} 
                variant="outline" 
                size="sm"
                onClick={() => onAccept(type)}
              >
                {type}
              </Button>
            ))}
          </div>
        )}
      </CardContent>
      
      <CardActions>
        <Button onClick={onAccept} variant="primary">
          Accept Classification
        </Button>
        <Button onClick={onReject} variant="outline">
          Choose Different
        </Button>
        <Button onClick={() => {}} variant="ghost">
          Ignore Suggestion
        </Button>
      </CardActions>
    </Card>
  );
};

3. Similar Objects Warning Component

Component: AISimilarObjectsWarning.tsx

const AISimilarObjectsWarning = ({ 
  similarObjects, 
  onViewObject, 
  onCreateAnyway,
  onMarkDifferent 
}: SimilarObjectsWarningProps) => {
  const highSimilarity = similarObjects.filter(obj => obj.similarity > 0.8);
  
  if (highSimilarity.length === 0) return null;
  
  return (
    <Alert variant="warning" className="similar-objects-warning">
      <AlertTriangleIcon />
      <AlertTitle>Possible Duplicate Detected</AlertTitle>
      <AlertDescription>
        We found {highSimilarity.length} similar object(s) that might be 
        the same as what you're creating:
      </AlertDescription>
      
      <div className="similar-objects-list">
        {highSimilarity.map(obj => (
          <div key={obj.objectId} className="similar-object-item">
            <img src={obj.thumbnail} alt={obj.name} />
            <div className="object-info">
              <h4>{obj.name}</h4>
              <p>{obj.organization.name}</p>
              <Badge variant="secondary">
                {Math.round(obj.similarity * 100)}% match
              </Badge>
            </div>
            <Button 
              variant="outline" 
              size="sm"
              onClick={() => onViewObject(obj.objectId)}
            >
              View Object
            </Button>
          </div>
        ))}
      </div>
      
      <AlertActions>
        <Button onClick={onCreateAnyway} variant="primary">
          Create Anyway
        </Button>
        <Button onClick={onMarkDifferent} variant="outline">
          This is Different
        </Button>
      </AlertActions>
    </Alert>
  );
};

4. AI Settings Panel

Component: AISettingsPanel.tsx

const AISettingsPanel = ({ 
  preferences, 
  onUpdatePreferences 
}: AISettingsPanelProps) => {
  return (
    <Card className="ai-settings-panel">
      <CardHeader>
        <CardTitle>AI Assistant Settings</CardTitle>
      </CardHeader>
      
      <CardContent>
        <div className="settings-grid">
          <Toggle
            checked={preferences.autoAcceptClassification}
            onCheckedChange={(checked) => 
              onUpdatePreferences({
                ...preferences,
                autoAcceptClassification: checked
              })
            }
          >
            Auto-accept object classifications
          </Toggle>
          
          <Toggle
            checked={preferences.autoAcceptNameSuggestions}
            onCheckedChange={(checked) =>
              onUpdatePreferences({
                ...preferences, 
                autoAcceptNameSuggestions: checked
              })
            }
          >
            Auto-apply name suggestions
          </Toggle>
          
          <Toggle
            checked={preferences.showSimilarObjectWarnings}
            onCheckedChange={(checked) =>
              onUpdatePreferences({
                ...preferences,
                showSimilarObjectWarnings: checked
              })
            }
          >
            Warn about similar objects
          </Toggle>
          
          <div className="slider-setting">
            <label>Duplicate Detection Sensitivity</label>
            <Slider
              value={preferences.duplicateDetectionThreshold}
              onValueChange={(value) =>
                onUpdatePreferences({
                  ...preferences,
                  duplicateDetectionThreshold: value
                })
              }
              min={0.5}
              max={0.95}
              step={0.05}
            />
          </div>
          
          <Select
            value={preferences.preferredAIMode}
            onValueChange={(mode) =>
              onUpdatePreferences({
                ...preferences,
                preferredAIMode: mode
              })
            }
          >
            <SelectTrigger>
              <SelectValue placeholder="AI Mode" />
            </SelectTrigger>
            <SelectContent>
              <SelectItem value="suggestions">Show Suggestions</SelectItem>
              <SelectItem value="auto">Auto-Apply</SelectItem>
              <SelectItem value="disabled">Disabled</SelectItem>
            </SelectContent>
          </Select>
        </div>
      </CardContent>
    </Card>
  );
};

Performance Requirements

AI Processing Performance

  • Analysis Completion: <30 seconds for comprehensive analysis
  • Classification Speed: <5 seconds for basic object type
  • Similar Object Search: <10 seconds for organization-wide search
  • Name Generation: <3 seconds for suggestion list
  • Progressive Results: Show partial results as they complete

Frontend Performance

  • State Updates: Real-time progress without blocking UI
  • Memory Usage: Efficient handling of AI result data
  • Network Optimization: Minimal API calls for results polling
  • Fallback Performance: Graceful degradation when AI unavailable

Backend Performance

  • Concurrent Processing: Handle multiple AI analyses simultaneously
  • Rate Limiting: Prevent abuse of AI services
  • Caching Strategy: Cache results for identical images
  • Cost Management: Monitor and limit AI service usage

Error Handling and Fallbacks

AI Service Failures

Graceful Degradation Strategy:

interface AIErrorHandling {
  // Service unavailable
  onServiceUnavailable: () => void; // Continue without AI
  
  // Analysis timeout
  onAnalysisTimeout: () => void; // Show partial results
  
  // Invalid image
  onInvalidImage: () => void; // Skip AI, proceed normally
  
  // Rate limit exceeded
  onRateLimitExceeded: () => void; // Queue for later processing
  
  // Authentication failure
  onAuthFailure: () => void; // Disable AI temporarily
}

User Experience During Errors

  • Transparent Communication: Clear error messages
  • Continued Functionality: Object creation works without AI
  • Retry Options: Manual retry for failed analyses
  • Settings Override: Disable AI if repeatedly failing

Security and Privacy

Image Processing Security

  • Secure Transmission: All AI service calls over HTTPS
  • No Data Retention: Configure AI services for zero retention
  • Access Control: Only authorized users can trigger AI analysis
  • Audit Logging: Log all AI processing requests and results

Privacy Controls

  • User Opt-out: Complete disable of AI processing
  • Organization Settings: Admin control over AI features
  • Data Locality: Option for local AI processing
  • Consent Management: Clear user consent for AI analysis

API Security

  • Authentication: Secure AI service API keys
  • Rate Limiting: Prevent abuse and cost overruns
  • Input Validation: Sanitize all inputs to AI services
  • Error Sanitization: Don’t expose internal errors to frontend

Testing Requirements

AI Integration Testing

describe('AI Image Processing', () => {
  test('should trigger AI analysis after image upload', async () => {
    // Test AI analysis initiation
  });
  
  test('should handle AI analysis results progressively', async () => {
    // Test progressive result display
  });
  
  test('should fallback gracefully when AI fails', async () => {
    // Test error handling and fallbacks
  });
  
  test('should respect user AI preferences', async () => {
    // Test preference application
  });
  
  test('should detect similar objects accurately', async () => {
    // Test duplicate detection
  });
});

Performance Testing

  • Load Testing: Multiple concurrent AI analyses
  • Stress Testing: High-volume image processing
  • Timeout Testing: Long-running AI operations
  • Memory Testing: AI result data handling

User Experience Testing

  • A/B Testing: With vs without AI suggestions
  • Usability Testing: AI suggestion interfaces
  • Accessibility Testing: AI components with screen readers
  • Mobile Testing: AI features on mobile devices

Deployment and Configuration

Environment Configuration

# AI Service Configuration
ai_services:
  vision_api:
    provider: "openai"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4-vision-preview"
    timeout: 30000
    
  similarity_search:
    enabled: true
    threshold: 0.75
    max_results: 5
    
  brand_detection:
    enabled: true
    ocr_provider: "google"
    logo_detection: true
    
ai_features:
  enabled: true
  auto_trigger: true
  max_concurrent_analyses: 10
  cache_results: true
  cache_duration: "24h"

Feature Flags

interface AIFeatureFlags {
  aiProcessingEnabled: boolean;
  classificationEnabled: boolean;
  brandDetectionEnabled: boolean;
  similarObjectSearchEnabled: boolean;
  nameGenerationEnabled: boolean;
  realTimeProgressEnabled: boolean;
}

Monitoring and Analytics

  • AI Processing Metrics: Success rates, processing times, costs
  • User Engagement: Suggestion acceptance rates, feature usage
  • Error Monitoring: Failed analyses, service outages, timeouts
  • Cost Tracking: AI service usage and spending

Integration References

Implementation Dependencies

  • Backend API: AI service integration and GraphQL schema
  • Image Upload: Extension of existing background upload system
  • State Management: Enhancement of current upload state tracking
  • UI Components: New suggestion and settings components

Requirements Document Created: Mån 7 Jul 2025 10:55:32 CEST - Technical specifications for Background AI Image Processing integration