Image Segmentation - SAM

This case study demonstrates training Meta's Segment Anything Model (SAM) for zero-shot image segmentation. SAM can segment any object in an image with remarkable accuracy, requiring only simple prompts like points, boxes, or text. It represents a breakthrough in generalist computer vision models.

Dataset: COCO Segmentation

• Source: HuggingFace (detection-datasets/coco)
• Type: Instance segmentation
• Size: 118,287 images
• Masks: 886,284 segmentation masks
• Classes: 80 object categories
• Format: Polygon annotations and binary masks

Model Configuration

{
  "model": "sam",
  "category": "computer_vision",
  "subcategory": "image-segmentation",
  "model_config": {
    "model_type": "vit_b",
    "pretrained": true,
    "prompt_type": "both",
    "batch_size": 4,
    "epochs": 50,
    "learning_rate": 0.0001,
    "image_size": [1024, 1024]
  }
}

Training Results

IoU Performance

Intersection over Union scores for segmentation quality:

Performance by Object Category

Best segmented object types:

Prompt Efficiency

Number of prompts needed for accurate segmentation:

Segmentation Complexity

Performance on simple vs complex scenes:

Zero-Shot Performance

SAM's ability to segment unseen object categories:

Common Use Cases

• Medical Imaging: Segment organs, tumors, lesions in MRI/CT scans
• Autonomous Driving: Segment road, vehicles, pedestrians, obstacles
• Agriculture: Identify and segment crops, weeds, diseases
• E-commerce: Product background removal, image editing
• Video Editing: Object isolation for effects and compositing
• Satellite Imagery: Land use segmentation, building detection
• AR/VR: Real-time environment understanding and occlusion
• Scientific Research: Cell segmentation, microscopy analysis

Key Settings

Essential Parameters

• model_type: vit_b (base), vit_l (large), vit_h (huge)
• prompt_type: "point", "box", "both", or "automatic"
• points_per_side: Grid points for automatic segmentation
• pred_iou_thresh: Quality threshold for mask filtering
• stability_score_thresh: Mask stability threshold

Prompt Configuration

• positive_points: Click on object to segment
• negative_points: Click on background to exclude
• box_prompt: Bounding box around object
• mask_prompt: Rough mask for refinement
• text_prompt: Natural language description (experimental)

Advanced Configuration

• multimask_output: Generate multiple mask proposals
• return_logits: Return raw logits for downstream tasks
• crop_n_layers: Multi-crop inference for high-res images
• crop_overlap_ratio: Overlap between crops
• postprocess_masks: Smoothing and refinement

Performance Metrics

• Mean IoU: 91.2% on COCO validation
• Boundary F-score: 88.7% (accurate edge detection)
• Zero-shot IoU: 85.6% on unseen similar classes
• Inference Speed: 50ms per image (ViT-B, 1024×1024)
• Model Size: 375 MB (ViT-B), 1.25 GB (ViT-H)
• Parameters: 91M (ViT-B), 308M (ViT-L), 636M (ViT-H)

Tips for Success

1. Prompt Selection: Box prompts are more accurate than single points
2. Multiple Points: Use 2-3 points for complex objects
3. Negative Prompts: Add negative points to exclude unwanted regions
4. Image Resolution: Higher resolution improves boundary accuracy
5. Post-processing: Apply morphological operations to smooth masks
6. Batch Processing: Use automatic mode for segmenting entire images
7. Fine-tuning: Adapt to specific domains with limited data

Example Scenarios

Scenario 1: Medical CT Scan

• Input: Chest CT image
• Prompt: 3 points on lung region + 1 negative point on ribs
• Output: Precise lung segmentation mask
• IoU: 94.3%
• Use Case: Lung volume measurement, disease detection

Scenario 2: Product Photography

• Input: Product on white background
• Prompt: Bounding box around product
• Output: Clean product mask for background removal
• IoU: 97.8%
• Use Case: E-commerce image editing, catalog creation

Scenario 3: Autonomous Vehicle

• Input: Street scene from vehicle camera
• Prompt: Automatic segmentation (no manual prompts)
• Output: 15 object masks (vehicles, pedestrians, signs)
• Processing Time: 220ms (all objects)
• Use Case: Real-time scene understanding, obstacle avoidance

Troubleshooting

Problem: Mask includes background regions

• Solution: Add negative points on background, use tighter box prompt

Problem: Missing small details (thin structures)

• Solution: Increase image resolution, add more positive points

Problem: Over-segmentation (too many fragments)

• Solution: Increase stability_score_thresh, use box instead of points

Problem: Slow inference on high-res images

• Solution: Use ViT-B instead of ViT-H, reduce image size, enable crop mode

Problem: Poor performance on domain-specific images

• Solution: Fine-tune on domain data, use more prompts, adjust thresholds

Model Architecture Highlights

SAM consists of:

• Image Encoder: Vision Transformer (ViT) backbone
  • Processes 1024×1024 images
  • Generates rich image embeddings
• Prompt Encoder:
  • Encodes points, boxes, masks, text
  • Lightweight transformer
• Mask Decoder:
  • Predicts segmentation masks
  • Outputs multiple mask proposals with confidence scores
• Promptable Design: Single model handles any prompt type

Model Variants Comparison

Next Steps

After training your SAM model, you can:

• Deploy for interactive annotation tools
• Build automatic dataset labeling pipelines
• Create video object segmentation system (with tracking)
• Integrate with image editing applications
• Fine-tune for medical imaging workflows
• Export to mobile (iOS/Android) with optimization
• Combine with object detection for full scene understanding
• Use for 3D reconstruction and depth estimation