​

Vision Capabilities

​

Overview

• Image description - Generate detailed descriptions of images
• Visual Q&A - Answer questions about image content
• OCR/Text extraction - Read text from images
• Object detection - Identify objects and entities
• Scene understanding - Understand context and relationships
• Chart analysis - Interpret graphs and visualizations

​

Vision-Capable Models

​

Basic Image Analysis

​

Simple Image Description

​

Visual Question Answering

​

Advanced Vision Use Cases

​

OCR and Text Extraction

​

Object and Entity Detection

​

Chart and Graph Analysis

​

Screenshot Analysis

​

Logo and Brand Detection

​

Multi-Image Analysis

​

Before/After Comparison

​

Multi-Image Context

​

Provider Comparison

​

OpenAI (GPT-4o, GPT-4-turbo)

• Fast processing
• Excellent general-purpose vision
• Strong multi-image capabilities
• Reliable OCR
• Good detail detection

• Real-time applications
• Multi-image analysis
• General image understanding
• Screenshot analysis

"model": "openai/gpt-4o"

​

Anthropic (Claude 3 Family)

• Superior reasoning about images
• Excellent document analysis
• Strong at complex visual tasks
• Detailed, thoughtful responses
• Multi-language support

• Document processing
• Complex reasoning tasks
• Detailed analysis
• Academic/research content

"model": "anthropic/claude-3-5-sonnet-20241022"

​

Google (Gemini 1.5)

• Extremely long context (up to 2GB)
• Fast processing (Flash variant)
• Strong multilingual capabilities
• Excellent for large documents
• Cost-effective (Flash)

• Large document processing
• Multi-page PDFs
• Video frame analysis
• High-volume applications

"model": "google/gemini-1.5-flash"

​

Mistral (Pixtral)

• European data residency
• Efficient processing
• Good price/performance
• Privacy-focused

• European compliance needs
• Cost-sensitive applications
• Privacy requirements

"model": "mistral/pixtral-12b"

​

Image Input Formats

​

HTTP(S) URLs

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/image.jpg"
    }
}

​

Base64 Data URLs

import base64

with open("image.jpg", "rb") as f:
    image_data = base64.b64encode(f.read()).decode('utf-8')

{
    "type": "image_url",
    "image_url": {
        "url": f"data:image/jpeg;base64,{image_data}"
    }
}

​

Uploaded File UUIDs

# Upload first
upload_response = requests.post(
    "https://api.edenai.run/v3/upload",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    files={"file": open("image.jpg", "rb")}
)
file_id = upload_response.json()["file_id"]

# Use in vision request
{
    "type": "file",
    "file": {"file_id": file_id}
}

​

Best Practices

​

Prompting for Vision

# Vague
"What's in this image?"

# Specific
"List all furniture items visible in this room photo, including their approximate positions and colors."

"Extract the following from this business card and format as JSON:
- name
- title
- company
- email
- phone"

"This is a medical X-ray of a chest. Identify any abnormalities or concerning features."

​

Image Quality Tips

• Use high-quality images (min 1024px on longest side)
• Avoid excessive compression
• Ensure text is legible

• Well-lit images work best
• Avoid glare and shadows
• Ensure good contrast

• Center subjects of interest
• Avoid clutter when possible
• Crop to relevant content

​

Temperature Settings

# Factual tasks (OCR, counting, detection)
"temperature": 0.1

# General description
"temperature": 0.5

# Creative interpretation
"temperature": 0.8

​

Cost Optimization

• Use gemini-1.5-flash for high-volume tasks
• Reserve claude-3-opus for complex analysis
• Use gpt-4o for balanced performance

• Resize images to minimum needed resolution
• Compress without losing critical details
• Use URLs instead of base64 when possible

​

Error Handling

​

Common Vision Errors

{
  "error": {
    "code": "unsupported_format",
    "message": "Image format .bmp is not supported"
  }
}

{
  "error": {
    "code": "image_too_large",
    "message": "Image size exceeds 20 MB limit for this provider"
  }
}

{
  "error": {
    "code": "invalid_image",
    "message": "Unable to process image data"
  }
}

​

Handling Vision Errors

​

Supported Image Formats

​

Next Steps

• Working with Media Files - Complete media guide
• File Attachments - Handle documents and PDFs
• Chat Completions - Core LLM features
• Streaming Responses - Handle SSE streams
• Upload Files - File management