Ovis-U1 Documentation

Complete guide to installing, configuring, and using Ovis-U1 for your multimodal AI projects and research.

Quick Start Guide

Prerequisites

System Requirements

  • Python 3.10 or higher
  • CUDA-compatible GPU (recommended)
  • 8GB+ GPU memory (for optimal performance)
  • 16GB+ system RAM

Required Dependencies

  • PyTorch 2.4.0+
  • Transformers 4.51.3+
  • DeepSpeed 0.15.4+
  • Conda or pip package manager

Step 1: Environment Setup

# Create and activate conda environment
conda create -n ovis-u1 python=3.10 -y
conda activate ovis-u1

Create a dedicated conda environment to avoid dependency conflicts.

Step 2: Repository Clone

# Clone the repository
git clone https://github.com/AIDC-AI/Ovis-U1.git
cd Ovis-U1

Download the complete Ovis-U1 codebase and navigate to the project directory.

Step 3: Dependency Installation

# Install required packages
pip install -r requirements.txt
pip install -e .

Install all required dependencies and the Ovis-U1 package in development mode.

Step 4: Verification

# Test installation
python -c "import ovis_u1; print('Installation successful!')"

Verify that the installation completed successfully and the module can be imported.

Core Functions

Image Understanding

# Single image understanding
python test_img_to_txt.py

Analyze and interpret individual images with comprehensive understanding capabilities.

Parameters:

  • • --image_path: Path to input image
  • • --prompt: Text prompt for understanding
  • • --max_tokens: Maximum output tokens

Multi-Image Understanding

# Multi-image understanding
python test_multi_img_to_txt.py

Process and analyze multiple images simultaneously for complex visual reasoning.

Parameters:

  • • --image_paths: List of image paths
  • • --prompt: Comparative analysis prompt
  • • --batch_size: Processing batch size

Text-to-Image Generation

# Generate images from text
python test_txt_to_img.py \
  --prompt "your description" \
  --height 1024 --width 1024

Create high-quality images from textual descriptions with customizable parameters.

Parameters:

  • • --prompt: Text description
  • • --height/width: Output dimensions
  • • --steps: Generation steps (50 default)
  • • --txt_cfg: Text guidance scale

Image Editing

# Edit existing images
python test_img_edit.py \
  --input_image "path/to/image" \
  --edit_prompt "edit description"

Perform sophisticated modifications on existing images with precise control.

Parameters:

  • • --input_image: Source image path
  • • --edit_prompt: Edit instructions
  • • --img_cfg: Image guidance scale
  • • --steps: Editing steps

Advanced Configuration

Model Configuration

Customize Ovis-U1 behavior through configuration files and environment variables.

# config.yaml example
model:
  max_length: 2048
  temperature: 0.7
  batch_size: 1
generation:
  default_steps: 50
  guidance_scale: 7.5
  default_resolution: [1024, 1024]

Memory Optimization

Low Memory Mode

export OVIS_LOW_MEMORY=true
python test_txt_to_img.py

Enable memory-efficient processing for systems with limited GPU memory.

Performance Tuning

Mixed Precision

export OVIS_MIXED_PRECISION=fp16
python test_img_to_txt.py

Use mixed precision for faster inference with minimal quality loss.

Python API Reference

OvisU1Model Class

from ovis_u1 import OvisU1Model

# Initialize model
model = OvisU1Model.from_pretrained("AIDC-AI/Ovis-U1-3B")

# Image understanding
response = model.understand_image(
  image_path="image.jpg",
  prompt="Describe this image"
)

# Text-to-image generation
image = model.generate_image(
  prompt="A sunset over mountains",
  height=1024, width=1024
)

# Image editing
edited_image = model.edit_image(
  image_path="input.jpg",
  edit_prompt="Add sunglasses to the person"
)

Method: understand_image()

Parameters

  • image_path (str): Path to image file
  • prompt (str): Understanding prompt
  • max_tokens (int): Max response length

Returns

str: Generated text description

Method: generate_image()

Parameters

  • prompt (str): Text description
  • height (int): Image height
  • width (int): Image width
  • steps (int): Generation steps

Returns

PIL.Image: Generated image

Troubleshooting

Common Issues

CUDA Out of Memory

Reduce batch size or enable low memory mode:

export OVIS_LOW_MEMORY=true

Slow Generation Speed

Enable mixed precision and optimize steps:

export OVIS_MIXED_PRECISION=fp16
python test_txt_to_img.py --steps 25

Import Errors

Verify installation and dependencies:

pip install -r requirements.txt --upgrade
pip install -e . --force-reinstall

Support and Community

GitHub Issues

Report bugs, request features, and get technical support through our GitHub repository.

Open an Issue →

Community Discussions

Join discussions, share experiences, and collaborate with other users.

Join Discussion →

Documentation

Access comprehensive guides, tutorials, and API documentation.

View Docs →