Multimodal LLM Workflows in Vertex AI Training Course

Introduction to Multimodal LLMs in Vertex AI

• An overview of multimodal capabilities within Vertex AI
• Exploration of Gemini models and their supported modalities
• Application scenarios in enterprise and research contexts

Configuring the Development Environment

• Setting up Vertex AI for multimodal workflow execution
• Managing datasets that span multiple modalities
• Practical lab: environment configuration and dataset preparation

Long Context Windows and Advanced Reasoning

• Understanding the mechanics of long-context workflows
• Applications in strategic planning and decision-making processes
• Practical lab: implementing long-context analysis techniques

Designing Cross-Modal Workflows

• Synthesising text, audio, and image analysis
• Chaining multimodal steps to form cohesive pipelines
• Practical lab: designing a functional multimodal pipeline

Managing Gemini API Parameters

• Configuring inputs and outputs for multimodal interactions
• Optimising inference speed and operational efficiency
• Practical lab: tuning Gemini API parameters for best results

Advanced Applications and Integrations

• Creating interactive multimodal agents and virtual assistants
• Integrating with external APIs and third-party tools
• Practical lab: constructing a complete multimodal application

Evaluation and Iteration

• Testing the performance of multimodal systems
• Applying metrics for accuracy, alignment, and model drift
• Practical lab: evaluating the effectiveness of multimodal workflows

Summary and Next Steps