Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 35 hours
Course Outline
Introduction and Diagnostic Foundations
- Overview of LLM system failure modes and common Ollama-specific challenges.
- Establishing reproducible experiments and controlled testing environments.
- Debugging toolkit: local logs, request/response captures, and sandboxing techniques.
Reproducing and Isolating Failures
- Methods for generating minimal failing examples and seeds.
- Stateful versus stateless interactions: isolating context-related defects.
- Managing determinism, randomness, and nondeterministic behaviors.
Behavioral Evaluation and Metrics
- Quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies.
- Qualitative assessments: human-in-the-loop scoring and rubric design.
- Task-specific fidelity checks and acceptance criteria definition.
Automated Testing and Regression
- Unit tests for prompts and components, as well as scenario and end-to-end tests.
- Developing regression suites and golden example baselines.
- CI/CD integration for Ollama model updates and automated validation gates.
Observability and Monitoring
- Structured logging, distributed tracing, and correlation ID management.
- Key operational metrics: latency, token usage, error rates, and quality signals.
- Alerting, dashboards, and SLIs/SLOs for model-backed services.
Advanced Root Cause Analysis
- Tracing through graphed prompts, tool calls, and multi-turn conversation flows.
- Comparative A/B diagnosis and ablation studies.
- Data provenance, dataset debugging, and mitigation of dataset-induced failures.
Safety, Robustness, and Remediation Strategies
- Mitigation techniques: filtering, grounding, retrieval augmentation, and prompt scaffolding.
- Rollback, canary, and phased rollout patterns for model updates.
- Post-mortems, lessons learned, and continuous improvement cycles.
Summary and Next Steps
Requirements
- Extensive experience in developing and deploying LLM applications.
- Proficiency with Ollama workflows and model hosting practices.
- Working knowledge of Python, Docker, and fundamental observability tools.
Target Audience
- AI Engineers
- MLOps Professionals
- QA Teams overseeing production LLM systems
Custom Corporate Training
Training solutions designed exclusively for businesses.
- Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
- Flexible Schedule: Dates and times adapted to your team's agenda.
- Format: Online (live), In-company (at your offices), or Hybrid.
Price per private group, online live training, starting from 6500 € + VAT*
Contact us for an exact quote and to hear our latest promotions