Roberto Amoroso

Roberto Amoroso

Senior Research Engineer @ NVIDIA
VLMs & Multimodal Agents

NVIDIA

About me

Ciao! I am Roberto Amoroso, a Senior Research Engineer at NVIDIA in Munich, Germany 🇩🇪, where I build Multimodal Video Understanding and Vision-Language Models (VLMs) systems for large-scale retrieval, with a focus on Autonomous Vehicle applications.

My work centers on VLM-based retrieval architectures that align language with visual signals at scale, enabling users and AI agents to surface relevant moments, objects, and scenes across massive image and video collections.

Currently, I am training VLMs on fleet-scale video data to produce rich scene descriptions and retrieval-quality embeddings, enabling both search over large video collections and downstream agentic systems that reason about dynamic scenes — with a current focus on autonomous driving.

I completed my PhD through the ELLIS program and the International Doctorate in ICT at the AImageLab research group of the University of Modena and Reggio Emilia (UNIMORE) 🇮🇹, under the supervision of Prof. Rita Cucchiara and Prof. Lorenzo Baraldi.

During my PhD, I also completed an internship at LMU — Ludwig-Maximilians-Universität of Munich, Germany 🇩🇪, working on Multimodal LLMs for Video Question Answering and Open-vocabulary Segmentation under the co-supervision of Prof. Volker Tresp.

Earlier, I was a Research Scholar at the Networking Research Group in Saint Louis, USA 🇺🇸, working on super-resolution techniques applied to Internet traffic matrices.

Beyond my current focus, my past research has also covered the pre-training and optimization of Transformer architectures, image classification, self-supervised learning, deepfake detection of synthetic images, and image watermarking.

Feel free to reach out if you have any questions or curiosities! :)

Interests
  • Computer Vision
  • Deep Learning
  • Machine Learning
  • Multimodal Video Understanding
  • Vision-Language Models
  • Multimodal Agents & Agentic Retrieval
  • Text-to-Visual Retrieval
Education

  • ELLIS PhD in AI and Computer Vision, 2024

    UNIMORE, Italy 🇮🇹 | LMU, Germany 🇩🇪 | NVIDIA, Germany 🇩🇪

  • MS in Artificial Intelligence, 2020

    UNIMORE, Italy 🇮🇹 | AGH, Poland 🇵🇱 | Saint Louis University, USA 🇺🇸

  • BS in Computer Engineering, 2018

    UNIMORE, Italy 🇮🇹

Recent News

All news »

  • [Jun. 2025] Presenting @ NVIDIA GTC Paris 🇫🇷 a live demo of VLM-powered multimodal video–text retrieval systems for autonomous vehicles.

  • [Jan. 2025] Started a new position as Senior Research Engineer @ NVIDIA in Munich, Germany 🇩🇪, working on Multimodal Video Understanding for Autonomous Vehicles

  • [Oct. 2024] Our paper “Video Search: A Large-scale Video-Text Retrieval System for AV” has been accepted @ NTECH 2024 NVIDIA conference

  • [Feb. 2024] Started a new position as Machine Learning Engineering Intern @ NVIDIA in Munich, Germany 🇩🇪, working on Multimodal Video Understanding for Autonomous Vehicles

All news »

Experience

 
 
 
 
 
NVIDIA
Senior Research Engineer
NVIDIA
Jan 2024 – Present Munich, Germany 🇩🇪
Research activity focused on the engineering, development, and deployment of Multimodal Video Understanding techniques for Autonomous Vehicles.
 
 
 
 
 
NVIDIA
Machine Learning Engineering Intern
NVIDIA
Jan 2024 – Oct 2024 Munich, Germany 🇩🇪
Research activity focused on Video-Text Retrieval techniques for Autonomous Vehicles.
 
 
 
 
 
LMU @ Ludwig-Maximilians-Universität of Munich
PhD Intern
LMU @ Ludwig-Maximilians-Universität of Munich
Jun 2023 – Nov 2023 Munich, Germany 🇩🇪
Research activity focused on the development of novel Multimodal LLM for Video Question Answering and Open-vocabulary Image Segmentation techniques, under the co-supervision of Prof. Volker Tresp.
 
 
 
 
 
AImageLab @ University of Modena and Reggio Emilia
ELLIS PhD Student | International Doctorate in ICT
AImageLab @ University of Modena and Reggio Emilia
Nov 2021 – Oct 2024 Modena, Italy 🇮🇹
  • Successfully defended cum laude (with highest honors) my PhD thesis titled “Multimodal Attentive Deep Learning Architectures for Visual-Semantic Understanding”.
  • The European Laboratory for Learning and Intelligent Systems (ELLIS) supports cutting-edge machine learning research in Europe. ELLIS PhD students (<5% 2021 acceptance rate) are selected on the basis of academic achievement. My research activity is focused on multimodal machine learning, image segmentation, image classification, self-supervised learning, video retrieval, and video question answering.
  • 1st in the ranking of student candidates for the International Doctorate in ICT.
 
 
 
 
 
AImageLab @ University of Modena and Reggio Emilia
Research Fellow
AImageLab @ University of Modena and Reggio Emilia
Feb 2021 – Nov 2021 Modena, Italy 🇮🇹
Research activity under the supervision of Prof. Rita Cucchiara and Prof. Lorenzo Baraldi, aimed at the study, analysis, and development of novel Computer Vision and Deep Learning techniques.
 
 
 
 
 
CINI - Consorzio Interuniversitario Nazionale per l’Informatica
Research Engineer
CINI - Consorzio Interuniversitario Nazionale per l’Informatica
Nov 2020 – Jan 2021 Modena, Italy 🇮🇹
Development of a web platform for the management of data concerning the activities of European research centers, as part of the HumanE-AI-NET project, funded by the EU Framework Programme for Research and Innovation Horizon 2020.
 
 
 
 
 
Saint Louis University
Research Scholar
Saint Louis University
Mar 2020 – Sep 2020 St. Louis, USA 🇺🇸
  • Conducted research to develop my MS thesis, winner of the Best Poster Award at CoNEXT 2020.
 
 
 
 
 
AGH Akademia Górniczo-Hutnicza
Erasmus+
AGH Akademia Górniczo-Hutnicza
Sep 2019 – Feb 2020 Krakow, Poland 🇵🇱
  • Completed the following courses: Advanced Python Programming | Computer Vision | Cybersecurity and Cryptography | Programming in Javascript | Mobile App Development

Honors and Awards

  • [Sep. 2024] Outstanding Reviewer Award @ ECCV 2024

  • [Jul. 2022] ICVSS 2022 Reading Group Competition Award sponsored by Amazon Web Services (AWS) @ ICVSS 2022 for participation in the reading group led by Prof. Dr. Stefano Soatto (AWS and University of California Los Angeles)

Publications

Please see my Google Scholar for the complete publication list.
Quickly discover relevant content by filtering publications.
Scalable Parallel Prompting for Complex AV Video Captioning
[ CVPRW 2026 ] We propose pSVLMs, a scalable video captioning framework based on small VLMs that produces diverse intermediate captions, which are consolidated into a comprehensive unified description for autonomous driving datasets.
April Yang, Roberto Amoroso, Nikita Durasov, Devansh Bisla, Sandipan Kundu, Elmar Haussmann, Ruchi Bhargava, Maying Shen, Nadine Chang, Jose M. Alvarez
Cite
Scalable Parallel Prompting for Complex AV Video Captioning
Cosmos-Embed1: A Joint Video-Text Embedder for Physical AI
Cosmos-Embed1 is a joint video-text embedder tailored for physical AI applications, including autonomous vehicle (AV) and robotics. It can be used for text-to-video retrieval, inverse video search, semantic deduplication, zero-shot and k-nearest-neighbors (kNN) classification, and as a base model for video curation tasks.
Francesco Ferroni, Prithvijit Chattopadhyay, Greg Heinrich, Mike Ranzinger, Roberto Amoroso, Alice Luo, Andrew Wang, Ming-Yu Liu
Cite HuggingFace Website Demo
Cosmos-Embed1: A Joint Video-Text Embedder for Physical AI
Multimodal Attentive Deep Learning Architectures for Visual-Semantic Understanding
PhD Thesis investigating critical challenges associated with multimodal attentive architectures, including improving semantic segmentation accuracy, enabling open-vocabulary segmentation, and advancing video question answering.
Roberto Amoroso
PDF Cite
Multimodal Attentive Deep Learning Architectures for Visual-Semantic Understanding
MaPeT: Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training
[ CVIU 2025 ] We propose a novel self-supervised pre-training technique for Vision Transformer called MaPeT and a novel image tokenizer called k-CLIP which directly employs discretized CLIP features.
Lorenzo Baraldi, Roberto Amoroso, Marcella Cornia, Lorenzo_Baraldi, Andrea Pilzer, Rita Cucchiara
PDF Cite Code ArXiv
MaPeT: Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training
Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
[ WACV 2025 ] We propose PQR, a novel LLM-based framework for video question answering that introduces T-Former, a question-guided temporal querying Transformer designed to efficiently extract and integrate video-specific features tailored to a given question.
Roberto Amoroso, Gengyuan Zhang, Rajat Koner, Lorenzo Baraldi, Rita Cucchiara, Volker Tresp
PDF Cite ArXiv
Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
FreeDA: Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
[ CVPR 2024 ] We present FreeDA, a novel training-free diffusion-augmented method for open-vocabulary segmentation, which leverages diffusion models to visually localize generated concepts and local-global similarities to match superpixel-based class-agnostic regions with semantic classes.
Luca Barsellotti, Roberto Amoroso, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Cite Project
FreeDA: Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
FOSSIL: Free Open-Vocabulary Semantic Segmentation through Synthetic References Retrieval
[ WACV 2024 ] We present FOSSIL, a novel Unsupervised Open-Vocabulary Semantic Segmentation model that enables a self-supervised visual backbone to perform open-vocabulary segmentation directly on the visual modality by retrieving a support set of generated synthetic references.
Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, Rita Cucchiara
PDF Cite
FOSSIL: Free Open-Vocabulary Semantic Segmentation through Synthetic References Retrieval
What’s Outside the Intersection? Fine-grained Error Analysis for Semantic Segmentation Beyond IoU
[ WACV 2024 ] We present a novel method for enhancing semantic segmentation models evaluation by categorizing errors, offering insights into false positives/negatives, and improving performance through the combination of model strengths.
Maximilian Bernhard, Roberto Amoroso, Yannic Kindermann, Lorenzo Baraldi, Rita Cucchiara, Volker Tresp, Matthias Schubert
PDF Cite Code
What’s Outside the Intersection? Fine-grained Error Analysis for Semantic Segmentation Beyond IoU
Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images
[ TOMM 2024 ] We propose a novel deepfake detection method for images generated through Diffusion Models and created a new dataset COCO-Fake consisting of 650K generated fake images.
Roberto Amoroso, Davide Morelli, Marcella Cornia, Lorenzo Baraldi, Alberto Del Bimbo, Rita Cucchiara
PDF Cite Dataset ArXiv
Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images
Superpixel Positional Encoding to Improve ViT-based Semantic Segmentation Models
[ BMVC 2023 ] We present a novel superpixel-based positional encoding technique that combines Vision Transformer (ViT) features with superpixels priors to improve the performance of semantic segmentation architectures.
Roberto Amoroso, Matteo Tomei, Lorenzo Baraldi, Rita Cucchiara
PDF Cite
Superpixel Positional Encoding to Improve ViT-based Semantic Segmentation Models
Enhancing Open-Vocabulary Semantic Segmentation with Prototype Retrieval
[ ICIAP 2023 ] We propose a novel open-vocabulary semantic segmentation paradigm based on weakly supervised visual prototypes extracted from image-caption pairs and adopt a retrieval-based approach to combine visual and textual features to enhance segmentation performance.
Luca Barsellotti, Roberto Amoroso, Lorenzo Baraldi, Rita Cucchiara
Cite
Enhancing Open-Vocabulary Semantic Segmentation with Prototype Retrieval
A Federated Learning Approach to Traffic Matrix Estimation using Super-resolution Techniques
[ IEEE CCNC 2023 ] In this work, we propose a super-resolution technique for traffic matrix estimation. We also expand our design by employing a federated learning model to address scalability and improve performance.
Roberto Amoroso, Lorenzo Pappone, Flavio Esposito
Cite
Investigating Bidimensional Downsampling in Vision Transformer Models
[ ICIAP 2021 | Best Paper Award sponsored by NVIDIA ] We explore the application of a 2D max-pooling operator to improve the efficiency of Transformer-based architecture for classification.
Paolo Bruno, Roberto Amoroso, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, Rita Cucchiara
PDF Cite Video
Investigating Bidimensional Downsampling in Vision Transformer Models
Assessing the Role of Boundary-Level Objectives in Indoor Semantic Segmentation
[ CAIP 2021 | Oral ] We test and devise variants of both the Boundary and Active Boundary losses, two recent proposals which deal with the prediction of semantic boundaries.
Roberto Amoroso, Lorenzo Baraldi, Rita Cucchiara
PDF Cite
Assessing the Role of Boundary-Level Objectives in Indoor Semantic Segmentation
Improving Indoor Semantic Segmentation with Boundary-Level Objectives
[ IWANN 2021 | Oral ] We propose two loss functions that improve the semantic segmentation accuracy at the boundary level.
Roberto Amoroso, Lorenzo Baraldi, Rita Cucchiara
PDF Cite
Improving Indoor Semantic Segmentation with Boundary-Level Objectives
Estimation of Traffic Matrices via Super-resolution and Federated Learning
[ CoNEXT 2020 | Best Poster Award ] Inspired by the field of image processing, we propose a super-resolution technique for Internet traffic matrix inference.
Roberto Amoroso, Flavio Esposito, Maria Luisa Merani
PDF Cite
Estimation of Traffic Matrices via Super-resolution and Federated Learning

Professional Activities

Reviewer

  • International Conference on Computer Vision and Pattern Recognition (CVPR)
  • International Conference on Computer Vision (ICCV)
  • European Conference on Computer Vision (ECCV)
  • Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
  • Association for the Advancement of Artificial Intelligence (AAAI)
  • IEEE Transactions on Multimedia (TMM)
  • Pattern Recognition Letters (PRL)
  • ACM Multimedia (ACMMM)
  • International Conference on Pattern Recognition (ICPR)

Contact