Deep Learning is a core paradigm for modern agents in basically all fields of application for AI. The SAG research here investigates advanced neural learning paradigms, with a focus on optimizing efficiency, scalability, and interpretability of the resulting DL-based AI agents. We explore a variety of models—from kernel-based approaches to large-scale transformer architectures—and develop methods for their effective training, adaptation, and deployment on real world tasks. Our work addresses both theoretical and applied challenges, aiming to balance model complexity with real-world constraints. We are particularly interested in sustainable AI, prompting strategies, and techniques that enable intelligent behavior with limited resources or supervision. These efforts contribute to building resilient and efficient AI systems ready for broad deployment.
Kernel Machines
The SAG group has a long-standing tradition in applying kernel methods to structured and linguistic data. Through the KeLP framework, we have designed efficient and flexible kernel-based models that can process syntactic trees, semantic roles, and relational data representations. These approaches allow us to capture fine-grained similarities between complex structures, providing robust alternatives to deep learning when interpretability and structure sensitivity are key. Our work blends the precision of kernel methods with modern learning paradigms, enabling hybrid models that combine symbolic and statistical reasoning. This line of research remains fundamental in tackling tasks where data structure and grammar play a central role.
Computational Methods for Model Optimization
The SAG researches develop and evaluate scalable optimization strategies for training and adapting neural architectures. Our work addresses challenges such as memory efficiency, convergence speed, and training stability across a wide range of NLP and multimodal tasks. We integrate techniques like parameter-efficient fine-tuning (e.g., LoRA), knowledge distillation, and multitask training to improve adaptability without compromising performance. These methods are especially relevant when deploying large models on limited hardware or in dynamic application contexts. Optimization research is at the core of our efforts to make advanced AI systems more practical and sustainable.
Transformer-based Models
Our research makes extensive use of Transformer-based models to address a broad spectrum of NLP tasks, including semantic parsing, relation classification, stance detection, and question answering. We investigate how pre-trained transformer encoders and decoders can be fine-tuned or instructed to handle linguistic complexity, domain specificity, and multilingual variation. In addition to standard fine-tuning, we explore instruction-based prompting and continual learning paradigms to make these models more versatile. We also study their limitations and opportunities in tasks requiring world knowledge, logic, or grounding. Our work contributes to the practical and theoretical understanding of transformers in real-world applications.
Large Language Models
Our research makes extensive use of Transformer-based models to address a broad spectrum of NLP tasks, including semantic parsing, relation classification, stance detection, and question answering. We investigate how pre-trained transformer encoders and decoders can be fine-tuned or instructed to handle linguistic complexity, domain specificity, and multilingual variation. In addition to standard fine-tuning, we explore instruction-based prompting and continual learning paradigms to make these models more versatile. We also study their limitations and opportunities in tasks requiring world knowledge, logic, or grounding. Our work contributes to the practical and theoretical understanding of transformers in real-world applications.
Sustainable Deep Learning
Sustainability is a key concern in our deep learning research, not just in computational terms but also in social and ethical impact. We investigate how to reduce energy consumption and training costs through lightweight adaptation methods, efficient architectures, and data-efficient training protocols. We also study the ethical behavior of AI systems, including fairness, bias mitigation, and alignment with human values in sensitive tasks. Our goal is to develop AI that is not only powerful, but also responsible, transparent, and accessible. Sustainable deep learning is a cross-cutting theme that guides how we build and deploy intelligent systems.
Prompt Engineering
Prompt engineering is central to our work with modern LLMs. We design strategies for guiding model behavior through carefully crafted instructions and task formulations, enabling effective zero-shot and few-shot learning. Our research includes building prompt collections, evaluating prompt transferability across tasks, and studying relationships and dependencies between model responses and prompt structures. We are particularly focused on applying prompt engineering in multilingual and low-resource settings, where it serves as a bridge between pre-trained capabilities and specific user needs. These methods allow us to scale intelligent behavior with minimal supervision and maximal adaptability.