SAML
Society-Aware Machine Learning (SAML)
SAML is a European Research Council (ERC) Starting Grant project focused on developing machine learning systems that are not only accurate and efficient but also fair, transparent, and socially responsible. By incorporating the perspectives of multiple stakeholders throughout the AI development process, SAML aims to create trustworthy and human-centered AI technologies.
This work is funded by the European Union through the ERC Starting Grant programme (Grant Agreement No. 101040177).
ERC Summary & Objectives
Objective 1: Developing a Society-Aware Machine Learning Framework
SAML establishes the methodological foundations for society-aware machine learning, where algorithms are designed and evaluated by explicitly accounting for the utilities, preferences, and welfare of all relevant stakeholders. This required addressing several fundamental technical challenges, including estimating the causal consequences of decisions and actions, comparing competing stakeholder objectives, and extending fairness assessment beyond traditional prediction-based metrics.
We developed new methods for causal inference that estimate the downstream consequences of decisions under realistic conditions, including hidden confounding and temporal dynamics (DeCaFlow, A Practical Approach to Causal Inference over Time).
We further introduced methods for comparing and jointly optimizing diverse objectives, integrating preferences and allowing practitioners to systematically explore Pareto-optimal trade-offs (COPA, Hellinger Multimodal Variational Autoencoders, Hyper-Transforming Latent Diffusion Models).
Building on these technical foundations, we established new frameworks for fairness assessment that move beyond binary decision outcomes by considering treatment decisions, their downstream consequences, long-term societal effects, and the welfare of multiple stakeholders (A Causal Framework to Measure and Mitigate Non-binary Treatment Discrimination, Fairness Beyond Binary Decisions, Designing Long-term Group Fair Policies in Dynamical Systems, Long-term Fairness with Selective Labels, First-See-Then-Design).
Objective 2: Achieving Consensual Machine Learning
Building on methodological advances (summarized under Objective 1), SAML develops machine learning methods that seek solutions benefiting all involved stakeholders.
We introduced decision-making frameworks that explicitly balance the utilities of system deployers, affected individuals, and society, enabling transparent exploration of performance–fairness trade-offs and the design of policies that remain fair over time (First-See-Then-Design, Designing Long-term Group Fair Policies in Dynamical Systems, Long-term Fairness with Selective Labels).
Beyond institutional decision-making, SAML empowers individuals affected by automated decisions. We develop a framework allowing individuals to overcome negative automated decisions through personalized algorithmic recourse that accounts for their preferences and constraints (From Universal to Individualized Actionability).
Finally, SAML improves the transparency and interpretability of machine learning by making trade-offs between competing objectives explicit and by developing inherently interpretable models that better expose the reasoning behind automated decisions (COPA, First-See-Then-Design, Towards Reasonable Concept Bottleneck Models).
Objective:3 Addressing Ethical Challenges in Machine Learning
The methodological advances developed throughout SAML translate into practical solutions for addressing ethical challenges arising from the deployment of machine learning in society.
We developed methods to identify and mitigate unfairness in high-stakes applications such as lending by uncovering previously overlooked forms of discrimination, extending fairness assessment beyond conventional metrics, and evaluating fairness in modern AI systems such as large language models (A Causal Framework to Measure and Mitigate Non-binary Treatment Discrimination, Fairness Beyond Binary Decisions, Accept or Deny? Evaluating LLM Fairness and Performance in Loan Approval, On the Misalignment Between Legal Notions and Statistical Metrics of Intersectional Fairness).
In parallel, the project increases individuals’ ability to respond to automated decisions through personalized recourse while improving the transparency and accountability of AI systems through interpretable models and explicit analysis of trade-offs between competing objectives (From Universal to Individualized Actionability, Towards Reasonable Concept Bottleneck Models).
SAML Publications
2026
Gupta, Kavya; Kalampalikis, Nektarios; Heitz, Christoph; Valera, Isabel
First-See-Then-Design: A Multi-Stakeholder View for Optimal Performance-Fairness Trade-Offs Proceedings Article
In: Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency, pp. 218–248, ACM, 2026.
Abstract | Links | BibTeX | Tags: isabel, kavya, nektarios, saml
@inproceedings{Gupta_2026,
title = {First-See-Then-Design: A Multi-Stakeholder View for Optimal Performance-Fairness Trade-Offs},
author = {Kavya Gupta and Nektarios Kalampalikis and Christoph Heitz and Isabel Valera},
url = {http://dx.doi.org/10.1145/3805689.3806541},
doi = {10.1145/3805689.3806541},
year = {2026},
date = {2026-06-01},
urldate = {2026-06-01},
booktitle = {Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency},
pages = {218–248},
publisher = {ACM},
series = {FAccT ’26},
abstract = {Fairness in algorithmic decision-making is often defined in the predictive space, where predictive performance - used as a proxy for decision-maker (DM) utility - is traded off against prediction-based fairness notions, such as demographic parity or equality of opportunity. This perspective, however, ignores how predictions translate into decisions and ultimately into utilities and welfare for both DM and decision subjects (DS), as well as their allocation across social-salient groups.
In this paper, we propose a multi-stakeholder framework for fair algorithmic decision-making grounded in welfare economics and distributive justice, explicitly modeling the utilities of both the DM and DS, and defining fairness via a social planner's utility that captures inequalities in DS utilities across groups under different justice-based fairness notions (e.g., Egalitarian, Rawlsian). We formulate fair decision-making as a post-hoc multi-objective optimization problem, characterizing the achievable performance-fairness trade-offs in the two-dimensional utility space of DM utility and the social planner's utility, under different decision policy classes (deterministic vs. stochastic, shared vs. group-specific). Using the proposed framework, we then identify conditions (in terms of the stakeholders' utilities) under which stochastic policies are more optimal than deterministic ones, and empirically demonstrate that simple stochastic policies can yield superior performance-fairness trade-offs by leveraging outcome uncertainty. Overall, we advocate a shift from prediction-centric fairness to a transparent, justice-based, multi-stakeholder approach that supports the collaborative design of decision-making policies.},
keywords = {isabel, kavya, nektarios, saml},
pubstate = {published},
tppubtype = {inproceedings}
}
Fairness in algorithmic decision-making is often defined in the predictive space, where predictive performance - used as a proxy for decision-maker (DM) utility - is traded off against prediction-based fairness notions, such as demographic parity or equality of opportunity. This perspective, however, ignores how predictions translate into decisions and ultimately into utilities and welfare for both DM and decision subjects (DS), as well as their allocation across social-salient groups.
In this paper, we propose a multi-stakeholder framework for fair algorithmic decision-making grounded in welfare economics and distributive justice, explicitly modeling the utilities of both the DM and DS, and defining fairness via a social planner's utility that captures inequalities in DS utilities across groups under different justice-based fairness notions (e.g., Egalitarian, Rawlsian). We formulate fair decision-making as a post-hoc multi-objective optimization problem, characterizing the achievable performance-fairness trade-offs in the two-dimensional utility space of DM utility and the social planner's utility, under different decision policy classes (deterministic vs. stochastic, shared vs. group-specific). Using the proposed framework, we then identify conditions (in terms of the stakeholders' utilities) under which stochastic policies are more optimal than deterministic ones, and empirically demonstrate that simple stochastic policies can yield superior performance-fairness trade-offs by leveraging outcome uncertainty. Overall, we advocate a shift from prediction-centric fairness to a transparent, justice-based, multi-stakeholder approach that supports the collaborative design of decision-making policies.
In this paper, we propose a multi-stakeholder framework for fair algorithmic decision-making grounded in welfare economics and distributive justice, explicitly modeling the utilities of both the DM and DS, and defining fairness via a social planner's utility that captures inequalities in DS utilities across groups under different justice-based fairness notions (e.g., Egalitarian, Rawlsian). We formulate fair decision-making as a post-hoc multi-objective optimization problem, characterizing the achievable performance-fairness trade-offs in the two-dimensional utility space of DM utility and the social planner's utility, under different decision policy classes (deterministic vs. stochastic, shared vs. group-specific). Using the proposed framework, we then identify conditions (in terms of the stakeholders' utilities) under which stochastic policies are more optimal than deterministic ones, and empirically demonstrate that simple stochastic policies can yield superior performance-fairness trade-offs by leveraging outcome uncertainty. Overall, we advocate a shift from prediction-centric fairness to a transparent, justice-based, multi-stakeholder approach that supports the collaborative design of decision-making policies.
Majumdar, Ayan; Kanubala, Deborah Dormah; Gupta, Kavya; Valera, Isabel
A Causal Framework to Measure and Mitigate Non-binary Treatment Discrimination Journal Article
In: CoRR, vol. abs/2503.22454, 2026.
Abstract | Links | BibTeX | Tags: ayanm, deborah, isabel, kavya, saml
@article{DBLP:journals/corr/abs-2503-22454,
title = {A Causal Framework to Measure and Mitigate Non-binary Treatment Discrimination},
author = {Ayan Majumdar and Deborah Dormah Kanubala and Kavya Gupta and Isabel Valera},
url = {https://doi.org/10.48550/arXiv.2503.22454},
doi = {10.48550/ARXIV.2503.22454},
year = {2026},
date = {2026-03-19},
urldate = {2026-03-19},
journal = {CoRR},
volume = {abs/2503.22454},
abstract = {Fairness studies of algorithmic decision-making systems often simplify complex decision processes, such as bail or lending decisions, into binary classification tasks (e.g., approve or not approve). However, these approaches overlook that such decisions are not inherently binary; they also involve non-binary treatment decisions (e.g., loan or bail terms) that can influence the downstream outcomes (e.g., loan repayment or reoffending). We argue that treatment decisions are integral to the decision-making process and, therefore, should be central to fairness analyses. Consequently, we propose a causal framework that extends and complements existing fairness notions by explicitly distinguishing between decision-subjects’ covariates and the treatment decisions. Our framework leverages path-specific counterfactual reasoning to: (i) measure treatment disparity and its downstream effects in historical data; and (ii) mitigate the impact of past unfair treatment decisions when automating decision-making. We use our framework to empirically analyze four widely used loan approval datasets to reveal potential disparity in non-binary treatment decisions and their discriminatory impact on outcomes, highlighting the need to incorporate treatment decisions in fairness assessments. Finally, by intervening in treatment decisions, we show that our framework effectively mitigates treatment discrimination from historical loan approval data to ensure fair risk score estimation and (non-binary) decision-making processes that benefit all stakeholders.},
keywords = {ayanm, deborah, isabel, kavya, saml},
pubstate = {published},
tppubtype = {article}
}
Fairness studies of algorithmic decision-making systems often simplify complex decision processes, such as bail or lending decisions, into binary classification tasks (e.g., approve or not approve). However, these approaches overlook that such decisions are not inherently binary; they also involve non-binary treatment decisions (e.g., loan or bail terms) that can influence the downstream outcomes (e.g., loan repayment or reoffending). We argue that treatment decisions are integral to the decision-making process and, therefore, should be central to fairness analyses. Consequently, we propose a causal framework that extends and complements existing fairness notions by explicitly distinguishing between decision-subjects’ covariates and the treatment decisions. Our framework leverages path-specific counterfactual reasoning to: (i) measure treatment disparity and its downstream effects in historical data; and (ii) mitigate the impact of past unfair treatment decisions when automating decision-making. We use our framework to empirically analyze four widely used loan approval datasets to reveal potential disparity in non-binary treatment decisions and their discriminatory impact on outcomes, highlighting the need to incorporate treatment decisions in fairness assessments. Finally, by intervening in treatment decisions, we show that our framework effectively mitigates treatment discrimination from historical loan approval data to ensure fair risk score estimation and (non-binary) decision-making processes that benefit all stakeholders.
2025
Kalampalikis, Nektarios; Gupta, Kavya; Vitanov, Georgi; Valera, Isabel
Towards Reasonable Concept Bottleneck Models Journal Article
In: CoRR, vol. abs/2506.05014, 2025.
Abstract | Links | BibTeX | Tags: isabel, kavya, nektarios, saml
@article{DBLP:journals/corr/abs-2506-05014,
title = {Towards Reasonable Concept Bottleneck Models},
author = {Nektarios Kalampalikis and Kavya Gupta and Georgi Vitanov and Isabel Valera},
url = {https://doi.org/10.48550/arXiv.2506.05014},
doi = {10.48550/ARXIV.2506.05014},
year = {2025},
date = {2025-01-01},
urldate = {2025-01-01},
journal = {CoRR},
volume = {abs/2506.05014},
abstract = {In this paper, we propose textbf{C}oncept textbf{REA}soning textbf{M}odels (CREAM), a novel family of Concept Bottleneck Models (CBMs) that: (i) explicitly encodes concept-concept ({texttt{C-C}}) and concept-task ({texttt{C$rightarrow$Y}}) relationships to enforce a desired model reasoning; and (ii) use a regularized side-channel to achieve competitive task performance, while keeping high concept importance. Specifically, CREAM architecturally embeds (bi)directed concept-concept, and concept to task relationships specified by a human expert, while severing undesired information flows (e.g., to handle mutually exclusive concepts). Moreover, CREAM integrates a black-box side-channel that is regularized to encourage task predictions to be grounded in the relevant concepts, thereby utilizing the side-channel only when necessary to enhance performance. Our experiments show that: (i) CREAM mainly relies on concepts while achieving task performance on par with black-box models; and (ii) the embedded {texttt{C-C}} and {texttt{C$rightarrow$Y}} relationships ease model interventions and mitigate concept leakage.},
keywords = {isabel, kavya, nektarios, saml},
pubstate = {published},
tppubtype = {article}
}
In this paper, we propose textbf{C}oncept textbf{REA}soning textbf{M}odels (CREAM), a novel family of Concept Bottleneck Models (CBMs) that: (i) explicitly encodes concept-concept ({texttt{C-C}}) and concept-task ({texttt{C$rightarrow$Y}}) relationships to enforce a desired model reasoning; and (ii) use a regularized side-channel to achieve competitive task performance, while keeping high concept importance. Specifically, CREAM architecturally embeds (bi)directed concept-concept, and concept to task relationships specified by a human expert, while severing undesired information flows (e.g., to handle mutually exclusive concepts). Moreover, CREAM integrates a black-box side-channel that is regularized to encourage task predictions to be grounded in the relevant concepts, thereby utilizing the side-channel only when necessary to enhance performance. Our experiments show that: (i) CREAM mainly relies on concepts while achieving task performance on par with black-box models; and (ii) the embedded {texttt{C-C}} and {texttt{C$rightarrow$Y}} relationships ease model interventions and mitigate concept leakage.
2024
Kanubala, Deborah Dormah; Valera, Isabel; Gupta, Kavya
Fairness Beyond Binary Decisions: a Case Study on German Credit Proceedings Article
In: Cerrato, Mattia; Coronel, Alesia Vallenas; Ahrweiler, Petra; Loi, Michele; Pechenizkiy, Mykola; Tamò-Larrieux, Aurelia (Ed.): Proceedings of the 3rd European Workshop on Algorithmic Fairness, Mainz, Germany, July 1st to 3rd, 2024, CEUR-WS.org, 2024.
Abstract | Links | BibTeX | Tags: deborah, isabel, kavya, saml
@inproceedings{DBLP:conf/ewaf/KanubalaVG24,
title = {Fairness Beyond Binary Decisions: a Case Study on German Credit},
author = {Deborah Dormah Kanubala and Isabel Valera and Kavya Gupta},
editor = {Mattia Cerrato and Alesia Vallenas Coronel and Petra Ahrweiler and Michele Loi and Mykola Pechenizkiy and Aurelia Tamò-Larrieux},
url = {https://ceur-ws.org/Vol-3908/paper_15.pdf},
year = {2024},
date = {2024-01-01},
urldate = {2024-01-01},
booktitle = {Proceedings of the 3rd European Workshop on Algorithmic Fairness,
Mainz, Germany, July 1st to 3rd, 2024},
volume = {3908},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
abstract = {Data-driven approaches are increasingly used to (partially) automate decision-making in credit scoring
by predicting whether an applicant is “creditworthy or not” based on a set of features about the applicant,
such as age and income, along with what we refer here to as treatment decisions, e.g., loan amount and
duration. Existing data-driven approaches for automating and evaluating the accuracy and fairness of
such credit decisions ignore that treatment decisions (here, loan terms) are part of the decision and
thus may be subject to discrimination. This discrimination can propagate to the final outcome (repaid
or not) of positive decisions (granted loans). In this extended abstract, we rely on causal reasoning
and a broadly studied fair machine-learning dataset, the German credit, to i) show that the current fair
data-driven approach neglects discrimination in treatment decisions (i.e., loan terms) and its downstream
consequences on the decision outcome (i.e., ability to repay); and ii) argue for the need to move beyond
binary decisions in fair data-driven decision-making in consequential settings like credit scoring},
keywords = {deborah, isabel, kavya, saml},
pubstate = {published},
tppubtype = {inproceedings}
}
Data-driven approaches are increasingly used to (partially) automate decision-making in credit scoring
by predicting whether an applicant is “creditworthy or not” based on a set of features about the applicant,
such as age and income, along with what we refer here to as treatment decisions, e.g., loan amount and
duration. Existing data-driven approaches for automating and evaluating the accuracy and fairness of
such credit decisions ignore that treatment decisions (here, loan terms) are part of the decision and
thus may be subject to discrimination. This discrimination can propagate to the final outcome (repaid
or not) of positive decisions (granted loans). In this extended abstract, we rely on causal reasoning
and a broadly studied fair machine-learning dataset, the German credit, to i) show that the current fair
data-driven approach neglects discrimination in treatment decisions (i.e., loan terms) and its downstream
consequences on the decision outcome (i.e., ability to repay); and ii) argue for the need to move beyond
binary decisions in fair data-driven decision-making in consequential settings like credit scoring
by predicting whether an applicant is “creditworthy or not” based on a set of features about the applicant,
such as age and income, along with what we refer here to as treatment decisions, e.g., loan amount and
duration. Existing data-driven approaches for automating and evaluating the accuracy and fairness of
such credit decisions ignore that treatment decisions (here, loan terms) are part of the decision and
thus may be subject to discrimination. This discrimination can propagate to the final outcome (repaid
or not) of positive decisions (granted loans). In this extended abstract, we rely on causal reasoning
and a broadly studied fair machine-learning dataset, the German credit, to i) show that the current fair
data-driven approach neglects discrimination in treatment decisions (i.e., loan terms) and its downstream
consequences on the decision outcome (i.e., ability to repay); and ii) argue for the need to move beyond
binary decisions in fair data-driven decision-making in consequential settings like credit scoring
2023
Gupta, Kavya
University of Paris-Saclay, France, 2023.
Abstract | Links | BibTeX | Tags: kavya
@phdthesis{DBLP:phd/hal/Gupta23,
title = {Stability Quantification of Neural Networks. (Quantification de la stabilité des réseaux de neurones)},
author = {Kavya Gupta},
url = {https://tel.archives-ouvertes.fr/tel-04047901},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
school = {University of Paris-Saclay, France},
abstract = {Artificial neural networks are at the core of recent advances in Artificial Intelligence. One of the main challenges faced today, especially by companies likeThales designing advanced industrial systems is to ensure the safety of newgenerations of products using these technologies. In 2013 in a key observation, neural networks were shown to be sensitive to adversarial perturbations, raising serious concerns about their applicability in critically safe environments. In the last years, publications studying the various aspects of this robustness of neural networks, and rising questions such as "Why adversarial attacks occur?", "How can we make the neural network more robust to adversarial noise?", "How to generate stronger attacks?" etc., have grown exponentially. The contributions of this thesis aim to tackle such problems. The adversarial machine learning community concentrates majorly on classification scenarios, whereas studies on regression tasks are scarce. Our contributions bridge this significant gap between adversarial machine learning and regression applications.The first contribution in Chapter 3 proposes a white-box attackers designed to attack regression models. The presented adversarial attacker is derived from the algebraic properties of the Jacobian of the network. We show that our attacker successfully fools the neural network and measure its effectiveness in reducing the estimation performance. We present our results on various open-source and real industrial tabular datasets. Our analysis relies on the quantification of the fooling error as well as different error metrics. Another noteworthy feature of our attacker is that it allows us to optimally attack a subset of inputs, which may help to analyze the sensitivity of some specific inputs. We also, show the effect of this attacker on spectrally normalised trained models which are known to be more robust in handling attacks.The second contribution of this thesis (Chapter 4) presents a multivariate Lipschitz constant analysis of neural networks. The Lipschitz constant is widely used in the literature to study the internal properties of neural networks. But most works do a single parametric analysis, which do not allow to quantify the effect of individual inputs on the output. We propose a multivariate Lipschitz constant-based stability analysis of fully connected neural networks allowing us to capture the influence of each input or group of inputs on the neural network stability. Our approach relies on a suitable re-normalization of the input space, intending to perform a more precise analysis than the one provided by a global Lipschitz constant. We display the results of this analysis by a new representation designed for machine learning practitioners and safety engineers termed as a Lipschitz star. We perform experiments on various open-access tabular datasets and an actual Thales Air Mobility industrial application subject to certification requirements.The use of spectral normalization in designing a stability control loop is discussed in Chapter 5. A critical part of the optimal model is to behave according to specified performance and stability targets while in operation. But imposing tight Lipschitz constant constraints while training the models usually leads to a reduction of their accuracy. Hence, we design an algorithm to train "stable-by-design" neural network models using our spectral normalization approach, which optimizes the model by taking into account both performance and stability targets. We focus on Small Unmanned Aerial Vehicles (UAVs). More specifically, we present a novel application of neural networks to detect in real-time elevon positioning faults to allow the remote pilot to take necessary actions to ensure safety.},
keywords = {kavya},
pubstate = {published},
tppubtype = {phdthesis}
}
Artificial neural networks are at the core of recent advances in Artificial Intelligence. One of the main challenges faced today, especially by companies likeThales designing advanced industrial systems is to ensure the safety of newgenerations of products using these technologies. In 2013 in a key observation, neural networks were shown to be sensitive to adversarial perturbations, raising serious concerns about their applicability in critically safe environments. In the last years, publications studying the various aspects of this robustness of neural networks, and rising questions such as "Why adversarial attacks occur?", "How can we make the neural network more robust to adversarial noise?", "How to generate stronger attacks?" etc., have grown exponentially. The contributions of this thesis aim to tackle such problems. The adversarial machine learning community concentrates majorly on classification scenarios, whereas studies on regression tasks are scarce. Our contributions bridge this significant gap between adversarial machine learning and regression applications.The first contribution in Chapter 3 proposes a white-box attackers designed to attack regression models. The presented adversarial attacker is derived from the algebraic properties of the Jacobian of the network. We show that our attacker successfully fools the neural network and measure its effectiveness in reducing the estimation performance. We present our results on various open-source and real industrial tabular datasets. Our analysis relies on the quantification of the fooling error as well as different error metrics. Another noteworthy feature of our attacker is that it allows us to optimally attack a subset of inputs, which may help to analyze the sensitivity of some specific inputs. We also, show the effect of this attacker on spectrally normalised trained models which are known to be more robust in handling attacks.The second contribution of this thesis (Chapter 4) presents a multivariate Lipschitz constant analysis of neural networks. The Lipschitz constant is widely used in the literature to study the internal properties of neural networks. But most works do a single parametric analysis, which do not allow to quantify the effect of individual inputs on the output. We propose a multivariate Lipschitz constant-based stability analysis of fully connected neural networks allowing us to capture the influence of each input or group of inputs on the neural network stability. Our approach relies on a suitable re-normalization of the input space, intending to perform a more precise analysis than the one provided by a global Lipschitz constant. We display the results of this analysis by a new representation designed for machine learning practitioners and safety engineers termed as a Lipschitz star. We perform experiments on various open-access tabular datasets and an actual Thales Air Mobility industrial application subject to certification requirements.The use of spectral normalization in designing a stability control loop is discussed in Chapter 5. A critical part of the optimal model is to behave according to specified performance and stability targets while in operation. But imposing tight Lipschitz constant constraints while training the models usually leads to a reduction of their accuracy. Hence, we design an algorithm to train "stable-by-design" neural network models using our spectral normalization approach, which optimizes the model by taking into account both performance and stability targets. We focus on Small Unmanned Aerial Vehicles (UAVs). More specifically, we present a novel application of neural networks to detect in real-time elevon positioning faults to allow the remote pilot to take necessary actions to ensure safety.
Gupta, Kavya; Verma, Sagar
CertViT: Certified Robustness of Pre-Trained Vision Transformers Journal Article
In: CoRR, vol. abs/2302.10287, 2023.
Abstract | Links | BibTeX | Tags: kavya
@article{DBLP:journals/corr/abs-2302-10287,
title = {CertViT: Certified Robustness of Pre-Trained Vision Transformers},
author = {Kavya Gupta and Sagar Verma},
url = {https://doi.org/10.48550/arXiv.2302.10287},
doi = {10.48550/ARXIV.2302.10287},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2302.10287},
abstract = {Lipschitz bounded neural networks are certifiably robust and have a good trade-off between clean and certified accuracy. Existing Lipschitz bounding methods train from scratch and are limited to moderately sized networks (< 6M parameters). They require a fair amount of hyper-parameter tuning and are computationally prohibitive for large networks like Vision Transformers (5M to 660M parameters). Obtaining certified robustness of transformers is not feasible due to the non-scalability and inflexibility of the current methods. This work presents CertViT, a two-step proximal-projection method to achieve certified robustness from pre-trained weights. The proximal step tries to lower the Lipschitz bound and the projection step tries to maintain the clean accuracy of pre-trained weights. We show that CertViT networks have better certified accuracy than state-of-the-art Lipschitz trained networks. We apply CertViT on several variants of pre-trained vision transformers and show adversarial robustness using standard attacks},
keywords = {kavya},
pubstate = {published},
tppubtype = {article}
}
Lipschitz bounded neural networks are certifiably robust and have a good trade-off between clean and certified accuracy. Existing Lipschitz bounding methods train from scratch and are limited to moderately sized networks (< 6M parameters). They require a fair amount of hyper-parameter tuning and are computationally prohibitive for large networks like Vision Transformers (5M to 660M parameters). Obtaining certified robustness of transformers is not feasible due to the non-scalability and inflexibility of the current methods. This work presents CertViT, a two-step proximal-projection method to achieve certified robustness from pre-trained weights. The proximal step tries to lower the Lipschitz bound and the projection step tries to maintain the clean accuracy of pre-trained weights. We show that CertViT networks have better certified accuracy than state-of-the-art Lipschitz trained networks. We apply CertViT on several variants of pre-trained vision transformers and show adversarial robustness using standard attacks
2022
Gupta, Kavya; Kaakai, Fateh; Pesquet-Popescu, Béatrice; Pesquet, Jean-Christophe
Safe Design of Stable Neural Networks for Fault Detection in Small UAVs Proceedings Article
In: Trapp, Mario; Schoitsch, Erwin; Guiochet, Jérémie; Bitsch, Friedemann (Ed.): Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops - DECSoS, DepDevOps, SASSUR, SENSEI, USDAI, and WAISE, Munich, Germany, September 6-9, 2022, Proceedings, pp. 263–275, Springer, 2022.
Abstract | Links | BibTeX | Tags: kavya
@inproceedings{DBLP:conf/safecomp/GuptaKPP22,
title = {Safe Design of Stable Neural Networks for Fault Detection in Small UAVs},
author = {Kavya Gupta and Fateh Kaakai and Béatrice Pesquet-Popescu and Jean-Christophe Pesquet},
editor = {Mario Trapp and Erwin Schoitsch and Jérémie Guiochet and Friedemann Bitsch},
url = {https://doi.org/10.1007/978-3-031-14862-0_19},
doi = {10.1007/978-3-031-14862-0_19},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops
- DECSoS, DepDevOps, SASSUR, SENSEI, USDAI, and WAISE, Munich, Germany,
September 6-9, 2022, Proceedings},
volume = {13415},
pages = {263–275},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
abstract = {Stability of a machine learning model is the extent to which a model can continue to operate correctly despite small perturbations in its inputs. A formal method to measure stability is the Lipschitz constant of the model which allows to evaluate how small perturbations in the inputs impact the output variations. Variations in the outputs may lead to high errors for regression tasks or unintended changes in the classes for classification tasks. Verification of the stability of ML models is crucial in many industrial domains such as aeronautics, space, automotive etc. It has been recognized that data-driven models are intrinsically extremely sensitive to small perturbation of the inputs. Therefore, the need to design methods for verifying the stability of ML models is of importance for manufacturers developing safety critical products.
In this work, we focus on Small Unmanned Aerial Vehicles (UAVs) which are in the frontage of new technology solutions for intelligent systems. However, real-time fault detection/diagnosis in such UAVs remains a challenge from data collection to prediction tasks. This work presents application of neural networks to detect in real-time elevon positioning faults. We show the efficiency of a formal method based on the Lipschitz constant for quantifying the stability of neural network models. We also present how this method can be coupled with spectral normalization constraints at the design phase to control the internal parameters of the model and make it more stable while keeping a high level of performance (accuracy-stability trade-off).},
keywords = {kavya},
pubstate = {published},
tppubtype = {inproceedings}
}
Stability of a machine learning model is the extent to which a model can continue to operate correctly despite small perturbations in its inputs. A formal method to measure stability is the Lipschitz constant of the model which allows to evaluate how small perturbations in the inputs impact the output variations. Variations in the outputs may lead to high errors for regression tasks or unintended changes in the classes for classification tasks. Verification of the stability of ML models is crucial in many industrial domains such as aeronautics, space, automotive etc. It has been recognized that data-driven models are intrinsically extremely sensitive to small perturbation of the inputs. Therefore, the need to design methods for verifying the stability of ML models is of importance for manufacturers developing safety critical products.
In this work, we focus on Small Unmanned Aerial Vehicles (UAVs) which are in the frontage of new technology solutions for intelligent systems. However, real-time fault detection/diagnosis in such UAVs remains a challenge from data collection to prediction tasks. This work presents application of neural networks to detect in real-time elevon positioning faults. We show the efficiency of a formal method based on the Lipschitz constant for quantifying the stability of neural network models. We also present how this method can be coupled with spectral normalization constraints at the design phase to control the internal parameters of the model and make it more stable while keeping a high level of performance (accuracy-stability trade-off).
In this work, we focus on Small Unmanned Aerial Vehicles (UAVs) which are in the frontage of new technology solutions for intelligent systems. However, real-time fault detection/diagnosis in such UAVs remains a challenge from data collection to prediction tasks. This work presents application of neural networks to detect in real-time elevon positioning faults. We show the efficiency of a formal method based on the Lipschitz constant for quantifying the stability of neural network models. We also present how this method can be coupled with spectral normalization constraints at the design phase to control the internal parameters of the model and make it more stable while keeping a high level of performance (accuracy-stability trade-off).
Gupta, Kavya; Kaakai, Fateh; Pesquet-Popescu, Beatrice; Pesquet, Jean-Christophe; Malliaros, Fragkiskos D.
Multivariate Lipschitz Analysis of the Stability of Neural Networks Journal Article
In: Frontiers in Signal Processing, vol. Volume 2 - 2022, 2022, ISSN: 2673-8198.
Abstract | Links | BibTeX | Tags: kavya
@article{10.3389/frsip.2022.794469,
title = {Multivariate Lipschitz Analysis of the Stability of Neural Networks},
author = {Kavya Gupta and Fateh Kaakai and Beatrice Pesquet-Popescu and Jean-Christophe Pesquet and Fragkiskos D. Malliaros},
url = {https://www.frontiersin.org/journals/signal-processing/articles/10.3389/frsip.2022.794469},
doi = {10.3389/frsip.2022.794469},
issn = {2673-8198},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Frontiers in Signal Processing},
volume = {Volume 2 - 2022},
abstract = {The stability of neural networks with respect to adversarial perturbations has been extensively studied. One of the main strategies consist of quantifying the Lipschitz regularity of neural networks. In this paper, we introduce a multivariate Lipschitz constant-based stability analysis of fully connected neural networks allowing us to capture the influence of each input or group of inputs on the neural network stability. Our approach relies on a suitable re-normalization of the input space, with the objective to perform a more precise analysis than the one provided by a global Lipschitz constant. We investigate the mathematical properties of the proposed multivariate Lipschitz analysis and show its usefulness in better understanding the sensitivity of the neural network with regard to groups of inputs. We display the results of this analysis by a new representation designed for machine learning practitioners and safety engineers termed as a Lipschitz star. The Lipschitz star is a graphical and practical tool to analyze the sensitivity of a neural network model during its development, with regard to different combinations of inputs. By leveraging this tool, we show that it is possible to build robust-by-design models using spectral normalization techniques for controlling the stability of a neural network, given a safety Lipschitz target. Thanks to our multivariate Lipschitz analysis, we can also measure the efficiency of adversarial training in inference tasks. We perform experiments on various open access tabular datasets, and also on a real Thales Air Mobility industrial application subject to certification requirements.},
keywords = {kavya},
pubstate = {published},
tppubtype = {article}
}
The stability of neural networks with respect to adversarial perturbations has been extensively studied. One of the main strategies consist of quantifying the Lipschitz regularity of neural networks. In this paper, we introduce a multivariate Lipschitz constant-based stability analysis of fully connected neural networks allowing us to capture the influence of each input or group of inputs on the neural network stability. Our approach relies on a suitable re-normalization of the input space, with the objective to perform a more precise analysis than the one provided by a global Lipschitz constant. We investigate the mathematical properties of the proposed multivariate Lipschitz analysis and show its usefulness in better understanding the sensitivity of the neural network with regard to groups of inputs. We display the results of this analysis by a new representation designed for machine learning practitioners and safety engineers termed as a Lipschitz star. The Lipschitz star is a graphical and practical tool to analyze the sensitivity of a neural network model during its development, with regard to different combinations of inputs. By leveraging this tool, we show that it is possible to build robust-by-design models using spectral normalization techniques for controlling the stability of a neural network, given a safety Lipschitz target. Thanks to our multivariate Lipschitz analysis, we can also measure the efficiency of adversarial training in inference tasks. We perform experiments on various open access tabular datasets, and also on a real Thales Air Mobility industrial application subject to certification requirements.
2021
Lassau, Nathalie; Ammari, Samy; Chouzenoux, Emilie; Gortais, Hugo; Herent, Paul; Devilder, Matthieu; Soliman, Samer; Meyrignac, Olivier; Talabard, Marie-Pauline; Lamarque, Jean-Philippe; Dubois, Remy; Loiseau, Nicolas; Trichelair, Paul; Bendjebbar, Etienne; Garcia, Gabriel; Balleyguier, Corinne; Merad, Mansouria; Stoclin, Annabelle; Jegou, Simon; Griscelli, Franck; Tetelboum, Nicolas; Li, Yingping; Verma, Sagar; Terris, Matthieu; Dardouri, Tasnim; Gupta, Kavya; Neacsu, Ana; Chemouni, Frank; Sefta, Meriem; Jehanno, Paul; Bousaid, Imad; Boursin, Yannick; Planchet, Emmanuel; Azoulay, Mikael; Dachary, Jocelyn; Brulport, Fabien; Gonzalez, Adrian; Dehaene, Olivier; Schiratti, Jean-Baptiste; Schutte, Kathryn; Pesquet, Jean-Christophe; Talbot, Hugues; Pronier, Elodie; Wainrib, Gilles; Clozel, Thomas; Barlesi, Fabrice; Bellin, Marie-France; Blum, Michael G. B.
Integrating deep learning CT-scan model, biological and clinical variables to predict severity of COVID-19 patients Journal Article
In: Nat Commun, vol. 12, no. 1, 2021, ISSN: 2041-1723.
Abstract | Links | BibTeX | Tags: kavya
@article{Lassau2021,
title = {Integrating deep learning CT-scan model, biological and clinical variables to predict severity of COVID-19 patients},
author = {Nathalie Lassau and Samy Ammari and Emilie Chouzenoux and Hugo Gortais and Paul Herent and Matthieu Devilder and Samer Soliman and Olivier Meyrignac and Marie-Pauline Talabard and Jean-Philippe Lamarque and Remy Dubois and Nicolas Loiseau and Paul Trichelair and Etienne Bendjebbar and Gabriel Garcia and Corinne Balleyguier and Mansouria Merad and Annabelle Stoclin and Simon Jegou and Franck Griscelli and Nicolas Tetelboum and Yingping Li and Sagar Verma and Matthieu Terris and Tasnim Dardouri and Kavya Gupta and Ana Neacsu and Frank Chemouni and Meriem Sefta and Paul Jehanno and Imad Bousaid and Yannick Boursin and Emmanuel Planchet and Mikael Azoulay and Jocelyn Dachary and Fabien Brulport and Adrian Gonzalez and Olivier Dehaene and Jean-Baptiste Schiratti and Kathryn Schutte and Jean-Christophe Pesquet and Hugues Talbot and Elodie Pronier and Gilles Wainrib and Thomas Clozel and Fabrice Barlesi and Marie-France Bellin and Michael G. B. Blum},
doi = {10.1038/s41467-020-20657-4},
issn = {2041-1723},
year = {2021},
date = {2021-12-00},
urldate = {2021-12-00},
journal = {Nat Commun},
volume = {12},
number = {1},
publisher = {Springer Science and Business Media LLC},
abstract = {The SARS-COV-2 pandemic has put pressure on intensive care units, so that identifying predictors of disease severity is a priority. We collect 58 clinical and biological variables, and chest CT scan data, from 1003 coronavirus-infected patients from two French hospitals. We train a deep learning model based on CT scans to predict severity. We then construct the multimodal AI-severity score that includes 5 clinical and biological variables (age, sex, oxygenation, urea, platelet) in addition to the deep learning model. We show that neural network analysis of CT-scans brings unique prognosis information, although it is correlated with other markers of severity (oxygenation, LDH, and CRP) explaining the measurable but limited 0.03 increase of AUC obtained when adding CT-scan information to clinical variables. Here, we show that when comparing AI-severity with 11 existing severity scores, we find significantly improved prognosis performance; AI-severity can therefore rapidly become a reference scoring approach.},
keywords = {kavya},
pubstate = {published},
tppubtype = {article}
}
The SARS-COV-2 pandemic has put pressure on intensive care units, so that identifying predictors of disease severity is a priority. We collect 58 clinical and biological variables, and chest CT scan data, from 1003 coronavirus-infected patients from two French hospitals. We train a deep learning model based on CT scans to predict severity. We then construct the multimodal AI-severity score that includes 5 clinical and biological variables (age, sex, oxygenation, urea, platelet) in addition to the deep learning model. We show that neural network analysis of CT-scans brings unique prognosis information, although it is correlated with other markers of severity (oxygenation, LDH, and CRP) explaining the measurable but limited 0.03 increase of AUC obtained when adding CT-scan information to clinical variables. Here, we show that when comparing AI-severity with 11 existing severity scores, we find significantly improved prognosis performance; AI-severity can therefore rapidly become a reference scoring approach.
Gupta, Kavya; Pesquet-Popescu, Béatrice; Kaakai, Fateh; Pesquet, Jean-Christophe
A Quantitative Analysis Of The Robustness Of Neural Networks For Tabular Data Proceedings Article
In: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pp. 8057–8061, IEEE, 2021.
Abstract | Links | BibTeX | Tags: kavya
@inproceedings{DBLP:conf/icassp/GuptaPKP21,
title = {A Quantitative Analysis Of The Robustness Of Neural Networks For Tabular Data},
author = {Kavya Gupta and Béatrice Pesquet-Popescu and Fateh Kaakai and Jean-Christophe Pesquet},
url = {https://doi.org/10.1109/ICASSP39728.2021.9413858},
doi = {10.1109/ICASSP39728.2021.9413858},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing,
ICASSP 2021, Toronto, ON, Canada, June 6-11, 2021},
pages = {8057–8061},
publisher = {IEEE},
abstract = {This paper presents a quantitative approach to demonstrate the robustness of neural networks for tabular data. These data form the backbone of the data structures found in most industrial applications. We analyse the effect of various widely used techniques we encounter in neural network practice, such as regularization of weights, addition of noise to the data, and positivity constraints. This analysis is performed by using three state-of-the-art techniques, which provide mathematical proofs of robustness in terms of Lipschitz constant for feed-forward networks. The experiments are carried out on two prediction tasks and one classification task. Our work brings insights into building robust neural network architectures for safety critical systems that require certification or approval from a competent authority.},
keywords = {kavya},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper presents a quantitative approach to demonstrate the robustness of neural networks for tabular data. These data form the backbone of the data structures found in most industrial applications. We analyse the effect of various widely used techniques we encounter in neural network practice, such as regularization of weights, addition of noise to the data, and positivity constraints. This analysis is performed by using three state-of-the-art techniques, which provide mathematical proofs of robustness in terms of Lipschitz constant for feed-forward networks. The experiments are carried out on two prediction tasks and one classification task. Our work brings insights into building robust neural network architectures for safety critical systems that require certification or approval from a competent authority.
Gupta, Kavya; Pesquet, Jean-Christophe; Pesquet-Popescu, Béatrice; Kaakai, Fateh; Malliaros, Fragkiskos D.
An Adversarial Attacker for Neural Networks in Regression Problems Proceedings Article
In: Espinoza, Huáscar; McDermid, John A.; Huang, Xiaowei; Castillo-Effen, Mauricio; Chen, Xin Cynthia; Hernández-Orallo, José; hÉigeartaigh, Seán Ó; Mallah, Richard; Pedroza, Gabriel (Ed.): Proceedings of the Workshop on Artificial Intelligence Safety 2021 co-located with the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI 2021), Virtual, August, 2021, CEUR-WS.org, 2021.
Abstract | Links | BibTeX | Tags: kavya
@inproceedings{DBLP:conf/ijcai/GuptaPPKM21,
title = {An Adversarial Attacker for Neural Networks in Regression Problems},
author = {Kavya Gupta and Jean-Christophe Pesquet and Béatrice Pesquet-Popescu and Fateh Kaakai and Fragkiskos D. Malliaros},
editor = {Huáscar Espinoza and John A. McDermid and Xiaowei Huang and Mauricio Castillo-Effen and Xin Cynthia Chen and José Hernández-Orallo and Seán Ó hÉigeartaigh and Richard Mallah and Gabriel Pedroza},
url = {https://ceur-ws.org/Vol-2916/paper_17.pdf},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {Proceedings of the Workshop on Artificial Intelligence Safety 2021
co-located with the Thirtieth International Joint Conference on Artificial
Intelligence (IJCAI 2021), Virtual, August, 2021},
volume = {2916},
publisher = {CEUR-WS.org},
series = {CEUR Workshop Proceedings},
abstract = {Adversarial attacks against neural networks and their defenses have been mostly investigated in classification scenarios. However, adversarial attacks in a regression setting remain understudied, although they play a critical role in a large portion of safety-critical applications. In this work, we present an adversarial attacker for regression tasks, derived from the algebraic properties of the Jacobian of the network. We show that our attacker successfully fools the neural network, and we measure its effectiveness in reducing the estimation performance. We present a white-box adversarial attacker to support engineers in designing safety-critical regression machine learning models. We present our results on various open-source and real industrial tabular datasets. In particular, the proposed adversarial attacker outperforms attackers based on random perturbations of the inputs. Our analysis relies on the quantification of the fooling error as well as various error metrics. A noteworthy feature of our attacker is that it allows us to optimally attack a subset of inputs, which may be helpful to analyse the sensitivity of some specific inputs.},
keywords = {kavya},
pubstate = {published},
tppubtype = {inproceedings}
}
Adversarial attacks against neural networks and their defenses have been mostly investigated in classification scenarios. However, adversarial attacks in a regression setting remain understudied, although they play a critical role in a large portion of safety-critical applications. In this work, we present an adversarial attacker for regression tasks, derived from the algebraic properties of the Jacobian of the network. We show that our attacker successfully fools the neural network, and we measure its effectiveness in reducing the estimation performance. We present a white-box adversarial attacker to support engineers in designing safety-critical regression machine learning models. We present our results on various open-source and real industrial tabular datasets. In particular, the proposed adversarial attacker outperforms attackers based on random perturbations of the inputs. Our analysis relies on the quantification of the fooling error as well as various error metrics. A noteworthy feature of our attacker is that it allows us to optimally attack a subset of inputs, which may be helpful to analyse the sensitivity of some specific inputs.
Project Contributors
- Batuhan Koyuncu
- Deborah Kanubala
- Georgi Vitanov
- Huyen Thuc Khanh Vo
- Isabel Valera
- Kavya Gupta
- Lena Marie Budde
- María Martínez García