Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PGD attack on multi-modal CLIP model #2340

Closed
wants to merge 46 commits into from

Conversation

GiulioZizzo
Copy link
Collaborator

Description

We include an experimental feature on handling multimodal inputs in ART and demonstrate its use for attacking CLIP. This is a new type of attack for ART and this PR shows a potential route for integrating multimodal inputs in ART without a full re-write of the backend code.

Fixes # (issue)

Type of change

Please check all relevant options.

  • Improvement (non-breaking)
  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

  • New test in tests/estimators/classification/test_multimodal.py
  • New test in tests/attacks/evasion/classification/test_multimodal_attack.py

Test Configuration:

  • OS: Mac OS
  • Python version: 3.8
  • ART version or commit number: ART 1.16
  • TensorFlow / Keras / PyTorch / MXNet version: torch 1.13.1

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • My changes have been tested using both CPU and GPU devices

Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
@GiulioZizzo GiulioZizzo marked this pull request as draft December 1, 2023 11:16
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
Signed-off-by: GiulioZizzo <[email protected]>
…starts for classical classification tasks

Signed-off-by: GiulioZizzo <[email protected]>
@GiulioZizzo GiulioZizzo marked this pull request as ready for review December 12, 2023 08:35
@beat-buesser beat-buesser self-assigned this Dec 13, 2023
@beat-buesser beat-buesser added this to the ART 1.18.0 milestone Jan 9, 2024
@beat-buesser beat-buesser changed the base branch from dev_1.17.0 to dev_1.18.0 January 9, 2024 12:26
@beat-buesser
Copy link
Collaborator

This is more duplicate code in the experimental module than I have expected. I think we need to extend the primary implementation of the attack to be input data agnostic or design a general data API.

Signed-off-by: GiulioZizzo <[email protected]>
@beat-buesser beat-buesser removed this from the ART 1.18.0 milestone Jun 16, 2024
@beat-buesser beat-buesser deleted the branch Trusted-AI:dev_1.18.0 October 1, 2024 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants