[Fleet] Migration of saved objects do not trigger a policy update #193352
Labels
bug
Fixes for quality problems that affect the customer experience
Team:Fleet
Team label for Observability Data Collection Fleet team
Comments
jen-huang
added
bug
|
Pinging @elastic/fleet (Team:Fleet) |
|
We have a mechanism to bump agent policies by increasing the FLEET_AGENT_POLICIES_SCHEMA_VERSION. We could potentially do something similar on package policy level. |
1 task
gergoabraham
added a commit
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing #195797 on Serverless As backfilled package policies are not automatically redeployed (see #193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in #195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in #193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if #193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in #195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (elastic#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing elastic#195797 on Serverless As backfilled package policies are not automatically redeployed (see elastic#193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in elastic#195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in elastic#193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if elastic#193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in elastic#195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1429979)
gergoabraham
added a commit
to gergoabraham/kibana
that referenced
this issue
Oct 18, 2024
…eployment issue mitigation (elastic#196708) ## Summary closes elastic/security-team#10851 > [!note] >⚠️ needs to be included in v8.16 >⚠️ needs to be merged this week to avoid releasing elastic#195797 on Serverless As backfilled package policies are not automatically redeployed (see elastic#193352), this PR's goal is to provide quick mitigation in the following matters: - update default values in the descriptions of advanced options added in elastic#195797, to harmonize with latest Endpoint changes (elastic/endpoint-dev#15109) - remove backfill/migration of those default values: - we should be _able_ to safely remove the backfills, as they have not yet been released to serverless. and, - we _should_ remove them to make sure that when we update the defaults in the future and apply the backfill, there will be a data change that could trigger policy re-deployment, in case data change is what the trigger will be in elastic#193352. - example scenario of what could go wrong: - if we'd apply backfill now, the package won't be redeployed. - if the user does not touch it until the next release - no redeploy. - if elastic#193352 is implemented and uses data comparison when running migrations - again, no redeploy because we already backfilled the data months before. - cc @ferullo @nfritts - hide banner describing event volume reduction (added in elastic#195177, already released to serverless, but it is what it is) ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit 1429979)
kibanamachine
added a commit
that referenced
this issue
Oct 18, 2024
…cy re-deployment issue mitigation (#196708) (#196835) # Backport This will backport the following commits from `main` to `8.16`: - [[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)](#196708) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-18T10:15:00Z","message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","Team:Defend Workflows","v8.16.0","backport:version"],"title":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation","number":196708,"url":"https://github.com/elastic/kibana/pull/196708","mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},"sourceBranch":"main","suggestedTargetBranches":["8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196708","number":196708,"mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Gergő Ábrahám <[email protected]>
gergoabraham
added a commit
that referenced
this issue
Oct 18, 2024
…y re-deployment issue mitigation (#196708) (#196843) # Backport This will backport the following commits from `main` to `8.x`: - [[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)](#196708) <!--- Backport version: 9.6.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-18T10:15:00Z","message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Fleet","v9.0.0","Team:Defend Workflows","v8.16.0","backport:version","v8.17.0"],"title":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation","number":196708,"url":"https://github.com/elastic/kibana/pull/196708","mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/196708","number":196708,"mergeCommit":{"message":"[Defend Workflows] Endpoint advanced options migration vs policy re-deployment issue mitigation (#196708)\n\n## Summary\r\n\r\ncloses https://github.com/elastic/security-team/issues/10851\r\n\r\n> [!note]\r\n>⚠️ needs to be included in v8.16\r\n>⚠️ needs to be merged this week to avoid releasing\r\nhttps://github.com//pull/195797 on Serverless\r\n\r\nAs backfilled package policies are not automatically redeployed (see\r\nhttps://github.com//issues/193352), this PR's goal is to\r\nprovide quick mitigation in the following matters:\r\n- update default values in the descriptions of advanced options added in\r\nhttps://github.com//pull/195797, to harmonize with latest\r\nEndpoint changes (https://github.com/elastic/endpoint-dev/issues/15109)\r\n- remove backfill/migration of those default values:\r\n- we should be _able_ to safely remove the backfills, as they have not\r\nyet been released to serverless. and,\r\n- we _should_ remove them to make sure that when we update the defaults\r\nin the future and apply the backfill, there will be a data change that\r\ncould trigger policy re-deployment, in case data change is what the\r\ntrigger will be in #193352.\r\n - example scenario of what could go wrong:\r\n - if we'd apply backfill now, the package won't be redeployed.\r\n- if the user does not touch it until the next release - no redeploy.\r\n- if #193352 is implemented and uses data comparison when running\r\nmigrations - again, no redeploy because we already backfilled the data\r\nmonths before.\r\n - cc @ferullo @nfritts \r\n- hide banner describing event volume reduction (added in\r\nhttps://github.com//pull/195177, already released to\r\nserverless, but it is what it is)\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios","sha":"142997925e5aafac306056b00be1789271aa5dd0"}},{"branch":"8.16","label":"v8.16.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/196835","number":196835,"state":"OPEN"},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When agent policy or package policy saved objects are migrated, they do not trigger a revision bump and thus the updated policies are never sent out to the agents. The policies are only bumped if they get updated again by another means.
We often migrate these saved objects so we need to implement some mechanism here to push the changes. This is frequently an issue for endpoint package policy migrations in particular.
We do not often migrate other objects that are used by policies such as proxies, outputs, etc but theoretically the same issue can occur when those are migrated too.
The text was updated successfully, but these errors were encountered: