Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS credential with #11581

Open
dyno opened this issue Nov 18, 2024 · 2 comments
Open

AWS credential with #11581

dyno opened this issue Nov 18, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@dyno
Copy link

dyno commented Nov 18, 2024

Apache Iceberg version

1.7.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

spark.sql.catalog.spark_catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
# ...
spark.sql.catalog.spark_catalog.client.assume-role.arn: <redacted>
spark.sql.catalog.spark_catalog.client.assume-role.region: us-west-2
spark.sql.catalog.spark_catalog.client.factory: org.apache.iceberg.aws.AssumeRoleAwsClientFactory

we use S3FileIO and AssumeRoleAwsClientFactory in a spark/kubernetes environment.
we provide a AWS_SHARED_CREDENTIALS_FILE, so eventually the initial credential provider is software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider. but the credential in the profile will rotate every 12 hours.
after 12 hours, the job is error out with software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is expired. what we have observed this problem from 1.5.0 but also tried 1.6.1 and latest 1.7 snapshot all have the same problem.

full error stack.

software.amazon.awssdk.services.sts.model.StsException: The security token included in the request is expired (Service: Sts, Status Code: 403, Request ID: 204449d9-191f-403e-b39b-70ee4060cac4)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.sts.DefaultStsClient.assumeRole(DefaultStsClient.java:272) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider.getUpdatedCredentials(StsAssumeRoleCredentialsProvider.java:73) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.updateSessionCredentials(StsCredentialsProvider.java:92) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.lambda$jitteredPrefetchValueSupplier$8(CachedSupplier.java:300) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier$PrefetchStrategy.fetch(CachedSupplier.java:448) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:208) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:135) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.sts.auth.StsCredentialsProvider.resolveCredentials(StsCredentialsProvider.java:105) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.lambda$trySelectAuthScheme$3(S3AuthSchemeInterceptor.java:152) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:77) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.trySelectAuthScheme(S3AuthSchemeInterceptor.java:152) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.selectAuthScheme(S3AuthSchemeInterceptor.java:83) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.beforeExecution(S3AuthSchemeInterceptor.java:63) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
        at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:239) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:130) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:62) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:5174) ~[iceberg-aws-bundle-1.5.0.jar:?]
        at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:192) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.aws.s3.S3InputStream.positionStream(S3InputStream.java:177) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.aws.s3.S3InputStream.read(S3InputStream.java:107) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.ensureLoaded(ByteSourceJsonBootstrapper.java:539) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.detectEncoding(ByteSourceJsonBootstrapper.java:133) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.core.json.ByteSourceJsonBootstrapper.constructParser(ByteSourceJsonBootstrapper.java:256) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.core.JsonFactory._createParser(JsonFactory.java:1685) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:1084) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3714) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:280) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:273) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:189) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:208) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:208) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:185) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:180) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:166) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49) ~[iceberg-hive-runtime-1.5.0.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:843) ~[iceberg-spark-runtime-3.5_2.12-1.5.0.jar:?]
        at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:170) ~[iceberg-spark-runtime-3.5_2.12-1.5.0.jar:?]
        at org.apache.iceberg.spark.SparkSessionCatalog.loadTable(SparkSessionCatalog.java:139) ~[iceberg-spark-runtime-3.5_2.12-1.5.0.jar:?]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.getTable(CatalogV2Util.scala:355) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:336) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$3(Analyzer.scala:1268) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$resolveRelation$1(Analyzer.scala:1267) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at scala.Option.orElse(Option.scala:447) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$resolveRelation(Analyzer.scala:1259) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1108) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$14.applyOrElse(Analyzer.scala:1087) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76) ~[spark-sql-api_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$1(AnalysisHelper.scala:138) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.allowInvokingTransformsInAnalyzer(AnalysisHelper.scala:323) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning(AnalysisHelper.scala:134) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.resolveOperatorsUpWithPruning$(AnalysisHelper.scala:130) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperatorsUpWithPruning(LogicalPlan.scala:32) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1087) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1046) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126) ~[scala-library-2.12.18.jar:?]
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122) ~[scala-library-2.12.18.jar:?]
        at scala.collection.immutable.List.foldLeft(List.scala:91) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:219) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at scala.collection.immutable.List.foreach(List.scala:431) ~[scala-library-2.12.18.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:226) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:222) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:173) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:222) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:188) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:209) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:208) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:?]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138) ~[spark-catalyst_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:546) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:219) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:218) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:77) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:85) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:83) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:142) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.DataFrameWriterV2.runCommand(DataFrameWriterV2.scala:196) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at org.apache.spark.sql.DataFrameWriterV2.overwrite(DataFrameWriterV2.scala:167) ~[spark-sql_2.12-3.5.0-sfdc-0.5.jar:3.5.0-sfdc-0.5]
        at com.salesforce.uip.data.pipeline.ingestion.coreapp.CoreAppLogTransformHelper$.$anonfun$processPartition$5(CoreAppLogTransformHelper.scala:167) ~[ingestion-app_2.12-5.5.4-770-sfspark.jar:?]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.18.jar:?]


our client 

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
@dyno dyno added the bug Something isn't working label Nov 18, 2024
@dyno
Copy link
Author

dyno commented Nov 18, 2024

we also tried s3 magic committer and there is no expiration problem.
it feels like the credentials retrieved from profile is not refreshed in iceberg S3FileIO S3 client.

@munendrasn
Copy link
Contributor

@dyno This might be related to bug on AWS SDK aws/aws-sdk-java-v2#4268, where the refreshed credentials are not picked up..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dyno @munendrasn