u/Leather-Gene4160

Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data, same works fine with normal compute storage and external location, credentials, IAM policy and serverless egress rules all seems to be correct because normal compute is working. I am on a premium plan and have free credits for now, do we need enterprise plan for this? or is it something else?

This is the error I am getting when trying to list the s3 bucket items in serverless compute:

ExecutionError: [UNAUTHORIZED_ACCESS] Unauthorized access: s3://<raw-data-bucket>/<partition-path>: getFileStatus on s3://<raw-data-bucket>/<partition-path>: 
shaded.databricks.awssdk.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; 
request: HEAD https://<raw-data-bucket>.s3-us-west-2.amazonaws.com/<partition-path> {} 
Hadoop 3.4.2, aws-sdk-java/1.12.681 Linux/6.1.174-217.345.amzn2023.x86_64 OpenJDK_64-Bit_Server_VM/17.0.18+8-LTS java/17.0.18 scala/2.13.16 kotlin/1.9.10 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy 
shaded.databricks.awssdk.com.amazonaws.services.s3.model.GetObjectMetadataRequest; 
Request ID: null, Extended Request ID: null, Cloud Provider: AWS, Instance ID: unknown 
credentials-provider: shaded.databricks.awssdk.com.amazonaws.auth.BasicSessionCredentials 
credential-header: AWS4-HMAC-SHA256 Credential=<redacted-access-key-id>/<date>/us-west-2/s3/aws4_request 
signature-present: true 
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: <redacted>), 
S3 Extended Request ID: null: 403 Forbidden

SQLSTATE: 42501

Caused by: com.databricks.sql.io.CloudAccessDeniedException: getFileStatus / listStatus 403 on same path as above.

[Trace ID: <redacted>]