Skip to content

You are viewing documentation for Immuta version 2024.1.

For the latest version, view our documentation for Immuta SaaS or the latest self-hosted version.

cancel

Amazon S3 Integration

Private preview

The Amazon S3 integration is available to select accounts. Reach out to your Immuta representative for details.

Getting started

Immuta's Amazon S3 integration allows users to apply subscription policies to data in S3 to restrict what prefixes, buckets, or objects users can access. To enforce access controls on this data, Immuta creates S3 grants that are administered by S3 Access Grants, an AWS feature that defines access permissions to data in S3.

Requirement

No location is registered in your S3 Access Grants instance before configuring the integration in Immuta

Permissions

  • APPLICATION_ADMIN Immuta permission to configure the integration
  • CREATE_S3_DATASOURCE Immuta permission to register S3 prefixes

  • The AWS account credentials or optional AWS IAM role you provide Immuta when configuring the integration must

    • have ownership of the buckets Immuta will enforce policies on

    • have the permissions to perform the following actions to create locations and issue grants:

      • accessgrantslocation resource:
        • s3:CreateAccessGrant
        • s3:DeleteAccessGrantsLocation
        • s3:GetAccessGrantsLocation
      • accessgrantsinstance resource:
        • s3:CreateAccessGrantsLocation
        • s3:GetAccessGrantsInstance
        • s3:GetAccessGrantsInstanceForPrefix
        • s3:GetAccessGrantsInstanceResourcePolicy
        • s3:ListAccessGrants
        • s3:ListAccessGrantsLocations
      • bucket resource: s3:ListBucket
      • accessgrant resource:
        • s3:DeleteAccessGrant
        • s3:GetAccessGrant
      • role resource:
        • iam:GetRole
        • iam:PassRole
      • all resources: s3:ListAccessGrantsInstances

Set up S3 Access Grants instance

  1. Follow AWS documentation to create an Access Grants instance using the S3 console, AWS CLI, AWS SDKs, or the REST API. AWS supports one Access Grants instance per region per AWS account.

  2. Follow the instructions at the top of the "Register a location" page in AWS documentation to create an AWS IAM role and give the S3 Access Grants service principal access to this role in the resource policy file. You will add this role to your integration configuration in Immuta so that Immuta can register this role with your Access Grants location. The AWS documentation linked above gives a complete policy example, but your policy should include the following permissions:

    • sts:AssumeRole
    • sts:SetSourceIdentity
    • sts:SetContext

  3. Follow the instructions at the top of the "Register a location" page in AWS documentation to create an IAM policy with the following permissions, and attach the policy to the IAM role you created to grant the permissions to the role. The AWS documentation linked above gives a complete example, but the policy should at least include the following permissions:

    • s3:GetObject
    • s3:GetObjectVersion
    • s3:GetObjectAcl
    • s3:GetObjectVersionAcl
    • s3:ListMultipartUploadParts
    • s3:PutObject
    • s3:PutObjectAcl
    • s3:PutObjectVersionAcl
    • s3:DeleteObject
    • s3:DeleteObjectVersion
    • s3:AbortMultipartUpload
    • s3:ListBucket
    • s3:ListAllMyBuckets
    • iam:passRole

    If you use server-side encryption with AWS Key Management Service (AWS KMS) keys to encrypt your data, the following permissions are required for the IAM role in the policy. If you do not use this feature, do not include these permissions in your IAM policy:

    • kms:Decrypt
    • kms:GenerateDataKey

  4. Opt to create an AWS IAM role that Immuta can use to create Access Grants locations and issue grants. This role must have the S3 permissions listed in the permissions section.

Configure the integration in Immuta

  1. In Immuta, click App Settings in the navigation menu and click the Integrations tab.
  2. Click + Add Native Integration.
  3. Select Amazon S3 from the dropdown menu and click Continue Configuration.

  4. Complete the connection details fields, where

    • Friendly Name is a name for the integration that is unique across all Amazon S3 integrations configured in Immuta.
    • AWS Account ID is the ID of your AWS account.
    • AWS Region is the AWS region to use.
    • S3 Access Grants Location IAM Role ARN is the role the S3 Access Grants service assumes to vend credentials to the grantee. When a grantee accesses S3 data, the Access Grants service attaches session policies and assumes this role in order to vend credentials scoped to a prefix or bucket to the grantee. This role needs full access to all paths under the S3 location prefix.
    • S3 Access Grants S3 Location Scope is the base S3 location that Immuta will use for this connection when registering S3 prefixes. This path must be unique across all S3 integrations configured in Immuta. During data source registration, this prefix is prepended to the data source prefixes to build the final path used to grant or revoke access to that data in S3. For example, a location prefix of s3://research-data would be prepended to the data source prefix /demographics to generate a final path of s3://research-data/demographics.

  5. Select your authentication method:

    • Automatic: Searches and obtains credentials using the AWS SDK's default credential provider chain. This method requires a configured IAM role for a service account. This option is only supported if you're using a self-managed deployment of Immuta. Work with your Immuta representative to customize your deployment and set up an IAM role for a service account that can give Immuta the credentials to set up the integration.
    • Access Key: Provide your AWS Access Key ID and AWS Secret Access Key.

  6. Click Verify Credentials.

  7. Click Next to review and confirm your connection information, and then click Complete Setup.

Register S3 data

Follow the Create an S3 data source guide to register prefixes in Immuta.

To create an S3 data source using the API, see the Configure an S3 integration and create an S3 data source API guide.

Protect data

Requirements: USER_ADMIN Immuta permission and either the GOVERNANCE or CREATE_S3_DATASOURCE Immuta permission

  1. Build subscription policies to enforce access controls.

  2. Map AWS IAM principals to each Immuta user to ensure Immuta properly enforces policies:

    1. Click People and select Users in the navigation menu.
    2. Navigate to the user's page and click the more actions icon next to their username.
    3. Select Change S3 User or AWS IAM Role from the dropdown menu.
    4. Enter the AWS IAM User or Role ARN. User and role names are case-sensitive. See the AWS documentation for details.
    5. Click Save.

    See the Mapping IAM principals in Immuta section for details about supported principals.

Access data

Requirement: User must be subscribed to the data source in Immuta

  1. Request access to Amazon S3 data through S3 Access Grants. If you're accessing S3 data through one of the supported S3 Access Grants integrations (such as Amazon EMR on EC2), that application will make this request on your behalf, so you can skip this step.
  2. Use the temporary credentials you received in the previous step to access the data in S3.

S3 integration overview

Immuta's Amazon S3 integration allows users to apply subscription policies to data in S3 to restrict what prefixes, buckets, or objects users can access. To enforce access controls on this data, Immuta creates S3 grants that are administered by S3 Access Grants, an AWS feature that defines access permissions to data in S3.

With this integration, users can avoid

  • hand-writing AWS IAM policies
  • managing AWS IAM role limits
  • manually tracking what user or role has access to what files in AWS S3 and verifying those are consistent with intent

S3 Access Grants components

To enforce controls on S3 data, Immuta interacts with several S3 Access Grants components:

  • Access Grants instance: An Access Grants instance is a logical container for individual grants that specify who can access what level of data in S3 in your AWS account and region. AWS supports one Access Grants instance per region per AWS account.
  • Location: A location specifies what data the Access Grants instance can grant access to. For example, registering a location with a scope of s3:// allows Access Grants to manage access to all S3 buckets in that AWS account and region, whereas setting the bucket s3://research-data as the scope limits Access Grants to managing access to that single bucket for that location. When you configure the S3 integration in Immuta, you specify a location's scope and IAM assumed role, and Immuta registers the location in your Access Grants instance and associates it with the provided IAM role for you. Each S3 integration you configure in Immuta is associated with one location, and Immuta manages all grants in that location. Therefore, grants cannot be manually created by users in an Access Grants instance location that Immuta has registered and manages. During data source registration, this location scope is prepended to the data source prefixes to build the final path used to grant or revoke access to that data in S3. For example, a location scope of s3://research-data would be prepended to the data source prefix /demographics to generate a final path of s3://research-data/demographics.
  • Individual grants: Individual permission grants in S3 Access Grants specify the identity that can access the data, the access level, and the location of the S3 data. Immuta creates a grant for each user subscribed to a prefix, bucket, or object by interacting with the Access Grants API. Each grant has its own ID and gives the user or role principle access to the data.
  • IAM assumed role: This is an IAM role you create in S3 that has full access to all prefixes, buckets, and objects in the Access Grants location registered by Immuta. This IAM role is used to vend temporary credentials to users or applications. When a grantee requests temporary credentials, the S3 Access Grants service assumes this role to vend credentials scoped to the prefix, bucket, or object specified in the grant to the grantee. The grantee then uses these credentials to access S3 data. When configuring the integration in Immuta, you specify this role, and then Immuta associates this role with the registered location in the Access Grants instance.
  • Temporary credentials: These just-in-time access credentials provide access to a prefix, bucket, or object with a permission level of READ or READWRITE in S3. When a user or application requests temporary credentials to access S3 data, the S3 Access Grants instance evaluates the request against the grants Immuta has created for that user. If a matching grant exists, S3 Access Grants assumes the IAM role associated with the location of the matching grant and scopes the permissions of the IAM session to the S3 prefix, bucket, or object specified by the grant and vends these temporary credentials to the requester. These credentials have a default timeout of 1 hour, but this duration can be changed by the requester.

The diagram below illustrates how these S3 Access Grants components interact.

Access Grants components

For more details about these Access Grants concepts, see the S3 Access Grants documentation.

How does the integration work?

After an administrator creates an Access Grants instance and an assumed IAM role in their AWS account, an application administrator configures the Amazon S3 integration in Immuta. During configuration, the administrator provides the following connection information so that Immuta can create and register a location in that Access Grants instance:

  • AWS account ID and region
  • ARN for the existing Access Grants instance
  • ARN for the assumed IAM role

When Immuta registers this location, it associates the assumed IAM role with the location. This allows the IAM role to create temporary credentials with access scoped to a particular S3 prefix, bucket, or object in the location. The IAM role you create for this location must have all the object- and bucket-level permissions listed in the set up S3 Access Grants instance section on all buckets and objects in the location; if it is missing permissions, the IAM role will not be able to grant those missing permissions to users or applications requesting temporary credentials.

In the example below, an application administrator registers the following location prefix and IAM role for their Access Grants instance in AWS account 123456:

  • Location path: s3://. This path allows a single Amazon S3 integration to manage all objects in S3 in that AWS account and region. Data owners can scope down access further when registering specific S3 prefixes and applying policies.
  • Location IAM role: The arn:aws:iam::123456:role/access-grants-role IAM role will be used to vend temporary credentials to users and applications.

Immuta registers this location and associated IAM role in the user's Access Grants instance:

Access grants configuration

After the S3 integration is configured, a data owner can register S3 prefixes and buckets that are in the configured Access Grants location path to enforce access controls on resources. Immuta stores the connection information for the prefix so that the metadata can be used to create and enforce subscription policies on S3 data.

A data owner or governor can apply a subscription policy to a registered prefix, bucket, or object to control who can access objects beginning with that prefix or in that bucket after it is registered in Immuta. Once a subscription policy is created and Immuta users are subscribed to the prefix, bucket, or object, Immuta calls the Access Grants API to create a grant for each subscribed user, specifying the following parameters in the payload so that Access Grants can create and store a grant for each user:

  • Access Grants location
  • READ access
  • User or role principle
  • Registered prefix, bucket, or object

In the example below, a data owner registers the s3://research-data/* bucket, and Immuta stores the connection information in the Immuta metadata database. Once the user, Taylor, is subscribed to s3://research-data/*, Immuta calls the Access Grants API to create a grant for that user to allow them to read S3 data in that bucket:

Access grants configuration

Accessing S3 data

To access S3 data registered in Immuta, users must be subscribed to the prefix, bucket, or object in Immuta, and their principals must be mapped to their Immuta user accounts. Once users are subscribed, they request temporary credentials from S3 Access Grants. Access Grants looks up the grant ID associated with the requester. If no matching grant exists, they receive an access denied error. If one exists, Access Grants assumes the IAM role associated with the location and requests temporary credentials that are scoped to the prefix, bucket, or object and permissions specified by the individual grant. Access Grants vends the credentials to the requester, who uses those temporary credentials to access the data in S3.

In the example below, Taylor requests temporary credentials from S3 Access Grants. Access Grants looks up the grant ID (1) for that user, assumes the arn:aws:iam::123456:role/access-grants-role IAM role for the location, and vends temporary credentials to Taylor, who then uses the credentials to access the research-data bucket in S3:

S3 integration

Note that when accessing data through S3 Access Grants, the user or application interacts directly with the Access Grants API to request temporary credentials; Immuta does not act in this process at all. See the diagram below for an illustration of the process for accessing data through S3 Access Grants.

Access S3 Process

AWS services that support S3 Access Grants will request temporary credentials for users automatically. If users are not using a service that supports S3 Access Grants, they must have the permissions listed in the AWS documentation to call the Access Grants API directly themselves to request temporary credentials to access data through the access grant.

For a list of AWS services that support S3 Access Grants, see the AWS documentation.

Policy enforcement

Immuta's S3 integration allows data owners and governors to apply object-level access controls on data in S3 through subscription policies. When a user is subscribed to a registered prefix, bucket, or object, Immuta calls the Access Grants API to create an individual grant that narrows the scope of access within the location to that registered prefix, bucket, or object. See the diagram below for a visualization of this process.

S3 integration process diagram

When a user's entitlements change or a subscription policy is added to, updated, or deleted from a prefix, Immuta performs one of the following processes for each user subscribed to the registered prefix:

  • User added to the prefix: Immuta specifies a permission (READ or READWRITE) for each user and uses the Access Grants API to create an individual grant for each user.
  • User updated: Immuta deletes the current grant ID and creates a new one using the Access Grants API.
  • User deleted: Immuta deletes the grant ID using the Access Grants API.

Data policies, which provide more granular controls by redacting or masking values in a table, are not supported for S3.

Prefix registration

Data owners can register an S3 prefix at any level in the S3 path by creating an Immuta data source. During this process, Immuta stores the connection information for use in subscription policies.

Each prefix added in the data registration workflow is created as a single Immuta data source, and a subscription policy added to a data source applies to any objects in that bucket or beginning with that prefix:

Registered prefixes 1

Therefore, data owners should register prefixes or buckets at the lowest level of access control they need for that data. Using the example above, if the data owner needed to allow different users to access s3://yellow-bucket/research-data/* than those who should access s3://yellow-bucket/analyst-data/*, the data owner must register the research-data/* and analyst-data/* prefixes separately and then apply a subscription policy to those prefixes:

Registered prefixes 2

Deleting registered prefixes

When an S3 data source is deleted, Immuta deletes all the grants associated with that prefix, bucket, or object in that location.

Mapping IAM principals in Immuta

Names are case-sensitive

The IAM role name and IAM user name are case-sensitive. See the AWS documentation for details.

Immuta supports mapping an Immuta user to one of the following AWS IAM principals:

  • IAM role principals: Only a single Immuta user can be mapped to an IAM role. This restriction prohibits enforcing policies on AWS users who could assume that role. Therefore, if using role principals, create a new user in Immuta that represents the role so that the role then has the permissions applied specifically to it.
  • IAM user principals

See the protect data section for instructions on mapping principals to user accounts in Immuta.

Existing S3 integrations

The Amazon S3 integration will not interfere with existing legacy S3 integrations, and multiple S3 integrations can exist in a single Immuta tenant.

Supported AWS services

AWS services that support S3 Access Grants will request temporary credentials for users automatically. If users are not using a service that supports S3 Access Grants, they must have the permissions listed in the AWS documentation to call the Access Grants API directly themselves to request temporary credentials to access data through the access grant.

For a list of AWS services that support S3 Access Grants, see the AWS documentation.

Limitations

  • During private preview, Immuta supports up to 500 prefixes (data sources) and up to 20 Immuta users that are mapped to S3 identities principals. This is a preview limitation that will be removed in a future phase of the integration.
  • S3 Access Grants allows 100,000 grants per region per account. Thus, if you have 5 Immuta users with access to 20,000 registered prefixes, you would reach this limit. See AWS documentation for details.

  • The following Immuta features are not currently supported by the integration in private preview:

    • Audit
    • Automatically syncing Immuta with AWS IAM identities: you cannot set the S3 User Type field to AWS IAM User when configuring your identity provider (IdP) in Immuta
    • Data policies
    • Schema monitoring
    • Tag ingestion