Blog

You are currently viewing Amazon MSK IAM authentication now helps all programming languages

Amazon MSK IAM authentication now helps all programming languages


The AWS Id and Entry Administration (IAM) authentication function in Amazon Managed Streaming for Apache Kafka (Amazon MSK) now helps all programming languages. Directors can simplify and standardize entry management to Kafka assets utilizing IAM. This help is predicated on SASL/OUATHBEARER, an open customary for authorization and authentication. Each Amazon MSK provisioned and serverless cluster varieties help the brand new Amazon MSK IAM growth to all programming languages.

On this put up, we present how one can join your functions to MSK clusters with minimal code adjustments utilizing the open-sourced shopper helper libraries and code samples for fashionable languages, together with Java, Python, Go, JavaScript, and .NET. You may as well use customary IAM entry controls corresponding to momentary role-based credentials and exactly scoped permission insurance policies extra broadly with the a number of language help on Amazon MSK.

For shoppers that want to attach from different VPCs to an MSK cluster, whether or not in a identical or a special AWS account, you possibly can allow multi-VPC non-public connectivity and cluster coverage help. IAM entry management through cluster coverage helps you handle all entry to the cluster and subjects in a single place. For instance, you possibly can management which IAM principals have write entry to sure subjects, and which principals can solely learn from them. Customers who’re utilizing IAM shopper authentication can even add permissions for required kafka-cluster actions within the cluster useful resource coverage.

Resolution overview

You may get began through the use of IAM principals as identities to your Apache Kafka shoppers and outline id insurance policies to offer them exactly scoped entry permissions. After IAM authentication is enabled to your cluster, you possibly can configure shopper functions to make use of the IAM authentication with minimal code adjustments.

The code adjustments enable your shoppers to make use of SASL/OAUTHBEARER, a Kafka supported token-based entry mechanism, to go the credentials required for IAM authentication. On this put up, we present how one can make these code adjustments through the use of the offered code libraries and examples.

With this launch, new code libraries for the next programming languages can be found within the AWS GitHub repo:

The next diagram reveals the conceptual course of circulate of utilizing SASL/OAUTHBEARER with IAM entry management for non-Java shoppers.

The workflow accommodates the next steps:

  1. The shopper generates an OAUTHBEARER token with the assistance of the offered library. The token accommodates a signed base64 encoded transformation of your IAM id credentials.
  2. The shopper sends this to Amazon MSK utilizing the IAM bootstrap dealer addresses together with its request to entry Apache Kafka assets.
  3. The MSK dealer decodes the OATHBEARER token, validates the credentials, and checks if the shopper is permitted to carry out the requested motion in accordance with the coverage connected to the IAM id.
  4. When the token expires, the shopper Kafka library routinely refreshes the token by making one other name to the desired token supplier.

Create IAM identities and insurance policies

IAM entry management for non-Java shoppers is supported for MSK clusters with Kafka model 2.7.1 and above. Earlier than you begin, you want to configure the IAM identities and insurance policies that outline the shopper’s permissions to entry assets on the cluster. The next is an instance authorization coverage for a cluster named MyTestCluster. To know the semantics of the motion and useful resource parts, see Semantics of actions and assets.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeCluster"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:cluster/MyTestCluster/abcd1234-0123-abcd-5678-1234abcd-1"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:*Topic*",
                "kafka-cluster:WriteData",
                "kafka-cluster:ReadData"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:topic/MyTestCluster/*"
            ]
        },
        {
            "Impact": "Permit",
            "Motion": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Useful resource": [
                "arn:aws:kafka:us-east-1:0123456789012:group/MyTestCluster/*"
            ]
        }
    ]
}

Arrange the MSK cluster

You’ll want to allow the IAM entry management authentication scheme to your MSK provisioned cluster and wait till the cluster finishes updating and turns to the Lively state. It is because SASL/OAUTHBEARER makes use of the identical dealer addresses for IAM authentication.

Configure the shopper

It is best to make code adjustments to your utility that enable the shoppers to make use of SASL/OAUTHBEARER to go the credentials required for IAM authentication. Subsequent, replace your utility to make use of the bootstrap server addresses for IAM authentication. You additionally want to verify the safety group related together with your MSK cluster has an inbound rule permitting the site visitors from the shopper functions in the identical VPC because the cluster to the TCP port 9098.

It’s essential to use a Kafka shopper library that gives help for SASL with OAUTHBRARER authentication.

On this put up, we use the JavaScript programming language. We additionally use https://github.com/tulios/kafkajs as our Kafka shopper library.

Amazon MSK offers you with a brand new code library per every language that generates the OAUTHBEARER token.

  1. To get began working with the Amazon MSK IAM SASL signer for JavaScript together with your Kafka shopper library, run the next command: 
    npm set up https://github.com/aws/aws-msk-iam-sasl-signer-js

  2. You’ll want to import the put in Amazon MSK IAM SASL signer library in your code: 
    const { Kafka } = require('kafkajs')

const { generateAuthToken } = require('aws-msk-iam-sasl-signer-js')

  3. Subsequent, your utility code must outline a token supplier that wraps the operate that generates new tokens: 
    async operate oauthBearerTokenProvider(area) {
    // Makes use of AWS Default Credentials Supplier Chain to fetch credentials
    const authTokenResponse = await generateAuthToken({ area });
    return {
        worth: authTokenResponse.token
    }
}

  4. Specify security_protocol as SASL_SSL and sasl_mechanism as oauthbearer in your JavaScript Kafka shopper properties, and go the token supplier within the configuration object: 
    const run = async () => {
    const kafka = new Kafka({
        clientId: 'my-app',
        brokers: [bootstrap server addresses for IAM],
        ssl: true,
        sasl: {
            mechanism: 'oauthbearer',
            oauthBearerProvider: () => oauthBearerTokenProvider('us-east-1')
        }
    })

    const producer = kafka.producer()
    const shopper = kafka.shopper({ groupId: 'test-group' })

    // Producing
    await producer.join()
    await producer.ship({
        matter: 'test-topic',
        messages: [
            { value: 'Hello KafkaJS user!' },
        ],
    })

    // Consuming
    await shopper.join()
    await shopper.subscribe({ matter: 'test-topic', fromBeginning: true })

    await shopper.run({
        eachMessage: async ({ matter, partition, message }) => {
            console.log({
                partition,
                offset: message.offset,
                worth: message.worth.toString(),
            })
        },
    })
}

run().catch(console.error)

You are actually completed with all of the code adjustments. For extra examples of producing auth tokens or for extra troubleshooting ideas, discuss with the next GitHub repo.

Conclusion

IAM entry management for Amazon MSK lets you deal with each authentication and authorization to your MSK cluster. This eliminates the necessity to use one mechanism for authentication and one other for authorization. For instance, when a shopper tries to write down to your cluster, Amazon MSK makes use of IAM to test whether or not that shopper is an authenticated id and likewise whether or not it’s approved to supply to your cluster.

With at the moment’s launch, Amazon MSK IAM authentication now helps all programming languages. This implies you possibly can join your functions in all languages with out worrying about implementing separate authentication and authorization mechanisms. For workloads that require Amazon MSK multi-VPC non-public connectivity and cluster coverage help, now you can simplify connectivity to your MSK brokers and handle all entry to the cluster and subjects in a single place that’s your cluster coverage.

For additional studying on Amazon MSK, go to the official product web page.

Concerning the creator

Ali Alemi is a Streaming Specialist Options Architect at AWS. Ali advises AWS clients with architectural greatest practices and helps them design real-time analytics information programs which might be dependable, safe, environment friendly, and cost-effective. He works backward from buyer’s use instances and designs information options to resolve their enterprise issues. Previous to becoming a member of AWS, Ali supported a number of public sector clients and AWS consulting companions of their utility modernization journey and migration to the cloud.

Leave a Reply