Configuring an ElasticSearch domain with Amazon SDK

We prefer to create a shared (between all dev environments in the team) ES domain because ES domains can take 15-20 mins to be set up. Having each member of the team deploys an ES cluster hurts productivity as well as consumes resources in the cloud (higher costs).

Creating an ES domain can be done using the CDK or the SDK. Some settings are not configurable via the CDK (e.g. Advanced Security Options). So, we prefer to have a script that uses the SDK to create and destroy our ES domain on-demand.

process.env.AWS_SDK_LOAD_CONFIG = 'true';

import { ES } from 'aws-sdk';
import { ElasticsearchDomainStatus } from 'aws-sdk/clients/es';

const es = new ES();

export async function getDomainStatus(name: string): Promise<ElasticsearchDomainStatus> {
  const domain = await es
    .describeElasticsearchDomain({
      DomainName: name,
    })
    .promise();

  return domain?.DomainStatus;
}

/**
 * @param name - The name of your ES Domain
 */
async function checkIfESDomainExists(name: string): Promise<boolean> {
  let doesDomainExist: boolean;
  try {
    doesDomainExist = !!(await getDomainStatus(ES_DOMAIN_NAME));
  } catch (e) {
    // NOT_FOUND_STATUS_CODE: 404
    // CONFLICT_STATUS_CODE: 409
    if (e.statusCode !== statusCodes.NOT_FOUND_STATUS_CODE && e.statusCode !== statusCodes.CONFLICT_STATUS_CODE) {
      error(`ES domain ${name} status can't be retrieved`);
    }

    doesDomainExist = false;
  }

  return doesDomainExist;
}

Creating your ES domain using the SDK

You will need to create your ES domain once at the beginning when you need to provision search into your application for the entire team. You can write a script with the following code to create your ES domain using the SDK. After provisioning your domain, you will not need to run this script again unless you want to make changes to your ES domain (which we expect to be infrequent).

process.env.AWS_SDK_LOAD_CONFIG = 'true';

import { ES } from 'aws-sdk';

import { ES_DOMAIN_NAME } from '../util/consts';

const es = new ES();

async function createESDomain(roleArn: string): Promise<void> {
  await es
    .createElasticsearchDomain({
      DomainName: ES_DOMAIN_NAME,
      ElasticsearchVersion: '7.7',
      ElasticsearchClusterConfig: {
        // Smaller instance types, e.g. t2 small do not support encryption at rest and 
        // fine-grained access control cannot be enabled when encryption at rest is disabled.
        InstanceType: 'r5.large.elasticsearch',
        InstanceCount: 1,
        DedicatedMasterEnabled: false,
        ZoneAwarenessEnabled: false,
      },
      AdvancedSecurityOptions: {
        Enabled: true,
        InternalUserDatabaseEnabled: true,
        MasterUserOptions: {
          // See the following section for an example on how to create this role
          MasterUserARN: roleArn,
        },
      },
      EBSOptions: {
        EBSEnabled: true,
        VolumeType: 'gp2',
        VolumeSize: 10,
      },
      AccessPolicies:
        '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"*"},"Action":"es:*","Resource":"arn:aws:es:*:*:domain/*/*"}]}',
      CognitoOptions: {
        Enabled: false,
      },
      EncryptionAtRestOptions: {
        Enabled: true,
      },
      NodeToNodeEncryptionOptions: {
        Enabled: true,
      },
      DomainEndpointOptions: {
        EnforceHTTPS: true,
        TLSSecurityPolicy: 'Policy-Min-TLS-1-0-2019-07',
      },
      AdvancedOptions: {
        'rest.action.multi.allow_explicit_index': 'true',
      },
    })
    .promise();
} 

Creating role to access domain

You need to create a role that gives your ES domain user the necessary permissions to work correctly. Here is an example of role created for enabling a glue job to gain MasterUser access into the ES domain,

async function createJobRole(): Promise<CreateRoleResponse> {
  const jobRole = iam
    .createRole({
      AssumeRolePolicyDocument: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Principal: {
              Service: 'glue.amazonaws.com',
            },
            Action: 'sts:AssumeRole',
          },
        ],
      }),
      RoleName: JOB_ROLE_NAME,
    })
    .promise();

  // Add a delay for the created role to propagate into AWS list of roles before attaching inline policies
  await delay(5000);

  await iam
    .putRolePolicy({
      PolicyDocument: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Action: ['s3:ListBucket', 's3:GetObject', 's3:PutObject', 's3:DeleteObject'],
            Resource: 'arn:aws:s3:::*',
          },
        ],
      }),
      PolicyName: 'AllowS3ReadAndWrite',
      RoleName: JOB_ROLE_NAME,
    })
    .promise();

  await iam
    .putRolePolicy({
      PolicyDocument: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Action: 'glue:*',
            Resource: '*',
          },
        ],
      }),
      PolicyName: 'AllowGlue',
      RoleName: JOB_ROLE_NAME,
    })
    .promise();

  await iam
    .putRolePolicy({
      PolicyDocument: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Action: 'logs:*',
            Resource: '*',
          },
        ],
      }),
      PolicyName: 'AllowCWLogs',
      RoleName: JOB_ROLE_NAME,
    })
    .promise();

  await iam
    .putRolePolicy({
      PolicyDocument: JSON.stringify({
        Version: '2012-10-17',
        Statement: [
          {
            Effect: 'Allow',
            Action: 'sts:*',
            Resource: '*',
          },
        ],
      }),
      PolicyName: 'AllowSTS',
      RoleName: JOB_ROLE_NAME,
    })
    .promise();

  return jobRole;
}

We can then pass this job role ARN to the domain creation function: await createESDomain(jobRole.Role.Arn)

Deleting your ES domain