AWS EMR

4 min

Resumo

Execute ações ou gerencie objetos no AWS EMR.

Parâmetros

Parâmetro	Tipo	Escolhas \| Valor Padrão	Comentários
region - required	string		A região AWS para criar ou gerenciar um EMR.
mode - required	string	- create - describe - get-cluster-id - get-cluster-ids - terminate - terminate-all - check-status - get-master-ip - get-core-ips - get-slave-ips - get-collection-id-by-name - add-instance-group - scale-out - scale-in	Operação dada ao EMR.
cluster_name	string		Nome do cluster EMR para buscar informações / executar uma ação.
id	string		ID do cluster. Se o ID do cluster não for fornecido, o nome do cluster também é aceito.
auto_scaling_role	string	Valor default: EMR_AutoScaling_DefaultRole	A função de escalonamento automático para criar cluster EMR
applications	list	Valor default: ['Hadoop', 'Spark']	As aplicações necessárias para criar um cluster EMR.
log_url	string		URL para manter registros de clusters EMR e lotes em execução no EMR.
release_label	string	Valor default: emr-5.8.0	Versão do EMR.
service_role	string	Valor default: iam-role-emr	A role IAM do cluster EMR.
ec2_service_role	string	Valor default: iam-role-emr-ec2	A role IAM para as instâncias ec2 do EMR.
scale_down_behavior	string	Valor default: TERMINATE_AT_TASK_COMPLETION	O comportamento de redução quando uma instância é dimensionada.
tags	string		Tags para instâncias ec2 ao criar um cluster EMR.
ec2_key_file_name	string		O nome do arquivo pem para acesso ssh à instância EMR.
emr_master_security_group	string		ID do grupo de segurança do EMR master.
emr_slave_security_group	string		ID do grupo de segurança do EMR slave.
emr_service_security_group	string		ID do grupo de segurança do EMR service access.
key_alive_when_no_steps	bool	Valor default: True	Se o EMR deve estar ativo após as etapas serem concluídas.
termination_protection	bool	Valor default: True	Se será necessário encerrar a proteção para o cluster EMR criado.
ec2_subnet	string		Id de sub-rede para criar um cluster EMR.
ec2_subnets	list		Lista de Ids de sub-rede para criar um cluster EMR.
bootstrap_actions	string		Caminho para um arquivo json contendo ações bootstrap.
configurations	string		Caminho para um arquivo json contendo configurações de aplicativo.
instances	string		Um caminho para um arquivo json contendo propriedades de instância.
instance_collection_name	string		Nome para grupo de instância.
instance_group_id	string		ID do grupo de instâncias.
scale_out_instance_type	string		Tipo de instância para grupo de instâncias de expansão.
scale_out_instance_count	string		Contagem de instâncias para escalar grupo de instâncias / frota.
enable_fleet	bool		Se você precisar criar um cluster EMR por instance fleet ou grupo de instâncias.
scale_in_instance_count			O número para scale.

Extende fragmentos da documentação do módulo aws.

Exemplos

# Example 1: Create EMR cluster by instance groups
- name: create emr cluster by instance groups
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: create
    name: EMR_exmaple_name
    auto_scaling_role: EMR_AutoScaling_DefaultRole
    applications: ['Hadoop', 'Spark']
    log_url: 's3://emr-bucket-example/logs'
    tags: [{'Name': 'EMR_exmaple_name'}]
    ec2_key_file_name: key-exmaple
    emr_master_security_group: 'sg-example1'
    emr_slave_security_group: 'sg-example2'
    emr_service_security_group: 'sg-example3'
    instances: "{{ playbook_dir }}/roles/emr/init-create/files/instances.json"
    bootstrap_actions: "{{ playbook_dir }}/roles/emr/init-create/files/bootstrapactions.json"
    configurations: "{{ playbook_dir }}/roles/emr/init-create/files/emr_config.json"
    ec2_subnet: 'subnet-example'

# Example 2: Create EMR cluster by instance fleets
- name: create emr cluster by instance fleets, cross different AZs.
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: create
    name: EMR_exmaple_name
    auto_scaling_role: EMR_AutoScaling_DefaultRole
    applications: ['Hadoop', 'Spark']
    log_url: 's3://emr-bucket-example/logs'
    tags: [{'Name': 'EMR_exmaple_name'}]
    ec2_key_file_name: key-exmaple
    emr_master_security_group: 'sg-example1'
    emr_slave_security_group: 'sg-example2'
    emr_service_security_group: 'sg-example3'
    instances: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/instances.json"
    bootstrap_actions: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/bootstrapactions.json"
    configurations: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/emr_config.json"
    ec2_subnets: ['subnet-example1', 'subnet-example2']
    enable_fleet: true

# Example 3: Describe cluster by cluster Id
- name: Describe cluster by cluster Id
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: describe
    id: "{{ CLUSTER_ID }}"

# Example 4: Get cluster Id of the active cluster by cluster name
- name: Get cluster Id of the active cluster by cluster name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-cluster-id
    name: "{{ CLSUTER_NAME }}"

# Example 5: Get cluster Id of all active clusters by cluster name
- name: Get cluster Id of all active clusters by cluster name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-cluster-ids
    name: "{{ CLSUTER_NAME }}"

# Example 6: Terminate EMR by cluster Id
- name: Terminate EMR by cluster Id
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: terminate
    id: "{{ CLUSTER_ID }}"

# Example 7: Terminate EMR by cluster name. There should be only one active cluster found by the given cluster name
- name: Terminate EMR by cluster name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: terminate
    name: "{{ CLSUTER_NAME }}"

# Example 8: Terminate all active cluster by given cluster name
- name: Terminate all active cluster by given cluster name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: terminate-all
    name: "{{ CLSUTER_NAME }}"

# Example 9: Check EMR clusters by cluster Id
- name: Check EMR clusters by cluster Id
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: check-status
    id: "{{ CLUSTER_ID }}"

# Example 10: Get the IP address of Master instance by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of master instance
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-master-ip
    id: "{{ CLUSTER_ID }}"

# Example 11: Get the IP address of Master instance by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of master instance
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-master-ip
    name: "{{ CLSUTER_NAME }}"

# Example 12: Get the IP address of CORE instances by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of CORE instances
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-core-ips
    id: "{{ CLUSTER_ID }}"

# Example 13: Get the IP address of CORE instances by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of CORE instances
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-core-ips
    name: "{{ CLSUTER_NAME }}"

# Example 14: Get the IP address of Slave instances by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of TASK instances
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-slave-ips
    id: "{{ CLUSTER_ID }}"

# Example 15: Get the IP address of Slave instances by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of TASK instances
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-slave-ips
    name: "{{ CLSUTER_NAME }}"
    
# Example 16: Get instance group/fleet id by cluster id and instance group/fleet name
- name: Get instance group/fleet id by name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-collection-id-by-name
    id: "{{ CLUSTER_ID }}"
    instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"

# Example 17: Get instance group/fleet id by cluster name and instance group/fleet name
- name: Get instance group/fleet id by name
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: get-collection-id-by-name
    name: "{{ CLSUTER_NAME }}"
    instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"

# Example 18: Add an instance group for a given EMR cluster Id (Not used). EMR cluster name is also acceptable.
- name: Add an instance group
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: add-instance-group
    id: "{{ CLUSTER_ID }}"
    scale_out_instance_type: m4.large
    scale_out_instance_count: 4
    instance_collection_name: 'New_EMR_Task_IG'

# Example 19: Scale out on instance fleet. The example will add one on-demand request on Task Instance Fleet
- name: Scale out on instance fleet
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: scale-out
    id: "{{ CLUSTER_ID }}"
    scale_out_instance_count: 1

# Example 20: Scale out by adding a new instance group, if the given instance group by name was not found.
- name: Scale out on instance group
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: scale-out
    id: "{{ CLUSTER_ID }}"
    scale_out_instance_count: 1
    instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"
    scale_out_instance_type: r4.xlarge

# Example 21: Scale in on instance fleet. The on-demand target will be reduced by 1. (0 is the minimum)
- name: Scale in
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: scale-in
    id: "{{ CLUSTER_ID }}"
    scale_in_instance_count: 1

# Example 22: Scale in on instance fleet. The on-demand target will be reduced by 1. (0 is the minimum) If scale_in_instance_count is undefined, the on-demand target will be set to 0 directly.
- name: Scale in
  aws_emr:
    aws_access_key: "{{ AWS_ACCESS_KEY }}"
    aws_secret_key: "{{ AWS_SECURITY_KEY }}"
    region: "{{ AWS_REGION }}"
    mode: scale-in
    id: "{{ CLUSTER_ID }}"
    scale_in_instance_count: 1

Valores Retornados

Chave	Tipo	Retornado quando	Descrição