AWS EMR
Resumo
- Execute ações ou gerencie objetos no AWS EMR.
Parâmetros
Parâmetro | Tipo | Escolhas | Valor Padrão | Comentários |
|---|---|---|---|
region - required | string | | A região AWS para criar ou gerenciar um EMR. |
mode - required | string | - create - describe - get-cluster-id - get-cluster-ids - terminate - terminate-all - check-status - get-master-ip - get-core-ips - get-slave-ips - get-collection-id-by-name - add-instance-group - scale-out - scale-in | Operação dada ao EMR. |
cluster_name | string | | Nome do cluster EMR para buscar informações / executar uma ação. |
id | string | | ID do cluster. Se o ID do cluster não for fornecido, o nome do cluster também é aceito. |
auto_scaling_role | string | Valor default: EMR_AutoScaling_DefaultRole | A função de escalonamento automático para criar cluster EMR |
applications | list | Valor default: ['Hadoop', 'Spark'] | As aplicações necessárias para criar um cluster EMR. |
log_url | string | | URL para manter registros de clusters EMR e lotes em execução no EMR. |
release_label | string | Valor default: emr-5.8.0 | Versão do EMR. |
service_role | string | Valor default: iam-role-emr | A role IAM do cluster EMR. |
ec2_service_role | string | Valor default: iam-role-emr-ec2 | A role IAM para as instâncias ec2 do EMR. |
scale_down_behavior | string | Valor default: TERMINATE_AT_TASK_COMPLETION | O comportamento de redução quando uma instância é dimensionada. |
tags | string | | Tags para instâncias ec2 ao criar um cluster EMR. |
ec2_key_file_name | string | | O nome do arquivo pem para acesso ssh à instância EMR. |
emr_master_security_group | string | | ID do grupo de segurança do EMR master. |
emr_slave_security_group | string | | ID do grupo de segurança do EMR slave. |
emr_service_security_group | string | | ID do grupo de segurança do EMR service access. |
key_alive_when_no_steps | bool | Valor default: True | Se o EMR deve estar ativo após as etapas serem concluídas. |
termination_protection | bool | Valor default: True | Se será necessário encerrar a proteção para o cluster EMR criado. |
ec2_subnet | string | | Id de sub-rede para criar um cluster EMR. |
ec2_subnets | list | | Lista de Ids de sub-rede para criar um cluster EMR. |
bootstrap_actions | string | | Caminho para um arquivo json contendo ações bootstrap. |
configurations | string | | Caminho para um arquivo json contendo configurações de aplicativo. |
instances | string | | Um caminho para um arquivo json contendo propriedades de instância. |
instance_collection_name | string | | Nome para grupo de instância. |
instance_group_id | string | | ID do grupo de instâncias. |
scale_out_instance_type | string | | Tipo de instância para grupo de instâncias de expansão. |
scale_out_instance_count | string | | Contagem de instâncias para escalar grupo de instâncias / frota. |
enable_fleet | bool | | Se você precisar criar um cluster EMR por instance fleet ou grupo de instâncias. |
scale_in_instance_count | | | O número para scale. |
Extende fragmentos da documentação do módulo aws.
Exemplos
# Example 1: Create EMR cluster by instance groups
- name: create emr cluster by instance groups
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: create
name: EMR_exmaple_name
auto_scaling_role: EMR_AutoScaling_DefaultRole
applications: ['Hadoop', 'Spark']
log_url: 's3://emr-bucket-example/logs'
tags: [{'Name': 'EMR_exmaple_name'}]
ec2_key_file_name: key-exmaple
emr_master_security_group: 'sg-example1'
emr_slave_security_group: 'sg-example2'
emr_service_security_group: 'sg-example3'
instances: "{{ playbook_dir }}/roles/emr/init-create/files/instances.json"
bootstrap_actions: "{{ playbook_dir }}/roles/emr/init-create/files/bootstrapactions.json"
configurations: "{{ playbook_dir }}/roles/emr/init-create/files/emr_config.json"
ec2_subnet: 'subnet-example'
# Example 2: Create EMR cluster by instance fleets
- name: create emr cluster by instance fleets, cross different AZs.
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: create
name: EMR_exmaple_name
auto_scaling_role: EMR_AutoScaling_DefaultRole
applications: ['Hadoop', 'Spark']
log_url: 's3://emr-bucket-example/logs'
tags: [{'Name': 'EMR_exmaple_name'}]
ec2_key_file_name: key-exmaple
emr_master_security_group: 'sg-example1'
emr_slave_security_group: 'sg-example2'
emr_service_security_group: 'sg-example3'
instances: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/instances.json"
bootstrap_actions: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/bootstrapactions.json"
configurations: "{{ playbook_dir }}/roles/emr/init-create-fleet/files/emr_config.json"
ec2_subnets: ['subnet-example1', 'subnet-example2']
enable_fleet: true
# Example 3: Describe cluster by cluster Id
- name: Describe cluster by cluster Id
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: describe
id: "{{ CLUSTER_ID }}"
# Example 4: Get cluster Id of the active cluster by cluster name
- name: Get cluster Id of the active cluster by cluster name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-cluster-id
name: "{{ CLSUTER_NAME }}"
# Example 5: Get cluster Id of all active clusters by cluster name
- name: Get cluster Id of all active clusters by cluster name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-cluster-ids
name: "{{ CLSUTER_NAME }}"
# Example 6: Terminate EMR by cluster Id
- name: Terminate EMR by cluster Id
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: terminate
id: "{{ CLUSTER_ID }}"
# Example 7: Terminate EMR by cluster name. There should be only one active cluster found by the given cluster name
- name: Terminate EMR by cluster name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: terminate
name: "{{ CLSUTER_NAME }}"
# Example 8: Terminate all active cluster by given cluster name
- name: Terminate all active cluster by given cluster name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: terminate-all
name: "{{ CLSUTER_NAME }}"
# Example 9: Check EMR clusters by cluster Id
- name: Check EMR clusters by cluster Id
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: check-status
id: "{{ CLUSTER_ID }}"
# Example 10: Get the IP address of Master instance by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of master instance
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-master-ip
id: "{{ CLUSTER_ID }}"
# Example 11: Get the IP address of Master instance by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of master instance
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-master-ip
name: "{{ CLSUTER_NAME }}"
# Example 12: Get the IP address of CORE instances by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of CORE instances
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-core-ips
id: "{{ CLUSTER_ID }}"
# Example 13: Get the IP address of CORE instances by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of CORE instances
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-core-ips
name: "{{ CLSUTER_NAME }}"
# Example 14: Get the IP address of Slave instances by cluster Id. Work for both Instance groups and instance fleets.
- name: get IP address of TASK instances
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-slave-ips
id: "{{ CLUSTER_ID }}"
# Example 15: Get the IP address of Slave instances by cluster name. Work for both Instance groups and instance fleets.
- name: get IP address of TASK instances
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-slave-ips
name: "{{ CLSUTER_NAME }}"
# Example 16: Get instance group/fleet id by cluster id and instance group/fleet name
- name: Get instance group/fleet id by name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-collection-id-by-name
id: "{{ CLUSTER_ID }}"
instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"
# Example 17: Get instance group/fleet id by cluster name and instance group/fleet name
- name: Get instance group/fleet id by name
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: get-collection-id-by-name
name: "{{ CLSUTER_NAME }}"
instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"
# Example 18: Add an instance group for a given EMR cluster Id (Not used). EMR cluster name is also acceptable.
- name: Add an instance group
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: add-instance-group
id: "{{ CLUSTER_ID }}"
scale_out_instance_type: m4.large
scale_out_instance_count: 4
instance_collection_name: 'New_EMR_Task_IG'
# Example 19: Scale out on instance fleet. The example will add one on-demand request on Task Instance Fleet
- name: Scale out on instance fleet
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: scale-out
id: "{{ CLUSTER_ID }}"
scale_out_instance_count: 1
# Example 20: Scale out by adding a new instance group, if the given instance group by name was not found.
- name: Scale out on instance group
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: scale-out
id: "{{ CLUSTER_ID }}"
scale_out_instance_count: 1
instance_collection_name: "{{ INSTANCE_GROUP_NAME }}"
scale_out_instance_type: r4.xlarge
# Example 21: Scale in on instance fleet. The on-demand target will be reduced by 1. (0 is the minimum)
- name: Scale in
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: scale-in
id: "{{ CLUSTER_ID }}"
scale_in_instance_count: 1
# Example 22: Scale in on instance fleet. The on-demand target will be reduced by 1. (0 is the minimum) If scale_in_instance_count is undefined, the on-demand target will be set to 0 directly.
- name: Scale in
aws_emr:
aws_access_key: "{{ AWS_ACCESS_KEY }}"
aws_secret_key: "{{ AWS_SECURITY_KEY }}"
region: "{{ AWS_REGION }}"
mode: scale-in
id: "{{ CLUSTER_ID }}"
scale_in_instance_count: 1
Valores Retornados
Chave | Tipo | Retornado quando | Descrição |
|---|