Skip to content

Latest commit

 

History

History
315 lines (236 loc) · 12 KB

AWSAdapter.org

File metadata and controls

315 lines (236 loc) · 12 KB

AWSAdapter for ADS

Introduction

An adapter is an ADS component, which is responsible for creating and managing VMs on a particular platform.

The AWSAdapter is an adapter for the Amazon Web Services(AWS) platform.

In the following document, the words instance and VM have been used interchangeably.

Functional Requirements

  1. AWSAdapter will create a VM on AWS based on the specification given in the lab spec for a particular lab.
  2. AWSAdapter will copy the VMManager component to the newly created VM and start the VMManager service.

Design

AWSAdpater is required to manage VMs - create, stop, start, restart and destroy, on the AWS platform and also initialize the VM, which requires installing one component of ADS into the VM and start that service, for further communication.

AWSAdapter internally uses the boto Python library which interfaces with AWS APIs to create, restart, stop and destroy VMs. For more information on creation of a VM, see Strategic Decisions.

AWSAdapter uses the SSH service to initially install the ADS component and start the service. After the service is up, other components of ADS can communicate with the VM through this service.

Pre-requisites for the AWSAdapter :

Before detailing out the design, it is important to list the pre-requisites for the AWSAdapter :

  • The AdapterServer which communicates with this adapter is running on AWS.
  • The AdapterServer is configured to use the AWSAdapter.
  • Appropriate parameters for the adapter are configured in the configuration file (explained below).
  • The AWS key pair is placed in the appropriate location, which is same as what is configured in the configuration file.

Interfaces

In the current design of ADS, a fixed set of interfaces to an adapter is already defined. An adapter is required to implement these interfaces. This enables other components/services of ADS to communicate with the adapter. The interfaces implemented by the AWSAdapter are listed below:

  1. create_vm (lab_spec, dry_run=False)

    Creates a new VM on Amazon Cloud platform (AWS).

    Parameters:

    • lab_spec(dict) - The specification of the lab
    • dry_run(bool) - Set to True if the operation should not actually run.

    Returns: Instance id of the newly created Amazon VM.

  2. start_vm (vm_id, dry_run=False)

    Starts the VM identified by vm_id.

    Parameters:

    • vm_id(string) - VM/instance id of the VM to be started.
    • dry_run(bool) - Set to True if the operation should not actually run.

    Returns: List of the instance that got started.

  3. start_vm_manager (vm_id )

    Starts the VMManager service inside the VM identified by vm_id.

    Parameters:

    • vm_id(string) - VM/instance id of the VM.

    Returns: True if the VMManager service command was successful, else False

  4. is_running_vm (vm_ip)

    Checks if the VM, given by the IP addresses, is running or not. To perform the check, internally it uses boto AWS APIs to find out from the instance’s state.

    Parameters:

    • vm_ip(string) - The IP address of VM.

    Returns: True if the VM is running, else False.

  5. destroy_vm (vm_id, dry_run=False)

    Terminates the VM identified by vm_id.

    Parameters:

    • vm_id(string) - VM/instance id of the VM to be terminated.
    • dry_run(bool) - Set to True if the operation should not actually run.

    Returns: List of instances that got terminated.

NOTE: All these public interfaces will throw an exception if they encounter any error or exception. It is the responsibility of the callee to handle those exceptions.

Configuration parameters of the adapter

Following are the parameters which are needed to be configured in this adapter, for the adapter to be able to communicate with AWS APIs.

  1. region: The data centre region in which an AWS connection is to be made to create instances/VMs. Each region is a separate geographic area. More information about AWS regions here.
  2. credentials: Credentials are used to authenticate the API to AWS. The REST API uses access keys as the credentials. The access key has two parts:
    • Access Key ID - It identifies the party responsible for service requests. It is included in each request, so it’s not a secret.
    • Secret Access Key - Each Access Key ID has a Secret Access Key associated with it. This key is just a long string of characters (and not a file) that is used to calculate the digital signature included in the request. More information here.

    Both region and credentials are required whenever a new connection is established. Within this connection, a number of VMs could be created.

  3. subnet_id: The id of the subnet to be associated with an instance or VM. This is used while creating an instance or VMs. More information here.
  4. security_group_ids: A list of ids of the security groups to be associated with an instance or VM. This is used while creating an instance or VMs. More information about security groups here.
  5. key_name: The name of the key pair with which to launch instances. More information here.

These parameters come from a configuration file which is not checked in to the repository. Instead, a sample configuration file is checked in. A deployer is responsible for modifying this file and filling the appropriate values.

To configure one has to copy the sample config file into the actual config file(which is not version controlled) and edit the values accordingly.

From the top level of ADS project directory,

cp src/adapters/sample_aws_config.py src/adapters/aws_config.py

Edit the values in aws_config.py to have the appropriate configuration parameters.

Scenarios of AWSAdapter

Scenario 1

This deals only with the creation and initialization of a VM. TODO: Handle exception in the diagram? ./Sequence diagram of AWSAdapter.png

Following is the sequence of events that occur during lab deployment on AWS using AWSAdapter:

  1. VMPoolManager sends an HTTP Request to AdapterServer indicating that a VM needs to be created. It also passes the lab spec that includes VM requirements.
  2. On receiving this request, the AdapterServer calls the create_vm() function of AWSAdapter and hands over the lab spec to it.
  3. Based on lab spec(as mentioned below), the AWSAdapter selects an AMI and instance type for the VM.
  4. It creates a VM on AWS Cluster.
  5. The VM/instance id of the newly created VM is returned to the AdapterServer.
  6. The AdapterServer now calls the init_vm() function of AWSAdapter.
  7. This initializes the newly created VM on AWS Cluster by copying relevant ADS component (VMManager) and lab sources, and starts the VMManager service.
  8. Once this service has started, the AWSAdapter sends back a tuple to the AdapterServer consisting of a boolean value(that indicates success/failure), instance/VM id of the VM on AWS Cluster, IP address of the VM, and port on which the VMManager service is running.
  9. The latter three are forwarded by AdapterServer as the HTTP response to the VMPoolManager.

Internal Design

How to select an AMI based on the lab spec input?

TODO

How to select an instance type based on the lab spec input?

TODO

Strategic decisions <<Strategic Decisions>>

What OS/AMIs to support?

[As of :<2015-05-01 Fri>] AWS uses OS images to associate an OS with an instance when one brings up an instance. AWS calls these OS images as AMIs. (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) To provision instances we need to choose AMIs, hence we needed to decide on what OS and AMIs to support.

  • In the existing implementation of ADS on CentOS-OpenVZ platform - Ubuntu, CentOS and Debian were supported. The decision is to go ahead with these OSes, considering almost all the current labs(level 5 and hosted labs on IIIT infrastructure) use Ubuntu 10.04/12.04.
  • Regarding versions of the OS, we decided:
    • Ubuntu::
      • Only LTS versions are supported.
      • Ubuntu 10.04:: Has reached end of life of support and hence we also cannot support it, even though some lab developers have communicated that they have a hard dependency on this version of Ubuntu.
      • Ubuntu 12.04:: Supported, as its end of life is 2017.
      • Ubuntu 14.04:: Supported, as its end of life 2019.
    • CentOS::
      • CentOS 6.6:: Supported. This is the current stable version.
      • CentOS 6.5:: Requests to use CentOS 6.5 will automatically be upgraded to use CentOS 6.6.
    • Debian::
      • Debian 7.x:: Supported, as end of life is 2016.
  • Hence the list of supported OS: [As of :<2015-05-01 Fri>]
    OSVersionTill
    Ubuntu12.042017
    Ubuntu14.042019
    CentOS6.6
    Debian7.x2016

What machine configuration/instance types to support?

[As of :<2015-05-01 Fri>] As AWS has the concept of instance types (which is essentially a set of machine configuration), we need to select a set of instance types that we support. All the instance types available on AWS are listed here: https://aws.amazon.com/ec2/instance-types/

Looking at most of the lab requirements, we concluded that having instance types with 2GB and 4GB RAM would suffice them. For storage, AWS has a concept of EBS(https://aws.amazon.com/ebs/getting-started/). The default EBS is of 8GB with our current instance types. Our conclusion was for the current labs, the default EBS should be fine. In a later version, the adapter can add the storage dynamically.

Hence, the current supported AWS instance types by AWSAdapter are:

  • t2.micro (1GB RAM)
  • t2.small (2GB RAM)

Implementation

The implementation of the AWSAdapter can be found here: ../src/adapters/AWSAdapter.py

Other related documentation

How to create an AMI for lab deployment?

TODO

Testing

Objective

  • To validate the creation of a VM on AWS platform.
  • To ensure that the VMManager service is running on this VM.

Assumptions

  1. A VM is provisioned on AWS, with internet access.
  2. ADS is installed and configured in that VM and its services are up.
  3. The source code of the lab to be tested for deployment is copied in /root/labs folder. However, the test module also takes care of this prerequisite.

Test Cases

Case 01: To test the creation of a Virtual Machine on AWS platform.

This is achieved by checking the state of the newly created instance. The test case checks with Amazon API, passing the vm_id returned from AWSAdapter and asserts if the state is either pending or running (corresponding to integers 0 or 16).

Case 02: To test the initialization of VM

The objective of this test is to validate if the lab sources and VMManager is copied and VMManager service is started. This is tested by asserting the boolean value returned after successful initialization of the VM.

Case 03: To test if the VMManager service is running

After the instance is initialized, it is necessary to validate if the VMManager service is actually running or not. This is tested by asserting the response of an HTTP request sent to the VMManager service.

The implementation of test cases can be found here