Using the AWS Asset Manager

The AWS Asset Manager can be used in multiple operational scenarios. For example, it can launch a Broker and its Engines in EC2. It’s also possible to launch Engines in EC2 from a Broker that is not launched in EC2. To use the AWS Asset Manager, perform the following tasks.

Prerequisites

You must have a valid AWS account.

Create a Broker AMI

This task only needs to be performed if both the Broker and Engines are to be run within EC2.

Procedure

  1. Launch an OS AMI (such as SLES 12 64-bit) making sure public IP is enabled, so yum updates to the OS will work. An m4.large AMI with 200 GB of disk space is ideal. Make sure there is enough swap space (about 8 GB) and enough file descriptors set on Broker for the fabric user (at least 200000).
  2. Install the Broker and configure it normally, starting with the JDK. Install relevant distributions, Enablers, and Skyway templates as needed. See the Installing Brokers section of the Silver Fabric Installation Guide for more information.
  3. Install an rc script to start and stop the Broker, as shown in Example Unix Broker Startup Script.
  4. Install any other items you may want in the Broker AMI, such as a reporting database, Maven repository, or LDAP server.
  5. Optionally, you can configure and run the NFS server on the Broker and export NFS directories. This makes it convenient to export the Grid Library location and a data directory used by Components when they need a shared file system. Refer to the Silver Fabric Cloud Administration Guide for more information on configuring an NFS mount as an alternate shared Grid Library location.
  6. Clean up the /tmp directory and various system logs.
  7. Create the AMI with the Broker shut down and snapshot the AMI without system reboot.

Create an Engine AMI

Procedure

  1. Launch an OS AMI (such as Centos 7 64-bit) making sure public IP is enabled so yum updates to the OS will work. An m4.large AMI with 200 GB of disk space is ideal.
  2. Extract the Engine TAR archive under /opt, but don't configure or run the Engine.
  3. Use the /opt/sfengine script as the start stop script and configure start/stop with the systemd service. See the Example AWS Sfengine Script for an example centos/RedHat start script.
  4. Enable the systemd sfengine.service. See the Example Sfengine.service Script for an example script.
  5. The standard engine.sh and configure.sh scripts may not always work for various OSes, and may require modification. For example, see the Example Engine.sh Script and Example Configure.sh Script for centos7/RedHat7.
  6. Create a user account named fabric.
  7. Clean up the /tmp directory and various system logs.
  8. Create the AMI with the Engine shut down and snapshot the AMI without system reboot.

Configure EC2 Security

Procedure

  1. Log in to the AWS Management Console and create a new Key Pair (See the AWS documentation on details), or use an existing AWS Key Pair.
  2. If you do not plan to use AWS VPC, create an AWS Security Group for the Silver Fabric Broker, named, for example, FSBroker. In this security group, grant access to ports 22, 8000 and 8443, and add more ports as needed.
  3. If you want your Silver Fabric Engine AWS instances to be in a different security group than the Silver Fabric Broker, create a separate Security Group for Silver Fabric Engines, named, for example, FSEngine. Allow public access to ports you want to access for running applications. It is easiest to grant access to all ports in the range 1-65000 from your Public Host IP address, but you may choose a narrower range.
  4. Ensure that the one or more security groups you created can access AWS instances within each other’s groups across all TCP/IP and UDP ports (1-65000) and also access instances within the same security group across all TCP/IP and UDP ports (1-65000).
  5. Go to the AWS Management Console and under EC2 > AMIs > Owned by me > Private images, ensure you have the Silver Fabric Broker and Engine AMIs.

Configure the Broker

Procedure

  1. Launch an Amazon AWS instance of the Silver Fabric Broker AMI in the Silver Fabric Broker security group, for example FSBroker, using the Key Pair you created above.
    Once the Silver Fabric Broker AWS instance is running, it will take approximately 45 minutes for the Silver Fabric Broker to initialize and be accessible by browser.
  2. You can connect to the Silver Fabric Broker AWS instance using your private key PEM file (make sure it is read-only by root) from a Linux machine using the command: ssh ec2-user@<hostname/ip_address> -I /<path_to_key_file>/<your_key_file>.pem
    While connecting to AWS instance, if it prompts you for a password, you missed a step, because you should not be prompted for a password when using your private key.
  3. Once you successfully log in into the AWS instance running the Silver Fabric Broker, you can find the current Silver Fabric Broker log file named fabricserver.log under /opt/SFversion/fabric/webapps/livecluster/WEB-INF/log/server/. Once the Silver Fabric Broker is successfully initialized, you will see following message in the fabricserver.log: file: INFO: [FabricEngineFileUpdateServer] Director->Broker synchronization finished, update server no longer suspended
  4. Once the Silver Fabric Broker is initialized, log in to the Silver Fabric administration tool in a browser at https://AWSInstancePublicHostname:8443/. Ignore the SSL certificate warning because the certificate is self-signed.

Configure Asset Manager

Procedure

  1. Go to Admin > Assets and update the Silver Fabric Asset Manager for AWS with the following configuration properties: Access Key, Secret Key, and Key Pair.
  2. For the Key Pair field, create the key pair at AWS > EC2 > Key Pair > Create new KeyPair and upload it to your Broker at the path given in this value.
  3. The Access key and Secret key values can also be provided in a file by doing the following:
    1. Create an AwsConfig.properties file.
    2. Enter the values of the access key and secret value in the following format. Note that the file contents are case-sensitive. AccessKey=accessKeyValue SecretKey=secretKeyValue
    3. Upload the file on the Broker at SF_HOME/webapps/livecluster/WEB-INF/assets/aws.
    4. Disable and re-enable the Asset Manager to apply the changes.
  4. If you are not using an AWS VPC (note that for all AWS accounts created after Dec. 4, 2013, AWS EC2 includes a default VPC, so you may be in a VPC without being aware of it), edit the Security Groups and set it to the Security Groups you created earlier, such as FSBroker, if you want to use a single security group for Broker and Engines or FSEngine, if you want to use a different security group for the Broker and Engines. More than one Security Group can be provided in a comma-separated list.
  5. To use Silver Fabric manager AMI in an AWS VPC (note that EC2 includes a default VPC for all accounts created after Dec. 4, 2013) configure the SubnetId and Security group Ids fields in the AWS Asset Manager configuration, but do not configure the Security Groups field. When using the security group ID of a VPC, configure the security group associated with the VPC appropriately to allow the Silver Fabric Broker to be accessible to Engines, and vice-versa. More than one Security Group ID can be provided through a comma-separated list.
  6. Select the Sweep Volumes option if you want detached volumes to be automatically deleted. Be careful while using this option because it will delete all detached volumes with status "available" in your AWS account.
  7. Select the Terminate option if you want idle Silver Fabric Engine AWS instances (the default minimum idle timeout is 15 minutes, and can be set with the Engine Idle Age variable) to be automatically terminated, otherwise, they are stopped, but not terminated.
  8. Set the Ec2 Url property to the relevant EC2 endpoint URL shown at http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region
  9. Go to Engines > Daemons and select Global Actions > Upgrade all Daemons if it is shown. If at any time your Engines turn red on the Engines > Engines page, you will need to do this step, which will restart all Engines and running components.
  10. Log into EC2 Management Console with your account, go to https://aws.amazon.com/marketplace/pp/B00O7WM7QW and launch an instance of the Centos 7 x86_64 AMI in your AWS account. The only purpose of this launch is to accept the Centos 7 AWS Marketplace license. You only have to do this once. You can terminate the instance as soon as you successfully launch it.
  11. Go to Admin > Assets and enable the Silver Fabric Asset Manager for AWS.
  12. Each Stack running within the Silver Fabric Broker can be configured to use its own specific Silver Fabric Engine AMI and a compatible AWS instance type.
  13. If the Require Policy Rule is selected, AssetManagerId is a must Component rule.
  14. If the AssetManagerId component rule is selected, it will always be validated against the value given in the Asset Manager configuration irrespective of the Require Policy Rule value.
  15. Resource preference Rules can be set with an equal operator for the Engine properties AssetManagerId, ec2AmiId, ec2InstanceType, ec2Zone, or vpcSubnetId. If there is a match, the respective properties values from the Asset Manager configuration will be overriden with the values defined in the Component policy rules.
  16. Initially launched Engines will have the Engine Group property value set to initial. In order to use the initially launched Engines for allocation, the Component Policy rule Group should be set to initial with the equal operator, or else the new Engine would be launched with the Engine Group property value as null.
  17. Select Aws Standard Image if the Engine installation and provisioning needs to be done from the Asset Manager on the launched instances.
  18. Provide Volume Size along with Device Name if the default root volume size needs to be updated. Note that volume size has to be greater than the original AMI volume size. Device name is the root device name of the AMI.
  19. If the Broker is SSL-enabled, make sure that the required certificates are available on the Broker. If you are using a custom AMI, make sure that certificates are available with the AMI. If the Broker is SSL-enabled and a standard AMI is used, SSL certificates need to be copied to Engines. They can be specified in the AwsConfig.properties file. If these values are not provided, the Asset Manager will copy the certificates from the default location (SF_HOME/webapps/livecluster/engineUpdate/shared). To specify them, add the following to the AwsConfig.properties file:
    1. Create an AwsConfig.properties file.
    2. Enter the values of the key and keystore files in the following format. Note that the file contents are case-sensitive. SslKeyFile=ssl key file path including file name (for example, /opt/SF/fabric/webapps/livecluster/WEB-INF/certs/ssl.pem) SslKeystoreFile=ssl keystore file path including file name (for example, /opt/SF/fabric/webapps/livecluster/WEB-INF/certs/ssl.keystore)
    3. Upload the AwsConfig.properties file to the Broker in the SF_HOME/webapps/livecluster/WEB-INF/assets/aws directory.

After Configuration

If changes needs to be done to an enabled Asset Manager, disable the respective Asset Manager first, make changes and then enable it again. If several configuration changes needs to be done, it is advisable to create another Asset Manager instance with the different configuration.

After your Broker is running, you can:

  • Log into Skyway at https://brokerhost:8443/skyway and launch a Stack.
  • Log into the Silver Fabric Broker at https://brokerhost:8443 and create Components and Stacks you would like to manage with Silver Fabric.

Important General Notes

The following notes apply to the AWS Asset Manager:

  • Do not use an Elastic IP for the Broker.
  • You do not need to manually launch a Silver Fabric Engine AWS instance. The Silver Fabric Engine AMI is automatically used by the Broker AWS Asset Manager to create AWS instances, as needed.
  • You can connect to any running Silver Fabric Engine AWS instance by using the command ssh –i your-key.pem user@aws-public-hostname
  • Once the Silver Fabric Broker AWS instance has been successfully started, it can be safely stopped and restarted at a later time. However, before you stop the Silver Fabric Broker AWS instance, make sure all running stacks in the Silver Fabric Administration Tool are completely stopped.
  • Do not manually start or stop AWS instances for Silver Fabric Engines.