AWS Cloud Cost Optimization using Lambda Function

Introduction to Serverless Computing

Today, we're going to embark on an exciting journey into the world of serverless computing and explore AWS Lambda, a powerful service offered by Amazon Web Services.

So, what exactly is "serverless computing"? Don't worry; it's not about getting rid of servers entirely. Instead, serverless computing is a cloud computing model where you, as a developer, don't have to manage servers directly. You get to focus on writing and deploying your code, while the cloud provider handles all the underlying infrastructure for you.

AWS Lambda:

AWS Lambda is a part of the compute family of services and addresses the need for serverless computing.

The main difference between EC2 and Lambda functions is that Lambda functions follow serverless principles. This means that AWS Lambda automatically manages the necessary infrastructure such as operating system, instance type, and storage. When you deploy a Lambda function, it automatically provisions the required resources to run your code and scales them according to the workload. When the code is not running, Lambda automatically scales down, effectively removing the resources and thus reducing costs.

In contrast, with EC2, you are responsible for provisioning and managing the virtual servers, including selecting the OS, instance type, and managing the scaling and termination of instances. After your application has finished running on EC2, you need to manually terminate the instances to stop incurring charges. Lambda abstracts away this manual management, making it a more efficient and cost-effective solution for many use cases.

Scenario:

Let's say a DevOps engineer has created an EC2 instance and a volume, and their organization feels that the application running on the EC2 instance contains important/sensitive information. To safeguard this data, the DevOps engineer takes daily snapshots (backups) of the volume. One day, the organization decides that the usage of the application is no longer needed, so they delete the EC2 instance and volumes. However, they forget to delete the EBS snapshots. Since the EC2 instance and volumes are deleted, the EBS snapshots are no longer useful, and if they are not deleted, AWS will continue to charge for them. To resolve this issue, a Lambda function can be used to automatically identify and delete snapshots that are not associated with any existing EC2 instances or volumes.

Step 1: Create an EC2 Instance and EBS Volume

To get started, we need to create an EC2 instance with ubuntu as an operating system. We will be using a t2.small instance type.

Configuring EC2 Instance:

Choose Ubuntu as the operating system (OS).
Select t2.large as the instance type.
Set up a key pair for SSH access to the EC2 instance. If you don't have a key pair, click "Create new key pair".

Click on the "Launch instance" button to proceed.

The instance is up and running, now check the EBS volume attached to this instance.

Step 2: Create a Snapshot for the EBS Volume

In the EC2 dashboard, head over to the Snapshots section and make a snapshot for the new EBS volume.

Now, check if the snapshot was created.

Step 3: Create a Lambda Function

Next, let's create a Lambda function to find and delete old EBS snapshots. Head over to AWS Lambda: Go to Services -> Lambda -> Create Function.

Now, at the "Create Function" screen, choose "Author from scratch." Fill in the basic information and select Python as the runtime.

select "Create a new role with basic Lambda permissions" and click on create function

Now copy and paste the python code in the code source section

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Get all EBS snapshots
    response = ec2.describe_snapshots(OwnerIds=['self'])

    # Get all active EC2 instance IDs
    instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    active_instance_ids = set()

    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
    for snapshot in response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        if not volume_id:
            # Delete the snapshot if it's not attached to any volume
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
        else:
            # Check if the volume still exists
            try:
                volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                if not volume_response['Volumes'][0]['Attachments']:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
            except ec2.exceptions.ClientError as e:
                if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                    # The volume associated with the snapshot is not found (it might have been deleted)
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

Next, click on "Test" to create an event. Then, in the test event action section, select "Create new event."

Now lets configure the Lambda function, set the runtime timeout to 10 seconds to minimize costs, as the longer a Lambda function runs, the more you will be charged.

Now, if you try to test the Lambda function, you will receive an error stating that you are not authorized to perform this operation. This is because we didn't assign the roles for this Lambda function. A role is essentially a set of permissions that allow a service to communicate with other services.

to resolve the above issue we need to assign a role to the created lambda function.

to do that

Go to Configuration -> Permissions.
Click on the role name to open it in a new tab.
Add the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeSnapshots",
                "ec2:DeleteSnapshot",
                "ec2:DescribeVolumes"
            ],
            "Resource": "*"
        }
    ]
}

The above role grants the following permissions to the Lambda function:

"ec2:DescribeInstances,ec2:DescribeSnapshots,ec2:DeleteSnapshot, ec2:DescribeVolumes"

Now, if you test the Lambda function again, it will run successfully. However, it will not delete any snapshots because there is currently an EC2 instance with its volume running.

Delete the EC2 instance and make sure its volume is also gone.

Run the Lambda function again to check that the snapshot is now marked as stale and gets deleted..

The function has been successfully executed and now let’s see if the Snapshot has been deleted.

You will notice that an EBS volume is deleted because its volume is not found.

Let's verify whether the snapshot has been deleted.

The Snapshot has been successfully deleted.

Conclusion

Congratulations! You have successfully created an AWS Lambda function to optimize storage costs by identifying and deleting stale EBS snapshots. This automated approach not only helps maintain an efficient cloud infrastructure but also reduces unnecessary storage expenses. Great job!

This is just a demonstration of one service that Lambda can handle. The possibilities with AWS Lambda are virtually limitless—you can create Lambda functions for a wide variety of tasks depending on your requirements and the number of services you are using.