For me, such a tool became Boto3 — the official AWS SDK for Python.

https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

Boto3 allows managing all AWS services via API, in other words, it provides programmatic access to many AWS capabilities, including EC2, S3, IAM, RDS, Lambda, EKS, CloudWatch, Organizations, and more.

In my work as a DevOps engineer, the main tasks consist of automating operations, managing the resource lifecycle, integrating services, and creating internal platform tools, which are often difficult to accomplish with Terraform. Terraform describes infrastructure declaratively, but very often there are tasks that do not fit well into a declarative model, for example: deleting all EBS snapshots older than 30 days; finding unused Elastic IPs; generating reports on IAM users without MFA; applying tags in bulk based on complex logic; responding to dynamic events.

In this regard, such tasks are best solved with procedural code, which is convenient using Boto3.
Below is an example of a script that automatically finds and deletes EBS snapshots older than 30 days:

import boto3

from datetime import datetime, timedelta

import logging

# Set up logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Create EC2 client

ec2 = boto3.client('ec2')

# Get list of own snapshots

snapshots = ec2.describe_snapshots(OwnerIds=['self'])

# Define threshold date (30 days ago)

threshold = datetime.utcnow() - timedelta(days=30)

# Counter for deleted snapshots

deleted_count = 0

for snapshot in snapshots['Snapshots']:

snapshot_time = snapshot['StartTime'].replace(tzinfo=None)

snapshot_id = snapshot['SnapshotId']

if snapshot_time < threshold:

try:

ec2.delete_snapshot(SnapshotId=snapshot_id)

logging.info(f"Deleted snapshot {snapshot_id} from {snapshot_time}")

deleted_count += 1

except ec2.exceptions.ClientError as e:

logging.error(f"Failed to delete snapshot {snapshot_id}: {e}")

logging.info(f"Total snapshots deleted: {deleted_count}")

Closing Public S3 Buckets (Public Access Block)

import boto3

import logging

# Set up logging

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

s3 = boto3.client('s3')

# Get a list of all buckets

buckets = s3.list_buckets()['Buckets']

for bucket in buckets:

bucket_name = bucket['Name']

try:

# Get current public access settings

pab = s3.get_public_access_block(Bucket=bucket_name)['PublicAccessBlockConfiguration']

# If any settings are open, close them

if not all(pab.values()):

s3.put_public_access_block(

Bucket=bucket_name,

PublicAccessBlockConfiguration={

'BlockPublicAcls': True,

'IgnorePublicAcls': True,

'BlockPublicPolicy': True,

'RestrictPublicBuckets': True

}

)

logging.info(f"Updated public access settings for bucket: {bucket_name}")

except s3.exceptions.NoSuchPublicAccessBlockConfiguration:

# If the configuration does not exist, create it

s3.put_public_access_block(

Bucket=bucket_name,

PublicAccessBlockConfiguration={

'BlockPublicAcls': True,

'IgnorePublicAcls': True,

'BlockPublicPolicy': True,

'RestrictPublicBuckets': True

}

)

logging.info(f"Created public access block for bucket: {bucket_name}")

except Exception as e:

logging.error(f"Error processing bucket {bucket_name}: {e}")

Based on my experience using Boto3 for effectively solving DevOps tasks, I have outlined in the table below the main directions, methods, and examples that I consider important.

Based on the above, I want to add that, in my opinion, Boto3 is not just an SDK; it is an operational tool for a DevOps engineer that:
allows you to write infrastructure logic, helps build internal platforms, automates security, and also scales together with the organization.

And my advice to you is that when you first start using Boto3, begin with small scripts, and then gradually build internal platform solutions.

Recently, I completed an introductory AWS Boto course and applied these skills and knowledge in practice. It is well known that Boto or AWS SDK for Python — is the official Python library for managing AWS, allowing you to manage AWS from code just like regular Python objects.

Previously, I handled most tasks in AWS through the AWS Management Console, which essentially represents a daily routine scenario:
go to AWS Console → find the desired service → click 20 times → get the result

But this creates a lot of problems, including repeating the same actions, inability to integrate this into pipelines, and increased risk of errors due to manual operations. With Boto, however, it is enough to write just a few lines of code:

s3.upload_file("model.pkl", "ml-models", "v1/model.pkl")

From my practice, I have seen significant benefits from AWS Boto for two directions: ML Engineer and DevOps. But I will talk about DevOps and AWS Boto in the next article.

For an ML Engineer, working with data in S3 is the foundation of everything. Boto helps automate processes for reading and writing data in S3, loading datasets, saving models and checkpoints, versioning artifacts — and all this with simple code, whether locally, on a server, or in a pipeline:

import boto3

s3 = boto3.client("s3")
s3.download_file("datasets", "train.csv", "train.csv")

This eliminates the use of FTP, scp, and manual uploads. Boto also helps ML Engineers automate model storage and deployment: it saves the model in S3, then triggers deployment and updates the version:

s3.upload_file("model.pt", "models", "fraud/v2/model.pt")

For me, practical experience using Boto has shown that everyday AWS tasks can be made reproducible, automated, and manageable.

Although monitoring tools are typically used by DevOps engineers, cloud engineers, and system administrators, they are also extremely valuable for Data Engineers and Data Scientists. In this article, we will explore how popular monitoring systems integrate with AWS and how they help maintain performance and reliability in AWS-based projects.

Amazon CloudWatch serves as the foundation for monitoring within AWS, allowing the collection of metrics, logs, and events from nearly all AWS services.
Official Amazon CloudWatch documentation:
https://docs.aws.amazon.com/cloudwatch/index.html

Prometheus + Grafana is a popular monitoring stack where Prometheus collects metrics using a powerful query model, and Grafana visualizes the data. In AWS, Prometheus can be easily used to gather metrics through the CloudWatch Exporter.

How it works:
The CloudWatch Exporter retrieves metrics from CloudWatch and exposes them in a format that Prometheus can understand. Grafana then connects either to Prometheus or directly to CloudWatch via a built-in data source to display the data visually.

Example configuration of CloudWatch Exporter:

region: us-east-1

metrics:

• aws_namespace: AWS/EC2

  aws_metric_name: CPUUtilization

  aws_dimensions: [InstanceId]

  aws_statistics: [Average]

  period_seconds: 300

  range_seconds: 600

Setting Up Grafana to Connect to CloudWatch

Configuring Grafana to connect to CloudWatch is quite straightforward:

1. In Grafana, go to Configuration > Data Sources.

2. Add a new data source — AWS CloudWatch.

3. Enter your AWS access parameters (Access Key, Secret Key, Region).

Documentation for Prometheus CloudWatch Exporter:
https://github.com/prometheus/cloudwatch_exporter

Documentation for Grafana AWS CloudWatch Data Source:
https://grafana.com/docs/grafana/latest/datasources/cloudwatch/

Another popular monitoring system is Zabbix, which supports integration with AWS CloudWatch via API and automatic discovery of AWS resources using IAM roles. This is convenient for monitoring services such as EC2, RDS, ELB, S3, and EMR.

Setup steps:

1. Create an IAM role with permissions to access CloudWatch.

2. Configure Zabbix to use the AWS API with this role.

3. Set up data items and triggers to track metrics.

Official Zabbix AWS Integration Guide:
https://www.zabbix.com/documentation/current/manual/config/items/aws

Another popular monitoring system, New Relic, supports integration with AWS through AWS Integration and OpenTelemetry for collecting metrics, logs, and traces. This allows monitoring of Lambda, ECS, SageMaker, and other AWS services.

The main benefits for Data Engineers and Data Scientists include the ability to monitor ML inference performance as well as latency and errors in data pipelines.

New Relic AWS Integration Documentation:
https://docs.newrelic.com/docs/integrations/amazon-integrations/aws-integrations-list/aws-integration-installation/

AWS provides native integration with Datadog, enabling monitoring of services such as Lambda, SageMaker, Glue, EMR, and Step Functions — making it ideal for DataOps and MLOps workflows.

Example of Datadog Integration with AWS Lambda:

# Installing the Datadog Lambda Layer

aws lambda update-function-configuration \

--function-name my-function \

--layers arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Python37:32

Datadog AWS Integration Documentation:
https://docs.datadoghq.com/integrations/amazon_web_services/

The SolarWinds Orion monitoring system also supports AWS monitoring through CloudWatch and the AWS API, and is primarily focused on infrastructure control — such as EC2, RDS, and DynamoDB — but is less suitable for ML-related tasks.

SolarWinds AWS Monitoring Documentation:
https://documentation.solarwinds.com/en/success_center/orionplatform/content/core-orion-platform-aws-monitoring.htm

In conclusion, monitoring is the foundation of reliable operation for analytical and ML systems. Using tools such as Prometheus + Grafana, Zabbix, New Relic, SolarWinds Orion and Datadog in combination with AWS services not only enables infrastructure health tracking but also provides detailed metrics on the performance of ETL processes and ML models.