Combatting the Clutter: A Fun, Yet Effective Guide to Managing Server Logs on a Windows EC2 Server

As Cloud, IT, or DevOps engineers, it is not uncommon to experience instances(pun not intended) where server logs consume significant storage space on our EC2 servers, potentially slowing down operations.

Server logs: the gift that keeps on giving… storage space.

Server logs play a crucial role in any server infrastructure, providing valuable information on system performance, security, and user activity. However, as these logs accumulate over time, they can take up a significant amount of storage space.

To mitigate this issue, it is essential to regularly clean out and manage these logs, similar to how one would maintain a diary. And don’t worry, it’s not as daunting as it sounds. By utilizing the right tools, such as a PowerShell script and Windows service, it is possible to periodically move server logs from a Windows EC2 server to S3 buckets for storage.

This not only frees up storage space on the server, but also allows for easy searching, filtering, and analyzing of logs for insights, both now and in the future.

To follow this process, it is necessary to have an AWS account, an s3 bucket, a running EC2 server with Windows as the AMI, and a basic understanding of PowerShell scripts, Windows services, and AWS.

Designing an automated log file transfer solution using a Windows service and Amazon S3 storage

Check if AWS CLI is installed

# run command:
aws --help

If it is not already installed, copy the following link to install the AWS CLI on the server CLI or PowerShell CLI:

msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

Confirm it has been properly installed with this command:

aws --version

Check if the server has Choco with this command

choco --help

If you receive an error, install choco with this command on server CLI or PowerShell CLI:

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

Confirm it has been properly installed and run this command

choco --help

Create a PowerShell script and give it a name

<NameOfService>.ps1

In the script write:

while ($true) {
aws s3 mv <SpecifyTheDirectoryTheLogsAre> s3://<SpecifyTheNameOfThes3BucketCreated>/ -exclude "*.log" -recursive -region <SpecifyRegion>
Start-Sleep -Seconds 172800
}

This script moves the server logs file to the s3 bucket every 2 days(172800 seconds). You can change it to suit how frequently you want the server logs to be moved.

Create an IAM role on AWS and give it full access to S3bucket. Give the role a descriptive name.

Follow this roadmap to attach the role to the Instance where you will be running the windows service:

On your instance table >> Click on the instance >> Click on action >> Security >> Modify IAM role >> Attach the role you created.

Install nssm(Non-Sucking Service Manager) that will create the window service on the instance. Run this command.

choco install nssm -y

Set the location to the path where the dot exe (.exe) file is. The command should look like this:

Set-Location C:\ProgramData\chocolatey\lib\NSSM\tools

Run command:

# to list the file in the directory. 
ls

Confirm you can see nssm.exe in the directory.

Run this command to create the windows service with the PowerShell script you created in the nssm.exe directory:

.\nssm install <NameOfService> powershell.exe <DirectoryOfTheScriptCreated>

You should see:

<ServiceName> created succcesfully

Run this command to get the service’s current state:

Get-Service <ServiceName>

Run this command to start the service:

Start-Service <ServiceName>

Congratulations! you have just created a windows service that moves log files to an s3 bucket.

Bonus:

Run this command to stop service:

Stop-Service <ServiceName>

Run this command to uninstall/delete the service:

nssm remove <ServiceName> confirm

In conclusion, server logs play an important role in monitoring and troubleshooting server activity. However, as these logs accumulate over time, they can take up a significant amount of storage space. To prevent this, it is important to regularly review and move logs to s3 storage. By keeping server logs under control, you can ensure that your server’s storage is used efficiently and your server runs smoothly. And remember, server logs can also be a source of entertainment and insight, so don’t be afraid to have some fun while keeping your server in check.