Home › OffTopic › Processing CloudFront logs

Processing CloudFront logs

Posted on Posted in OffTopic

If you want a nice graphical view of who’s hitting your CloudFront edge servers, then you can use GoAccess.

On a Mac, it’s as simple as

$ brew install goaccess

If you configure CloudFront logs to drop logs to an S3 bucket with a prefix, you can download all the directories using something like CyberDuck, and then save and run this script to parse them all:-

#!/bin/bash
 
# CloudFront stores it's web logs to a bucket with a prefix.
# 'brew install goaccess' - to install the web log parser prior to use.
# Download all the directories in the CloudFront logs bucket.
# Run this script to process all the logs in the subdirs, creating a HTML report
# for each CloudFront distribution.
 
for d in */; do
    cfdist="${d%?}"
    echo "$cfdist"
    gunzip -c "$cfdist"/*.gz | goaccess -a --time-format %H:%M:%S --date-format %Y-%m-%d --log-format %d\t%t\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^ > $cfdist.html
done

# CloudFront stores it’s web logs to a bucket with a prefix.
# ‘brew install goaccess’ – to install the web log parser prior to use.
# Download all the directories in the CloudFront logs bucket.
# Run this script to process all the logs in the subdirs, creating a HTML report
# for each CloudFront distribution.

for d in */; do
cfdist="${d%?}"
echo "$cfdist"
gunzip -c "$cfdist"/*.gz | goaccess -a –time-format %H:%M:%S –date-format %Y-%m-%d –log-format %d\t%t\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^ > $cfdist.html
done

About

This is my personal blog - all views are my own.

Tagged with: , , ,