Getting Usage Info From CloudFront Logs

Say you have a bunch of sites that are hosted on an EC2 instances, and for various reasons you’ve setup CloudFront to help handle the traffic. You have your normal webserver logs to give you part of the picture, but you probably will want to dig into the CloudFront logs as well in order to get a better picture of your actual throughput.

Thankfully, this is pretty easy. And somewhat annoyingly complicated.
Continue reading

Using S3 as a File System

Ignoring for the fact that there are very good reasons not to do this (latency, eventual consistency, etc), using S3 buckets as filesystems actually fills a niche that AWS doesn’t currently even try. Sure you could just setup an NFS instance on EC2, or even use the storage gateway service to go an off-service data store, but that is overkill for many use cases.

So here’s how I got it done.

First of all, there are some options on how to implement this. Here are the ones I tried out :

http://code.google.com/p/s3ql/

https://code.google.com/p/s3backer/

http://code.google.com/p/s3fs/wiki/FuseOverAmazon

https://github.com/tongwang/s3fs-c

I ended up choosing s3fs-c. I used s3fs at first, but the lack of other-client compatibility was a killer. I’m still not sure why this is even a problem to be solved, given that the s3fs-c fork didn’t make major changes to get that feature. Or maybe there are major changes that I didn’t notice on my, admittedly cursory, review.

So I took an S3 bucket, and EC2 instance running ubuntu 12.04, and did the following.
Continue reading

taking the joyent plunge

i’ve been happy with my AWS instances that i’m running, but decided tonight to start up a joyent instance to see how that goes. so far i’m fairly disappointed with the documentation, but it isn’t any worse than the AWS EC2 stuff.

still, the whole process was pretty easy to go through, even if it wasn’t easy to actually login to my new ubuntu VM.

i’ll be starting up a mail server on this thing to see how that goes.

dancing with perl – part 2

Almost a year after I first wrote about dancer, I finally got around to building my first site with it. Overall, it is a pretty nice little framework. If you are already familiar with perl and/or the template toolkit, then I highly recommend this for quickly deploying projects.

In my case, I was able to build and deploy a management interface for a smartos zone server in just a few hours. I fleshed it out over the course of a week or so, and how it is in production managing dozens of zones quite easily. The interface talks to a mysql database, a node.js server I wrote as part of the same project, and even manages a local haproxy instance. All of this in such a tiny space that I keep getting weirded out.

In the future I’ll document some of the pitfalls and oddities that I experienced. I found that once I came around to the dancer way of thinking, it got pretty easy and I’ve rarely had to address the documentation since.