Using S3 as a File System

Ignoring for the fact that there are very good reasons not to do this (latency, eventual consistency, etc), using S3 buckets as filesystems actually fills a niche that AWS doesn’t currently even try. Sure you could just setup an NFS instance on EC2, or even use the storage gateway service to go an off-service data store, but that is overkill for many use cases.

So here’s how I got it done.

First of all, there are some options on how to implement this. Here are the ones I tried out :
http://code.google.com/p/s3ql/
https://code.google.com/p/s3backer/
http://code.google.com/p/s3fs/wiki/FuseOverAmazon
https://github.com/tongwang/s3fs-c

I ended up choosing s3fs-c. I used s3fs at first, but the lack of other-client compatibility was a killer. I’m still not sure why this is even a problem to be solved, given that the s3fs-c fork didn’t make major changes to get that feature. Or maybe there are major changes that I didn’t notice on my, admittedly cursory, review.

So I took an S3 bucket, and EC2 instance running ubuntu 12.04, and did the following.
Continue reading “Using S3 as a File System”

Using S3 as a File System

taking the joyent plunge

i’ve been happy with my AWS instances that i’m running, but decided tonight to start up a joyent instance to see how that goes. so far i’m fairly disappointed with the documentation, but it isn’t any worse than the AWS EC2 stuff.

still, the whole process was pretty easy to go through, even if it wasn’t easy to actually login to my new ubuntu VM.

i’ll be starting up a mail server on this thing to see how that goes.

taking the joyent plunge

building a new mail server – planning

it has recently been heavily hinted to me that i might be tasked with building a new mail system soon, which got me thinking about how to do it better than i have before. i haven’t built a mail system from scratch in about a year, and there are some things i want to try that i either didn’t want to do last time, or didn’t have time to implement. my planning, design, and implementation process will be put here so that i have a future reference, and also in case it helps others.
Continue reading “building a new mail server – planning”

building a new mail server – planning

ruminations : network monitoring (data collection part one)

network monitoring is, in theory, a relatively simple thing : watch X for Y conditions, and tell Z what you find. most systems break this down into collection, analysis, and alerting. today i’m looking at data collection for monitoring systems.
Continue reading “ruminations : network monitoring (data collection part one)”

ruminations : network monitoring (data collection part one)