networked day to day technical issues


A Tool to Backup Files to Amazon S3

For the past year I've been working on and off on a little project to create a tool which:

  • runs on at least Linux, MacOS X and FreeBSD
  • allows to backup your files to Amazon S3 while providing optional server side encryption (AES-256)
  • is cost effective for large numbers of files (the problem with things like s3cmd or aws s3 sync is that they need to compare local files with metadata retrieved on the fly from AWS and this can get expensive)
  • is easy to install
  • provides meaningful error messages and the possibility to debug

I've ended up creating a tool called S3backuptool (yeah, not that original) which does the above and in order to run it requires Python 2.7 , PyCrypto and the Boto library.

Details are available on the project's page and it can be installed from prebuilt packages (deb or rpm) for several Linux distributions or from Python's PyPi for far more Linux distributions and OSes.

So far it's been quite the educative enterprise while also catering to my needs.

Metadata about all backed up files is stored locally in SQLite database(s) and in S3 as metadata for each uploaded file. When a backup job runs it compares the state of files with the one stored in the local SQLite database(s) and action is needed on S3 only then actual S3 api calls are performed (those cost money). In case the local SQLite databases are lost then they can be reconstructed from the S3 stored metadata.


Secure and Scalable WordPress In the Cloud (Amazon S3 for content delivery and EC2 for authoring)

Several months ago I decided to move all of the stuff running on my server (Droplet on Digital Ocean) to various cloud providers. My main motivation was that I did not have time any more to manage my email server which was made up on Postfix + Zarafa + MailScanner + SpamAssassin + ClamAV + Pyzor/Razor/DCC + Apache2 + Mysql . Then I was also dealing with monitoring + backups.
Anyway moving the mail was easy as there are plenty of cloud solutions which are mature.

With my blog (which I did not post to since a long time ago) I decided to try something a bit more interesting so I decided to move it to Amazon S3 as a static website.
In order to achieve this I had to solve the following:

  • convert WordPress from dynamically generated pages to static ones. This was easy using the plugin "WP Static HTML Output" which does what it says
  • find a solution for the comments as with a static page you won't be able to add comments. The solution was to start using Disqus. I've installed the plugin "Disqus Comment System", created a Disqus account and then using the plugin proceeded to import all of the comments which were stored in WordPress' database
  • find a solution for search. Again this was not hard and I've moved to using Google Search (plugin "WP Google Search")
  • once I had the above I generated the a static release which was a .zip file.
  • I've created an S3 bucket called . The bucket must be named as your site/blog and bucket names are unique across all of AWS S3 which means that if someone else is already having a bucket called like that then you're out of luck and your remaining option then is to use CloudFront together with a differently named S3 bucket