Total Pageviews

Wednesday, December 22, 2010

Amazon S3 mounting


http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Installation Notes:
http://code.google.com/p/s3fs/wiki/InstallationNotes

In case link fails for installation notes, please refer bottom of this page.


TO install FUSE 2.X
 wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.8.5/fuse-2.8.5.tar.gz
tar zxvf fuse-2.8.5
./configure
make
make isntall

create password file (eg: .passwd-s3fs) with your amazon keys in /home/ubuntu;

The s3fs password file has this format (use this format if you have only one set of credentials):
accessKeyId:secretAccessKey

If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well:
bucketName:accessKeyId:secretAccessKey
 

The password file shall not have others permissions; so execute the following command:


chmod 600 .passwd-s3fs

to mount s3 bucket as a drive:
s3fs <s3 bucket name> <local Path> -ouse_cache=/tmp -opasswd_file=.passwd-s3fs -ouse_rrs=1 -o allow_other

To mount on booting:

echo 's3fs <bucketname> <loal path> -ouse_cache=/tmp -opasswd_file=.passwd-s3fs -ouse_rrs=1 -o allow_other' > /etc/init.d/s3_byte

chmod 755 /etc/init.d/s3_byte

update-rc.d s3_byte defaults

Command to know S3fs version installed: S3fs version: s3fs --version

To check errors in log:
grep s3fs /var/log/syslog OR
grep s3fs /var/log/messages

Update:

  Make sure updatedb is not indexing the mounted s3 bucket, else the cost of s3 will shootsup exponentially. To de-activate "updatedb", edit /etc/updatedb.conf with
1. PRUNEFS = fuse-s3fs
2. PRUNEPATH = your s3 bucket mount point.

http://manpages.ubuntu.com/manpages/natty/man8/updatedb.8.html



General Instructions

From released tarball

Download: http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
Download SHA1 checksum: 8f6561ce00b41c667b738595fdb7b42196c5eee6
Download size: 154904

  • tar xvzf s3fs-1.61.tar.gz
  • cd s3fs-1.61/
  • ./configure --prefix=/usr
  • make
  • make install (as root)

From subversion repository

Notes for Specific Operating Systems

Debian / Ubuntu

Tested on Ubuntu 10.10
Install prerequisites before compiling:
  • apt-get install build-essential
  • apt-get install libfuse-dev
  • apt-get install fuse-utils
  • apt-get install libcurl4-openssl-dev
  • apt-get install libxml2-dev
  • apt-get install mime-support

Fedora / CentOS

Tested on Fedora 14 Desktop Edition and CentOS 5.5 (Note: on Nov 25, 2010 with s3fs version 1.16, newer versions of s3fs have not been formally tested on these platforms)
Note: See the comment below on how to get FUSE 2.8.4 installed on CentOS 5.5
Install prerequisites before compiling:
  • yum install gcc
  • yum install libstdc++-devel
  • yum install gcc-c++
  • yum install fuse
  • yum install fuse-devel
  • yum install curl-devel
  • yum install libxml2-devel
  • yum install openssl-devel
  • yum install mailcap

1 comment:

Anonymous said...

I used the following command line (change the bucket and mount point names, btw)
s3fs -o allow_other,uid=33,gid=33,umask=227,use_cache=/root/cache MY-BUCKET-NAME /mnt/MY-MOUNT-POINT

Two other important things I had to figure out :

For mounting via fstab, using the "fuse _netdev" option is necessary, or it will fail.
Here's the line I added :
s3fs#MY-BUCKET-NAME /mnt/MY-MOUNT-POINT fuse _netdev,allow_other,umask=227,uid=33,gid=33,use_cache=/root/cache 0 0

Also, if your system includes locate and/or mlocate (e.g. Ubuntu), you may want to add an exception, so that it does NOT scan your bucket.
I modified both my /etc/updatedb.conf and /etc/cron.daily/locate, adding " /mnt/my-bucket-name" to PRUNEPATHS and " fuse.s3fs" to PRUNEFS
Adding fuse.s3fs should be enough, I guess, but I got no time experiment now :)

More details here :
https://github.com/s3fs-fuse/s3fs-fuse/issues/333#issuecomment-291541457


Finally, be careful with use_rrs=1, I guess it stands for "reduced redundancy storage", which not everyone would want !