S3 storage classes expansion

Added support for storageclass=STANDARD_IA to use Amazon's
new Infrequently Accessed storage.

Also allows using storageclass=NEARLINE to use Google's NearLine storage.

The necessary changes to aws to support this are in
https://github.com/aristidb/aws/pull/176

Assistant/WebApp/Configurators/AWS.hs

@@ -41,11 +41,12 @@ glacierConfigurator a = do
   where
 	needglaciercli = $(widgetFile "configurators/needglaciercli")
 
-data StorageClass = StandardRedundancy | ReducedRedundancy
+data StorageClass = StandardRedundancy | StandardInfrequentAccess | ReducedRedundancy
 	deriving (Eq, Enum, Bounded)
 
 instance Show StorageClass where
 	show StandardRedundancy = "STANDARD" 
+	show StandardInfrequentAccess = "STANDARD_IA"
 	show ReducedRedundancy = "REDUCED_REDUNDANCY"
 
 data AWSInput = AWSInput
@@ -75,6 +76,7 @@ s3InputAForm defcreds = AWSInput
 	storageclasses :: [(Text, StorageClass)]
 	storageclasses =
 		[ ("Standard redundancy", StandardRedundancy)
+		, ("Infrequent access (cheaper for backups and archives)", StandardInfrequentAccess)
 		, ("Reduced redundancy (costs less)", ReducedRedundancy)
 		]
 

Remote/S3.hs

@@ -499,8 +499,8 @@ getBucketName = map toLower <$$> M.lookup "bucket"
 
 getStorageClass :: RemoteConfig -> S3.StorageClass
 getStorageClass c = case M.lookup "storageclass" c of
-	Just "REDUCED_REDUNDANCY" -> S3.ReducedRedundancy
+	Just s -> S3.OtherStorageClass (T.pack s)
-	_ -> S3.Standard
+	Nothing -> S3.Standard
 
 getPartSize :: RemoteConfig -> Maybe Integer
 getPartSize c = readSize dataUnits =<< M.lookup "partsize" c

debian/changelog

@@ -2,6 +2,11 @@ git-annex (5.20150917) UNRELEASED; urgency=medium
 
   * info: Don't allow use in a non-git-annex repository, since it
     uses the git-annex branch and would create it if it were missing.
+  * S3: Added support for storageclass=STANDARD_IA to use Amazon's
+    new Infrequently Accessed storage.
+    (Needs aws-0.13.0.)
+  * S3: Use aws-0.13.0, which allows using storageclass=NEARLINE
+    to use Google's NearLine storage.
 
  -- Joey Hess <id@joeyh.name>  Wed, 16 Sep 2015 12:23:33 -0400
 

doc/bugs/Support_non-default_storage_classes_with_Google_Cloud_Storage/comment_1_a10aaa758daee3ca0b064c60c0382ce8._comment

@@ -0,0 +1,18 @@
+[[!comment format=mdwn
+ username="joey"
+ subject="""comment 1"""
+ date="2015-09-17T21:09:24Z"
+ content="""
+It's now possible to use storageclass=NEARLINE, when git-annex is built
+with aws-0.13.0. So, the approach of manually creating the bucket with the
+desired storage class should work now.
+
+I'm unsure if the first method, of letting git-annex create the bucket,
+will work now. Can you test? It may work now too with
+storageclass=NEARLINE. While no storage class is currently specified when
+creating the bucket (that's not in the S3 api at all); but once the bucket
+exists, with whatever storage class is default, git-annex will specify
+NEARLINE when storing objects in it. Seems a good chance this will work,
+and it'd be easier than extending the aws library with google-specific
+features.
+"""]]

doc/special_remotes/S3.mdwn

@@ -37,9 +37,19 @@ the S3 remote.
   "us-west-1", "us-west-2", "ap-southeast-1", "ap-southeast-2", and
   "sa-east-1".
 
-* `storageclass` - Default is "STANDARD". If you have configured git-annex
+* `storageclass` - Default is "STANDARD".  
-  to preserve multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
+  Consult S3 provider documentation for pricing details and available
-  to save money.
+  storage classes.
+
+  When using Amazon S3, if you have configured git-annex to preserve
+  multiple [[copies]], consider setting this to "REDUCED_REDUNDANCY"
+  to save money. Or, if the remote will be used for backup or archival,
+  and so its files are Infrequently Accessed, "STANDARD_IA" is also a
+  good choice to save money.
+
+  Note that changing the storage class of an existing S3 remote will
+  affect new objects sent to the remote, but not objects already
+  stored there.
 
 * `host` and `port` - Specify in order to use a different, S3 compatable
   service.

git-annex.cabal

@@ -178,7 +178,7 @@ Executable git-annex
     Build-Depends: regex-compat
 
   if flag(S3)
-    Build-Depends: conduit, conduit-extra, aws (>= 0.9.2), http-client
+    Build-Depends: conduit, conduit-extra, aws (>= 0.13.0), http-client
     CPP-Options: -DWITH_S3
 
   if flag(WebDAV)