git-annex/doc/todo/lockContent_for_special_remotes.mdwn

Support lockContent in more special remotes. This allows dropping
content from a special remote when the only copies are in other special
remotes.

(All the easy ones, eg read-only special remotes, are implemented already.)

* It might be possible for an external remote to lock content, but it would
  be a tricky extension to the protocol, since lockContent needs to keep it
  locked while running another action. There would need to be separate
  actions for locking and unlocking.

  If this were implemented in git-annex, and some special remote program
  didn't used to support it, and implemented REMOVE w/o checking a lock,
  then making that program support lockContent would run the risk
  of a misture of the old and new version being in use for the same remote,
  which could result in data loss.

  To avoid that, the author of the special remote would need to either
  a) always do lock checking from the beginning or
  b) wait long enough or document well enough to be sure that situation
  never happens.

* directory could use fcntl locking

  This would need a transition, because dropping from directory first needs
  to support checking for locks. Only after we're sure that any git-annex
  binaries that might be in use on a machine do that is it safe for
  lockContent to take the lock. So git-annex would need to be released with
  the first change, then wait several months or years before making the
  second change.

  Also, the directory might be on a filesystem that does not support
  locking, with various failure modes. And unlike a git-annex repo,
  there's nowhere in a directory special remote to record information about
  locking problems with it. Getting this right seems hard..

* S3 has an object lock feature, I don't know if it would be usable for
  this.

  It would need a transition, with dropKey first failing when the object
  lock was in place, and then once that git-annex was in use everywhere,
  lockContent setting the object lock.
  
  (S3 with versioning=yes already supports lockContent.)

* adb could use some shell trick perhaps.. But it would depend on doing
  locking in /sdcard, which seems likely to be a massive ball of POSIX
  incompliance and pain.

----

I was thinking about something related to this recently. Dropping currently
guaranteeds numcopies will be maintained for remotes that support
lockContent. But when some of the copies are on special remotes, this
is not guaranteed. It only makes sure lockContent is keeping one copy
locked, and can verify the existence of the other copies less stringently.

So perhaps it would be good to make this explicit in the configuration,
by adding a mincopies. (Analagous to required content configs.)
Defaulting to 1 as now, but if the user wants to they can set it higher,
perhaps as high as their numcopies, or higher.
all easy cases done bup can't do it after all, because removeKey deletes the git branch. And the rest seem too hard to tackle today. 2020-06-26 18:24:48 +00:00			`Support lockContent in more special remotes. This allows dropping`
			`content from a special remote when the only copies are in other special`
			`remotes.`

			`(All the easy ones, eg read-only special remotes, are implemented already.)`
comment and todo 2020-06-11 18:05:01 +00:00
			`* It might be possible for an external remote to lock content, but it would`
			`be a tricky extension to the protocol, since lockContent needs to keep it`
all easy cases done bup can't do it after all, because removeKey deletes the git branch. And the rest seem too hard to tackle today. 2020-06-26 18:24:48 +00:00			`locked while running another action. There would need to be separate`
			`actions for locking and unlocking.`

			`If this were implemented in git-annex, and some special remote program`
			`didn't used to support it, and implemented REMOVE w/o checking a lock,`
			`then making that program support lockContent would run the risk`
			`of a misture of the old and new version being in use for the same remote,`
			`which could result in data loss.`

			`To avoid that, the author of the special remote would need to either`
			`a) always do lock checking from the beginning or`
			`b) wait long enough or document well enough to be sure that situation`
			`never happens.`
update 2020-06-26 17:54:23 +00:00
			`* directory could use fcntl locking`

			`This would need a transition, because dropping from directory first needs`
			`to support checking for locks. Only after we're sure that any git-annex`
			`binaries that might be in use on a machine do that is it safe for`
			`lockContent to take the lock. So git-annex would need to be released with`
			`the first change, then wait several months or years before making the`
			`second change.`

			`Also, the directory might be on a filesystem that does not support`
			`locking, with various failure modes. And unlike a git-annex repo,`
			`there's nowhere in a directory special remote to record information about`
			`locking problems with it. Getting this right seems hard..`
all easy cases done bup can't do it after all, because removeKey deletes the git branch. And the rest seem too hard to tackle today. 2020-06-26 18:24:48 +00:00
			`* S3 has an object lock feature, I don't know if it would be usable for`
			`this.`

			`It would need a transition, with dropKey first failing when the object`
			`lock was in place, and then once that git-annex was in use everywhere,`
			`lockContent setting the object lock.`

			`(S3 with versioning=yes already supports lockContent.)`

			`* adb could use some shell trick perhaps.. But it would depend on doing`
			`locking in /sdcard, which seems likely to be a massive ball of POSIX`
			`incompliance and pain.`
thought 2020-11-30 17:27:45 +00:00
			`----`

			`I was thinking about something related to this recently. Dropping currently`
			`guaranteeds numcopies will be maintained for remotes that support`
			`lockContent. But when some of the copies are on special remotes, this`
			`is not guaranteed. It only makes sure lockContent is keeping one copy`
			`locked, and can verify the existence of the other copies less stringently.`

			`So perhaps it would be good to make this explicit in the configuration,`
mincopies This is conceptually very simple, just making a 1 that was hard coded be exposed as a config option. The hard part was plumbing all that, and dealing with complexities like reading it from git attributes at the same time that numcopies is read. Behavior change: When numcopies is set to 0, git-annex used to drop content without requiring any copies. Now to get that (highly unsafe) behavior, mincopies also needs to be set to 0. It seemed better to remove that edge case, than complicate mincopies by ignoring it when numcopies is 0. This commit was sponsored by Denis Dzyubenko on Patreon. 2021-01-06 18:11:08 +00:00			`by adding a mincopies. (Analagous to required content configs.)`
thought 2020-11-30 17:27:45 +00:00			`Defaulting to 1 as now, but if the user wants to they can set it higher,`
mincopies This is conceptually very simple, just making a 1 that was hard coded be exposed as a config option. The hard part was plumbing all that, and dealing with complexities like reading it from git attributes at the same time that numcopies is read. Behavior change: When numcopies is set to 0, git-annex used to drop content without requiring any copies. Now to get that (highly unsafe) behavior, mincopies also needs to be set to 0. It seemed better to remove that edge case, than complicate mincopies by ignoring it when numcopies is 0. This commit was sponsored by Denis Dzyubenko on Patreon. 2021-01-06 18:11:08 +00:00			`perhaps as high as their numcopies, or higher.`