Merge branch 'master' of ssh://git-annex.branchable.com
This commit is contained in:
commit
093110d997
3 changed files with 39 additions and 0 deletions
22
doc/forum/Revisiting_migration_and_multiple_keys.mdwn
Normal file
22
doc/forum/Revisiting_migration_and_multiple_keys.mdwn
Normal file
|
@ -0,0 +1,22 @@
|
||||||
|
I have several workflows that rely on regular key migrations, and I would love to explore some ways that migrating keys could be improved.
|
||||||
|
|
||||||
|
I see there has already been discussion about this:
|
||||||
|
https://git-annex.branchable.com/todo/alternate_keys_for_same_content/
|
||||||
|
|
||||||
|
I don't know how often this comes up, but it comes up a lot for me. I have several data sources that I regularly index and mirror by constructing keys based on md5 and size, and assemble a repo with the known filename. (gdrive, many software distribution sites, and others).
|
||||||
|
|
||||||
|
So, I have a queue-repo, and I have the flexibilty of populating it later. I could even have a queue repo with just URL keys. Then I can handle ingestion and migration later.
|
||||||
|
|
||||||
|
|
||||||
|
I would love to have a simple programatic way of recording that one key is the authoritative key for another key, like for MD5 -> SHA256 migrations.
|
||||||
|
|
||||||
|
There don't seem to be any really great solutions to the prolem of obsolete keys. Merging will often re-introduce them, even if they have been excised. Marking them as dead stil keeps them around, and doesn't preserve information about what key now represents the same object.
|
||||||
|
|
||||||
|
I have written helper scripts, and tools like git-annex-filter-branch are also very helpful. But I like having the flexibility of many repos that may not regularly be in sync with each other, and a consistent history.
|
||||||
|
|
||||||
|
|
||||||
|
This would break things for sure, but what if during a migration, a symlink was made in the git-annex branch from the prev key to the migrated key. The union merge driver could defer to the upgraded or prefered backend. If an out of date repo tries syncing with an already upgraded key, the merge driver can see that the migration for that key has already happened, merge the obsolete key entries, and overwrite it back to a symlink during merge.
|
||||||
|
|
||||||
|
A less drastic approach might be to expand the location log format to indiciate a canonical "successor" key, instead of just being dead.
|
||||||
|
|
||||||
|
It might seem like a lot of complexity, but it would also in my opinion make a more consistent and flexible data model.
|
|
@ -0,0 +1,10 @@
|
||||||
|
[[!comment format=mdwn
|
||||||
|
username="nobodyinperson"
|
||||||
|
avatar="http://cdn.libravatar.org/avatar/736a41cd4988ede057bae805d000f4f5"
|
||||||
|
subject="comment 1"
|
||||||
|
date="2023-11-30T06:51:53Z"
|
||||||
|
content="""
|
||||||
|
Hi! If I understand you correctly, your problem is that you often migrate keys to another backend, and there are situations involving merges of repos far away from each other in history that cause merge conflicts, which results in the dead old pre-migration key being reintroduced?
|
||||||
|
|
||||||
|
I never use key backend migration and I don't fully understand your workflow. Could you provide a reproducible example of your problem (incl all commands)? This would help a lot.
|
||||||
|
"""]]
|
7
doc/users/unqueued.mdwn
Normal file
7
doc/users/unqueued.mdwn
Normal file
|
@ -0,0 +1,7 @@
|
||||||
|
Hello.
|
||||||
|
|
||||||
|
I have been using git-annex since 2017, and it has been a huge boon.
|
||||||
|
|
||||||
|
I mostly consider myself a hobbyist, but I have used git-annex professionally. Along the way, I've learned a lot about unix system programming, git, and automation.
|
||||||
|
|
||||||
|
I personally find managing large sets of files to be very annoying. So being able to wrangle giant sets of files with the precision of git is awesome. I used to spend a lot of time checksumming files, making download queues, and trying to catalog where stuff was.
|
Loading…
Reference in a new issue