mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
36903 commits 13 branches 442 tags 103 MiB
Commit graph

3010 commits

Author SHA1 Message Date
yarikoptic
9b5240ce83 Added a comment: reference original bug report 2019-11-29 17:58:29 +00:00
Joey Hess
d7833def66
use ByteString for git config 
The parser and looking up config keys in the map should both be faster
due to using ByteString.

I had hoped this would speed up startup time, but any improvement to
that was too small to measure. Seems worth keeping though.

Note that the parser breaks up the ByteString, but a config map ends up
pointing to the config as read, which is retained in memory until every
value from it is no longer used. This can change memory usage
patterns marginally, but won't affect git-annex.
 2019-11-27 17:40:09 -04:00
Ilya_Shlyakhter
7d3750682b git-annex-cat 2019-11-27 18:16:35 +00:00
Joey Hess
d830386ab2
update based on profiling 
While L.toStrict copies, profiling showed it was only around 0.3% of
git-annex find runtime. Does not seem worth optimising that, which would
probably involve either a major refactoring, or a use of
UnsafeInterleaveIO.

Also, it seems to me that the latter would need to read chunks, and
preappend the leftover part to the next chunk. But a strict ByteString
append itself is a copy, so I'm not convinced that would be faster than
L.toStrict.
 2019-11-27 14:09:11 -04:00
Ilya_Shlyakhter
9f4b99a0e7 Added a comment: parallelization 2019-11-27 17:23:15 +00:00
Ilya_Shlyakhter
a27ffd3aec Added a comment: representing paths 2019-11-27 15:08:41 +00:00
Joey Hess
a2b566be29
Merge branch 'master' of ssh://git-annex.branchable.com 2019-11-26 16:12:53 -04:00
Joey Hess
3361edfb61
todo for bs branch 2019-11-26 16:11:55 -04:00
Ilya_Shlyakhter
dd58cfd8e1 Added a comment: use named pipes? 2019-11-25 16:45:27 +00:00
yarikoptic
70172712a5 initial idea on joint "get+checksum" 2019-11-25 03:26:23 +00:00
Ilya_Shlyakhter
1f035c0d66 Added a comment: even git mv -f seems to work correctly 2019-11-24 17:25:32 +00:00
Ilya_Shlyakhter
20da59f62f Added a comment: moving unlocked file onto locked file isn't possible 2019-11-24 16:36:24 +00:00
Joey Hess
4cc6985494
todo 2019-11-22 19:47:53 -04:00
Joey Hess
7263aafd2b
Merge branch 'master' into sqlite 2019-11-22 12:49:35 -04:00
Joey Hess
93789cbf40
close as dup 2019-11-22 12:11:04 -04:00
Joey Hess
cf2e23d39c
close not viable 2019-11-22 12:10:59 -04:00
Joey Hess
d4661959de
Merge branch 'master' into sqlite 2019-11-21 17:26:50 -04:00
Joey Hess
64e21f683c
Merge branch 'master' of ssh://git-annex.branchable.com 2019-11-19 14:13:22 -04:00
Joey Hess
c0117044ae
todo 2019-11-19 14:13:01 -04:00
Ilya_Shlyakhter
0822ee52dd Added a comment: import/export 2019-11-19 17:39:47 +00:00
Joey Hess
3b64e35694
comments 2019-11-19 13:19:45 -04:00
Joey Hess
581eb7a3de
comments 2019-11-19 13:07:54 -04:00
Joey Hess
49cf86cfc0
Merge branch 'master' of ssh://git-annex.branchable.com 2019-11-18 16:27:17 -04:00
Joey Hess
b207d944f3
sync, assistant: Pull and push from git-lfs remotes. 
Oversight, forgot to add it to gitSyncableRemote
 2019-11-18 16:13:21 -04:00
Joey Hess
5877de5e80
git-lfs: remember urls, and autoenable remotes using known urls 
* git-lfs: The url provided to initremote/enableremote will now be
  stored in the git-annex branch, allowing enableremote to be used without
  an url. initremote --sameas can be used to add additional urls.
* git-lfs: When there's a git remote with an url that's known to be
  used for git-lfs, automatically enable the special remote.
 2019-11-18 16:09:09 -04:00
Joey Hess
d06b5bcd7b
todo 2019-11-18 13:57:16 -04:00
stefan.huchler@5f110ab85a15aec7e2b38586eb7ec37b12e2e6aa
7fd870d78b 2019-11-16 05:53:02 +00:00
Ilya_Shlyakhter
4b5040818b added todo "change git-annex-import not to delete original files by default" 2019-11-15 18:53:16 +00:00
yarikoptic
58be3af084 Added a comment 2019-11-13 04:41:56 +00:00
Joey Hess
3129923867
comment 2019-11-12 16:37:24 -04:00
Ilya_Shlyakhter
41bb14a885 added todo about documenting sqlite database schemas 2019-11-10 23:27:57 +00:00
Ilya_Shlyakhter
b7a3ba9434 Added a comment: limiting clean/smudge filter to unlocked files 2019-11-08 19:48:11 +00:00
yarikoptic
ea7959353e Added a comment: RFC: how would it work for regular git remote + special remote 2019-11-08 19:19:19 +00:00
Joey Hess
446ce47038
Merge branch 'master' into sqlite 2019-11-07 13:26:45 -04:00
Joey Hess
8f276f33b7
update status of sqlite branch 2019-11-07 13:26:05 -04:00
Joey Hess
3553867b66
v7 to v8 auto-upgrade 
bump version to 8

and update NEWS about it
 2019-11-07 13:24:16 -04:00
Joey Hess
cefdea8073
results of more testing 2019-11-07 13:10:39 -04:00
Joey Hess
2f94b5419a
use new name for new format export dbs 
Delete the old export dbs on upgrade.

Testing this an exporting to a directory with both exporttree=yes and
importtree=yes, it refused to let an interrupted export proceed after
upgrade, with "unsafe to overwrite file". An import resolved the
problem.
 2019-11-06 17:34:15 -04:00
Joey Hess
3b820f08f7
use new name for new format content identifier db 
It will be populated automatically by the next command that needs data
from it, the same way it gets populated in a fresh clone. That may be a
little expensive, but it's a one time cost, and no slower than in a
fresh clone.
 2019-11-06 16:43:52 -04:00
Joey Hess
1b5f4b67b5
use new name for new format fsck db 
The old db is cleaned up when a new incremental fsck is started.

The incremental fsck won't pick up where the old one left off, but I
consider this a minor enough thing that it can just be documented and
won't be a problem.
 2019-11-06 16:27:25 -04:00
Joey Hess
89bdcffdfa
found a way to extract InodeCache from git index 
This will allow a race-free database transition. It is somewhat hairy in
that it depends on an unspecified git output format.
 2019-11-06 14:23:00 -04:00
Joey Hess
6147130e86
Merge branch 'master' into sqlite 2019-11-05 12:59:28 -04:00
Joey Hess
41825195ba
Merge branch 'master' of ssh://git-annex.branchable.com 2019-11-05 12:51:50 -04:00
Joey Hess
fccfcba89f
update to document current state of sqlite branch 2019-11-05 12:50:53 -04:00
Joey Hess
b1cf0869db
link to todo 2019-11-04 11:48:45 -04:00
Joey Hess
96c81fa4f1
comment 2019-11-04 11:47:21 -04:00
Ilya_Shlyakhter
59fa021c3d added todo re: git-annex-sync handling of linked worktrees 2019-11-03 23:06:39 +00:00
Ilya_Shlyakhter
72252e1be4 more notes on communicating breaking changes 2019-10-31 15:52:10 +00:00
Joey Hess
d3e4de0175
fix test suite 
The test suite found a bug; select_ can fail now because a uniqueness
constrain has been added.

Now the test suite passes.

Also, I'm satisfied the changed PersistField instances work.
Looking over what changed, and what I've already tested, Key, FilePath,
and InodeCache are known working; ContentIdentifier is trivial
ByteString to blob; and SSha is trivial String to varchar. Both are
tested by the test suite. I've also tested the new FileSize and
EpochTime instances already, and they work.
 2019-10-30 15:51:37 -04:00
Joey Hess
d7ecac66d2
notes 2019-10-30 15:31:16 -04:00
Joey Hess
4940a135af
eliminate raw sql LIKE query 2019-10-30 15:19:52 -04:00
Joey Hess
09c7cbbaa8
update for things already fixed in this branch 2019-10-30 13:57:22 -04:00
Joey Hess
9085a2cfec
make sure all sqlite selects have indexes 
Bearing in mind that these indexes are really uniqueness constraints
that just happen to also make sqlite generate indexes.

In Database.ContentIndentifier, the ContentIndentifiersKeyRemoteCidIndex
is fine as a uniqueness constraint because it contains all rows from the
table. The ContentIndentifiersCidRemoteIndex is also ok because there
can only be one key for a given (cid, uuid) combination.

In Database.Export, the new ExportTreeFileKeyIndex is the same pair as
the old ExportTreeKeyFileIndex (previously ExportTreeIndex). And
in Database.Keys.SQL, the new InodeCacheKeyIndex is the same pair as the
old KeyInodeCacheIndex.
 2019-10-30 13:46:52 -04:00
yarikoptic
cc62ba1da8 Added a comment: ACLs keep fighting back 2019-10-29 18:03:57 +00:00
Ilya_Shlyakhter
cd482f3cd6 added todo "warn of breaking changes same way git does" 2019-10-24 22:05:46 +00:00
Joey Hess
8fb1350a2c
Merge branch 'master' of ssh://git-annex.branchable.com 2019-10-23 15:30:55 -04:00
Joey Hess
bd197be3ad
annex.gitaddtoannex configuration 
Added annex.gitaddtoannex configuration. Setting it to false prevents
git add from usually adding files to the annex.
(Unless the file was annexed before, or a renamed annexed file is detected.)

Currently left at true; some users are encouraging it be set to false.
 2019-10-23 15:29:46 -04:00
Joey Hess
94efc400e9
horrible impementation of isInodeKnown 
The only good thing about it is it does not require a major version bump
to improve the database. That will need to happen at some point though.

Potentially very very slow in a large repository.

Ugly use of raw sql.
 2019-10-23 14:37:29 -04:00
Ilya_Shlyakhter
2d33c62965 Added a comment: configuring git add behavior 2019-10-23 17:57:03 +00:00
Ilya_Shlyakhter
bdf6d2eb48 Added a comment: preventing git add from annexing new files 2019-10-23 17:05:42 +00:00
Ilya_Shlyakhter
5f7036d0a7 Added a comment: searching by inode 2019-10-23 16:32:37 +00:00
spwhitton
f67285f079 Added a comment 2019-10-23 15:28:56 +00:00
Joey Hess
c4374e4aee
followup 2019-10-22 14:36:31 -04:00
Joey Hess
5581dc3318
comment 2019-10-22 13:30:04 -04:00
Joey Hess
913ac80d33
close a bug report redundant with an existing todo 
copied over the dandi tag

updated the todo, as now that sameas is done it's unblocked
 2019-10-21 15:23:04 -04:00
cds@be5df0987fc4e414401b86f980a60f7bf0288037
70702c63a8 Making a todo as requested on IRC. 2019-10-16 02:34:53 +00:00
Joey Hess
8ef5e79f79
sameas done and merged 2019-10-14 16:07:49 -04:00
Joey Hess
123d0d9add
Merge branch 'sameas' 2019-10-14 16:07:19 -04:00
Joey Hess
37f0abbca8
looked over all my old comments, current status of all concerns 2019-10-11 16:06:42 -04:00
Joey Hess
37f725a9f7
Merge branch 'master' into sameas 2019-10-11 15:56:00 -04:00
Joey Hess
debafcba2b
autoenable sameas remotes 2019-10-11 15:52:40 -04:00
Joey Hess
ec778888d2
got enableremote working for sameas 
Also the assistant can enable sameas remotes, should work, but not
tested.
 2019-10-11 15:11:08 -04:00
Joey Hess
35d7ffe128
initremote --sameas fully working 
And using sameas remotes is working.

Moved annex-config-uuid setting out of Remote.Helper.Special.
EnableRemote will also have to set it.
 2019-10-11 14:19:10 -04:00
Joey Hess
1b9c1d1737
fix sameas inherited key removal 2019-10-11 13:18:27 -04:00
Ilya_Shlyakhter
8432c3912f Added a comment: preventing inadvertently adding annexed files in unlocked form 2019-10-11 16:38:07 +00:00
Joey Hess
2bd6e81bb0
support annex-config-uuid when generating remote 
This is used by a special remote with sameas-uuid=
The remote's uuid is the sameas-uuid, but it needs to get
its RemoteConfig from the annex-config-uuid.
 2019-10-11 12:34:11 -04:00
Ilya_Shlyakhter
599d438fa6 Added a comment 2019-10-11 16:23:08 +00:00
Joey Hess
cb08cb0052
Merge branch 'master' of ssh://git-annex.branchable.com 2019-10-10 16:18:46 -04:00
Joey Hess
0dd5691951
update notes 2019-10-10 16:12:59 -04:00
Ilya_Shlyakhter
159ba92ea1 added todo for separate annex.largefiles.git-add and annex.largefiles.git-annex-add settings 2019-10-10 18:29:03 +00:00
Joey Hess
d1130ea04a
get rid of hardcoded "name" lookups 
Support "sameas-name" being set instead.

In RenameRemote, rename which ever of the two is set.
 2019-10-10 13:25:10 -04:00
Joey Hess
d9938b97c6
thought 2019-10-10 12:06:37 -04:00
jason.dixon.email@aa0e536a2ec2877d6f666108dbbc6e39bbe87ac0
7e692864de Added a comment: wanted content commands 2019-10-10 08:16:59 +00:00
Ilya_Shlyakhter
df42fc998c Added a comment 2019-10-08 20:23:07 +00:00
Joey Hess
30204cc493
comment 2019-10-08 14:36:21 -04:00
Joey Hess
7c91eb35f7
comment 2019-10-08 12:19:05 -04:00
Joey Hess
1ba99c31ca
comment 2019-10-08 12:11:19 -04:00
Joey Hess
560c873f50
Merge branch 'master' of ssh://git-annex.branchable.com 2019-10-08 12:09:26 -04:00
Ilya_Shlyakhter
2e8abdae5a Added a comment: automatic retries if index is locked 2019-10-08 16:01:52 +00:00
Joey Hess
08bf984921
more thoughts 2019-10-08 11:39:12 -04:00
Ilya_Shlyakhter
d2369894f0 Added a comment: automaticallly sync content on git push/pull 2019-10-08 06:43:33 +00:00
Ilya_Shlyakhter
a728b78ca2 added todo for operating on files touched by commits in a range 2019-10-08 05:13:23 +00:00
Ilya_Shlyakhter
3d677c0990 added suggestion for more fine-grained config of when files are added as locked vs unlocked 2019-10-07 18:08:44 +00:00
Joey Hess
350bb456c0
this looks messy 2019-10-07 13:10:04 -04:00
Joey Hess
ae86cfb5ef
couple clarifications 2019-10-07 12:20:49 -04:00
Ilya_Shlyakhter
79705f831b fix typo 2019-10-05 02:47:10 +00:00
Ilya_Shlyakhter
d4dafa0671 Added a comment: special remote that restores contents by running a command 2019-10-04 18:16:39 +00:00
Joey Hess
e4e4777a9b
Merge branch 'master' of ssh://git-annex.branchable.com 2019-10-03 09:54:24 -04:00
tobiasBora
d42b117f71 Added a comment 2019-10-03 00:48:23 +00:00
Ilya_Shlyakhter
7dce92bfca Added a comment 2019-10-03 00:36:54 +00:00
tobiasBora
1a2a0707a6 2019-10-02 21:21:01 +00:00
tobiasBora
71405c1c85 2019-10-02 21:20:36 +00:00
Joey Hess
4967f8f0ab
update 2019-10-02 11:24:30 -04:00
Ilya_Shlyakhter
f35b130f59 Added a comment: different repos with same uuid 2019-10-02 15:13:56 +00:00
Joey Hess
d7f4bb5462
Merge branch 'master' of ssh://git-annex.branchable.com 2019-10-01 16:05:15 -04:00
Joey Hess
3229cde14f
comments 2019-10-01 16:04:44 -04:00
Joey Hess
7245a86587
my, this is looking complicated 2019-10-01 13:11:49 -04:00
Joey Hess
61b384d2b7
add --sameas option, not yet used 2019-10-01 12:36:25 -04:00
branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e
115886df9f Added a comment: There are still benefits to commit throttling 2019-09-30 22:16:10 +00:00
Joey Hess
7b5ce2b330
expand 2019-09-30 17:35:38 -04:00
Joey Hess
ccf9f03c84
Merge branch 'master' of ssh://git-annex.branchable.com 2019-09-30 17:33:08 -04:00
Joey Hess
7302f12bf6
todo 2019-09-30 17:32:41 -04:00
yarikoptic
47245d7f4b Added a comment 2019-09-30 20:36:00 +00:00
Joey Hess
4200d2933b
comment 2019-09-30 15:31:55 -04:00
Joey Hess
2b55a2b882
remotedaemon: Don't list --stop in help since it's not supported. 
Also, move out of plumbing section. When using tor, the remotedaemon is
part of the user's workflow, as it runs the tor hidden service.
 2019-09-30 14:40:46 -04:00
Joey Hess
5fbf30301a
response 2019-09-30 14:27:23 -04:00
Joey Hess
a9f078da43
Merge branch 'master' of ssh://git-annex.branchable.com 2019-09-30 14:21:23 -04:00
Joey Hess
e52fb641a0
comment 2019-09-30 14:19:35 -04:00
branchable@bafd175a4b99afd6ed72501042e364ebd3e0c45e
2e1d2bc6d5 Added a comment: .gitattributes could solve this 2019-09-29 22:59:48 +00:00
yarikoptic
e51697fe06 initial idea on separating our hint about remotes 2019-09-27 20:37:40 +00:00
Joey Hess
090898a138
adjust --lock: This enters an adjusted branch where files are locked. 
Straightforward, except for the issue of how to reverse LockAdjustment.

With --unlock, a commit that modifies/adds unlocked files gets reverse
adjusted to use locked files. That's fairly reasonable, I think.

But reversing --lock by unlocking all modified files feels wrong. Maybe
that's just because repositories typically seem to still have mostly
locked files in them (unless one is in an adjusted unlocked branch of
course!)

It may be that eventually how to reverse both will need to be configurable,
I don't know.
 2019-09-27 14:23:25 -04:00
Joey Hess
3d5c1a4912
slightly improve design 2019-09-27 12:46:14 -04:00
Joey Hess
e5cb4f9dee
thoughts 2019-09-27 12:17:16 -04:00
Ilya_Shlyakhter
f9aae586a8 Added a comment 2019-09-25 17:34:33 +00:00
Joey Hess
789721b65f
Merge branch 'master' of ssh://git-annex.branchable.com 2019-09-24 14:46:43 -04:00
Joey Hess
6ae0a44c64
git-lfs: Added support for http basic auth 2019-09-24 14:46:20 -04:00
kyle
b2a2bc9597 Added a comment: git docs on "%f" 2019-09-24 18:10:16 +00:00
Ilya_Shlyakhter
dd99dfa301 added suggestion to only enable clean/smudge filter for unlocked annexed files 2019-09-24 17:33:06 +00:00
Joey Hess
9b5f014238
todo 2019-09-24 11:42:20 -04:00
Joey Hess
5e787b936b
Merge branch 'master' of ssh://git-annex.branchable.com 2019-09-21 21:27:06 -04:00
Yaroslav Halchenko
6eb8784e7c
Embed projects/{datalad,repronim} tags for the past todo/bugs 
Assignment between datalad and repronim might not be 100% right but
approx there depending on the date

Here is the command and its output:

	$> git grep -l '\[\[!meta author=\(yoh\|ben\|yarikoptic\|kyle\)\]\]' | grep -v comment | grep -v assure_correct_names___40 | while read f; do d=$(git log --date=iso $f| awk '/^Date/{print $2;}' | tail -n 1 ); p=datalad; if [[ "$d" =~ 2019-.* ]] || [[ "$d" =~ 2018-1.* ]]; then p=repronim; fi; echo $d $p: $f; sed -i -e "s,\(\[\[\!meta author.*\),\1\n\[\[\!tag projects/$p\]\],g" $f; done
	2017-10-10 datalad: doc/bugs/--shared_setting_of_git_causes_annex__39__ed_files_to_be_writeable__33__.mdwn
	2017-08-12 datalad: doc/bugs/2_ssh_connection_prompts_for_password.mdwn
	2018-10-22 repronim: doc/bugs/3_tests_fail_while_trying_to_build_fresh_git_annex_6.20181011+git109-gff9ba1f4d.mdwn
	2018-12-20 repronim: doc/bugs/7.20181211+git29-gab4a1bed9_fails_tests_during_neurodebian_build.mdwn
	2016-03-12 datalad: doc/bugs/OSX__58___addurl_--batch_--json_spits_out_shortened_output_string__dies_off_with_4.mdwn
	2018-11-16 repronim: doc/bugs/Please_include_version_of_used_to_build_ghc_within___34__git_annex_version__34__.mdwn
	2016-03-10 datalad: doc/bugs/R__47__O_permissions_are_stalking_me_somehow.mdwn
	2019-07-03 repronim: doc/bugs/Regression_in___96__find_--json__96___output.mdwn
	2017-01-27 datalad: doc/bugs/Too_difficult_if_not_impossible_to_explicitly_add__47__keep_file_under_git___40__not_annex__41___in_v6_without_employing_.gitattributes.mdwn
	2018-12-11 repronim: doc/bugs/Unable_to_addurl_file__58____47____47____47___on_Windows.mdwn
	2019-04-10 repronim: doc/bugs/Unable_to_get__47__addurl_to_http_link__58___download_failed__58___InvalidHeader___34__preload__34__.mdwn
	2019-09-19 repronim: doc/bugs/add_--json_should_provide_a___34__note__34___with_an_error_message_in_case_of_an_error.mdwn
	2016-11-20 datalad: doc/bugs/adds_file_destined_for_annex_into_git_in___39__addurl__39__.mdwn
	2016-01-12 datalad: doc/bugs/addurl_--batch__--with-files_doesn__39__t_add_file_into_git_until_pipe_is_closed.mdwn
	2016-01-15 datalad: doc/bugs/addurl_--batch___40__--json_or_not__41___doesn__39__t_report_failure_correctly_if_non-annexed_file_exists.mdwn
	2016-05-23 datalad: doc/bugs/addurl_--batch_decides_to_talk_to_ssh_remotes_for_some_reason.mdwn
	2016-01-11 datalad: doc/bugs/addurl_--file__causes_file_redownload_even_if_it_already_present.mdwn
	2019-03-18 repronim: doc/bugs/addurl_fails_with_non-youtube_url_with___34__This_url_is_supported_by_youtube-dl__44___but_youtube-dl_could_potentially_access_any_address__34__.mdwn
	2016-12-04 datalad: doc/bugs/annex_add_ignores_.-prefixed_directories.mdwn
	2017-03-22 datalad: doc/bugs/annex_doesn__39__t_fixup_symlinks_when___34__git_commit_path__95__to__95__repo__34___is_used.mdwn
	2016-07-12 datalad: doc/bugs/annex_drop_fails_to_determine_availability_on_a_http_url_redirecting_to_ftp.mdwn
	2016-04-26 datalad: doc/bugs/annex_drop_is_not___34__in_effect__34___for_load_which_was___34__addurl_--batch__34__ed_but_not_yet_committed.mdwn
	2018-09-25 datalad: doc/bugs/annex_get_-J_16_via_ssh_stalls_.mdwn
	2019-06-05 repronim: doc/bugs/annex_init_no_longer_generates_default_description.mdwn
	2018-03-27 datalad: doc/bugs/annex_metadata___40__not_--batch__39__ed__41___is_not_aware_of_files_added_via_addurls_--batch.mdwn
	2018-11-12 repronim: doc/bugs/annex_sets_remote___40__regular_http__41___to_be_ignored_while_running_in_docker_container.mdwn
	2018-02-23 datalad: doc/bugs/annex_sync___40__in_direct_mode_only__41___does_not_actually_update_local_branch__47__tree.mdwn
	2018-05-04 datalad: doc/bugs/annex_view_barfs__fatal__58___Unable_to_add___40__null__41___to_database.mdwn
	2016-09-09 datalad: doc/bugs/autoenable__61__true_seems_to_not_work_any_longer.mdwn
	2015-11-20 datalad: doc/bugs/autostart__47__git-annex.desktop_gets_resurrected.mdwn
	2016-03-29 datalad: doc/bugs/awkward_error_from_annex_whenever_operating_on_a_clone_with_submodules.mdwn
	2016-11-08 datalad: doc/bugs/cannot___34__install__34___standalone_git_annex_within_afs_mount.mdwn
	2017-03-17 datalad: doc/bugs/cannot___40__or_how__63____41___to_pass_socket_path_with_a_space_in_its_path_via_annex-ssh-options.mdwn
	2018-08-08 datalad: doc/bugs/cannot_commit___34__annex_add__34__ed_modified_file_which_switched_its_largefile_status_to_be_committed_to_git_now.mdwn
	2017-01-20 datalad: doc/bugs/clash_of_-j__in_copy_for_--json_--json-progress.mdwn
	2017-03-01 datalad: doc/bugs/copy_does_not_reflect_some_failed_copies_in_--json_output.mdwn
	2017-02-10 datalad: doc/bugs/could_standalone_copy_of_git_ship__47__deploy_hooks_samples_as_well_as_stock_git__63__.mdwn
	2015-11-12 datalad: doc/bugs/does_not_complain__47__fail_if_by_mistake_option_value_passed_within_arg_for__-c.mdwn
	2016-07-14 datalad: doc/bugs/drop_blows_on_lustre__58___SQLite3_returned_ErrorIO.mdwn
	2016-04-29 datalad: doc/bugs/dropkey_--batch_--json_--force_is_always_succesfull.mdwn
	2016-02-24 datalad: doc/bugs/duplicate_progress_reports_in_parallel___39__get__39__.mdwn
	2016-04-27 datalad: doc/bugs/encfs_support_--_shouldn__39__t_it_be_treated_as_crippled_already__63__.mdwn
	2019-08-03 repronim: doc/bugs/fails_to___96__get__96___in_parallel_for_a_freshly_clone_from_public_s3_bucket_where_versioning_info_was_forgotten.mdwn
	2019-09-02 repronim: doc/bugs/fails_to_init_under_a_directory_with_a___34__tricky__34___name.mdwn
	2017-07-26 datalad: doc/bugs/fails_to_verify_presence_via_http_while_wget_fetches_it_just_fine.mdwn
	2019-08-13 repronim: doc/bugs/fresh_build_for_neurodebian__58___test_failure.mdwn
	2017-09-13 datalad: doc/bugs/fsck_does_not_detect__47__fix_some_key_directories_correctly.mdwn
	2017-07-30 datalad: doc/bugs/get_--json_fails_whenever_plain_get_works___40__with_https_urls__41__.mdwn
	2018-03-09 datalad: doc/bugs/get_-JX__58____transfer_already_in_progress_..._for_some_files.mdwn
	2017-08-30 datalad: doc/bugs/get_-J___34__fails__34___to_get_files_with_the_same_key.mdwn
	2017-03-15 datalad: doc/bugs/get_-J_cannot_be_used_with_password-based_authentication.mdwn
	2018-02-10 datalad: doc/bugs/get_-J_from_ssh_remote_tries_to_lock_in_home_directory__63__.mdwn
	2019-07-11 repronim: doc/bugs/get_fails_to_place_v7_unlocked_file_content_into_the_file_tree_in_v7_in_repo_with_detached_HEAD.mdwn
	2018-05-06 datalad: doc/bugs/get_from_the_web_remote_fails___40__redirect__63____41___-_lack_of_further_debug_info.mdwn
	2018-09-22 datalad: doc/bugs/get_over_ssh_fails_with___fd__58__19__58___hClose__58___resource_vanished.mdwn
	2016-03-12 datalad: doc/bugs/git-annex__58___content_is_locked__while_trying_to_move_under_NFS_and_pidlock.mdwn
	2019-03-18 repronim: doc/bugs/git-annex_precommit_.__gets_stuck_after_likely_interrupted___96__annex_add__96___call.mdwn
	2016-06-02 datalad: doc/bugs/git_annex_adjust_--unlock_seems_to_cause_migration_of_a_file_to_another_backend.mdwn
	2017-02-06 datalad: doc/bugs/git_annex_info_fails_on_NFS__58___waitToSetLock__58___resource_exhausted___40__No_locks_available__41__.mdwn
	2017-10-25 datalad: doc/bugs/graft__47__graft_cleanup_commits_--_really_needed__63__.mdwn
	2018-03-05 datalad: doc/bugs/howto_guarantee_a_single_instance_of_a_special_remote__63__.mdwn
	2018-09-25 datalad: doc/bugs/impossible_to_login_to_the_website_at_times.mdwn
	2018-02-20 datalad: doc/bugs/impossible_to_perform___34__read-only__34___git_annex_info_without_write_permissions.mdwn
	2016-01-11 datalad: doc/bugs/inconsistent_output_upon_addurl_--batch_complicates_if_not_forbids_reliable_parsing_of_output.mdwn
	2015-12-21 datalad: doc/bugs/info_--json_lists_backend_usage_stats_as_a_list_of_lists.mdwn
	2018-10-11 repronim: doc/bugs/initial_get_-J_on_exported_S3_bucket_fails.mdwn
	2016-01-28 datalad: doc/bugs/lookupkey_started_to_spit_out___34__debug__34___messages_to_stdout.mdwn
	2016-02-09 datalad: doc/bugs/man_page_for_command_misses_actual_command_in_the_synopsis_for_git-annex-checkpresentkey.mdwn
	2018-11-08 repronim: doc/bugs/multiple_ssh_prompts__44___and_thread_blocked_indefinitely_in_an___63____63____63___transaction.mdwn
	2017-02-17 datalad: doc/bugs/mysterious_dependency_of_git_annex_status_output_of_the_added_file.mdwn
	2016-01-15 datalad: doc/bugs/new_whereis_--json_lost_information_about_web_urls_if_other_special_remotes_provide_them.mdwn
	2017-05-25 datalad: doc/bugs/parallel_get_can_fail_some_downloads_and_require_re-getting_.mdwn
	2016-05-01 datalad: doc/bugs/prematurely___40__can__39__t_check_offline__41___marks_remote_as_annex-ignore.mdwn
	2018-05-08 datalad: doc/bugs/regression_-_fails_to_drop_._Exit_code_11_wo_--debug__44___and_1_with_--debug.mdwn
	2017-12-06 datalad: doc/bugs/regression_-_yt__58___prefix_for___34__regular__34___urls.mdwn
	2019-08-23 repronim: doc/bugs/regression__58___fails_to_detect_need_for_pidlock_on_an_NSF_mount.mdwn
	2019-05-09 repronim: doc/bugs/regression__58___http_downloads_redirecting_to_ftp_are_no_longer_supported.mdwn
	2016-10-17 datalad: doc/bugs/remote_repository_must_be_version_6_as_well_to_have_copied_files_appear_correctly.mdwn
	2016-08-05 datalad: doc/bugs/reports_success_when_addurl_--batch__a_file_which_is_.gitignore__39__d.mdwn
	2017-09-26 datalad: doc/bugs/set_metadata_leaks_from_one___40__staged__41___key_to_another_during_rename_of_file.mdwn
	2016-02-15 datalad: doc/bugs/shouldn__39__t_keep_permissions_of_the_ssh_remote__63__.mdwn
	2016-03-09 datalad: doc/bugs/some_tests_fail_while_running_under_NFS.mdwn
	2018-09-18 datalad: doc/bugs/some_transfers_for_get_-J4_fail_over_ssh__47__rsync_and_require_retry.mdwn
	2018-09-25 datalad: doc/bugs/standalone_build_is_slightly_out_of_date.mdwn
	2015-10-28 datalad: doc/bugs/standalone_builds_shouldn__39__t_pollute___126____47__.ssh_with_helpers_merely_upon_annex_init.mdwn
	2016-05-25 datalad: doc/bugs/strips___95___from_extensions_in_E_backends__63__.mdwn
	2016-03-01 datalad: doc/bugs/thread_blocked_indefinitely_in_an_STM_transaction__while_moving_within__a_local_clone.mdwn
	2016-01-15 datalad: doc/bugs/treatment_of_largefiles_is_not_working_for_addurl_--fast___40__or_--relaxed__41__.mdwn
	2018-09-11 datalad: doc/bugs/unable_to_get_from_public_S3_remote_without_clear_reasoning_why.mdwn
	2018-09-25 datalad: doc/bugs/unclear_why_annex_fails_to_get_a_file_from_a_public_S3_bucket_where_it_exported_before.mdwn
	2016-02-08 datalad: doc/bugs/use_of_annex_in_submodule_replaces_.git_with_incorrect_symlink.mdwn
	2016-02-26 datalad: doc/bugs/using_regular_magic_file__warning_pollutes_stderr.mdwn
	2018-09-06 datalad: doc/bugs/v6_-_under_subdir__58___git_add___34__whines__34____44___git_commit___34__blows__34__.mdwn
	2018-09-12 datalad: doc/bugs/whereis_shouldn__39__t_just_crash_if_one_of_the_special_remotes___34__drivers__34___is_not_available.mdwn
	2019-01-21 repronim: doc/bugs/windows__58___autostart_for_annex_64bit_looks_under_x86.mdwn
	2016-01-18 datalad: doc/todo/--batch_for_add.mdwn
	2016-01-18 datalad: doc/todo/--batch_for_find.mdwn
	2016-01-05 datalad: doc/todo/--batch_for_info.mdwn
	2016-10-24 datalad: doc/todo/Allow_for_TRANSFER-SUCCESS_to_report_also_a_URL_where_key_could_now_be_obtained_from.mdwn
	2018-02-05 datalad: doc/todo/INFO_message_for_custom_special_remotes.mdwn
	2018-09-19 datalad: doc/todo/Natively_support_s3__58____47____47___urls___40__for_addurl__44___get__44___etc__41__.mdwn
	2016-07-28 datalad: doc/todo/__39__info_filename__39___to_provide_information_either_content_is_locally_present.mdwn
	2019-04-25 repronim: doc/todo/add_--json-progress_to___96__git_annex_add__96__.mdwn
	2016-04-22 datalad: doc/todo/add_option_to_whereis_to_avoid_network_interactions.mdwn
	2017-03-23 datalad: doc/todo/annex_add___40__-u__124__--update__41___mode.mdwn
	2016-08-05 datalad: doc/todo/annex_merge_--remotes.mdwn
	2017-02-06 datalad: doc/todo/be_able_to_specify_custom_commit_message_for_git-annex_branch_commit.mdwn
	2016-02-10 datalad: doc/todo/checkpresentkey_without_explicit_remote.mdwn
	2018-09-28 datalad: doc/todo/config_setting_to_force_--progress_reporting_for_some_git_calls.mdwn
	2018-09-27 datalad: doc/todo/configuration_option_for_default___34__mode__34___on_crippled_file_systems.mdwn
	2016-06-09 datalad: doc/todo/could_standalone___39__fixed__39___git-annex_binaries_be_prelinked__63__.mdwn
	2016-04-25 datalad: doc/todo/drop_--batch.mdwn
	2016-06-16 datalad: doc/todo/get_--batch.mdwn
	2017-07-21 datalad: doc/todo/git_annex_info___60__remote__62___does_not_list_all_the_parameters_for_the_remote.mdwn
	2016-06-09 datalad: doc/todo/git_annex_info_to_include_information_about_repo_version__63__.mdwn
	2019-04-05 repronim: doc/todo/globus_special_remote_as_a___34__transport__34___layer.mdwn
	2018-02-06 datalad: doc/todo/include_msg_with_possible_reason_why_command___40__e.g._add__41___failed_into_--json_output.mdwn
	2016-07-05 datalad: doc/todo/interface_to_the___34__progress__34___of_annex_operations.mdwn
	2018-05-22 datalad: doc/todo/machine_readable_information_about_reason_of_failure_in_enableremote.mdwn
	2015-11-02 datalad: doc/todo/make_addurl_respect_annex.largefiles_option.mdwn
	2016-02-15 datalad: doc/todo/make_annex_info_more_efficient.mdwn
	2016-05-19 datalad: doc/todo/make_copy_--fast__faster.mdwn
	2016-02-12 datalad: doc/todo/metadata_--batch.mdwn
	2017-02-16 datalad: doc/todo/output_of_wanted___40__and_possibly_group_etc__41___should_not_be_polluted_with___34__informational__34___messages.mdwn
	2014-12-18 datalad: doc/todo/parallel_get.mdwn
	2018-09-28 datalad: doc/todo/prevent_directly_printing_to_stderr_error_from_an_attempt_to_download_remote___47__config_file.mdwn
	2018-08-04 datalad: doc/todo/provide___39__file__39___in_--json-progress_record_for_addurl.mdwn
	2019-01-25 repronim: doc/todo/publicurl_config_for_all_special_remotes.mdwn
	2016-01-19 datalad: doc/todo/return___34__key__34___entry_in_--json_output_for_addurl___40__and_future_add__41___--batch.mdwn
	2019-01-25 repronim: doc/todo/some_way_to_get_a_list_of_options_for_a_special_remote_of_a_given_type.mdwn
	2018-04-28 datalad: doc/todo/support___126____47__.netrc_for_http_access_authentication_specs_as_was_done_when_wget_was_used.mdwn
	2018-01-12 datalad: doc/todo/support_ssh__58____47____47___or_sftp__58____47____47___urls_via___34__built-in__34___ssh_support.mdwn
	2016-08-05 datalad: doc/todo/sync_--branches__to_sync_only_specified_branches___40__e.g._git-annex__41__.mdwn
	2016-10-28 datalad: doc/todo/unlock_--json.mdwn
	2016-10-17 datalad: doc/todo/unlock_--read-only.mdwn
 2019-09-21 21:26:15 -04:00
yarikoptic
191dcb7c8e change from using !meta to !tag for project specification 2019-09-22 00:07:46 +00:00
Ilya_Shlyakhter
2c774b910c Added a comment 2019-09-20 20:46:41 +00:00
Ilya_Shlyakhter
a9d5c08b6a Added a comment 2019-09-20 20:41:25 +00:00
yarikoptic
fccbe038f3 Added a comment: found it 2019-09-20 18:56:52 +00:00
Joey Hess
a3f5158021
responses 2019-09-20 13:01:22 -04:00
Ilya_Shlyakhter
3fadb8e20f Added a comment 2019-09-20 15:58:08 +00:00
Ilya_Shlyakhter
0558c6de79 added suggestion to prevent pushed branches from referencing unavailable files 2019-09-19 21:03:30 +00:00
Ilya_Shlyakhter
f64f6f82d3 removed 2019-09-19 20:17:10 +00:00
Ilya_Shlyakhter
dc4d34355b added todo for optionally auto-locking unlocked files after one edit 2019-09-19 19:57:04 +00:00
Ilya_Shlyakhter
96e4c10cf9 Added a comment 2019-09-19 18:10:36 +00:00
Ilya_Shlyakhter
4f2e58b5cd added suggestion to auto-configure git diff driver 2019-09-19 17:43:41 +00:00
Joey Hess
b13a350556
added --unlocked and --locked 2019-09-19 12:33:13 -04:00
Ilya_Shlyakhter
d1ec2a1faa added todo for finding unlocked files 2019-09-19 01:47:10 +00:00
Ilya_Shlyakhter
31a218a6a3 Added a comment: git-annex REST API 2019-09-18 22:55:06 +00:00
Ilya_Shlyakhter
1fc9cae9fe Added a comment 2019-09-18 21:30:46 +00:00
Ilya_Shlyakhter
177b3f9dce added suggestion to split off clean and smudge filter into separate lightweight executables 2019-09-18 20:59:15 +00:00
Joey Hess
7722125e97
Merge branch 'master' of ssh://git-annex.branchable.com 2019-09-18 13:49:00 -04:00
Joey Hess
31ead4f0f3
comment 2019-09-18 13:17:39 -04:00
yarikoptic
fd3423deba initial TODO on special remotes parameter checking 2019-09-18 17:09:11 +00:00
Joey Hess
10b63050eb
add 2019-09-17 11:56:56 -04:00
Ilya_Shlyakhter
24a256d402 Added a comment 2019-09-16 23:44:48 +00:00
Ilya_Shlyakhter
d98e6a211a Added a comment 2019-09-16 21:12:21 +00:00
Joey Hess
15e35a80f5
comment 2019-09-16 13:26:16 -04:00
Joey Hess
81b77dd73d
comment and todo 2019-09-16 13:17:15 -04:00
forums+git-annex@d2ec818a5e37440c45fa5f3cbd82fe5110b32020
3c6ebf69ae Added a comment 2019-09-13 23:17:37 +00:00
Joey Hess
91caba25be
comment 2019-09-13 18:21:21 -04:00
forums+git-annex@d2ec818a5e37440c45fa5f3cbd82fe5110b32020
378974d502 2019-09-13 20:37:03 +00:00
Joey Hess
eae5e96473
Merge branch 'v7-default' 2019-09-12 13:47:55 -04:00
Joey Hess
5ba16c06ed
re-investigated sqlite encoding issues 
It's really persistent causing the problem, and BLOB really seems the
only way around it.
 2019-09-12 13:06:39 -04:00
Joey Hess
23ca1bcd99
close 2019-09-12 11:52:22 -04:00
Joey Hess
9317c4a5b3
close all old direct mode issues 
except for one issue that was not really specific to direct mode after
all
 2019-09-11 13:44:17 -04:00
Ilya_Shlyakhter
664897e3bf reinit repo with new uuid 2019-09-10 21:14:55 +00:00
Joey Hess
70f02df599
link to datalad issue that has some data collected 2019-09-10 14:50:58 -04:00
Ilya_Shlyakhter
1a7c06ea9d added suggestion for S3 export remote redirecting to key-value store 2019-09-02 17:41:46 +00:00
Joey Hess
31a6086a9a
v7-default branch 2019-08-30 14:14:42 -04:00
Joey Hess
6b27fc58d0
v7 bug review and cleanup 2019-08-28 12:20:30 -04:00
Joey Hess
a648c22b9b
clarification 2019-08-27 16:18:29 -04:00
Joey Hess
c21c0cd885
update with WSL findings 2019-08-27 15:45:43 -04:00
Joey Hess
56370c009c
Merge branch 'master' of ssh://git-annex.branchable.com 2019-08-27 14:14:52 -04:00
Joey Hess
45663f6f35
planning 2019-08-27 14:14:16 -04:00
Joey Hess
6b56c3ae11
close 2019-08-27 11:39:52 -04:00
Joey Hess
0b10d10753
add 2019-08-26 17:00:06 -04:00
Ilya_Shlyakhter
637ced91b9 Added a comment 2019-08-26 20:58:14 +00:00
Joey Hess
20741b1eb4
Automatically convert direct mode repositories to v7 with adjusted unlocked branches 
* Automatically convert direct mode repositories to v7 with adjusted
  unlocked branches and set annex.thin.
* init: When run on a crippled filesystem with --version=5,
  will error out, since version 7 is needed for adjusted unlocked branch.
* direct: This command always errors out as direct mode is no longer
  supported.
* indirect: This command has become a deprecated noop.
* proxy: This command is deprecated because it was only needed in direct
  mode. (But it continues to work.)

Also removed mentions of direct mode throughough the documentation.

I have not removed all the direct mode code yet.
 2019-08-26 15:05:25 -04:00
Joey Hess
2fd27c6df5
assistant: When creating a new repository use v7 adjusted branches with annex.thin 
Rather than direct mode, which this is a small step on the path to
removing.

Init on a crippled filesystem already used v7 adjusted branches,
and like that, this doesn't pose any interoperability issues with old
versions of git-annex that clone the same repo, because files are only
unlocked on the adjusted branch.
 2019-08-26 12:54:14 -04:00
Joey Hess
0c718763fd
planning 
several entangled things
 2019-08-26 12:29:43 -04:00
Joey Hess
9bf3a5c405
close as probably fixed in recent test suite improvements 2019-08-26 12:27:27 -04:00
Joey Hess
b627c5b4d0
followup 2019-08-26 12:20:57 -04:00
Joey Hess
cb318d4404
todo 2019-08-23 16:52:08 -04:00
Joey Hess
0dd6ba302e
close 2019-08-05 13:44:24 -04:00
Joey Hess
74a36b8262
comment 2019-08-01 12:36:06 -04:00
Joey Hess
3d301dfb9f
comment 2019-08-01 12:26:28 -04:00
Joey Hess
018b5b8173
Support building with socks-0.6 and persistant-template-2.7 
persistent-template now needs UndecidableInstances.

socks changed defaultSocksConf to take a SockAddr.
 2019-07-30 12:50:48 -04:00
Joey Hess
c634dae66b
wacky ideas 2019-07-30 12:13:00 -04:00
Joey Hess
ee3885d152
git-lfs branch 2019-07-29 15:48:52 -04:00
Joey Hess
724ebec3f0
lfs deep dive 2019-07-29 12:28:07 -04:00
Ilya_Shlyakhter
5759065e18 added suggestion to let git-annex-export export non-annexed files 2019-07-26 00:36:18 +00:00
Dan
8fcf9ff7c6 Added a comment: git annex sync commits changes, too? 2019-07-22 17:25:39 +00:00
Dan
adc2ac1a89 Added a comment: Still Wanted 2019-07-22 17:06:21 +00:00
Joey Hess
875c7b5cc9
windows long filenames should be fixed now by new ghc 2019-07-22 09:44:09 -04:00
Joey Hess
7ea98e00d0
todo 2019-07-19 14:49:26 -04:00
Joey Hess
05e3b93b89
comment 2019-07-19 13:22:09 -04:00
Joey Hess
cf6152575a
comment, open related todo 2019-07-19 13:10:17 -04:00
Ilya_Shlyakhter
8ba9aae481 added a request for more extensive retrying of transiently-failing operations 2019-07-08 16:30:37 +00:00
Joey Hess
8f26422943
Merge branch 'master' of ssh://git-annex.branchable.com 2019-07-05 09:49:57 -04:00
Ilya_Shlyakhter
1c25bfc88c Added a comment: status of v7 2019-07-02 18:45:52 +00:00
Ilya_Shlyakhter
47f16c8915 Added a comment 2019-07-02 18:36:16 +00:00
yarikoptic
067050c5c7 Added a comment: I wonder if "thin mode" could generalize beyond hardlinks 2019-07-01 16:19:47 +00:00
Ilya_Shlyakhter
607bb3c208 Added a comment: verifying and external backends 2019-06-28 17:14:18 +00:00
Ilya_Shlyakhter
3388722110 Added a comment: dockerized external backends 2019-06-28 16:36:10 +00:00
Ilya_Shlyakhter
24378d94e6 Added a comment: read-only unlock of only some files 2019-06-28 15:57:05 +00:00
Joey Hess
dfe30c8a6c
todo 2019-06-28 10:59:20 -04:00
Joey Hess
c05c73657c
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-27 19:08:46 -04:00
Joey Hess
a9f7e013a0
todo 2019-06-27 19:08:27 -04:00
Joey Hess
fb72f4db64
comment 2019-06-27 10:44:35 -04:00
Ilya_Shlyakhter
b168a06b82 Added a comment 2019-06-26 20:38:19 +00:00
Ilya_Shlyakhter
8ae3197940 Added a comment: some todos that external backends would obviate (fix formatting, add one) 2019-06-26 20:25:51 +00:00
Ilya_Shlyakhter
13794595d6 Added a comment 2019-06-26 20:15:19 +00:00
Joey Hess
732f03b202
comment 2019-06-26 11:58:53 -04:00
Joey Hess
3bb8b62699
comments 2019-06-26 11:23:41 -04:00
Joey Hess
42c386fc47
add: Display progress meter when hashing files. 
* add: Display progress meter when hashing files.
* add: Support --json-progress option.
 2019-06-25 13:12:47 -04:00
Joey Hess
191bdaafc5
comment 2019-06-25 11:08:45 -04:00
Joey Hess
9d36c826c0
use fine-grained WorkerStages when transferring and verifying 
This means that Command.Move and Command.Get don't need to
manually set the stage, and is a lot cleaner conceptually.

Also, this makes Command.Sync.syncFile use the worker pool better.
In the scenario where it first downloads content and then uploads it to
some other remotes, it will start in TransferStage, then enter VerifyStage
and then go back to TransferStage for each transfer to the remotes.
Before, it entered CleanupStage after the download, and stayed in it for
the upload, so too many transfer jobs could run at the same time.

Note that, in Remote.Git, it uses runTransfer and also verifyKeyContent
inside onLocal. That has a Annex state for the remote, with no worker pool.
So the resulting calls to enteringStage won't block in there.

While Remote.Git.copyToRemote does do checksum verification, I
realized that should not use a verification slot in the WorkerPool
to do it. Because, it's reading back from eg, a removable disk to checksum.
That will contend with other writes to that disk. It's best to treat
that checksum verification as just part of the transer. So, removed the todo
item about that, as there's nothing needing to be done.
 2019-06-19 13:24:20 -04:00
Joey Hess
e19408ed9d
Merge branch 'master' of ssh://git-annex.branchable.com 2019-06-17 15:26:57 -04:00
Joey Hess
04cc470201
run download checksum verification in separate job pool 
get, move, copy, sync: When -J or annex.jobs has enabled concurrency,
checksum verification uses a separate job pool than is used for
downloads, to keep bandwidth saturated.

Not yet done for upload checksum verification, but that only affects
remotes on local disks.
 2019-06-17 14:58:02 -04:00
Joey Hess
1a8d06d251
thought 2019-06-17 11:50:18 -04:00
jsag@f84637fe752e0235291a118b1cd007bafad0997e
ae9f2d5e6a 2019-06-17 12:43:17 +00:00
Joey Hess
e589a9b3fc
moving this to a bug 2019-06-12 15:00:14 -04:00
Joey Hess
8e5ea28c26
finish CommandStart transition 
The hoped for optimisation of CommandStart with -J did not materialize.
In fact, not runnign CommandStart in parallel is slower than -J3.
So, CommandStart are still run in parallel.

(The actual bad performance I've been seeing with -J in my big repo
has to do with building the remoteList.)

But, this is still progress toward making -J faster, because it gets rid
of the onlyActionOn roadblock in the way of making CommandCleanup jobs
run separate from CommandPerform jobs.

Added OnlyActionOn constructor for ActionItem which fixes the
onlyActionOn breakage in the last commit.

Made CustomOutput include an ActionItem, so even things using it can
specify OnlyActionOn.

In Command.Move and Command.Sync, there were CommandStarts that used
includeCommandAction, so output messages, which is no longer allowed.
Fixed by using startingCustomOutput, but that's still not quite right,
since it prevents message display for the includeCommandAction run
inside it too.
 2019-06-12 13:24:01 -04:00
Joey Hess
3893d84764
todo 2019-06-06 12:02:27 -04:00
Joey Hess
3eac4e01a4
idea 2019-06-05 19:43:01 -04:00
Joey Hess
659640e224
separate queue for cleanup actions 
When running multiple concurrent actions, the cleanup phase is run in a
separate queue than the main action queue. This can make some commands
faster, because less time is spent on bookkeeping in between each file
transfer.

But as far as I can see, nothing will be sped up much by this yet, because
all the existing cleanup actions are very light-weight. This is just groundwork
for deferring checksum verification to cleanup time.

This change does mean that if the user expects -J2 will mean that they see no
more than 2 jobs running at a time, they may be surprised to see 4 in some
cases (if the cleanup actions are slow enough to notice).

It might also make sense to enable background cleanup without the -J,
for at least one cleanup action. Indeed, that's the behavior that -J1
has now. At some point in the future, it make make sense to make the
behavior with no -J the same as -J1. The only reason it's not currently
is that git-annex can build w/o concurrent-output, and also any bugs
in concurrent-output (such as perhaps misbehaving on non-VT100 compatible
terminals) are avoided by default by only using it when -J is used.
 2019-06-05 17:54:35 -04:00
Joey Hess
7dcc815c29
more thoughts 2019-06-04 14:38:55 -04:00
Joey Hess
cd20dc4158
thoughts 2019-06-04 14:13:15 -04:00
Joey Hess
cd5e8be2dc
comment 2019-05-23 13:36:59 -04:00
Joey Hess
e06feb7316
honor preferred content when importing 
Importing from a special remote honors its preferred content too; unwanted
files are not imported. But, some preferred content expressions can't be
checked before files are imported, and trying to import with such an
expression will fail.

Tested this with scenarios including changing the preferred content
expression and making sure merging the import didn't delete files that were
no longer wanted.

There was one minor inefficiency mentioned in the todo that I punted on.
 2019-05-21 14:38:06 -04:00
Joey Hess
ec11575d17
hairyness 2019-05-21 12:54:57 -04:00
Joey Hess
3b9a19171a
Merge branch 'master' into preferred 2019-05-21 11:34:45 -04:00
Joey Hess
5e1221ad53
Improve shape of commit tree when importing from unversioned special remotes 
Make the import have the previous import as a parent, so eg `git log --stat`
displays a useful diff.

Also a minor optimisation, only calculate the depth of the imported history
once.
 2019-05-21 11:32:54 -04:00
Joey Hess
5af9e7f3d0
break out a todo 2019-05-21 11:10:13 -04:00
Joey Hess
97fd9da6e7
add back non-preferred files to imported tree 
Prevents merging the import from deleting the non-preferred files from
the branch it's merged into.

adjustTree previously appended the new list of items to the old, which
could result in it generating a tree with multiple files with the same
name. That is not good and confuses some parts of git. Gave it a
function to resolve such conflicts.

That allowed dealing with the problem of what happens when the import
contains some files (or subtrees) with the same name as files that were
filtered out of the export. The files from the import win.
 2019-05-20 16:43:52 -04:00
Joey Hess
7d177b78e4
docs for export preferred content 
This includes a note about how include= and exclude= match when exporting
a subtree. I don't know if the note is prominent enough, but the
behavior seems unsurprising enough.
 2019-05-20 12:06:02 -04:00
Joey Hess
12451ea010
Merge branch 'master' into preferred 2019-05-20 10:00:03 -04:00
Joey Hess
8958556fe3
thought 2019-05-16 20:41:17 -04:00
Joey Hess
24c8b1b15a
update 2019-05-14 15:25:09 -04:00
Joey Hess
9411a7c93c
matching preferred content before key is known 
This will let import try to match preferred content expressions before
downloading the content and generating its key.

If an expression needs a key, it preferredContentParser with
preferredContentKeylessTokens will fail to parse it.

standard and groupwanted are not in preferredContentKeylessTokens
because they may refer to an expression that refers to a key.
That needs further work to support them.
 2019-05-14 14:28:23 -04:00
Joey Hess
a3e24ed533
more design work 2019-05-14 11:49:23 -04:00
Joey Hess
c5a61ee808
closing in on final design for this 2019-05-14 10:52:00 -04:00
Ilya_Shlyakhter
0610789285 Added a comment: checksums of remote data 2019-05-13 22:03:38 +00:00
Joey Hess
0c7569bb6f
close 2019-05-10 13:59:39 -04:00
Joey Hess
c77d79d343
close old todo 2019-05-10 13:54:32 -04:00
Joey Hess
ae562ad4d7
update old todo item with what still needs doing 
removed old comments that are no longer relevant
 2019-05-10 13:52:40 -04:00
Joey Hess
daa0c6c1c6
close old todo 2019-05-10 13:35:55 -04:00
Joey Hess
d32143e7ad
close 2019-05-10 13:34:44 -04:00
Joey Hess
ccfb800fa6
Merge branch 'master' of ssh://git-annex.branchable.com 2019-05-10 13:31:49 -04:00
Joey Hess
82186ca58f
annex.jobs=cpus etc 
Added the ability to run one job per CPU (core), by setting annex.jobs=cpus,
or using option --jobs=cpus or -Jcpus.

Built with future expansion in mind, including not defaulting matching on
Concurrency so more constructors can later be added, and using "cpu"
instead of "0".
 2019-05-10 13:27:08 -04:00
Ilya_Shlyakhter
e0c73c7f29 Added a comment 2019-05-09 21:07:39 +00:00
Ilya_Shlyakhter
5638ae9688 Added a comment 2019-05-07 00:59:31 +00:00
Joey Hess
b03e65d260
Improved locking when multiple git-annex processes are writing to the .git/index file 2019-05-06 15:15:12 -04:00