mirrors/git-annex
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
24610 commits 13 branches 445 tags 110 MiB
Commit graph

1946 commits

Author SHA1 Message Date
Joey Hess
a812d598ef Take space that will be used by running downloads into account when checking annex.diskreserve. 2015-05-12 15:20:22 -04:00
Joey Hess
505d540360 allow building without ascii-progress, since it is not ready yet 
No progress bars with -J unless built with ascii-progress.
 2015-05-12 13:54:16 -04:00
Joey Hess
e27b97d364 Merge branch 'master' into concurrentprogress 
Conflicts:
	Command/Fsck.hs
	Messages.hs
	Remote/Directory.hs
	Remote/Git.hs
	Remote/Helper/Special.hs
	Types/Remote.hs
	debian/changelog
	git-annex.cabal
 2015-05-12 13:23:22 -04:00
Joey Hess
96d24a8161 note about git annex drop behavior change in bare repo 2015-05-12 13:02:22 -04:00
Joey Hess
4b2e3210ea drop: Now supports --all, --unused, and --key. 2015-05-12 13:00:06 -04:00
Joey Hess
db5d831d07 import: Refuse to import files that are within the work tree, as that does not make sense and could cause data loss. 2015-05-11 12:57:47 -04:00
Joey Hess
6ebf615995 cutting a point release, will be a cabal-only update 2015-05-11 12:09:02 -04:00
Joey Hess
26ac0753c1 more FlexibleContexts 2015-05-10 15:54:58 -04:00
Joey Hess
6a1814abf5 seems that webapp ssh password ssh-agent hack is broken on windows after recent ssh changes 2015-05-08 15:07:23 -04:00
Joey Hess
f1a81cfce2 prep release 2015-05-08 13:42:45 -04:00
Joey Hess
f395ca7a41 Improve quvi 0.4 output parsing to handle cases wher there is no known filename extension. This is currently the case when using quvi with youtube. In this case, the extension ".m" will be used. 2015-05-08 13:39:00 -04:00
Joey Hess
54bee7dad4 Revert "Android: Updated bundled ssh from 6.1p1 to 6.4p1." 
This reverts commit 27836a2af9.

Build didn't actually seem to succeed, for both versions of android, so
I'll need to revisit this later.
 2015-05-08 13:05:33 -04:00
Joey Hess
ee94a79690 Debian's ghc now supports TH on arm! Adjust build dependencies to build the webapp on arm, and enable DAV support on arm. \o/ 
Also, adjust some other arch specific build dependencies that are now
available on more architectures in Devian unstable.
 2015-05-07 19:09:44 -04:00
Joey Hess
e09816b0e6 reorder 2015-05-07 18:12:08 -04:00
Joey Hess
4aba1c74bd remaining dataenc to sandi conversions 
I've tested all the dataenc to sandi conversions except Assistant.XMPP,
and all have unchanged behavior, including behavior on large unicode code
points.
 2015-05-07 18:07:13 -04:00
Joey Hess
18fc05ff04 Windows: Roll back to an older version of rsync from cygwin. The newer version has some dependency on a newer ssh from cygwin. 2015-05-07 15:56:40 -04:00
Joey Hess
ce285ea6fa Windows: Remove cygwin ssh, the newer version of which has stopped honoring the setting of HOME. Instead, copy msysgit's ssh into PATH. 2015-05-07 15:53:03 -04:00
Joey Hess
469242ac4d fsck: Ignore error recording the fsck in the activity log, which can happen when running fsck in a read-only repository. Closes: #698559 (fsck can still need to write to the repository if it find problems, but a successful fsck can be done read-only) 2015-05-06 14:45:20 -04:00
Joey Hess
eeb0359a2e contentlocation, examinekey, lookupkey: Added --batch mode option. 2015-05-06 13:44:53 -04:00
Joey Hess
cf786f42a4 Support checking ftp urls for file presence. 2015-05-05 14:05:02 -04:00
Joey Hess
0b18228516 Work around wget bug #784348 which could cause it to clobber git-annex symlinks when downloading from ftp. 2015-05-05 13:53:06 -04:00
Joey Hess
27836a2af9 Android: Updated bundled ssh from 6.1p1 to 6.4p1. 
Tested it builds, but have not run it yet.
 2015-05-01 15:33:29 -04:00
Joey Hess
437d9db41d assistant: Added --autostop to complement --autostart. 2015-05-01 13:53:45 -04:00
Joey Hess
efb37e7c78 Improve behavior when a git-annex command is told to operate on a file that doesn't exist. It will now continue to other files specified after that on the command line, and only error out at the end. 2015-04-30 15:28:17 -04:00
Joey Hess
ac6b492711 import: Before removing a duplicate file in --deduplicate or --clean-duplicates mode, verify that enough copies of its content still exist. 2015-04-30 14:04:36 -04:00
Joey Hess
995643665a Improve integration with KDE's file manager to work with dolphin version 14.12.3 while still being compatable with 4.14.2. Thanks, silvio. 2015-04-29 15:53:16 -04:00
Joey Hess
d8ad1d5503 import: Don't stop entire import when one file fails due to being gitignored or conflicting with something in the work tree. 2015-04-29 13:56:41 -04:00
Joey Hess
2e54251c18 import: Check for gitignored files before moving them into the tree. (Needs git 1.8.4 or newer.) 2015-04-29 13:46:12 -04:00
Joey Hess
986e245ac1 changelog 2015-04-27 18:10:56 -04:00
Joey Hess
ee78958798 S3: Fix incompatability with bucket names used by hS3; the aws library cannot handle upper-case bucket names. git-annex now converts them to lower case automatically. 
For example, it failed to get files from a bucket named S3.

Also fixes `git annex initremote UPPERCASE type=S3`, which failed with the
new aws library, with a signing error message.
 2015-04-27 18:00:58 -04:00
Joey Hess
cfbeb1e7b7 Fix bogus failure of fsck --fast. 2015-04-27 17:40:21 -04:00
Joey Hess
22a4e92df7 S3: git annex enableremote will not create a bucket name, which failed since the bucket already exists. 2015-04-23 14:16:53 -04:00
Joey Hess
b3eccec68c S3: git annex info will show additional information about a S3 remote (endpoint, port, storage class) 2015-04-23 14:12:25 -04:00
Joey Hess
8f010bb55b also drop old dbus compat 2015-04-22 16:23:41 -04:00
Joey Hess
eb8ef44133 Dropped support for older versions of yesod and warp than the ones in Debian Jessie. 
466 lines of compat cruft deleted!
 2015-04-22 16:19:11 -04:00
Joey Hess
3b3aaf0d56 S3: Enable debug logging when annex.debug or --debug is set. 
To debug a bug report, but generally useful.
 2015-04-21 15:55:42 -04:00
Joey Hess
bf0e860617 changelog for https://github.com/joeyh/git-annex/pull/41 2015-04-20 14:53:14 -04:00
Joey Hess
b7be1680a7 prep release 2015-04-20 14:44:15 -04:00
Joey Hess
5948c148fb Make repo init more robust. 
The setDifferences that got added to initialize turns out to make a git
commit, and before ensureCommit has been used. Thus, repo init can fail
when the system has a broken hostname etc.

Move the ensureCommit to the very first thing to avoid this kind of breakage.
 2015-04-20 14:01:41 -04:00
Joey Hess
fdf0d708d2 reorder 2015-04-20 13:35:09 -04:00
Joey Hess
2665be44d5 update some windows libraries for newer version of cygwin 2015-04-20 11:08:32 -04:00
Joey Hess
5f73bc3a12 Removed dependency on haskell SHA library, instead using cryptohash >= 0.11.0. 2015-04-19 11:05:32 -04:00
Joey Hess
ef2202fd94 required: New command, like wanted, but for required content. 
Also refactored some code to reduce duplication.
 2015-04-18 16:04:35 -04:00
Joey Hess
8d685768d3 fsck --from remote: Avoid downloading a key if it would go over the annex.diskreserve limit. 2015-04-18 14:23:42 -04:00
Joey Hess
8489057e8d fsck --from remote: When bad content is found in the remote, and the local repo does not have a copy of the content, preserve the bad content in .git/annex/bad/ to avoid further data loss. 2015-04-18 14:13:07 -04:00
Joey Hess
0def1f0b53 Fix fsck --from a git remote in a local directory, and from a directory special remote. This was a reversion caused by the relative path changes in 5.20150113. 
The directory special remote was not affected in its normal configuration,
since annex-directory is an absolute path normally. But it could fail
when a relative path was used.

The git remote was affected even when an absolute path to it was used in
.git/config, since git-annex now converts all such paths to relative.
 2015-04-18 13:36:12 -04:00
Joey Hess
3a078ab357 When a key's size is unknown, still check the annex.diskreserve, and avoid getting content if the disk is too full. 
We can't check if there's enough disk space to download the content,
but we *can* check if there's certainly not enough!
 2015-04-17 21:29:15 -04:00
Joey Hess
86a2f9dc4d Merge branch 'master' into concurrentprogress 
Conflicts:
	debian/changelog
 2015-04-14 15:35:15 -04:00
Joey Hess
f84ccaa4e8 fix relPathDirToFileAbs on windows with different drive letters 
Since we started using this for git repos, when a remote was on another
drive, it resulted in a bogus relative path to it being used by git-annex,
which didn't work.
 2015-04-14 14:16:44 -04:00
Joey Hess
efd6a8e3aa bittorrent: Fix handling of magnet links. 2015-04-14 12:57:01 -04:00
Joey Hess
4b6dec6a12 Windows: Renamed start menu file to avoid loop in some versions of Windows where the menu file is treated as a git-annex program. 2015-04-13 14:24:07 -04:00
Joey Hess
2be4834822 info: Added --bytes option. 2015-04-12 14:08:40 -04:00
Joey Hess
b14ec45aa8 info dir: Added information about repositories that contain files in the specified directory. 
This is a nearly free feature; it piggybacks on the location log lookups
done for the numcopies stats. So, the only extra overhead is updating
the map of repository sizes.

However, I had to switch to Data.Map.Strict, which needs containers 0.5.
If backporting to wheezy, will probably need to revert this commit.
 2015-04-12 12:54:41 -04:00
Joey Hess
8077ccbd54 get, move, copy, mirror: Concurrent downloads and uploads are now supported! 
This works, and seems fairly robust. Clean get of 20 files at -J3. At -J10,
there are some messages about ssh multiplexing, probably due to a race
spinning up the ssh connection cacher. But, it manages to get all the files
ok regardless.

The progress bars are a scrambled mess though, due to bugs in
ascii-progress, which I've already filed. Particularly this one:
https://github.com/yamadapc/haskell-ascii-progress/issues/8
 2015-04-10 17:08:07 -04:00
Joey Hess
2879adc551 fix union merge to call diff-index with -- after the ref 
Otherwise, if there's a file in the repo with a name matching the ref,
git could get confused and the merge not work.
 2015-04-09 21:13:28 -04:00
Joey Hess
06211738c1 Fix activity log parsing. 
I had some cargo culting in there that used the wrong type, so it failed
to parse old logs, and overwrote them with the new log.
 2015-04-09 21:02:38 -04:00
Joey Hess
ce0a82f493 contentlocationn: New plumbing command. 2015-04-09 15:34:47 -04:00
Joey Hess
b22aacab44 prep release 2015-04-09 15:06:46 -04:00
Joey Hess
2736c59204 fix webapp reversion caused by relative path changes 
webapp: When adding another local repository, and combining it with the
current repository, the new repository's remote path was set to "." rather
than the path to the current repository. This was a reversion caused by the
relative path changes in 5.20150113.
 2015-04-09 14:59:08 -04:00
Joey Hess
0f740fd198 This fixes a bug in the assistant introduced by the literal pathspec changes in version 5.20150406. 
git-checkignore refuses to work if any pathspec options are set. Urgh.

I audited the rest of git, and no other commands used by git-annex have
such limitations. Indeed, AFAICS, *all* other commands support
--literal-pathspecs. So, worked around this where git-checkignore is
called.
 2015-04-09 13:37:06 -04:00
Joey Hess
3695094454 importfeed: Error out when passed a non-url. 2015-04-09 13:09:06 -04:00
Joey Hess
aa31af5594 importfeed: Fix feed download when curl is used. 
This bug used to only afflict file:/// urls, but became more exposed
by b8f0b7309f.
 2015-04-09 13:02:57 -04:00
Joey Hess
84dc5629c6 info: Display repository mode: bare when in a bare (non-direct mode) repo. 2015-04-09 12:28:26 -04:00
Joey Hess
d5a25bd567 proxy: Made it work when run in a new repository before initial commit. 2015-04-09 12:24:42 -04:00
Joey Hess
4da371af1e add: If annex.largefiles is set and does not match a file that's being added, the file will be checked into git rather than being added to the annex. Previously, git annex add skipped over such files; this new behavior is more useful in direct mode. 2015-04-08 16:14:23 -04:00
Joey Hess
279fa44cd0 fromkey, registerurl: When reading from stdin, allow the filename and url, respectively, to contain whitespace. 2015-04-07 00:58:51 -04:00
Joey Hess
42281f12d6 bring back --quiet filtering of stdout and stderr, with deadlock fixed 
I don't quite understand the cause of the deadlock. It only occurred
when git-annex-shell transferinfo was being spawned over ssh to feed
download transfer progress back. And if I removed this line from
feedprogressback, the deadlock didn't occur:
	bytes <- readSV v

The problem was not a leaked FD, as far as I could see. So what was it?
I don't know.

Anyway, this is a nice clean implementation, that avoids the deadlock.
Just fork off the async threads to handle filtering the stdout and stderr,
and let them clean up their handles whenever they decide to exit.

I've verified that the handles do get promptly closed, although a little
later than I would expect. Presumably that "little later" is what
was making waiting on the threads deadlock.

Despite the late exit, the last line of stdout and stderr appears where
I'd want it to, so I guess this is ok..
 2015-04-06 20:20:52 -04:00
Joey Hess
c83940393a emergency release 2015-04-06 17:13:20 -04:00
Joey Hess
0a89d55269 Fixes a bug in the last release that caused rsync and possibly other commands to hang at the end of a file transfer. 
Stderr reader blocks waiting for all stderr, and so blocks the process ever
exiting.

I tried several ways to get around this, but no success yet. For now,
disable the stderr reader entirely.
 2015-04-06 17:12:38 -04:00
Joey Hess
42e46a8701 avoid using --literal-pathspecs with git older than 1.8.1 which added it 
Windows is still building with an older git.
 2015-04-06 13:46:11 -04:00
Joey Hess
dd96043011 prep release 2015-04-06 12:50:22 -04:00
Joey Hess
9445556c97 rethought distributed fsck; instead add activity.log and expire command 
This is much more space efficient!
 2015-04-05 12:50:02 -04:00
Joey Hess
1d57f142f1 Merge branch 'concurrentprogress' 2015-04-04 15:01:00 -04:00
Joey Hess
20fb91a7ad WIP on making --quiet silence progress, and infra for concurrent progress bars 2015-04-03 16:48:30 -04:00
Joey Hess
bd110516c0 init: Improve fifo test to detect NFS systems that support fifos but not well enough for sshcaching. 
ssh tries to hard link a fifo, and if not, complains:

muxserver_listen: link mux listener .git/annex/ssh/SHARD1@iabak.archiveteam.org.QK8zOCbtNebI7q54 => .git/annex/ssh/SHARD1@iabak.archiveteam.org: Operation not permitted
 2015-04-03 14:57:10 -04:00
Joey Hess
7a44dacb74 version: Add --raw 2015-04-02 15:37:39 -04:00
Joey Hess
294991dacb Significantly sped up processing of large numbers of directories passed to a single git-annex command. (try 2) 
New approach is to do it the expensive way for the first 100 paths
on the command line, but then assume the user doesn't care about order too
much and fall back to the cheap way that does not preserve order.
 2015-04-02 01:44:32 -04:00
Joey Hess
f79502d377 Revert "Significantly sped up processing of large numbers of directories passed to a single git-annex command." 
This reverts commit 705112903e.

Whoops, git ls-files does not always output in the input ordering.
That's why all this work is needed. Urk.
 2015-04-02 01:24:37 -04:00
Joey Hess
8aa6b5f2a6 Fix truncation of parameters that could occur when using xargs git-annex. 
This will only ever result in a few more git-ls-files being run than were run
before. (Only 1 more is really needed, but around 10 more are currently run
for a max length command line.)

So, no need to worry about the extra zombie, or lost laziness due to concat.
 2015-04-02 01:20:07 -04:00
Joey Hess
705112903e Significantly sped up processing of large numbers of directories passed to a single git-annex command. 2015-04-02 00:19:49 -04:00
Joey Hess
656fc1c881 fsck: Added --distributed and --expire options, for distributed fsck. 2015-04-01 17:53:16 -04:00
Joey Hess
42bbed7ce5 import: --deduplicate and --cleanduplicates now output the keys corresponding to duplicated files they process. 2015-03-31 15:36:02 -04:00
Joey Hess
6b55974723 addurl --file: When used with a special remote that claims urls and checks their contents, don't override the user's provided filename with filenames that the special remote suggests. Also, don't allow adding the url if the special remote says it contains multiple files. 2015-03-31 15:20:29 -04:00
Joey Hess
f842888de3 test: Fix --list-tests 
Seems list order matters, I don't know why, but have reported it on #tasty
 2015-03-31 14:47:27 -04:00
Joey Hess
0a3f2757a7 Relax debian package dependencies to git >= 1:1.7.7.6 rather than needing >= 1:2.0. 
Git.Merge needs >= 1.7.7.6 if built with such a version; rest of code all
probes git version at runtime.
 2015-03-31 14:22:04 -04:00
Joey Hess
7ebd656eff remove changelog entry 
This is confusing because I updated the affected tarballs using the version
number of the last release.
 2015-03-31 13:30:39 -04:00
Joey Hess
9e25cbde20 importfeed: Avoid downloading a redundant item from a feed whose guid has been downloaded before, even when the url has changed. 
To support this, always store itemid in metadata; before this was only done
when annex.genmetadata was set.
 2015-03-31 13:30:13 -04:00
Joey Hess
f0195b2a43 Fix GETURLS in external special remote protocol to strip downloader prefix from logged url info before checking for the specified prefix. 
This doesn't change what GETURLS returns, but only whether it matches
any prefix that the external special remote asked for.
 2015-03-27 18:49:03 -04:00
Joey Hess
6b89692008 Fix make build target. Thanks, Justin Geibel. 2015-03-27 18:00:32 -04:00
Joey Hess
f35d0bf4b2 Prevent git-ls-files from double-expanding wildcards when an unexpanded wildcard is passed to a git-annex command like add or find. 
Note that previously, `git annex find *.jpg` would find eg, foo/bar.jpg.
That was never intended or documented behavior, so I'm going to change it.
But this is potentially a behavior change if someone discovered that
behavior and relied on it despite it being accidental. Oh well.. can't make
an omlette w/o breaking some eggs.
 2015-03-27 16:45:50 -04:00
Joey Hess
00b07d7155 Better fix for standalone tarball git-annex sync linker shim bug, that works for "git annex sync" as well as "git-annex sync". 2015-03-27 16:06:50 -04:00
Joey Hess
e473d59b92 prep release 2015-03-27 13:11:40 -04:00
Joey Hess
5be536e523 Fix bug introduced in the last release that broke git-annex sync when git-annex was installed from the standalone tarball. 
This was introduced by commit 450ee53ab6

However, the same problem could affect other calls to programPath,
specifically some on the assistant. So, I fixed it at a deeper level.
 2015-03-27 12:55:18 -04:00
Joey Hess
b8f0b7309f Work around curl bug when asked to download an empty url to a file. 
In this situation, curl -o exits successfully without creating the output
file.

There was already a workaround for curl file:/// but I did not realize this
also affected regular url downloads.

To fix it, pre-create the destination file before starting curl.
Since we cannot always know the size of an url before trying to download
it, let's always do this.

Note that since curl is told -C -, we have to consider if this
makes curl try to do a ranged download, which might fail on some servers
where a regular download would have succeeded. My testing indicates
this isn't a problem; since the file is empty, curl seems to not try to
do a ranged download.

Original report: https://github.com/datalad/datalad/issues/79
Curl bug report: https://github.com/bagder/curl/issues/183
 2015-03-27 10:22:36 -04:00
Joey Hess
2b7f3ee3f2 assistant: Committing a whole lot of files at once could overflow command-line length limits and cause the commit to fail. This only happened when using the assistant in an indirect mode repository. 2015-03-26 14:02:35 -04:00
Joey Hess
3af4691978 Improve error message when --in @date is used and there is no reflog for the git-annex branch. 2015-03-26 11:15:15 -04:00
Joey Hess
cd6b62f35e --auto is no longer a global option; only get, drop, and copy accept it. 
Not a behavior change unless you were passing it to a command that ignored it.
 2015-03-25 17:06:14 -04:00
Joey Hess
3cc7c03721 Man pages for individual commands now available, and can be opened using "git annex help <command>" 2015-03-23 17:50:03 -04:00
Joey Hess
c233f98564 migrate: --force will force migration of keys already using the destination backend. Useful in rare cases. 2015-03-23 12:11:16 -04:00
Joey Hess
798da6cf2e Added a post-update-annex hook, which is run after the git-annex branch is updated. Needed for git update-server-info. 
See https://github.com/datalad/datalad/issues/1#issuecomment-84094406
 2015-03-20 14:52:58 -04:00