Commit graph

985 commits

Author SHA1 Message Date
Joey Hess
d1bacccff4 importfeed: Also ignore transient problems with downloading content from feeds. 2013-09-03 14:32:26 -04:00
Joey Hess
0831e18372 forget --drop-dead: Completely removes mentions of repositories that have been marked as dead from the git-annex branch.
Wrote nice pure transition calculator, and ugly code to stage its results
into the git-annex branch. Also had to split up several Log modules
that Annex.Branch needed to use, but that themselves used Annex.Branch.

The transition calculator is limited to looking at and changing one file at
a time. While this made the implementation relatively easy, it precludes
transitions that do stuff like deleting old url log files for keys that are
being removed because they are no longer present anywhere.
2013-08-31 17:51:13 -04:00
Joey Hess
62beaa1a86 refactor git-annex branch log filename code into central location
Having one module that knows about all the filenames used on the branch
allows working back from an arbitrary filename to enough information about
it to implement dropping dead remotes and doing other log file compacting
as part of a forget transition.
2013-08-29 19:13:00 -04:00
Joey Hess
6cdac3a003 sync, assistant: Force push of the git-annex branch.
Necessary to ensure it gets pushed to remotes after being rewritten by forget.
See inline rationalles for why I think this is safe!
2013-08-29 14:27:53 -04:00
Joey Hess
4a915cd3cd add forget command
Works, more or less. --dead is not implemented, and so far a new branch
is made, but keys no longer present anywhere are not scrubbed.

git annex sync fails to push the synced/git-annex branch after a forget,
because it's not a fast-forward of the existing synced branch. Could be
fixed by making git-annex sync use assistant-style sync branches.
2013-08-28 16:41:13 -04:00
guilhem
f754779c02 Unused: bugfix
Detect staged files that are not in the working tree.
2013-08-26 13:50:09 -04:00
Joey Hess
88e2618e38 fix reversion in unused
The reversion was that, if a file was git rm'd, but still in branches, it
would not be seen as used. Looking at both the added and the removed (or
changed) files from the diff-index is a cheap way to fix that.
2013-08-26 00:19:19 -04:00
Joey Hess
36f5b10065 whitespace 2013-08-25 21:41:10 -04:00
Joey Hess
0963f92984 unnecessary do block 2013-08-25 21:38:01 -04:00
guilhem
f15fda60ed Speed up the 'unused' command.
Instead of populating the second-level Bloom filter with every key
referenced in every Git reference, consider only those which differ
from what's referenced in the index.

Incidentaly, unlike with its old behavior, staged
modifications/deletion/... will now be detected by 'unused'.

Credits to joeyh for the algorithm. :-)
2013-08-25 21:02:13 -04:00
Joey Hess
824241b6fb better cases 2013-08-22 23:44:13 -04:00
Joey Hess
46b6d75274 Youtube support! (And 53 other video hosts)
When quvi is installed, git-annex addurl automatically uses it to detect
when an page is a video, and downloads the video file.

web special remote: Also support using quvi, for getting files,
or checking if files exist in the web.

This commit was sponsored by Mark Hepburn. Thanks!
2013-08-22 18:50:43 -04:00
Joey Hess
6fd2935a5a unused: Pay attention to symlinks that are not yet staged in the index. 2013-08-22 10:20:03 -04:00
Joey Hess
0f921307e7 mirror: New command, makes two repositories contain the same set of files.
This is a simple approach for setting up a mirroring repository.

It will work with any type of remotes.

Mirror --from is more expensive than mirror --to in general.
OTOH, mirror --from will get the file from any remote that has it, not only
the named mirror remote. And if the named mirror remote is not the fastest
available remote with a file, that can speed things up.

It would be possible to make the assistant or watch command do a more
dynamic mirroring, that didn't need to scan every time.
2013-08-20 15:46:35 -04:00
Joey Hess
b46afa29ac implement import --deduplicate and import --clean-duplicates
Note that --deduplicate currently checksums each file twice,
once to see if it's a known key, and once when importing it.

Perhaps this could be revisited and the extra checksum gotten rid of,
at the cost of not locking down the file when adding it.
2013-08-20 11:00:52 -04:00
Joey Hess
e240cb99f7 Merge branch 'duplicate'
Conflicts:
	debian/changelog
2013-08-20 10:27:24 -04:00
Joey Hess
a6a047192e sync, merge: Bug fix: Don't try to merge into master when in a bare repo. 2013-08-17 21:29:44 +02:00
Joey Hess
d69da2bf22 implement import --duplicate
The other two options are harder, due to needing to get the key for a file
before adding it.
2013-08-11 20:31:54 +02:00
Joey Hess
b28023cb52 importfeed: Fix handling of dots in extensions. 2013-08-03 02:36:38 -04:00
Joey Hess
24c8a6042b importfeed: Ignores transient problems with feeds. Only exits nonzero when a feed has repeatedly had a problems for at least 1 day. 2013-08-03 01:40:21 -04:00
Joey Hess
dc3e0725f9 improve error message 2013-08-02 13:01:25 -04:00
Joey Hess
93f2371e09 get rid of __WINDOWS__, use mingw32_HOST_OS
The latter is harder for me to remember, but avoids build failures in code
used by the configure program.
2013-08-02 12:27:32 -04:00
Joey Hess
03c76b5a30 improve importfeed --force; try to match existing files to avoid unncessary duplication 2013-08-01 11:57:05 -04:00
Joey Hess
42ca8aaa61 importfeed --force: re-download urls that have been seen before 2013-07-31 12:19:00 -04:00
Joey Hess
9476355bc3 find: Avoid polluting stdout with progress messages. Closes: #718186 2013-07-30 20:24:27 -04:00
Joey Hess
ddd46db09a Fix a few bugs involving filenames that are at or near the filesystem's maximum filename length limit.
Started with a problem when running addurl on a really long url,
because the whole url is munged into the filename. Ended up doing
a fairly extensive review for places where filenames could get too large,
although it's hard to say I'm not missed any..

Backend.Url had a 128 character limit, which is fine when the limit is 255,
but not if it's a lot shorter on some systems. So check the pathconf()
limit. Note that this could result in fromUrl creating different keys
for the same url, if run on systems with different limits. I don't see
this is likely to cause any problems. That can already happen when using
addurl --fast, or if the content of an url changes.

Both Command.AddUrl and Backend.Url assumed that urls don't contain a
lot of multi-byte unicode, and would fail to truncate an url that did
properly.

A few places use a filename as the template to make a temp file.
While that's nice in that the temp file name can be easily related back to
the original filename, it could lead to `git annex add` failing to add a
filename that was at or close to the maximum length.

Note that in Command.Add.lockdown, the template is still derived from the
filename, just with enough space left to turn it into a temp file.
This is an important optimisation, because the assistant may lock down
a bunch of files all at once, and using the same template for all of them
would cause openTempFile to iterate through the same set of names,
looking for an unused temp file. I'm not very happy with the relatedTemplate
hack, but it avoids that slowdown.

Backend.WORM does not limit the filename stored in the key.
I have not tried to change that; so git annex add will fail on really long
filenames when using the WORM backend. It seems better to preserve the
invariant that a WORM key always contains the complete filename, since
the filename is the only unique material in the key, other than mtime and
size. Since nobody has complained about add failing (I think I saw it
once?) on WORM, probably it's ok, or nobody but me uses it.

There may be compatability problems if using git annex addurl --fast
or the WORM backend on a system with the 255 limit and then trying to use
that repo in a system with a smaller limit. I have not tried to deal with
those.

This commit was sponsored by Alexander Brem. Thanks!
2013-07-30 19:18:29 -04:00
Joey Hess
07a9910af7 improve comment 2013-07-28 20:15:20 -04:00
Joey Hess
ac08924ec3 fix bug in makeUnique
Returned the possibly non-unique file
2013-07-28 20:14:13 -04:00
Joey Hess
8c55970413 better extension handling
When there's no extension, don't use "none", but "".

When there is an extension, it starts with a dot, so don't put a redundant
dot in the default format.
2013-07-28 19:08:50 -04:00
Joey Hess
8c8488e01a if a feed cannot be downloaded or has no enclosures, fail 2013-07-28 18:16:24 -04:00
Joey Hess
18541bf3fa don't crash on encoding issues in feeds
filesystem encoding to the rescue once more!

IIRC this was the main bug in hpodder.
2013-07-28 17:24:30 -04:00
Joey Hess
66dfeaff44 show a side action when finding known urls 2013-07-28 17:19:21 -04:00
Joey Hess
7e66d260ea importfeed: git-annex becomes a podcatcher in 150 LOC 2013-07-28 16:55:42 -04:00
Joey Hess
c6100aa5cc unused: No longer shows as unused tmp files that are actively being transferred. 2013-07-25 19:51:08 -04:00
Joey Hess
822918089e dropunused behavior change: Now refuses to drop the last copy of a file, unless you use the --force.
This was the last place in git-annex that could remove data referred to by
the git history, without being forced.

Like drop, dropunused checks remotes, and honors the global annex.numcopies
setting. (However, .gitattributes settings cannot apply to unused files.)
2013-07-25 19:50:44 -04:00
Joey Hess
5e3a404d4f Support import in direct mode. 2013-07-22 20:18:00 -04:00
Joey Hess
f353f13c9d Support unannex and uninit in direct mode.
In direct mode, it's best to whenever possible not move direct mode files
out of the way, and so I made unannex avoid touching the direct mode file at
all.

That actually turns out to be easy, because in direct mode, unlike indirect
mode, the pre-commit hook won't get confused if the unannexed file later
gets added back by git add. So there's no need to commit the unannex right
away; it can be staged for the user to commit later. This also means that
unannex in direct mode is a lot faster than in indirect mode!

Another subtle bit is the bookkeeping that is done when unannexing a direct
mode file. The inode cache needs to be removed so that when uninit runs
getKeysPresent, it doesn't see the cache and think the key is still
present and crash when it's not.

This commit is sponsored by Douglas Butts. Thanks!
2013-07-22 17:28:53 -04:00
Joey Hess
3e422cb5fa fix uninit to delete content from annex when it ended up hard linked back to the work tree 2013-07-18 13:30:12 -04:00
Joey Hess
1d7d3ac325 uninit: Preserve .git/annex/objects at the end, if it still has content, so that old versions of files and deleted files are not deleted. Print a message with some suggested actions. 2013-07-16 15:00:25 -04:00
Joey Hess
c936384164 fix: Preserve the original mtime of fixed symlinks. 2013-07-11 11:39:42 -04:00
Joey Hess
207c9f3c4a dropunused, addunused: Complain when asked to operate on a number that does not correspond to any unused key. 2013-07-08 16:47:34 -04:00
Joey Hess
74ad3072e4 addurl --pathdepth: Fix failure when the pathdepth specified is deeper than the urls's path. 2013-07-05 12:46:38 -04:00
Joey Hess
7a7e426352 moved AssociatedFile definition 2013-07-04 02:36:02 -04:00
Joey Hess
980e9a15e0 merge: Now also merges synced/master or similar branches, which makes it useful to put in a post-receive hook to make a repository automatically update its working copy when git annex sync or the assistant sync with it. 2013-07-03 15:42:56 -04:00
Joey Hess
04d07f2c1f --unused: New switch that makes git-annex operate on all data found by the last run of git annex unused. Supported by fsck, get, move, copy. 2013-07-03 15:26:59 -04:00
Joey Hess
b337a8b4c7 --all for get, move, and copy 2013-07-03 13:55:50 -04:00
Joey Hess
def7cb706f Add --all option, and support it for fsck 2013-07-03 13:12:53 -04:00
Joey Hess
a35bdcb3f2 fsck: Ensures that direct mode is used for files when it's enabled.
A common failure mode for direct mode has been for files to end up still
stored in indirect mode. While I hope that doesn't happen anymore, fsck
should deal with it.
2013-06-24 16:26:00 -04:00
Joey Hess
53d52d57c1 check in configure if ionice -c3 works
On old systems, it may need to be run as root.
2013-06-21 13:43:04 -04:00
Joey Hess
d901ba1781 assistant --autostart: Automatically ionices the daemons it starts. 2013-06-21 13:23:20 -04:00
Joey Hess
bf72c2c7fe make dead output consistent with other trust setting commands 2013-06-18 15:41:19 -04:00
Joey Hess
64f8819ae4 fix build 2013-06-17 21:30:52 -04:00
Joey Hess
9ef09587dc fsck: Avoid getting confused by Windows path separators 2013-06-17 21:18:43 -04:00
Joey Hess
98be446d02 remove workaround for old bug that was only in one release
It's causing some problem on windows, see
http://git-annex.branchable.com/bugs/windows_port_-_repo_can__39__t_pull_newly_added_files_/#comment-45df9748bba687d95e3c96b3877ea925
And only affected WORM backend, and for one release well over a year ago,
so could well be bitrotted.
2013-06-17 20:51:36 -04:00
Joey Hess
2844e7175e status: No longer shows dead repositories.
This is because people continually whine about it.  Seemingly not aware
that data generally cannot be deleted from git anyway.
2013-06-17 12:35:33 -04:00
Joey Hess
9666addfaa sync: Better support for bare git remotes. Now pushes directly to the master branch on such a remote, instead of to synced/master. This makes it easier to clone from a bare git remote that has been populated with git annex sync or by the assistant. 2013-06-12 14:54:23 -04:00
Joey Hess
6dcf21db93 Direct mode: No longer temporarily remove write permission bit of files when adding them.
This write permission frobbing is very appropriate in indirect mode,
since annexed objects are stored as immutably as can be managed. But not
in direct mode, where files should be able to be modified at any time.

There are already sufficient guards that there's no need to prevent a file
being written to while it's being ingested, in direct mode. The inode cache
will detect (most) types of modifications, and the add will fail. Then a
re-add should be done. The assistant should get another inotify change
event, and automatically add the new version of the file.
2013-06-12 14:02:31 -04:00
Joey Hess
c46b263fde Android: Make the "Open webapp" menu item open the just created repository when a new repo is made. 2013-06-10 23:55:53 -04:00
Joey Hess
a64106dcef Supports indirect mode on encfs in paranoia mode, and other filesystems that do not support hard links, but do support symlinks and other POSIX filesystem features. 2013-06-10 13:11:33 -04:00
Joey Hess
92f036fcb4 avoid warnings when built with ghc 7.6 2013-06-02 15:01:58 -04:00
Joey Hess
91c9ae83f1 squash warning 2013-06-02 14:06:17 -04:00
Joey Hess
a48d340abd Android: Work around Android devices where the am command doesn't work. 2013-05-31 21:30:21 -04:00
Joey Hess
cba2942cda Revert "android dupped stderr workaround"
This reverts commit 4cc803c733.

The stderr fd is also trashed after `am` fails to open the web browser.
2013-05-30 16:27:10 -04:00
Joey Hess
4cc803c733 android dupped stderr workaround
Avoid using dupped stderr, since
http://git-annex.branchable.com/bugs/warning_-_WebApp_crashed:___60__file_descriptor_15__62__:_hPutStr:_illegal_operation___40__handle_is_closed__41___on_Android/#comment-a24c73803fb10bd35afdc10d50e071c8
seems to involve that handle not being dupped originally, or perhaps
getting closed when the web browser is started on Android.

Using the dupped stdout is known to work before starting the web browser,
so it should work after -- unless perhaps starting it closes both handles.

In any case, there's no real need to write to stderr here.
2013-05-30 13:55:22 -04:00
Joey Hess
3e2d50a336 Android: Added an "Open WebApp" item to the terminal's menu. Should work for Android devices that cannot auto-open the webapp on start. 2013-05-28 18:25:27 -04:00
Joey Hess
f1cce62283 fix merge conflict resolution when both sides have the same key
Still need to git rm the old file so git accepts the merge is resolved.
2013-05-26 18:32:11 -04:00
Joey Hess
2180068e30 correct recent fix
fc37456d0fe1fb0fd3e33338223977b3e7a940bb's fix caused it to try to stage a
symlink in .git/annex/tmp, oops
2013-05-26 18:10:07 -04:00
Joey Hess
919a7d7316 sync: Fix double merge conflict resolution handling.
Ie, when there'a a conflicted merge we may get foo.variant-xxxx
created in a merge. If a second merge conflict occurs on that new file,
it was not falling back to putting in the whole key (which should stop
the merge conflicts happening for good, but is ugly).
2013-05-26 17:42:15 -04:00
Joey Hess
469b3859fc reduce the amount of subdirectories created by the fuzz tester to saner limit 2013-05-26 16:15:25 -04:00
Joey Hess
9978269b55 make fuzztest honor annex.diskreserve 2013-05-26 16:04:52 -04:00
Joey Hess
fc37456d0f fuzz test discovered a way automatic merge resultion can fail; fix
It may be that the directory we need to make a symlink in, to resolve the
merge, doesn't exist locally.
2013-05-25 18:42:43 -04:00
Joey Hess
33fe3dac9e fuzz tester: avoid deleting entire repository (had to happen eventually, right?) 2013-05-25 18:15:34 -04:00
Joey Hess
377bc7bbb7 fix 2013-05-25 17:55:31 -04:00
Joey Hess
ce0ee2aa44 fuzz improvements 2013-05-25 17:52:33 -04:00
Joey Hess
4dc61d74c9 tune frequencies to avoid enormous trees being created 2013-05-25 17:18:37 -04:00
Joey Hess
8b5b4522d1 make fuzz test also output its actions to stdout 2013-05-25 15:52:28 -04:00
Joey Hess
b276857a7a content: New command line way to view and configure a repository's preferred content settings. 2013-05-25 12:44:58 -04:00
Joey Hess
2b14fe2c98 refactor 2013-05-24 23:07:26 -04:00
Joey Hess
a96e982bd3 fuzz tester 2013-05-23 19:00:46 -04:00
Joey Hess
cf7fec4b46 list standard groups in vicfg 2013-05-23 14:54:59 -04:00
Joey Hess
21d5489bd3 typo 2013-05-19 14:46:48 -04:00
Joey Hess
345ee4f37c Switch to MonadCatchIO-transformers for better handling of state while catching exceptions.
As seen in this bug report, the lifted exception handling using the StateT
monad throws away state changes when an action throws an exception.
http://git-annex.branchable.com/bugs/git_annex_fork_bombs_on_gpg_file/
  .. Which can result in cached values being redundantly calculated, or other
     possibly worse bugs when the annex state gets out of sync with reality.

This switches from a StateT AnnexState to a ReaderT (MVar AnnexState).
All changes to the state go via the MVar. So when an Annex action is
running inside an exception handler, and it makes some changes, they
immediately go into affect in the MVar. If it then throws an exception
(or even crashes its thread!), the state changes are still in effect.

The MonadCatchIO-transformers change is actually only incidental.
I could have kept on using lifted-base for the exception handling.
However, I'd have needed to write a new instance of MonadBaseControl
for the new monad.. and I didn't write the old instance.. I begged Bas
and he kindly sent it to me. Happily, MonadCatchIO-transformers is
able to derive a MonadCatchIO instance for my monad.

This is a deep level change. It passes the test suite! What could it break?

Well.. The most likely breakage would be to code that runs an Annex action
in an exception handler, and *wants* state changes to be thrown away.
Perhaps the state changes leaves the state inconsistent, or wrong. Since
there are relatively few places in git-annex that catch exceptions in the
Annex monad, and the AnnexState is generally just used to cache calculated
data, this is unlikely to be a problem.

Oh yeah, this change also makes Assistant.Types.ThreadedMonad a bit
redundant. It's now entirely possible to run concurrent Annex actions in
different threads, all sharing access to the same state! The ThreadedMonad
just adds some extra work on top of that, with its own MVar, and avoids
such actions possibly stepping on one-another's toes. I have not gotten
rid of it, but might try that later. Being able to run concurrent Annex
actions would simplify parts of the Assistant code.
2013-05-19 14:16:36 -04:00
Joey Hess
b8e5b9c645 test suite passes in direct mode
This fixes a bug with git annex add in direct mode. If some files already
existed in the tree pointing at the same key as a file that was just added,
and their content was not present, add neglected to copy the content to
those files.

I also changed the behavior of moveAnnex slightly: When content is moved
into the annex in direct mode, it does not overwrite any content already
present in direct mode files. That content may be modified after all.
2013-05-17 15:59:37 -04:00
Joey Hess
dc66b1f27d Merge branch 'master' into windows
Conflicts:
	Annex/Environment.hs
	Build/Configure.hs
	Git/Construct.hs
	Utility/FileMode.hs
2013-05-14 15:37:24 -04:00
Joey Hess
c553f56634 migrate: Detect if a file gets corrupted while it's being migrated. 2013-05-13 14:27:39 -04:00
Joey Hess
abe8d549df fix permission damage (thanks, Windows) 2013-05-11 23:54:25 -04:00
Joey Hess
1e2ddcb68a use setCurrentDirectory
On POSIX, this just calls changeWorkingDirectory.
2013-05-11 19:14:30 -04:00
Joey Hess
18bdff3fae clean up from windows porting 2013-05-11 18:23:41 -04:00
Joey Hess
3c7e30a295 git-annex now builds on Windows (doesn't work) 2013-05-11 15:03:00 -05:00
Joey Hess
763cbda14f fixup #if 0 stubs to use #ifndef mingw32_HOST_OS
That's needed in files used to build the configure program.
For the other files, I'm keeping my __WINDOWS__ define, as I find that much easier to type.
I may search and replace it to use the mingw32_HOST_OS thing later.
2013-05-10 16:57:21 -05:00
Joey Hess
6c74a42cc6 stub out POSIX stuff 2013-05-10 16:29:59 -05:00
Joey Hess
adde00f4f3 git-annex-shell: Ensure that received files can be read. Files transferred from some Android devices may have very broken permissions as received. 2013-05-06 17:30:57 -04:00
Joey Hess
886897f627 fix build 2013-05-02 20:38:57 -04:00
Joey Hess
29df1a6bec fix build 2013-05-02 19:10:30 -04:00
Joey Hess
6dca92defe need cpp 2013-05-02 19:00:50 -04:00
Joey Hess
1e9b8e1467 cannot open file:// url on Android, so don't use the htmlshim there 2013-05-02 16:46:47 -04:00
Joey Hess
906258ca5e include build flags in version info when not in a repo 2013-05-01 23:11:13 -04:00
Bruno Bigras
0fe8eb0953 Fix the URL not showing until the app closes 2013-05-01 15:48:16 +00:00
Joey Hess
0807211a67 thaw content directory in direct mode too
A content directory can be frozen in direct mode. One way this can happen
is if the content is transferred before direct mode has a mapping for it,
so it's stored in the content directory.

So, we need to thaw the content directory before doing things with it.
2013-04-30 19:33:43 -04:00
Joey Hess
85d83e7756 To enable an existing special remote, the new enableremote command must be used. The initremote command now is used only to create new special remotes. 2013-04-26 18:22:52 -04:00