Commit graph

1413 commits

Author SHA1 Message Date
Joey Hess
134fdefb8c fsck: When run with --all or --unused, while .gitattributes annex.numcopies cannot be honored since it's operating on keys instead of files, make it honor the global numcopies setting, and the annex.numcopies git config setting. 2014-02-20 14:45:17 -04:00
Joey Hess
db48b8a4a3 unused: Fix to actually detect unused keys when in direct mode. 2014-02-20 13:53:49 -04:00
Joey Hess
dd7b99c860 add tip about metadata driven views (and more flexible view filtering)
While writing this documentation, I realized that there needed to be a way
to stay in a view like tag=* while adding a filter like tag=work that
applies to the same field.

So, there are really two ways a view can be refined. It can have a new
"field=explicitvalue" filter added to it, which does not change the
"shape" of the view, but narrows the files it shows.
Or, it can have a new view added, which adds another level of
subdirectories.

So, added a vfilter command, which takes explicit values to add to the
filter, and rejects changes that would change the shape of the view.

And, made vadd only accept changes that change the shape of the view.

And, changed the View data type slightly; now components that can match
multiple metadata values can be visible, or not visible.

This commit was sponsored by Stelian Iancu.
2014-02-19 16:29:56 -04:00
Joey Hess
39ebfa1a2e pre-commit: Update metadata when committing changes to annexed files within a view.
So the user can now switch to a view and then move files around within it
to manage metadata. For example, moving a file into a new directory
when in the tags=* view adds a tag to it.

Implementation is fairly efficient. One diff-index, which is no more
expensive than the first stage of a git commit, followed by possibly
some cat-file --batch traffic to find the key (when deleting a file).
Very similar to what's done in direct mode when committing. And like
direct mode when updating the WC after a merge, it has to buffer the
diff-tree values in order to make 2 passes over them.

When not in a view, pre-commit now does one extra git symbolic-ref,
which is tiny overhead.

This commit was sponsored by Andrew Eskridge.
2014-02-19 14:17:58 -04:00
Joey Hess
67a5f02a0b add vcycle command 2014-02-18 20:16:28 -04:00
Joey Hess
f603692a72 add vadd command 2014-02-18 20:02:09 -04:00
Joey Hess
67fd06af76 add git annex view command
(And a vpop command, which is still a bit buggy.)

Still need to do vadd and vrm, though this also adds their documentation.

Currently not very happy with the view log data serialization. I had to
lose the TDFA regexps temporarily, so I can have Read/Show instances of
View. I expect the view log format will change in some incompatable way
later, probably adding last known refs for the parent branch to View
or something like that.

Anyway, it basically works, although it's a bit slow looking up the
metadata. The actual git branch construction is about as fast as it can be
using the current git plumbing.

This commit was sponsored by Peter Hogg.
2014-02-18 18:22:20 -04:00
Joey Hess
fd09798e40 windows webapp: fix rsync.net support 2014-02-14 15:47:54 -04:00
Joey Hess
d007d1ac0c windows: hack to ensure HOME is always set 2014-02-14 14:03:24 -04:00
Joey Hess
2075cdeb59
limiting files based on metadata
Note that there is currently no caching, so
	--metadata foo=bar --metadata tag=blah
will currently read the log 2x per file.
2014-02-13 02:24:30 -04:00
Joey Hess
cf44fc1acb Windows webapp: Can create repos on removable drives. 2014-02-11 16:15:40 -04:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
e885080d06 Add progress display for transfers to/from external special remotes. 2014-02-10 21:33:22 -04:00
Joey Hess
b3a002f21d prep release 2014-02-10 12:59:18 -04:00
Joey Hess
c95d0cf7a8 Windows: Fix handling of absolute unix-style git repository paths.
Note that on Windows a remote with a path like /home/foo/bar
is interpreted by git as being some screwy relative path (relative to what
exactly seems ill-defined -- it seemed relative to C:\Program Files\Git\ in
my tests!) So no attempt has been made to handle such a path sanely, just not
to crash when encountering it.

Note that "C:\\foo" </> "/home/foo/bar" yields /home/foo/bar even though
that is not absolute! I don't know what to make of all this,
except that I will be very happy when this crock of **** vanishes from
the face of the earth.
2014-02-08 15:39:04 -04:00
Joey Hess
4e7c65dca0 Fix build on platforms not supporting the webapp. 2014-02-08 14:25:18 -04:00
Joey Hess
250df4685c avoid broken version of DAV 2014-02-07 17:42:52 -04:00
Joey Hess
3cbaa68be8 Android: Avoid crashing when unable to set file mode for ssh config file due to Android filesystem horribleness. 2014-02-07 14:57:23 -04:00
Joey Hess
3a3fda5bc0 Fix initremote with encryption=pubkey to work with S3, glacier, webdav, and external special remotes. 2014-02-06 18:25:31 -04:00
Joey Hess
897d877472 work around absNormPath not working on Windows
When making git-annex links, we want unix-style paths in the link targets.
2014-02-06 17:17:35 -04:00
Joey Hess
28cabd9909 Revert "work around absNormPath not working on Windows"
This reverts commit 0fc3ad82c5.

That caused 2 other test cases to fail. Windows path slash issues are
horrible.
2014-02-06 16:59:39 -04:00
Joey Hess
0fc3ad82c5 work around absNormPath not working on Windows
Seems I punted on this while porting before. This hack relies on DOS not
using / in filenames, it's effectively an alternate path separatr in at
least current versions of windows..
2014-02-06 15:06:45 -04:00
Joey Hess
a44e01c29c --in can now refer to files that were located in a repository at some past date. For example, --in="here@{yesterday}" 2014-02-06 12:43:56 -04:00
Joey Hess
751f496c11 add test case & fix conflict resolution bug on Windows & FAT
Fix bug in automatic merge conflict resolution code when used
on a filesystem not supporting symlinks, which resulted in it losing
track of the symlink bit of annexed files.

This was the underlying bug that was causing another test to fail,
which got worked around in 1c997fd08c.
I've chosen to keep 2 separate test cases since the old test case only
detected the problem accidentially.

Test suite passes on FAT & in windows, as well as on proper unix systems.

This commit was sponsored by Ellis Whitehead.
2014-02-04 17:24:12 -04:00
Joey Hess
1c997fd08c fix test suite to pass on windows
On windows, the sync of the second cloned repo to origin failed, because
synced/master was a non-fast-forward. This may be a bug of its own, but
it's not the issue that this test was intended to test, so disconnect
the repos from origin before syncing.
2014-02-03 12:12:50 -04:00
Joey Hess
29bb04aa0d Windows: Fix deletion of repositories by test suite and webapp.
On Windows, a file that is not writable cannot be deleted even if in a
directory with write perms. So git object files were not getting deleted
when removing a git repository.
2014-02-03 11:19:41 -04:00
Joey Hess
f657e4070b reorg 2014-02-03 10:45:28 -04:00
Joey Hess
fded408b44 sync --content: Drop files from remotes that don't have them after getting them.
Need to include the uuid of the local repo in the list of belived locations
of a key after getting it, in order for the drop from remote to include it
in the numcopies calculation.
2014-02-02 22:48:45 -04:00
Joey Hess
4676706756 sync --content: Reuse smart copy code from copy command, including handling and repairing out of date location tracking info. Closes: #737480 2014-02-02 19:57:22 -04:00
Joey Hess
1858c1f44a Document in man page that sshcaching uses ssh ControlMaster. Closes: #737476 2014-02-02 19:27:47 -04:00
Joey Hess
089c0109a2 Added ways to configure rsync options to be used only when uploading or downloading from a remote. Useful to eg limit upload bandwidth. 2014-02-02 16:06:34 -04:00
Joey Hess
15ac2ea4de sync --content: Re-pull from remotes after downloading content, since that can take a while and other changes may be pushed in the meantime. 2014-02-01 10:49:50 -04:00
Joey Hess
bbef0cddfd improve sync with xmpp and annex-ignore
* sync --content: Honor annex-ignore configuration.
* sync: Don't try to sync with xmpp remotes, which are only currently
  supported when using the assistant.
2014-02-01 10:33:55 -04:00
Joey Hess
1669e80e85 Windows: Avoid using unix-compat's rename, which refuses to rename directories.
Opened a bug about this: https://github.com/jystic/unix-compat/issues/10
2014-01-29 15:19:03 -04:00
Joey Hess
721cc0cd22 rework annexed object locking in direct mode & support Windows
Seems that locking of annexed objects when they're being dropped was broken
in direct mode:

* When taking the lock before dropping, it created the .git/annex/objects
  file, as an empty file. It seems that the dropping code deleted that,
  but that is not right, and for all I know could in some situation cause
  a corrupted object to leak out.
* When the lock was checked, it actually tried to open each direct mode
  file, and checked if it was locked. Not the same lock used above, and
  could also fail if some consumer of the file locked it.

Fixed this, and added windows support by switching direct mode to lock a
.lck file.
2014-01-28 16:43:11 -04:00
Joey Hess
891c85cd88 use locking on Windows
This is all the easy cases, where there was already a separate lock file.
2014-01-28 14:42:03 -04:00
Joey Hess
998d76ae04 Revert "temporarily revert tasty-rerun support for this release"
This reverts commit 8fcf041159.
2014-01-27 14:22:44 -04:00
Joey Hess
8fcf041159 temporarily revert tasty-rerun support for this release
revert c340d9506c

tasty-rerun is stuck in NEW in debian
2014-01-27 13:58:43 -04:00
Joey Hess
e0224f01c1 prep release 2014-01-27 13:44:12 -04:00
Joey Hess
ec7443eb06 All commands that support --all also support a --key option, which limits them to acting on a single key. 2014-01-26 14:59:47 -04:00
Joey Hess
85398d068e whereis: Support --all. 2014-01-26 14:26:32 -04:00
Joey Hess
78d0fd389e webapp: After upgrading a git repository to git-annex, fix bug that made it temporarily not be synced with. 2014-01-26 13:57:05 -04:00
Joey Hess
5fc2d760ea Optimise non-bare http remotes; no longer does a 404 to the wrong url every time before trying the right url. Needs annex-bare to be set to false, which is done when initially probing the uuid of a http remote. 2014-01-26 13:03:25 -04:00
Joey Hess
b93e485ef1 added annex.secure-erase-command config option. 2014-01-24 12:58:52 -04:00
Joey Hess
c340d9506c tasty-rerun! make rerest runs much much faster than running whole test suite 2014-01-24 12:21:30 -04:00
Joey Hess
3da0064657 assistant unused file handling
Make sanity checker run git annex unused daily, and queue up transfers
of unused files to any remotes that will have them. The transfer retrying
code works for us here, so eg when a backup disk remote is plugged in,
any transfers to it are done. Once the unused files reach a remote,
they'll be removed locally as unwanted.

If the setup does not cause unused files to go to a remote, they'll pile
up, and the sanity checker detects this using some heuristics that are
pretty good -- 1000 unused files, or 10% of disk used by unused files,
or more disk wasted by unused files than is left free. Once it detects
this, it pops up an alert in the webapp, with a button to take action.

TODO: Webapp UI to configure this, and also the ability to launch an
immediate cleanup of all unused files.

This commit was sponsored by Simon Michael.
2014-01-22 22:53:18 -04:00
Joey Hess
ed7c61914c assistant: Run the periodic git gc in batch mode. 2014-01-22 17:11:41 -04:00
Joey Hess
4b55afe9e9 add "unused" preferred content expression
With a really nice optimisation that keeps it from having any overhead
in normal operation!

This commit was sponsored by Ulises Vitulli.
2014-01-22 16:35:32 -04:00
Joey Hess
f2713a3bb9 benchmarked numcopies .gitattributes in preferred content
Checking .gitattributes adds a full minute to a git annex find looking for
files that don't have enough copies. 2:25 increasts to 3:27. I feel this is
too much of a slowdown to justify making it the default. So, exposed two
versions of the preferred content expression, a slow one and a fast but
approximate one.

I'm using the approximate one in the default preferred content expressions
to avoid slowing down the assistant.
2014-01-21 18:49:25 -04:00
Joey Hess
78ead70ea4 repair: Check git version at run time. 2014-01-21 13:22:48 -04:00