Commit graph

1421 commits

Author SHA1 Message Date
Joey Hess
7498c5dd96 annex.genmetadata can be set to make git-annex automatically set metadata (year and month) when adding files 2014-02-23 00:08:29 -04:00
Joey Hess
079b35a1a8 views: add automatically constructed file location metadata
When constructing views, metadata is available about the location of the
file in the view's reference branch. Allows incorporating parts of the
directory hierarchy in a view.

For example `git annex view tag=* podcasts/=*` makes a view in the form
tag/showname.

Performance impact: I benchmarked git annex view tag=* in the conference
proceedings repo to take 6.459s before this change, and 6.544s after.

FWIW, I considered making the syntax for this be podcasts/*, which might
be easier for the user to learn. However, I think it's not as good:

* The user has to then juggle two different syntaxes, and podcasts/* will
  be expanded by the shell so they also need to quote it, while podcasts/=*
  is unlikely to be expanded by the shell.
* It would allow for things like podcasts/*/* and *.mp3 which do not
  map well into views.

This commit was sponsored by Aurélien Pinceaux.
2014-02-22 16:27:53 -04:00
Joey Hess
24f8136504 --metadata field=value can now use globs to match, and matches case insensatively, the same as git annex view field=value does.
Also refactored glob code into its own module.
2014-02-21 18:34:34 -04:00
Joey Hess
ded4ab5704 Fix handling of rsync remote urls containing a username, including rsync.net.
This breakage seems to have been caused way back in a1eded86,
but I am pretty sure rsync.net support has not been entirely
broken since last April. AFAICS, the generated .ssh/config
has not changed since then -- it has never included a Username setting
line. So, I am puzzled at when this reversion was introduced.

Note that the breakage only affected checkpresent and remove. Upload and
download use the ssh connection caching, which includes a -l username.
2014-02-21 13:20:57 -04:00
Joey Hess
1f6f034e90 prep release 2014-02-21 11:24:18 -04:00
Joey Hess
9edc3a735d fsck: Refuse to do anything if more than one of --incremental, --more, and --incremental-schedule are given, since it's not clear which option should win. 2014-02-20 15:56:45 -04:00
Joey Hess
7d288d83c9 glacier: Do not try to run glacier value create when an existing glacier remote is enabled. 2014-02-20 15:56:26 -04:00
Joey Hess
21610294da trust, untrust, semitrust, dead: Warn when the trust level is overridden in .git/config. 2014-02-20 15:17:39 -04:00
Joey Hess
134fdefb8c fsck: When run with --all or --unused, while .gitattributes annex.numcopies cannot be honored since it's operating on keys instead of files, make it honor the global numcopies setting, and the annex.numcopies git config setting. 2014-02-20 14:45:17 -04:00
Joey Hess
db48b8a4a3 unused: Fix to actually detect unused keys when in direct mode. 2014-02-20 13:53:49 -04:00
Joey Hess
dd7b99c860 add tip about metadata driven views (and more flexible view filtering)
While writing this documentation, I realized that there needed to be a way
to stay in a view like tag=* while adding a filter like tag=work that
applies to the same field.

So, there are really two ways a view can be refined. It can have a new
"field=explicitvalue" filter added to it, which does not change the
"shape" of the view, but narrows the files it shows.
Or, it can have a new view added, which adds another level of
subdirectories.

So, added a vfilter command, which takes explicit values to add to the
filter, and rejects changes that would change the shape of the view.

And, made vadd only accept changes that change the shape of the view.

And, changed the View data type slightly; now components that can match
multiple metadata values can be visible, or not visible.

This commit was sponsored by Stelian Iancu.
2014-02-19 16:29:56 -04:00
Joey Hess
39ebfa1a2e pre-commit: Update metadata when committing changes to annexed files within a view.
So the user can now switch to a view and then move files around within it
to manage metadata. For example, moving a file into a new directory
when in the tags=* view adds a tag to it.

Implementation is fairly efficient. One diff-index, which is no more
expensive than the first stage of a git commit, followed by possibly
some cat-file --batch traffic to find the key (when deleting a file).
Very similar to what's done in direct mode when committing. And like
direct mode when updating the WC after a merge, it has to buffer the
diff-tree values in order to make 2 passes over them.

When not in a view, pre-commit now does one extra git symbolic-ref,
which is tiny overhead.

This commit was sponsored by Andrew Eskridge.
2014-02-19 14:17:58 -04:00
Joey Hess
67a5f02a0b add vcycle command 2014-02-18 20:16:28 -04:00
Joey Hess
f603692a72 add vadd command 2014-02-18 20:02:09 -04:00
Joey Hess
67fd06af76 add git annex view command
(And a vpop command, which is still a bit buggy.)

Still need to do vadd and vrm, though this also adds their documentation.

Currently not very happy with the view log data serialization. I had to
lose the TDFA regexps temporarily, so I can have Read/Show instances of
View. I expect the view log format will change in some incompatable way
later, probably adding last known refs for the parent branch to View
or something like that.

Anyway, it basically works, although it's a bit slow looking up the
metadata. The actual git branch construction is about as fast as it can be
using the current git plumbing.

This commit was sponsored by Peter Hogg.
2014-02-18 18:22:20 -04:00
Joey Hess
fd09798e40 windows webapp: fix rsync.net support 2014-02-14 15:47:54 -04:00
Joey Hess
d007d1ac0c windows: hack to ensure HOME is always set 2014-02-14 14:03:24 -04:00
Joey Hess
2075cdeb59
limiting files based on metadata
Note that there is currently no caching, so
	--metadata foo=bar --metadata tag=blah
will currently read the log 2x per file.
2014-02-13 02:24:30 -04:00
Joey Hess
cf44fc1acb Windows webapp: Can create repos on removable drives. 2014-02-11 16:15:40 -04:00
Joey Hess
fa24ba2520 plumb creds from webapp to initremote
Avoids abusing setting environment variables, which was always a hack
and won't work on windows.
2014-02-11 14:07:56 -04:00
Joey Hess
e885080d06 Add progress display for transfers to/from external special remotes. 2014-02-10 21:33:22 -04:00
Joey Hess
b3a002f21d prep release 2014-02-10 12:59:18 -04:00
Joey Hess
c95d0cf7a8 Windows: Fix handling of absolute unix-style git repository paths.
Note that on Windows a remote with a path like /home/foo/bar
is interpreted by git as being some screwy relative path (relative to what
exactly seems ill-defined -- it seemed relative to C:\Program Files\Git\ in
my tests!) So no attempt has been made to handle such a path sanely, just not
to crash when encountering it.

Note that "C:\\foo" </> "/home/foo/bar" yields /home/foo/bar even though
that is not absolute! I don't know what to make of all this,
except that I will be very happy when this crock of **** vanishes from
the face of the earth.
2014-02-08 15:39:04 -04:00
Joey Hess
4e7c65dca0 Fix build on platforms not supporting the webapp. 2014-02-08 14:25:18 -04:00
Joey Hess
250df4685c avoid broken version of DAV 2014-02-07 17:42:52 -04:00
Joey Hess
3cbaa68be8 Android: Avoid crashing when unable to set file mode for ssh config file due to Android filesystem horribleness. 2014-02-07 14:57:23 -04:00
Joey Hess
3a3fda5bc0 Fix initremote with encryption=pubkey to work with S3, glacier, webdav, and external special remotes. 2014-02-06 18:25:31 -04:00
Joey Hess
897d877472 work around absNormPath not working on Windows
When making git-annex links, we want unix-style paths in the link targets.
2014-02-06 17:17:35 -04:00
Joey Hess
28cabd9909 Revert "work around absNormPath not working on Windows"
This reverts commit 0fc3ad82c5.

That caused 2 other test cases to fail. Windows path slash issues are
horrible.
2014-02-06 16:59:39 -04:00
Joey Hess
0fc3ad82c5 work around absNormPath not working on Windows
Seems I punted on this while porting before. This hack relies on DOS not
using / in filenames, it's effectively an alternate path separatr in at
least current versions of windows..
2014-02-06 15:06:45 -04:00
Joey Hess
a44e01c29c --in can now refer to files that were located in a repository at some past date. For example, --in="here@{yesterday}" 2014-02-06 12:43:56 -04:00
Joey Hess
751f496c11 add test case & fix conflict resolution bug on Windows & FAT
Fix bug in automatic merge conflict resolution code when used
on a filesystem not supporting symlinks, which resulted in it losing
track of the symlink bit of annexed files.

This was the underlying bug that was causing another test to fail,
which got worked around in 1c997fd08c.
I've chosen to keep 2 separate test cases since the old test case only
detected the problem accidentially.

Test suite passes on FAT & in windows, as well as on proper unix systems.

This commit was sponsored by Ellis Whitehead.
2014-02-04 17:24:12 -04:00
Joey Hess
1c997fd08c fix test suite to pass on windows
On windows, the sync of the second cloned repo to origin failed, because
synced/master was a non-fast-forward. This may be a bug of its own, but
it's not the issue that this test was intended to test, so disconnect
the repos from origin before syncing.
2014-02-03 12:12:50 -04:00
Joey Hess
29bb04aa0d Windows: Fix deletion of repositories by test suite and webapp.
On Windows, a file that is not writable cannot be deleted even if in a
directory with write perms. So git object files were not getting deleted
when removing a git repository.
2014-02-03 11:19:41 -04:00
Joey Hess
f657e4070b reorg 2014-02-03 10:45:28 -04:00
Joey Hess
fded408b44 sync --content: Drop files from remotes that don't have them after getting them.
Need to include the uuid of the local repo in the list of belived locations
of a key after getting it, in order for the drop from remote to include it
in the numcopies calculation.
2014-02-02 22:48:45 -04:00
Joey Hess
4676706756 sync --content: Reuse smart copy code from copy command, including handling and repairing out of date location tracking info. Closes: #737480 2014-02-02 19:57:22 -04:00
Joey Hess
1858c1f44a Document in man page that sshcaching uses ssh ControlMaster. Closes: #737476 2014-02-02 19:27:47 -04:00
Joey Hess
089c0109a2 Added ways to configure rsync options to be used only when uploading or downloading from a remote. Useful to eg limit upload bandwidth. 2014-02-02 16:06:34 -04:00
Joey Hess
15ac2ea4de sync --content: Re-pull from remotes after downloading content, since that can take a while and other changes may be pushed in the meantime. 2014-02-01 10:49:50 -04:00
Joey Hess
bbef0cddfd improve sync with xmpp and annex-ignore
* sync --content: Honor annex-ignore configuration.
* sync: Don't try to sync with xmpp remotes, which are only currently
  supported when using the assistant.
2014-02-01 10:33:55 -04:00
Joey Hess
1669e80e85 Windows: Avoid using unix-compat's rename, which refuses to rename directories.
Opened a bug about this: https://github.com/jystic/unix-compat/issues/10
2014-01-29 15:19:03 -04:00
Joey Hess
721cc0cd22 rework annexed object locking in direct mode & support Windows
Seems that locking of annexed objects when they're being dropped was broken
in direct mode:

* When taking the lock before dropping, it created the .git/annex/objects
  file, as an empty file. It seems that the dropping code deleted that,
  but that is not right, and for all I know could in some situation cause
  a corrupted object to leak out.
* When the lock was checked, it actually tried to open each direct mode
  file, and checked if it was locked. Not the same lock used above, and
  could also fail if some consumer of the file locked it.

Fixed this, and added windows support by switching direct mode to lock a
.lck file.
2014-01-28 16:43:11 -04:00
Joey Hess
891c85cd88 use locking on Windows
This is all the easy cases, where there was already a separate lock file.
2014-01-28 14:42:03 -04:00
Joey Hess
998d76ae04 Revert "temporarily revert tasty-rerun support for this release"
This reverts commit 8fcf041159.
2014-01-27 14:22:44 -04:00
Joey Hess
8fcf041159 temporarily revert tasty-rerun support for this release
revert c340d9506c

tasty-rerun is stuck in NEW in debian
2014-01-27 13:58:43 -04:00
Joey Hess
e0224f01c1 prep release 2014-01-27 13:44:12 -04:00
Joey Hess
ec7443eb06 All commands that support --all also support a --key option, which limits them to acting on a single key. 2014-01-26 14:59:47 -04:00
Joey Hess
85398d068e whereis: Support --all. 2014-01-26 14:26:32 -04:00
Joey Hess
78d0fd389e webapp: After upgrading a git repository to git-annex, fix bug that made it temporarily not be synced with. 2014-01-26 13:57:05 -04:00