The items will still match full-text word searches, but they won't match
phrase searches (because those require cache files for non-text
attachments) and the full-text won't sync to other computers, so they
should really be reindexed.
This should be tested, but we run tests in Firefox, and this doesn't
exist in Firefox... Easiest option is probably to add the submenu to the
Firefox menus for the purposes of testing.
If this works out I think we'll want to use this approach for
all data layer changes.
Previously, an unsaved change on an object would update its state
immediately, which was fine for synchronous code but breaks down if a
save involves multiple asynchronous calls, because modifying state after
the relevant data has been saved to the DB but before the `_changed`
object has been cleared would mean that new changes would be lost. Now,
changes are written to _changedData, and a get for the data first checks
_changedData before checking the state property (e.g., _tags) directly.
The changedData property is cleared as it's written, and once the object
is saved, the reload updates the state property with the new data.
If a standalone attachment existed in a collection and then was added to
a parent (e.g., via Create Parent Item), and attachment metadata was
also changed at the same time (e.g., due to file syncing), the
'collection item must be top level' trigger could throw on another
syncing computer. To work around this, remove collections first, then
make changes to the parentItemID columns, and then add new collections.
I think it might be worth having a tag management window that lets you
view tags as a grid, sort by column (e.g., type), select ranges, delete,
consolidate, etc., but until then, this fulfills a popular request.
And use them in new importTextAttachment() and importHTMLAttachment()
test support functions. These can be used to avoid needing a hidden
browser for determining the character set of the imported text
documents.
This is useful when trying to debug an error that only happens after a
number of other tests have run -- specify -e and run tests from either
an earlier file with -s or from the beginning.
I'm not sure if we need the browser tests at all -- we still translate
some things via hidden browsers, and I'm not sure what we have that
tests that -- but we definitely don't need to test saving before the
Zotero pane is opened.
* Use mocha, chai & sinon from the npm. As of sinon 2.0
sinon-as-promised is no longer required so it is removed
* Tweak code to re-use the same loader with the same environment
throghout the code
* Introduce browserify step for testing tools that only provide
node-compatible libraries (sinon, chai-as-promised)
* Introduce copy step for test data to resolve multiple issues with
tests depending on files not being symlinks
* Re-introduce custom implementation of setTimeout to resolve issues
with few tests
* Re-introduce custom Bluebird Promises config & monkey patch
While objects in the sync queue that fail to save should remain in the
queue, objects that just don't exist remotely need to be removed, or
else they'll be retried forever.
Instead of My Publications being a separate library, have it be a
special collection inside My Library. Top-level items can be dragged
into it as before, and child items can be toggled off and on with a
button in the item pane. Newly added child items won't be shown by
default.
For upgraders, items in the My Publications library will be moved into
My Library, which might result in their being duplicated if the items
weren't removed from My Library. The client will then upload those new
items into My Library.
The API endpoint will continue to show items in the separate My
Publications library until My Publications items are added to My
Library, so the profile page will continue to show them.
Contains a dummy doc plugin, which is useful for:
- Testing integration.js functionality
- Serving as succint documentation for development of new integration
plugins
The web library will probably still display the old tag in addition to
the new one, at least until browser restart. We'll have to deal with
that separately.
Closes#1205
When refreshing, if fewer than 100 tags to show, just create them from
scratch instead of updating the full set. Otherwise, remove the full set
from DOM and add it back in after updates to avoid reflows (from #1204).
There are various things that could be done to optimize this further
(avoiding unnecessary sorting during full refreshes, calculating a hash
of the full set and not updating it every time), but we should probably
just replace it with @tnajdek's React version first.
Closes#1204
Changes `libraryTreeView::addEventListener('load')` and similar to
`libraryTreeView::onLoad.addListener(listener, once)`, etc. `once` is an
optional boolean that, when true, causes the listener to fire once and
then be removed. This is implicit for 'load'.
'load' maintains its special behavior of running immediately if the
treeview has already been loaded.
Also adds `waitForLoad()` and `waitForSelect()` functions that return
promises on event completion, since most uses of those events were just
resolving deferreds.
When dragging an item to another library, we have to check if there's a
linked item in the target library, but items might not yet be laoded in
the other library, so item.getLinkedItem() can fail with "Item [n] not
yet loaded].
Fixing required asyncifying the follow functions:
- Zotero.Item::getLinkedItem()
- Zotero.Collection::getLinkedCollection()
- Zotero.URI.getURIItem()
- Zotero.URI.getURICollection()
- Various integration functions
As noted in 27cb099c82, import translators should be rewritten to return
a promise from doImport() and wait for promises from successive
item.complete() calls. They should then be marked as minVersion: "5.0"
to be handled properly by this new code.
(But this tries to account, albeit with somewhat worse behavior, for
translators that haven't been rewritten and sandboxes without Promise
(which is currently the case with child sandboxes in the client).)
(Oh, and I haven't tested this at all in the connectors.)
- Archive remotely missing that user chooses to keep
- Ignore archived groups that don't existing remotely
- Unarchive groups that become available again
Previously on Windows, where we don't have /bin/mv, we were recursing
into the data directory and copying files individually, which is very
slow, so automatic migration was disabled. Instead, try moving
directories with OS.File.move() with the `noCopy` flag. Moving
directories is technically unsupported by OS.File, but probably only
because of the possibility of a cross-volume copy (which is only
implemented for some platforms), and using `noCopy` hopefully prevents
that. If someone does have their data directory or storage directory on
a different volume, the migration might be quite slow, but leaving a
data directory behind in the Firefox profile directory (where it can be
easily misplaced with a seemingly unrelated Firefox reset) is worse.
Attachments are now saved before the connector server responds, because they're
no longer started out-of-band in saveItems(). This is necessary to prevent
transaction badness during imports, but it may not be what we want for the
connector, so we may want to revisit this after further testing.
E.g., moving 3,600 items to the trash now takes 4 seconds instead of 62
Instead of saving each item, update internal state and database directly
(which is more brittle but worth it). Also avoid unnecessary sorting
after removing an item from the items tree.
This reverts c6b78da69d, which changed it to expect numbers when I
noticed the type being undefined in debug output, but apparently the
only tests where the type actually mattered passed it as a name.
At some point we should just change all tests to pass as a name.
When adding many search conditions (e.g., when matching many items with the
`key` condition), the query can fail due to either the bound parameter limit or
the expression tree size limit.
To avoid this, add support for an 'inlineFilter' property on search conditions
when using the 'is' or 'isNot' operator. 'inlineFilter' is a function that
returns a quoted value suitable for direct embedding in the SQL statement, or
false if not valid. Multiple consecutive conditions for the same 'inlineFilter'
field are combined into an `IN (x, y, z)` condition.
The Firefox French language pack contains some mistakes regarding
the short form of months. As a consequence, French month parsing
didn't work.
Please note that these values aren't even the correct abbreviations
but only the three or four first letters of the correct abbreviations.
See the French CSL locale for the correct abbreviations.
Month values from the Firefox language packs are included in a
JSON file used by `Zotero.Date.getMonths()`. `getMonths(true)` includes
English months as well.
The JSON file should be bundled with the connectors as well, and
Zotero.Date.init() should be updated to populate the month data from
that.
* Mark feedItems read in a single batch SQL update
* Automatically remove old feed items
* User-facing preference globally and per-feed for feed item expiration
Improves proxy support
- Automatically detect and dehyphenise https proxies which use EZProxy
HttpsHyphens
- Web translators now pass around Zotero.Proxy instances which can
proxify/deproxify urls passed to `translate.setLocation()` before calling
`translate.getTranslators()`/ translate.detect()`. The proxy passing is
done within connector background/injected processes and between
standalone and connectors.
- Proxy protocol unified with connectors. Connectors can now pass
proxies to `/connector/save_items`. The proxies will be used to resolve
true item and attachment urls when saving.
Closeszotero/zotero#578, zotero/zotero#721
Relevant zotero/zotero#34, zotero/zotero#556
- Make Zotero.Attachments.createDirectoryForItem() delete existing
directory instead of moving it to orphaned-files; also now returns a
string path instead of an nsIFile
- Use above function during file sync instead of
_deleteExistingAttachmentFiles(), which was partly broken
- Fix throwing on errors when saving some attachment types
- Don't block the UI with a progress meter during imports. Instead, show
a popup in the bottom right when the import is done that shows how
many items were saved.
- Fix hang when importing some files
- Fix various problems with asynchronous operations/transactions
- Use the save queue for imports instead of creating concurrent
transactions that can time out
- Wait for the save to finish before returning from the translate()
promise. All save modes now use the save queue, so code that
handled the non-save-queue process can probably be removed.
- Serialize child attachments instead of running them concurrently.
This might make multi-attachment saves a little slower, since they
can't download at the same time, but it avoids problems with
concurrent transactions. We might be able to improve this to allow
concurrent downloads, or allow concurrent saves for a limited
number of items (e.g., from web saving) if not for larger imports.
- Change collection handling during import, since UI is now active
- Select the root collection at the beginning of the import
- Assign items and collections to the root during the import instead
of at the end
- Don't select other collections
- Change a few ItemSaver functions to use promises and remove
unnecessary callbacks. (This includes some connector code that needs
to be tested.)
- Change some `parentID` variables in ItemSaver to `parentItemID` for
clarity, since collections are now handled in more places
To-do:
- Save items in smaller batches instead of doing all in the same
transaction
- Show progress meter in a bottom-right popup during the import
Zotero.getString() now takes a third parameter, `num` (which should also
appear in `params`) to use when determining which plural form of the
string to use. Localized strings should include all forms in the order
specified in [1], separated by semicolons.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_and_Plurals
This reverts commit 60befe52e4 and adds a
better fix that leaves the notifier event in place. Feeds just don't
need to update after syncs during tests.
An endpoint can now take a single object containing 'method', 'pathname',
'query', 'headers', and 'data' and return an integer, an array containing
[statusCode, contentType, body], or a promise for either. This allows the
handlers to use the HTTP method and headers and removes the need for callbacks
when some handlers already use coroutine().
If init() returns a promise, it now has to use the new single-parameter
signature (because the check is done with Function.length, and combining
promises and callbacks doesn't make sense anyway).
If someone can manage to reproduce this locally, we can try a real fix:
✖ [FAIL] should show tags in alphabetical order
AssertionError: expected '' to equal 'A, B, C' at
@resource://zotero-unit-tests/noteeditorTest.js:61:4
The Zotero.DataDirectory equivalents return string paths instead of nsIFile
instances, so some of these calls now just use Zotero.File.pathToFile(), which
can be removed when the surrounding code is updated to OS.File,
Look for other profiles, from both apps (Firefox and Standalone), that
point to the data directory being migrated and update prefs.js in those
profiles to point to the new location.
Also reorganize code into Zotero.Profile and Zotero.DataDirectory
namespaces
This prevents us from moving the data directory if the other app
(Firefox or Standalone) is running from the same directory.
Also clean up stub code in migration tests
When an export translator is selected for Quick Copy, Quick Copy
initialization triggers translator initialization a few seconds after
startup, because the translator code needs to be available synchronously
for drag/drop. A Quick Copy test was changing the setting to BibTeX,
which was resulting in random timeouts after subsequent resetDB() calls
due to slow translator loading. This change skips initialization in test
mode. This might actually fix a lot of timeouts on Travis in the second
half of the tests...
This also resets the Quick Copy pref in those tests so that it's left at
the default, though really we should automatically reset all prefs after
all test groups and in resetDB().
This adds a new button to the Advanced prefs to migrate the data directory to
$HOME/Zotero. The button only appears if the data directory is set to the
default location within a profile directory (including the other program from
the one running, even though that's technically stored as a custom data
directory).
On Mac/Linux, directories within the data directory are moved with /bin/mv. On
Windows, or if that fails, they're copied recursively using OS.File.move()
(which annoyingly doesn't reliably support directory moving). The former should
be instantaneous on most systems (unless the data directory or 'storage' were
on a different filesystem from $HOME).
If the database fails to transfer, migration fails and the data directory
setting remains on the old directory. If the database transfers but other files
fail, the data directory setting is updated. In both cases, the user is
encouraged to migrate remaining files manually with a button that reveals the
directories and quits the program.
This isn't yet tested on Linux or Windows, and migration isn't yet suggested
automatically.
Adds Zotero.File.reveal(), Zotero.File.directoryIsEmpty(), and
Zotero.File.moveDirectory().
With icons to identify collections and searches
Also:
- `savedSearch` search condition in general
- Clean up some search window code
- Reorganize search tests
This is the recommended approach (since NetUtil can still do some main-thread
I/O for files) and avoids warnings in the console.
For getContentsAsync(), also sends nsIURIs and string URIs to
Zotero.HTTP.request(), which should be used instead.
This makes getBinaryContentsAsync() much slower (due to the conversion from an
array of bytes to a binary string), but it's only used in tests. For one test
that compares two large files, use MD5 instead.
Clicking it cancels the current window, opens the Cite pane of the
prefs, and selects the Styles tab. (This will be more useful once we
have inline style installation from that pane.)
If an object exists locally but not remotely and the local version has a
version number, that's an error. I don't think that should ever happen,
but it can if things somehow get out of sync due to other bugs.
To address, reprocess the API delete log during a full sync and then
reset the version number of all remaining local objects that don't exist
remotely (not just unmodified objects, as was the case previously) to 0
for uploading.
When remote deletions are reprocessed, delete local objects that haven't
been modified and show the conflict resolution window for any local
items that have.
Also:
- Clean up checking of last remote library version during download
syncs
- Add Zotero.DataObjects.getAllKeys()
The client skips synced storage properties (md5, mtime) when uploading items to
ZFS-enabled libraries, but since the API returns JSON with those values
included after writes, they do get saved to the sync cache. If the local
attachment is then modified and the client generates a diff from the cached
version with those properties skipped, they'll be included in the patch JSON as
empty strings in order to clear them. This changes Zotero.Item::toJSON() to
skip those properties in patch mode as well.
This fixes a sync error ("Cannot change 'md5' directly in group library") when
a group attachment is updated locally.
If the item was deleted on one side and moved to the trash on the other,
just delete the item on the trash side. Since trash emptying happens
automatically, this would otherwise result in a conflict even if the
user carefully avoided making changes before a manual sync.
On reset, items are overwritten with pristine versions if available and deleted
otherwise, and then the library is marked for a full sync. Unsynced/changed
files are deleted and marked for download.
Closes#1002
Todo:
- Handle API key access change (#953, in part)
- Handle 403 from data/file upload for existing users (#1041)
Previously they only showed for My Library by default, which I suspect
meant that most people didn't know you could get them for other
libraries...
This hides "Duplicate Items" and "Unfiled Items" from the context menu
when they're active, which may or may not be desirable (but we don't
show, say, "Trash" in the context menu).
Also tweaks selection behavior after hide to select next appropriate row
instead of the parent library.
If a field is open and the user right-clicks on another field (e.g., swap
names, creator type, transform text), any changed value in the open field was
lost.
Also:
- Don't show swap-names menu in single-field mode
I can't quite get programmatic access to context menus to work correctly, so
tests are disabled for now. (They work individually, but not together.)
I originally attempted this with zotero-persist and column attributes,
but there is no good way to make it succinct paramswise and the code was
painful to look at too. Thus different group settings are stored in
preferences.
Currently there are two view groups: "feed" and "default". Items view
columns have two new attributes:
`default-in` - a space separated list of views in which a column is
visible by default
`disabled-in` - a space separated list of views in which a column is
disabled by default (invisible + not possible to enable)
Fields not parsed for feeds are now disabled.
And don't skip alerts in Zotero.alert() during automated tests. (That
was intended to avoid long timeouts after unexpected failures, but,
e.g., PDF metadata lookups (which are currently disabled in automated
tests) should just be mocked so they don't intermittently fail.)
OS.File.DirectoryIterator, used by OS.File.removeDir(), isn't reliable
on Travis, returning entry.isDir == false for directories, so use
nsIFile instead
See also: 2c2a5a378
server_connectorTest - alternating port prevents
exceptions not catchable in JS for httpd server when the socket
and the port remains open after httpd.stop() callback
support - changes the window on which `setTimeout()` is ran in dialogs.
If the timeout is ran on the main window the `dialog` object appears to
lose certain properties and not respond to interactions completely.
Also changes Zotero.Item.prototype.clone() to take an `options` object for its
second parameter instead of a boolean `skipTags`. The object includes
`skipTags` as well as a new `includeCollections` property to add the new item
to the same collections.
Indexing currently happens a second after the 201 is returned to the
connector, so we have to wait for that before continuing tests, or else
a DB clear in a later test (e.g., storageLocal) will cause an error to
be logged when the indexing kicks off.
WPD code hasn't been updated in many years, and there was an issue with
document permissions in 5.0. We'll need to replace nsIWBP in Electron,
but this will do for now.
Attachments are opened using file:// URIs instead of
zotero://attachment, which is what Standalone does anyway. Ancient HTML
annotations and highlights won't be displayed anymore, but I'm not sure
they worked anyway, and it hasn't been possible to create them in years.
We might be able to write out existing annotations to notes.
iframes are skipped during saving, in an attempt to reduce the number of
junk ad files. JS can still cause problems with viewing, so we might
still want to either disable scripts or force the viewed page offline
(if such a thing is possible).
There might be issues with auxiliary filename length/characters during
cross-platform file syncing. (We modified the WPD code to shorten/clean
them.)
- Add test for recogning within a collection (follow-up from #1015)
- Update/remove some outdated code
These tests are still skipped by default, since we don't want to actually do
lookups on every test run.
If the API returns a modified item after an upload (e.g., to strip invalid
characters), don't update the Date Modified field when saving those changes to
the local version (though it would still be good to avoid API-side changes as
much as possible).
And omit in ZFS file sync requests
The API previously didn't allow these properties to be set for group items,
because they were set atomically during the file upload process, but 1) that's
not really necessary (makes a little sense for 'filename', but not really a big
deal if an old file is renamed on another computer before the new file is
synced down) and 2) skipping them results in the properties getting erased
after items are uploaded and the empty values returned by the server overwrite
the local values.
Before 5.0 we performed a regexp on new item data values to determine if
they were integers and saved them natively in SQLite if so. We no longer
do that, but setField() used strict equality when checking for changes,
so an item could be marked as changed when comparing to a new string
value (e.g., from a write response from the API, which always returns
strings). To avoid that, this converts all old values in the DB to
strings and saves all incoming values as strings automatically. (This
should also help with searching and some other things.)
Until we have a consistent way of sanitizing HTML on client and server, account
for differences manually. More differences between HTMLPurifier and TinyMCE
should be added as necessary.
- Fix upgrading of Mozilla-style attachments/storage file paths on upgrade
(requires re-upgrade)
- Save relative paths using forward slashes for consistency, and convert
to platform-appropriate slashes on use
- Fixes#994, 5.0: "+" doesn't expand all collections within a library
- If a container (library, collection) is closed directly, the open state of
all containers below it are now restored when it's reopened. Previously all
collections would be closed on a manual reopen (though they might have been
restored on the next Zotero restart).
- If "-" is pressed, all containers are closed, and reopening the library will
show only top-level collections.
- Use custom exception for user-initiated sync cancellations, which can bubble
up to the sync runner -- this should help with a sync stop button (#915)
- Separate out deletions-downloading code
- Refactor delay generator handling on library version mismatch
- Clearer variable names
Return false for single ids or skip for multiple ids. This is the original
behavior, but at some point it started throwing an UnloadedDataException. IDs
are always loaded at initialization, though, so we know whether the objects
actually exist.
This is necessary to get a library version after the write instead of an
item version. Otherwise after a full-text write, the main library
version is behind, so the next sync checks all object types for that
library instead of getting a 304.
Full text is batched up to 500K characters or 10 items, whichever is
less.
This also switches to using ?format=versions for /fulltext requests,
which isn't currently necessary but reflects what it's actually doing.
If a version is returned for an item's full-text content but a 404 is returned
for the content itself (because it's missing in Elasticsearch for some reason),
don't throw an error.
Also remove legacy array comprehensions in fulltext and syncFullTextEngine test
files, which apparently weren't being run.
1c19fe8d81 isn't sufficient for local files, because detection is run
twice, so a translator may not be available when the detect callback is
run. This changes the test to poll for the translate icon, which is a
bit of a hack but does the job.
Unfortunately this isn't perfect either, because it seems the RIS
detection sometimes just isn't run, which means that the icon never
changes and the test times out. Maybe @simonster has an idea why that's
happening.
