This adds selectItems() to ZoteroPane and collectionTreeView and removes
the ancient, unused 'expand' argument to selectItem(), which didn't
really make sense there. It also includes a new
itemTreeView::ensureRowsAreVisible() that tries to scroll to an
appropriate place (or, better yet, not scroll at all) given the
specified rows and page size.
- Fix incorrect results for ANY search with multiple "Attachment
Content" conditions and no other conditions
- Dramatically speed up single-word searches by avoiding unnecessary
text scans (which probably addresses #1595)
- Clean up code
- Added icon-button UI code for the menubutton
- Upgrade to React 16 to allow non-standard attrs, such as `tooltiptext`
to support XUL tooltips
- Add i18n support for React UI elements
- Update tests for reactified tag selector
This is hard to do currently because the natural place to do it (and
where the previous seeAlso stuff was done) is translate_item.js, but
with async import translators that now only gets one item at a time,
whereas saving item relations requires all items to be saved. So this
would probably need to be done in the import code in translate.js.
It might also require undoing
https://github.com/zotero/zotero/pull/453 so that getResourceURI() works
on notes and figuring out another solution for the problem that was
trying to solve.
Invalid paths, including Windows UNC paths on other OSes, caused exports
to fail. Now they're ignored, which is what we do for other missing
attachment files.
Fixes#1622
This fixes direct and VPN-based retrieval of PDFs for Elsevier (e.g.,
ScienceDirect) items that have a DOI but no URL, since Elsevier resolves
DOIs through an intermediate page.
Delay requests to the same domain by 1 second, respect a Retry-After
header if present for 429 and 503, and delay for 10 seconds on 429 or
5xx otherwise.
Currently only .status and .getResponseHeader() (for getting 'Location')
are available in the returned object, but we could make the body
available if necessary.
If there's no translated PDF or the translated PDF fails and the item
has a DOI, check Zotero's Unpaywall mirror for possible sources and try
to download one of those.
Unlike with "Add Item by Identifier" and "Find Available PDF" in the
item context menu, this does not try the DOI/URL page, since it would
result in more data leakage and most of the time you'd be saving from
the DOI page already. We could consider offering it as an option, but
for it to be useful, you'd have to have an institutional subscription,
be on-campus or connected via VPN (for now), and be saving from
somewhere other than the main page.
A new connector endpoint, sessionProgress, takes the place of
attachmentProgress. Unlike attachmentProgress, sessionProgress can show
new attachments that have been added to the save, and with a little more
work should also be able to show when a parent item has been recognized
for a directly saved PDF.
This also adds support for custom PDF resolvers, available to all PDF
retrieval methods. I'll document those separately.
Closes#1542
- Add the ability to extract a PDF URL from a given webpage using the
translation framework
- Add the ability to get open-access PDFs from landing pages from
Unpaywall data in addition to direct PDF URLs
- Use the above functionality to improve PDF retrieval for "Add Item by
Identifier"
- Add "Find Available PDFs" option to the item context menu to retrieve
PDFs for existing items from the DOI or URL page or using Unpaywall
data. The option appears for single items with a DOI or URL and no PDF,
and it always appears when selecting multiple top-level items (but
skips ineligible items).
PDF extraction from DOI/URL pages will currently only work with
unauthenticated access (i.e., on-campus or VPN, but not via a web-based
proxy).
Supersedes and closes#948
Use `getResource` in Zotero.Date.init (this turns it into a
synchronous function). Zotero.File.getResource makes it easier
to load local files on platforms that do not support the
`resource://` URLs.
When the associated-files pref is enabled, Add Item by Identifier uses a
Zotero Unpaywall mirror to find available open-access PDFs. No details
about the contents of searches are logged.
The progress percentage is based on the most recent transaction
(or undeterminate if this is the first session transaction)
Fix undefined function call error
Move an item and its attachments to another library. Attachments are
removed as necessary if linked files or all files aren't supported in
the target library.
Add newly added attachments to a queue, start processing it after five
seconds have passed since the last attachment was added, and process
another every half second after that unless another is added.
This queue won't survive a restart, so the queue should really be in the
DB, but this should avoid problems when adding multiple attachments at
once.
Addresses #1284
Applies to dragging to the collections pane or the items pane, adding
via New Item menu, or saving via the connector server
If the renaming pref is enabled, the PDF is renamed after recognition.
Can be disabled in the preferences
Closes#917
Do cleanup on 'unload' rather than 'close' (which is limited to a click
on the close button and doesn't get called for win.close()) and clear
the queue after each test.
Automatic renaming is now done for dragging of an external file onto an
item (as added in 7cb95f41) as well as dragging as a linked file,
dragging a non-native (not text or HTML) URL, "Attach Link to File…",
"Attach Stored Copy of File…", and "Retrieve Metadata for PDF". It only
applies if a single file is being added and if the parent item has no
non-HTML attachments. By default, the renaming only applies to PDFs, but
this can be changed with the renameAttachmentFiles.automatic.fileTypes
hidden pref.
A new General pref, "Automatically rename attachment files using parent
metadata", controls whether the renaming happens.
Files saved via web translators are renamed regardless of this pref,
because they would often be gibberish otherwise.
Closes#113
Check file-editing access for the group from the API before offering to
reset, update the filesEditable setting properly, and restart the sync
automatically after resetting.
If a transaction took over 30 seconds and another transaction timed out
waiting for it, the second transaction would reset the notifier queue,
but if the first transaction then tried to queue an event, it would fail
with this error and roll back. (It would be nice to figure out why
transactions are taking over 30 seconds, though.)
- Updates /saveItems and /saveSnapshot to take a sessionID
- Provides a list of editable collections in the current library
- Adds an /updateSession method that takes a sessionID and updates the
collection and tags of any items saved via that operation (and changes
the currently selected collection)
Cross-library changes are not yet supported
I'm not sure what this was for, but at least with an async test function
it seems to be causing spurious "the string 'x' was thrown, throw an
Error :)" messages that hide the real error.
This is a simplified version of the fix from #872. Unlike the proposal
in #36, this doesn't require all child items to be selected, since in a
search some children might be grayed out. If the child of an unselected
parent item is included, the drag isn't allowed.
Closes#36
- Moves a bunch of citation related processing from Integration.Session
- Replaces missing item handling with a function instead of exception
- Solves some really confusing flow issues in _processFields
Restores the "Restore to Zotero Server" functionality, now using the
API:
1. Get all remote keys and send `DELETE` for any that don't exist
locally.
2. Upload all local objects in full (non-patch) mode using only library
version so that the remotes are overwritten.
3. Reset file sync history, causing all files to be uploaded (or, more
likely, reassociated with existing remote files).
Since these are treated as regular updates on the server, they'll sync
down to other clients normally. Unsynced changes by other clients might
still trigger conflicts.
This and Reset File Sync History can also now be run on group libraries,
with a library selector in the Reset pane (which I forgot to do with
React).
The full sync option is now removed from the Reset pane, since there
wasn't ever really a reason to run it manually.
We should be able to reimplement Restore from Online Library (#1386)
using the inverse of this approach.
Closes#914
It would be better to handle this automatically in Sinon, but as it is
uploads are compressed if they're bigger than an arbitrary limit, which
can break tests unexpectedly if they check req.requestBody.
A XUL one for the current use in Advanced Search and an HTML one for
future uses. Sets the value to libraryID and adds data attributes for
editable/filesEditable on the HTML one.
f40b7ae6ac didn't help with people who've already upgraded, so check at
startup and show a warning if the profile is inaccessible until 1) the
profile has been accessed once or 2) the user checks "Don't show again"
in the warning dialog.
Also fix Zotero.Profile.getDefaultInProfilesDir() to properly throw an
error if it can't access the default directory.
In advance of #1356
We're not properly handling DOIs in parentheses or brackets (which would
require non-regex logic), so those tests are skipped for now.
Use Atom namespace when getting fields, and use `<updated>` date before
`<published>`. (The dates are also available on the nsIFeedContainer
(`feedEntry`), but we're getting them directly from the fields for some
reason.)
- Return `undefined` instead of throwing an error trying to access
`libraryTypeID` on a Zotero.Feed -- this fixes a test failure with
the latest Chai, which annoyingly runs inspect() on an object passed
to .include() regardless of whether the test succeeds
- Make some deprecated properties non-enumerable to avoid unnecessary
logging when the object is dumped
When an item is created, an active quick search is cleared, but that's
now an async operation. We weren't waiting for that, which meant that
new items weren't selected and depending on a race condition could even
show the welcome pane despite there being items in the library.
- Move identifier detection to `Zotero.Utilities.Internal.extractIdentifiers()`
so that it can be used for things other than Add Item by Identifier
(e.g., translation-server)
- Add a `Zotero.Translate.Search::setIdentifier()` function that takes an
identifier object produced by `extractIdentifiers()` (`{ DOI: "10/..." }`),
converts that to the search format expected by translators, and calls setSearch()
processDocuments() now uses an XHR 'document' request, wrapped to
provide a 'location' property, and uses promises for a simpler call
signature (though the old one will continue to work, for existing
translators). 'done' and 'exception' can now be handled via promises,
and in the translator sandbox an optional noCompleteOnError argument
instructs it not to automatically cancel the translation process with an
error (e.g., for supplementary materials).
Since we do need a hidden browser in some situations (e.g., for saving
snapshots), the old hidden-browser-based processDocuments() is still
available as Zotero.HTTP.loadDocuments().
This hopefully also fixes various problems with document property access
in translation-server.
Return a 500 for read-only libraries for all save modes. Read-only views
within editable libraries will save to the library root.
Addresses #185, RIS/BibTeX interception to read-only view behaves
differently from save button
The items will still match full-text word searches, but they won't match
phrase searches (because those require cache files for non-text
attachments) and the full-text won't sync to other computers, so they
should really be reindexed.
This should be tested, but we run tests in Firefox, and this doesn't
exist in Firefox... Easiest option is probably to add the submenu to the
Firefox menus for the purposes of testing.
If this works out I think we'll want to use this approach for
all data layer changes.
Previously, an unsaved change on an object would update its state
immediately, which was fine for synchronous code but breaks down if a
save involves multiple asynchronous calls, because modifying state after
the relevant data has been saved to the DB but before the `_changed`
object has been cleared would mean that new changes would be lost. Now,
changes are written to _changedData, and a get for the data first checks
_changedData before checking the state property (e.g., _tags) directly.
The changedData property is cleared as it's written, and once the object
is saved, the reload updates the state property with the new data.
If a standalone attachment existed in a collection and then was added to
a parent (e.g., via Create Parent Item), and attachment metadata was
also changed at the same time (e.g., due to file syncing), the
'collection item must be top level' trigger could throw on another
syncing computer. To work around this, remove collections first, then
make changes to the parentItemID columns, and then add new collections.
I think it might be worth having a tag management window that lets you
view tags as a grid, sort by column (e.g., type), select ranges, delete,
consolidate, etc., but until then, this fulfills a popular request.
And use them in new importTextAttachment() and importHTMLAttachment()
test support functions. These can be used to avoid needing a hidden
browser for determining the character set of the imported text
documents.
This is useful when trying to debug an error that only happens after a
number of other tests have run -- specify -e and run tests from either
an earlier file with -s or from the beginning.
I'm not sure if we need the browser tests at all -- we still translate
some things via hidden browsers, and I'm not sure what we have that
tests that -- but we definitely don't need to test saving before the
Zotero pane is opened.
* Use mocha, chai & sinon from the npm. As of sinon 2.0
sinon-as-promised is no longer required so it is removed
* Tweak code to re-use the same loader with the same environment
throghout the code
* Introduce browserify step for testing tools that only provide
node-compatible libraries (sinon, chai-as-promised)
* Introduce copy step for test data to resolve multiple issues with
tests depending on files not being symlinks
* Re-introduce custom implementation of setTimeout to resolve issues
with few tests
* Re-introduce custom Bluebird Promises config & monkey patch
While objects in the sync queue that fail to save should remain in the
queue, objects that just don't exist remotely need to be removed, or
else they'll be retried forever.
Instead of My Publications being a separate library, have it be a
special collection inside My Library. Top-level items can be dragged
into it as before, and child items can be toggled off and on with a
button in the item pane. Newly added child items won't be shown by
default.
For upgraders, items in the My Publications library will be moved into
My Library, which might result in their being duplicated if the items
weren't removed from My Library. The client will then upload those new
items into My Library.
The API endpoint will continue to show items in the separate My
Publications library until My Publications items are added to My
Library, so the profile page will continue to show them.
Contains a dummy doc plugin, which is useful for:
- Testing integration.js functionality
- Serving as succint documentation for development of new integration
plugins
The web library will probably still display the old tag in addition to
the new one, at least until browser restart. We'll have to deal with
that separately.
Closes#1205
When refreshing, if fewer than 100 tags to show, just create them from
scratch instead of updating the full set. Otherwise, remove the full set
from DOM and add it back in after updates to avoid reflows (from #1204).
There are various things that could be done to optimize this further
(avoiding unnecessary sorting during full refreshes, calculating a hash
of the full set and not updating it every time), but we should probably
just replace it with @tnajdek's React version first.
Closes#1204
Changes `libraryTreeView::addEventListener('load')` and similar to
`libraryTreeView::onLoad.addListener(listener, once)`, etc. `once` is an
optional boolean that, when true, causes the listener to fire once and
then be removed. This is implicit for 'load'.
'load' maintains its special behavior of running immediately if the
treeview has already been loaded.
Also adds `waitForLoad()` and `waitForSelect()` functions that return
promises on event completion, since most uses of those events were just
resolving deferreds.
When dragging an item to another library, we have to check if there's a
linked item in the target library, but items might not yet be laoded in
the other library, so item.getLinkedItem() can fail with "Item [n] not
yet loaded].
Fixing required asyncifying the follow functions:
- Zotero.Item::getLinkedItem()
- Zotero.Collection::getLinkedCollection()
- Zotero.URI.getURIItem()
- Zotero.URI.getURICollection()
- Various integration functions
As noted in 27cb099c82, import translators should be rewritten to return
a promise from doImport() and wait for promises from successive
item.complete() calls. They should then be marked as minVersion: "5.0"
to be handled properly by this new code.
(But this tries to account, albeit with somewhat worse behavior, for
translators that haven't been rewritten and sandboxes without Promise
(which is currently the case with child sandboxes in the client).)
(Oh, and I haven't tested this at all in the connectors.)
- Archive remotely missing that user chooses to keep
- Ignore archived groups that don't existing remotely
- Unarchive groups that become available again
Previously on Windows, where we don't have /bin/mv, we were recursing
into the data directory and copying files individually, which is very
slow, so automatic migration was disabled. Instead, try moving
directories with OS.File.move() with the `noCopy` flag. Moving
directories is technically unsupported by OS.File, but probably only
because of the possibility of a cross-volume copy (which is only
implemented for some platforms), and using `noCopy` hopefully prevents
that. If someone does have their data directory or storage directory on
a different volume, the migration might be quite slow, but leaving a
data directory behind in the Firefox profile directory (where it can be
easily misplaced with a seemingly unrelated Firefox reset) is worse.
Attachments are now saved before the connector server responds, because they're
no longer started out-of-band in saveItems(). This is necessary to prevent
transaction badness during imports, but it may not be what we want for the
connector, so we may want to revisit this after further testing.
E.g., moving 3,600 items to the trash now takes 4 seconds instead of 62
Instead of saving each item, update internal state and database directly
(which is more brittle but worth it). Also avoid unnecessary sorting
after removing an item from the items tree.
This reverts c6b78da69d, which changed it to expect numbers when I
noticed the type being undefined in debug output, but apparently the
only tests where the type actually mattered passed it as a name.
At some point we should just change all tests to pass as a name.
When adding many search conditions (e.g., when matching many items with the
`key` condition), the query can fail due to either the bound parameter limit or
the expression tree size limit.
To avoid this, add support for an 'inlineFilter' property on search conditions
when using the 'is' or 'isNot' operator. 'inlineFilter' is a function that
returns a quoted value suitable for direct embedding in the SQL statement, or
false if not valid. Multiple consecutive conditions for the same 'inlineFilter'
field are combined into an `IN (x, y, z)` condition.
The Firefox French language pack contains some mistakes regarding
the short form of months. As a consequence, French month parsing
didn't work.
Please note that these values aren't even the correct abbreviations
but only the three or four first letters of the correct abbreviations.
See the French CSL locale for the correct abbreviations.
Month values from the Firefox language packs are included in a
JSON file used by `Zotero.Date.getMonths()`. `getMonths(true)` includes
English months as well.
The JSON file should be bundled with the connectors as well, and
Zotero.Date.init() should be updated to populate the month data from
that.
* Mark feedItems read in a single batch SQL update
* Automatically remove old feed items
* User-facing preference globally and per-feed for feed item expiration
Improves proxy support
- Automatically detect and dehyphenise https proxies which use EZProxy
HttpsHyphens
- Web translators now pass around Zotero.Proxy instances which can
proxify/deproxify urls passed to `translate.setLocation()` before calling
`translate.getTranslators()`/ translate.detect()`. The proxy passing is
done within connector background/injected processes and between
standalone and connectors.
- Proxy protocol unified with connectors. Connectors can now pass
proxies to `/connector/save_items`. The proxies will be used to resolve
true item and attachment urls when saving.
Closeszotero/zotero#578, zotero/zotero#721
Relevant zotero/zotero#34, zotero/zotero#556
- Make Zotero.Attachments.createDirectoryForItem() delete existing
directory instead of moving it to orphaned-files; also now returns a
string path instead of an nsIFile
- Use above function during file sync instead of
_deleteExistingAttachmentFiles(), which was partly broken
- Fix throwing on errors when saving some attachment types
- Don't block the UI with a progress meter during imports. Instead, show
a popup in the bottom right when the import is done that shows how
many items were saved.
- Fix hang when importing some files
- Fix various problems with asynchronous operations/transactions
- Use the save queue for imports instead of creating concurrent
transactions that can time out
- Wait for the save to finish before returning from the translate()
promise. All save modes now use the save queue, so code that
handled the non-save-queue process can probably be removed.
- Serialize child attachments instead of running them concurrently.
This might make multi-attachment saves a little slower, since they
can't download at the same time, but it avoids problems with
concurrent transactions. We might be able to improve this to allow
concurrent downloads, or allow concurrent saves for a limited
number of items (e.g., from web saving) if not for larger imports.
- Change collection handling during import, since UI is now active
- Select the root collection at the beginning of the import
- Assign items and collections to the root during the import instead
of at the end
- Don't select other collections
- Change a few ItemSaver functions to use promises and remove
unnecessary callbacks. (This includes some connector code that needs
to be tested.)
- Change some `parentID` variables in ItemSaver to `parentItemID` for
clarity, since collections are now handled in more places
To-do:
- Save items in smaller batches instead of doing all in the same
transaction
- Show progress meter in a bottom-right popup during the import
Zotero.getString() now takes a third parameter, `num` (which should also
appear in `params`) to use when determining which plural form of the
string to use. Localized strings should include all forms in the order
specified in [1], separated by semicolons.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_and_Plurals
This reverts commit 60befe52e4 and adds a
better fix that leaves the notifier event in place. Feeds just don't
need to update after syncs during tests.
An endpoint can now take a single object containing 'method', 'pathname',
'query', 'headers', and 'data' and return an integer, an array containing
[statusCode, contentType, body], or a promise for either. This allows the
handlers to use the HTTP method and headers and removes the need for callbacks
when some handlers already use coroutine().
If init() returns a promise, it now has to use the new single-parameter
signature (because the check is done with Function.length, and combining
promises and callbacks doesn't make sense anyway).
If someone can manage to reproduce this locally, we can try a real fix:
✖ [FAIL] should show tags in alphabetical order
AssertionError: expected '' to equal 'A, B, C' at
@resource://zotero-unit-tests/noteeditorTest.js:61:4
The Zotero.DataDirectory equivalents return string paths instead of nsIFile
instances, so some of these calls now just use Zotero.File.pathToFile(), which
can be removed when the surrounding code is updated to OS.File,
Look for other profiles, from both apps (Firefox and Standalone), that
point to the data directory being migrated and update prefs.js in those
profiles to point to the new location.
Also reorganize code into Zotero.Profile and Zotero.DataDirectory
namespaces
This prevents us from moving the data directory if the other app
(Firefox or Standalone) is running from the same directory.
Also clean up stub code in migration tests
When an export translator is selected for Quick Copy, Quick Copy
initialization triggers translator initialization a few seconds after
startup, because the translator code needs to be available synchronously
for drag/drop. A Quick Copy test was changing the setting to BibTeX,
which was resulting in random timeouts after subsequent resetDB() calls
due to slow translator loading. This change skips initialization in test
mode. This might actually fix a lot of timeouts on Travis in the second
half of the tests...
This also resets the Quick Copy pref in those tests so that it's left at
the default, though really we should automatically reset all prefs after
all test groups and in resetDB().
This adds a new button to the Advanced prefs to migrate the data directory to
$HOME/Zotero. The button only appears if the data directory is set to the
default location within a profile directory (including the other program from
the one running, even though that's technically stored as a custom data
directory).
On Mac/Linux, directories within the data directory are moved with /bin/mv. On
Windows, or if that fails, they're copied recursively using OS.File.move()
(which annoyingly doesn't reliably support directory moving). The former should
be instantaneous on most systems (unless the data directory or 'storage' were
on a different filesystem from $HOME).
If the database fails to transfer, migration fails and the data directory
setting remains on the old directory. If the database transfers but other files
fail, the data directory setting is updated. In both cases, the user is
encouraged to migrate remaining files manually with a button that reveals the
directories and quits the program.
This isn't yet tested on Linux or Windows, and migration isn't yet suggested
automatically.
Adds Zotero.File.reveal(), Zotero.File.directoryIsEmpty(), and
Zotero.File.moveDirectory().
