This should dramatically improve import speed for large imports by
delaying UI updates until the import finishes.
Additional Zotero.Attachments methods now support `saveOptions` to
support `notifierQueue`.
`nsIFilePicker::show()` is removed in Firefox 60 in favor of `open()`,
which takes a callback (and apparently has been preferred for a long
time).
There's no point switching to that, so this module is a version of
nsIFilePicker with an async `show()` that returns a promise and some
XPCOM-isms replaced (e.g., string paths instead of nsIFile).
StopIteration is no longer supported in Firefox 60, so instead of taking
a generator function that might throw StopIteration for the second
parameter, take a function that is passed to iterator.forEach() that
receives an OS.File.DirectoryIterator.Entry for each directory entry. If
the function returns a promise, it's waited for.
Also update other direct uses of OS.File.DirectoryIterator to remove
StopIteration use.
If "Place imported collections and items into new collection" is
unchecked, previously imported collections will be reused when they're
in the right place in the hierarchy rather than creating new ones.
Use the new wizard for all imports (even if no Mendeley DB), and add a
page with a "Place imported collections and items into new collection"
option. If deselected, collections are added to the library root.
Accept Mendeley SQLite databases via File → Import… and perform a
direct import, including collections, timestamps, notes, attachments,
and extracted annotations.
When a Mendeley database is present, File → Import… shows a wizard that
lets you choose between a file and Mendeley for the source, and choosing
the latter shows a list of available databases in the Mendeley data
directory.
Known fields that aren't valid for a type are stored in Extra.
Files in the Mendeley 'Downloaded' folder are stored. Files elsewhere
are linked.
This allows for imported items to be saved individually instead of being
saved in a batch after processing the entire imported file (which for
large imports would hang the UI, even if the actual saving was
asynchronous). This also fixes the progress meter during asynchronous
saves.
To take advantage of this, import translators will need to return a
promise when available (using the native Promise object) from doImport()
and wait for optional promises from item.complete().
The logic here can probably be streamlined further. (E.g., we might be
able to say that item.complete() always returns a promise.) It's
complicated by the fact that, at the moment, Promise isn't available in
child sandboxes, though this can probably be fixed.
Tests forthcoming, but they require a translator that supports this,
which needs to be committed separately.
View with -w for a cleaner diff.
- Don't block the UI with a progress meter during imports. Instead, show
a popup in the bottom right when the import is done that shows how
many items were saved.
- Fix hang when importing some files
- Fix various problems with asynchronous operations/transactions
- Use the save queue for imports instead of creating concurrent
transactions that can time out
- Wait for the save to finish before returning from the translate()
promise. All save modes now use the save queue, so code that
handled the non-save-queue process can probably be removed.
- Serialize child attachments instead of running them concurrently.
This might make multi-attachment saves a little slower, since they
can't download at the same time, but it avoids problems with
concurrent transactions. We might be able to improve this to allow
concurrent downloads, or allow concurrent saves for a limited
number of items (e.g., from web saving) if not for larger imports.
- Change collection handling during import, since UI is now active
- Select the root collection at the beginning of the import
- Assign items and collections to the root during the import instead
of at the end
- Don't select other collections
- Change a few ItemSaver functions to use promises and remove
unnecessary callbacks. (This includes some connector code that needs
to be tested.)
- Change some `parentID` variables in ItemSaver to `parentItemID` for
clarity, since collections are now handled in more places
To-do:
- Save items in smaller batches instead of doing all in the same
transaction
- Show progress meter in a bottom-right popup during the import
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
Still need to make the progress indicator work again. Also there may be
some performance to be gained by pooling item saves into a transaction
if one is already open.
Notifier.trigger() needs to be async, since if it actually runs it waits for
promises returned from observers. But the vast majority of trigger() calls are
in transactions where they just queue and can therefore be synchronous. This
replaces all such calls with Notifier.queue().
This should fix a race condition that was causing the emptyTrash() test to fail
intermittently.
Get rid of data_access.js, at long last. Existing calls to
Zotero.getCollections() will need to be replaced with
Zotero.Collections.getByLibrary() or .getByParent().
Also removes Zotero.Collection::getCollections(), which is redundant
with Zotero.Collections.getByLibrary(), and Zotero.Collections.add().
The latter didn't didn't include a libraryID anyway, so code might as
well just use 'new Zotero.Collection' instead.
Note that this loses conflicting changes to translate_item.js from 849803473a,
so those will need to be reapplied if applicable. /cc @aurimasv, @mtd91429
