.contains() was removed in Firefox 48, but .includes() wasn't available until
40, so use indexOf() for now. We can start using .contains() once we no longer
need to support 38 ESR.
A queue can be created and passed as an option to data layer methods, which
will then queue events on that queue instead of the main internal queue. A
queue or an array of queues can then be passed to Zotero.Notifier.commit() to
commit those events.
Some auxiliary functions don't yet take a queue, so those events will still get
run on DB transaction commit.
Sync data processing now processes notifier events in batches to reduce
repaints, even though individual objects are processed within their own
transactions (so that failures don't roll back other objects' data).
Also remove some unused notifier code
Instead of getting batches of unused primary key ids, even if they're lower
than other ids, which for some reason seemed like a good idea in 2008, just do
a `MAX()` on the table at startup and return the next available id on each call
to `Zotero.ID.get()`. This is much simpler, and not reusing ids allows them to
be used as a chronological sort field.
While SQLite's `SELECT last_insert_rowid()` could return auto-increment values,
it's unsafe with async DB access, since a second `INSERT` can come in before
the first `last_insert_rowid()` is called. This is true even in a transaction
unless a function that calls it is never called in parallel (e.g., with
`Zotero.Promise.all()`, which can be faster than sequential `yield`s).
Note that the next id is always initialized as MAX() + 1, so if an object is
added and then deleted, after a restart the same id will be given. (This is
equivalent to (though unrelated to) SQLite's `INTEGER PRIMARY KEY` behavior,
as opposed to its `INTEGER PRIMARY KEY AUTOINCREMENT` behavior.)
Closes#993, Feed items out of order
Previously, objects were first downloaded and saved to the sync cache,
which was then processed separately to create/update local objects. This
meant that a server bug could result in invalid data in the sync cache
that would never be processed. Now, objects are saved as they're
downloaded and only added to the sync cache after being successfully
saved. The keys of objects that fail are added to a queue, and those
objects are refetched and retried on a backoff schedule or when a new
client version is installed (in case of a client bug or a client with
outdated data model support).
An alternative would be to save to the sync cache first and evict
objects that fail and add them to the queue, but that requires more
complicated logic, and it probably makes more sense just to buffer a few
downloads ahead so that processing is never waiting for downloads to
finish.
While trying to get translation and citing working with asynchronously
generated data, we realized that drag-and-drop support was going to
be...problematic. Firefox only supports synchronous methods for
providing drag data (unlike, it seems, the DataTransferItem interface
supported by Chrome), which means that we'd need to preload all relevant
data on item selection (bounded by export.quickCopy.dragLimit) and keep
the translate/cite methods synchronous (or maintain two separate
versions).
What we're trying instead is doing what I said in #518 we weren't going
to do: loading most object data on startup and leaving many more
functions synchronous. Essentially, this takes the various load*()
methods described in #518, moves them to startup, and makes them operate
on entire libraries rather than individual objects.
The obvious downside here (other than undoing much of the work of the
last many months) is that it increases startup time, potentially quite a
lot for larger libraries. On my laptop, with a 3,000-item library, this
adds about 3 seconds to startup time. I haven't yet tested with larger
libraries. But I'm hoping that we can optimize this further to reduce
that delay. Among other things, this is loading data for all libraries,
when it should be able to load data only for the library being viewed.
But this is also fundamentally just doing some SELECT queries and
storing the results, so it really shouldn't need to be that slow (though
performance may be bounded a bit here by XPCOM overhead).
If we can make this fast enough, it means that third-party plugins
should be able to remain much closer to their current designs. (Some
things, including saving, will still need to be made asynchronous.)
When called on an identified object (i.e., one with an id or
library/key), loadAllData() must be called first. When called on a new
object (which is more common anyway), fromJSON() can be called
immediately.
Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
Save uploaded data to cache, and update local object if necessary (which
it mostly shouldn't be except for invalid characters and HTML filtering
in notes)
Also add some upload and JSON tests
Notifier.trigger() needs to be async, since if it actually runs it waits for
promises returned from observers. But the vast majority of trigger() calls are
in transactions where they just queue and can therefore be synchronous. This
replaces all such calls with Notifier.queue().
This should fix a race condition that was causing the emptyTrash() test to fail
intermittently.
After saving a new object and reloading primary data and any changed
data (which we can maybe reconsider at some point), mark all other data
types as loaded, since there's no other data we don't have. For example,
this allows for item.save() to be followed by item.setField() without
needing to call item.loadItemData() first.
- Moved ::_get() and _set() from Collection/Search into DataObject, and
disabled in Item
- Don't disable new items after save. We now put new objects into the
DataObjects cache from save() so that changes made post-save are
picked up by other code using .get().
- Added 'skipCache' save() option to avoid reloading data on new objects
and adding them to the cache. (This will be used in syncing, where
objects might be in another library where they're not needed right
away.) Objects created with this option are instead disabled to
prevent reuse.
- Modified some tests to try to make sure we're reloading everything properly
after a save.
- Documented save() options
- Fixes some saving and erasing issues with collections and searches
- Adds Zotero.DataObject::eraseTx() to automatically start transaction,
and have .erase() log a warning like .save()
- Adds createUnsavedDataObject() and createDataObject() helper functions
for tests
- Add an 'exclusive' option to transactions that causes them to block other
transactions and wait for other transactions to finish before starting,
instead of nesting
- Resolve Zotero.DB.waitForTransaction() promise before returning from
executeTransaction()
- A side effect of the above: wait for a newly created item to be selected in
the middle pane and rendered in the right-hand pane before returning from
executeTransaction()
- Don't save items multiple times when adding/removing a non-final creator in
the Info pane
- Use a simpler, non-recursive method for focusing the next field in the Info
pane; this prevents "too much recursion" errors if something causes the
right-hand pane not to be rendered when expected
Show a wizard after items are dragged to My Publications choosing
whether to include files and notes and choosing sharing settings for the
items. Sharing options are Creative Commons licenses, CC0, "All rights
reserved", or keeping the existing Rights field if available.
Also blocks collections, searches, linked file attachments, and
top-level attachments/notes from My Publications at the data layer, but
not yet from the UI in all places (so it can crash if you try).
Todo:
- Block certain UI actions with nice messages
- Show a nice scrollable list of items in the wizard to allow selecting
specific files/notes, instead of just having checkboxes for files and
notes that apply to all dragged items
- Show an explanation of My Publications in the right-hand pane when no
items are selected
- Maybe adjust handling when no attached files/notes, since it might be
a bit alarming at the moment to see sharing options for metadata
entries
Object contains 'libraryID' and 'key' properties
This is due to changed array destructuring behavior in Firefox. Previously,
`var [foo, bar] = maybeArrayMaybeFalse()` always worked, leaving foo and bar
undefined if the function returned false. Now (with ES6, I assume), if the
function returns false it results in a "false[Symbol.iterator] is not a
function" error. But `var {libraryID, key} = false` works as expected, leaving
both values undefined, so instead we can just return an object with those
properties from getLibraryAndKeyFromID(). To assign to different variables, use
`var {libraryID, key: parentItemKey} = ...`.
Search condition ids are now indexed from 0, and always saved
contiguously (no more 'fixGaps' option), since they're just in an array
in the API. (They're still returned as an object from
Zotero.Search.prototype.getConditions() because it's easier for the
advanced search window to not have to deal with shifting ids between
saves.)
Also:
- Return an object from `Zotero.Search.prototype.getConditions()`
instead of an array.
- Add support function `getPromiseError(promise)` to return the error
thrown from a chain of promises, or false if none. (We could make an
`assert.throwsAsync()`, but this allows testing of various properties
such as `.name`, which even the built-in `assert.throws()` can't
test.)
- Clarify some search save errors
