A queue can be created and passed as an option to data layer methods, which
will then queue events on that queue instead of the main internal queue. A
queue or an array of queues can then be passed to Zotero.Notifier.commit() to
commit those events.
Some auxiliary functions don't yet take a queue, so those events will still get
run on DB transaction commit.
Sync data processing now processes notifier events in batches to reduce
repaints, even though individual objects are processed within their own
transactions (so that failures don't roll back other objects' data).
Also remove some unused notifier code
Items in a library are now loaded only when a library is clicked on and
at sync time. There might be some other areas where they need to be
loaded or where this causes problems (e.g., drag and drop, word
processor integration).
Otherwise the test could run scrapeThisPage() before translators were ready. It
would be good to make scrapeThisPage wait for detection to complete so that an
early press still uses a translator for saving, but this way tests can also
test for the proper icon (though they don't now).
We don't want auto-sync to try to run during unrelated tests.
Additionally, even though the sync process wouldn't actually start due
to the absence of an API key (which is injected directly into the sync
functions for sync tests), since auto-sync kicks off after a one-second
delay on pane open, during tests the Zotero object has often been torn
down by the time the check code runs, producing spurious errors in the
console.
Previously, objects were first downloaded and saved to the sync cache,
which was then processed separately to create/update local objects. This
meant that a server bug could result in invalid data in the sync cache
that would never be processed. Now, objects are saved as they're
downloaded and only added to the sync cache after being successfully
saved. The keys of objects that fail are added to a queue, and those
objects are refetched and retried on a backoff schedule or when a new
client version is installed (in case of a client bug or a client with
outdated data model support).
An alternative would be to save to the sync cache first and evict
objects that fail and add them to the queue, but that requires more
complicated logic, and it probably makes more sense just to buffer a few
downloads ahead so that processing is never waiting for downloads to
finish.
- Hide notes, tags and related for feed items in itembox
- Add feed support for <enclosure> elements
- Add feed syncing methods for synced settings (additional work is
needed on the sync architecture to download synced settings from the
server)
- Change feed item clear policy to be less aggressive
- Adjust for deasyncification
- Disable translate-on-select
- Closeadomasven/zotero#7, Remove context menu items from feeds
This reverts Zotero.Translate.ItemGetter.prototype.nextItem() to being
synchronous post-deasyncification. This will need to be made to work
asynchronously in the future if _attachmentToArray(), which is called by
nextItem, is changed to use async file access (which might be required
at some point).
Addresses #734, [Async DB] Import/export fails
This allows Zotero.Relations.getByPredicateAndObject()/getByObject() and
Zotero.Item::getLinkedItem()/Zotero.Collection::getLinkedCollection() to
be synchronous, which is necessary for word processor integration.
While trying to get translation and citing working with asynchronously
generated data, we realized that drag-and-drop support was going to
be...problematic. Firefox only supports synchronous methods for
providing drag data (unlike, it seems, the DataTransferItem interface
supported by Chrome), which means that we'd need to preload all relevant
data on item selection (bounded by export.quickCopy.dragLimit) and keep
the translate/cite methods synchronous (or maintain two separate
versions).
What we're trying instead is doing what I said in #518 we weren't going
to do: loading most object data on startup and leaving many more
functions synchronous. Essentially, this takes the various load*()
methods described in #518, moves them to startup, and makes them operate
on entire libraries rather than individual objects.
The obvious downside here (other than undoing much of the work of the
last many months) is that it increases startup time, potentially quite a
lot for larger libraries. On my laptop, with a 3,000-item library, this
adds about 3 seconds to startup time. I haven't yet tested with larger
libraries. But I'm hoping that we can optimize this further to reduce
that delay. Among other things, this is loading data for all libraries,
when it should be able to load data only for the library being viewed.
But this is also fundamentally just doing some SELECT queries and
storing the results, so it really shouldn't need to be that slow (though
performance may be bounded a bit here by XPCOM overhead).
If we can make this fast enough, it means that third-party plugins
should be able to remain much closer to their current designs. (Some
things, including saving, will still need to be made asynchronous.)
Unfortunately this will need to be partly redone, since retrieveItem(), and
therefore itemToCSLJSON(), and therefore itemToExportFormat(), need to be
synchronous. The item data load statements in itemToExportFormat() will
probably need to be performed earlier, when they can be async, and made
available to the session for retrieval by retrieveItem(), but I'll let someone
more familiar with the citation infrastructure do that.
This restores some code in retrieveItem() that may have been accidentally
removed in a merge, though it probably won't be useful anymore anyway.
Addresses #529
Now that 500 errors are retried in file downloads (ec28c5a3), we have to
override the default backoff schedule in order to get expected failures.
This also fixes an error that occurred on a retried download.
In Fx44, SQL queries must use '?' with LIKE and cannot concatenate a
placeholder string (e.g., 'foo%'). This is for Sqlite.jsm only, so it
doesn't affect 4.0.
Mozilla file functions (OS.File.move()/copy(),
NetUtil.asyncFetch/asyncCopy()) can leave file descriptors open for a
few seconds (even with an explicit inputStream.close() in the case of
the latter), so a source installation that copies ~500 translators and
styles (with fds for source and target) can exceed the default 1024
limit.
For some reason the confirmEx prompt behaves differently (as a modal
window?) on Ubuntu if it's the only window open, preventing the
setTimeout() callback within support.js::waitForWindow() from being run.
(Passing -c to the test runner fixed it.) This removes a win.close() in
the middle of the file, which was probably a mistake anyway, so that the
pref window stays open until the end of the file.
Tests should make no assumptions about the presence of bundled files and
should do a full resetDB() if they need them. But most tests don't need
them, and they're very slow to install. We can reconsider this if we
drastically speed up DB resetting in tests (e.g., by caching a pristine
data directory).
Also:
- Remove last-sync-time mechanism for both WebDAV and ZFS, since it can
be determined by storage properties (mtime/md5) in data sync
- Add option to include synced storage properties in item toJSON()
instead of local file properties
- Set "Fake-Server-Match" header in setHTTPResponse() test support
function, which can be used for request count assertions -- see
resetRequestCount() and assertRequestCount() in webdavTest.js
- Allow string (e.g., 'to_download') instead of constant in
Zotero.Sync.Data.Local.setSyncState()
- Misc storage tweaks
Improves UX of sync authentication.
The account is now linked and unlinked and an API key related to
the client is generated transparently in the background.
The API key is deleted on unlinking.
No sync options are allowed before linking an account.
Closes#864
This adds a 'channel' property to Zotero.HTTP.UnexpectedStatusException,
because the 'channel' property of the XHR can be garbage-collected
before handling, and the channel's 'securityInfo' property is necessary
to detect certificate errors.
This changes Zotero.Translate.Base.translate() to take an options object (in
order to take a 'collections' parameter, which is passed to the
Zotero.Translate.ItemSaver constructor). The old parameters are still supported
with a deprecation warning, and there may be other places that still need to be
updated.
This mostly gets ZFS file syncing and file conflict resolution working
with the API sync process. WebDAV will need to be updated separately.
Known issues:
- File sync progress is temporarily gone
- File uploads can result in an unnecessary 412 loop on the next data
sync
- This causes Firefox to crash on one of my computers during tests,
which would be easier to debug if it produced a crash log.
Also:
- Adds httpd.js for use in tests when FakeXMLHttpRequest can't be used
(e.g., saveURI()).
- Adds some additional test data files for attachment tests
- Different constructor parameters
- id property for logging
- fcall() -> start()
- add() to enqueue without starting
- runAll() to run down queue and return promises for all current tasks
- wait() to wait for all running tasks to finish
When called on an identified object (i.e., one with an id or
library/key), loadAllData() must be called first. When called on a new
object (which is more common anyway), fromJSON() can be called
immediately.
Absolute paths have been stored as strings on all platforms for a while,
but old Mac persistent descriptors (Base64-encoded opaque alias records)
could still exist in the DB. Additionally, relative paths for stored
files were stored as Mozilla-specific opaque strings rather than UTF-8
strings.
This adds a schema step to convert those to strings paths in the DB.
Since Mac persistent descriptors aren't converted if the file isn't
found, we still handle and (convert) old-style persistent descriptors if
necessary when reading paths from the DB.
This also moves path string handling -- converting a path to a prefixed
string for stored or base-dir-relative files -- to the
Zotero.Item#attachmentPath setter instead of save() so that reading it
back immediately returns the correct value. One consequence is that the
attachment link mode must now be set before setting the path.
Zotero.Item#getFile() is now deprecated in favor of getFilePath() and
getFilePathAsync() (which checks file existence).
Zotero.File.directoryContains() now takes string paths instead of files.
Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
This will appear much less frequently, since non-conflicting field changes on
both sides can be resolved automatically, but genuine field conflicts still
require manual conflict resolution.
The merge pane is no longer editable, since the itembox code to do that is
async and can't run in a modal window, but it's not really necessary,
particularly with conflicts happening less frequently.
TODO:
- Remote item deletions
- File conflicts
- Maybe handle some edge cases where the conflicted items fail to save
Save uploaded data to cache, and update local object if necessary (which
it mostly shouldn't be except for invalid characters and HTML filtering
in notes)
Also add some upload and JSON tests
There's a lot more to do, and this isn't ready for actual usage, but the
basic functionality is mostly in place and has decent test coverage. It
can successfully upgrade a library last used with classic syncing and
pull down changes via the API. Uploading mostly works but is currently
disabled for safety until it has better test coverage.
Downloaded JSON is first saved to a cache table, which is then used to
populate other tables and later for generating PATCH requests and
automatically resolving conflicts (since it shows what was changed
locally and what was changed remotely). Objects with unmet dependencies
or unknown fields are skipped for now but don't block the rest of the
sync.
Some of the bigger remaining to-dos:
- Tests for uploading
- Re-do the preferences to get an API key
- File sync integration
- Full-text syncing integration
- Manual conflict resolution (though this already includes much smarter
conflict handling that automatically resolves many conflicts)
And use it in resetDB() test support function, mainly to allow
skipBundledFiles for resetDB calls. Translator installation and
initialization can take a long time, but tests that need a clean DB
don't necessarily rely on translators. Without this, running resetDB()
in beforeEach() for many tests is prohibitively slow.
Previously, if .synced was already true, setting it to true and saving
would result in .synced == false unless skipSyncedUpdate was passed. Now
the value assigned to .synced is always used on the next save. If the
value hasn't changed and no other values have changed, a save will be a
no-op.
The default items cause problems with conflict resolution for existing
users (and not syncing them or ignoring conflicts for them is kind of
weird), and they require remote changes for new databases. I do like
there not being a completely empty library, but I think it's probably
better just to display a virtual welcome message with a link to the
Quick Start Guide somewhere else, such as in the right-hand pane. (A new
installation also opens the start page on zotero.org.)
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
- Simplified schema
- Tags are now added without reloading entire tag selector
- On my system, adding 400 tags to an item (separately, with the tag
selector updating each time) went from 59 seconds to 42. (Given that
it takes only 13 seconds with the tag selector closed, though,
there's clearly more work to be done.)
- Tag selector now uses HTML flexbox (in identical fashion, for now, but
with the possibility of fancier changes later, and with streamlined
logic thanks to the flexbox 'order' property)
- Various async fixes
- Tests
Still need to make the progress indicator work again. Also there may be
some performance to be gained by pooling item saves into a transaction
if one is already open.
Restore prepopulated charset table, but this time with just the
encodings from the WHATWG Encoding Standard. Assigning a charset to
Zotero.Item::attachmentCharset runs the value through
Zotero.CharacterSets.toCanonical() automatically.
This migrates attachment charsets to the new canonical values, clearing any
that are unsupported.
Other legacy mappings could still be added back, as disussed in #760.
This uses ISO 8601 dates for generateAllTypesAndFieldsData (and
changes populateDBWithSampleData to use Item#fromJSON), and makes
translators expect ISO 8601 accessDates, although SQL accessDates are
still supported with a deprecation warning. Canonicalization happens in
Zotero.Translate, so I need to remember to update connectors as well.
And add group.fromJSON(json, userID), which sets editable and
filesEditable properties based on the group JSON (libraryReading, role
lists, etc.) and the given user
Also adds Zotero.DataObjects::getLoaded(), which returns an array of all
loaded objects of the given type. This is useful for selective
reloading, for example with item.reload(['relations'], true).
Waits for an alert or confirmation dialog to open and closes it
automatically, optionally after running onOpen(dialog) to check its
contents (e.g., with dialog.document.documentElement.textContent) and
optionally clicking a button other than 'accept' (e.g., 'cancel',
extra1').
Supports delayed accept buttons
In particular, 0 is kept as a value, and passing undefined to setField
now throws an error.
I'm not sure if we actually want to return an empty string in all cases
for missing/invalid fields, but that's what we do currently.
And simplify tree view load event handling, which may or may not have
been contributing to intermittent test failures, but is cleaner this way
regardless.
Relations are now properties of collections and items rather than
first-class objects, stored in separate collectionRelations and
itemRelations tables with ids for subjects, with foreign keys to the
associated data objects.
Related items now use dc:relation relations rather than a separate table
(among other reasons, because API syncing won't necessarily sync both
items at the same time, so they can't be stored by id).
The UI assigns related-item relations bidirectionally, and checks for
related-item and linked-object relations are done unidirectionally by
default.
dc:isReplacedBy is now dc:replaces, so that the subject is an existing
object, and the predicate is now named
Zotero.Attachments.replacedItemPredicate.
Some additional work is still needed, notably around following
replaced-item relations, and migration needs to be tested more fully,
but this seems to mostly work.
getGroup() can be used to access a default group library for general
group tests. createGroup() can be used to create one for a particular
test or set of tests.
The test runner now downloads and caches the PDF tools for the current
platform within the test data directory and only redownloads them when
out of date, and it updates the download URL so that the full-text code
pulls from the cache directory via a file:// URL.
The installPDFTools() support function now installs the files directly
instead of going through the prefs, and a new uninstallPDFTools()
function removes the tools. Since the presence of the PDF tools can
affect other tests, tests that need the tools should install them in a
before() and uninstall them in an after(), leaving most tests to run
without PDF indexing.
This also adds a callback to the waitForWindow() support function. If a
modal dialog is opened, it blocks the next promise handler from running,
so a callback has to be used to interact with and close the dialog
immediately.
These previously returned an itemID, but now that new saved items can be edited
without a refetch, they should just return the new item.
(Possible I missed a few spots where these are called.)
(Mocha has a 'bail' config flag that's supposed to do this, but it doesn't seem
to work when passed to mocha.setup() (maybe because we're setting a custom fail
handler?), so this just calls abort() on the runner manually.)
This is arbitrary, and we could increase it more or make it configurable via
the command line if Travis continue to time out, but this allows all tests to
complete for me in a VM.
Not crazy about this, but (at least on my system) it's an easy way to
avoid DB errors due to interrupted transaction or query errors after the
DB connection was cleaned up. (I can reproduce those pretty reliably
right now by running collectionTreeView tests alone.)
