On reset, items are overwritten with pristine versions if available and deleted
otherwise, and then the library is marked for a full sync. Unsynced/changed
files are deleted and marked for download.
Closes#1002
Todo:
- Handle API key access change (#953, in part)
- Handle 403 from data/file upload for existing users (#1041)
Previously they only showed for My Library by default, which I suspect
meant that most people didn't know you could get them for other
libraries...
This hides "Duplicate Items" and "Unfiled Items" from the context menu
when they're active, which may or may not be desirable (but we don't
show, say, "Trash" in the context menu).
Also tweaks selection behavior after hide to select next appropriate row
instead of the parent library.
If a field is open and the user right-clicks on another field (e.g., swap
names, creator type, transform text), any changed value in the open field was
lost.
Also:
- Don't show swap-names menu in single-field mode
I can't quite get programmatic access to context menus to work correctly, so
tests are disabled for now. (They work individually, but not together.)
I originally attempted this with zotero-persist and column attributes,
but there is no good way to make it succinct paramswise and the code was
painful to look at too. Thus different group settings are stored in
preferences.
Currently there are two view groups: "feed" and "default". Items view
columns have two new attributes:
`default-in` - a space separated list of views in which a column is
visible by default
`disabled-in` - a space separated list of views in which a column is
disabled by default (invisible + not possible to enable)
Fields not parsed for feeds are now disabled.
And don't skip alerts in Zotero.alert() during automated tests. (That
was intended to avoid long timeouts after unexpected failures, but,
e.g., PDF metadata lookups (which are currently disabled in automated
tests) should just be mocked so they don't intermittently fail.)
OS.File.DirectoryIterator, used by OS.File.removeDir(), isn't reliable
on Travis, returning entry.isDir == false for directories, so use
nsIFile instead
See also: 2c2a5a378
server_connectorTest - alternating port prevents
exceptions not catchable in JS for httpd server when the socket
and the port remains open after httpd.stop() callback
support - changes the window on which `setTimeout()` is ran in dialogs.
If the timeout is ran on the main window the `dialog` object appears to
lose certain properties and not respond to interactions completely.
Also changes Zotero.Item.prototype.clone() to take an `options` object for its
second parameter instead of a boolean `skipTags`. The object includes
`skipTags` as well as a new `includeCollections` property to add the new item
to the same collections.
Indexing currently happens a second after the 201 is returned to the
connector, so we have to wait for that before continuing tests, or else
a DB clear in a later test (e.g., storageLocal) will cause an error to
be logged when the indexing kicks off.
WPD code hasn't been updated in many years, and there was an issue with
document permissions in 5.0. We'll need to replace nsIWBP in Electron,
but this will do for now.
Attachments are opened using file:// URIs instead of
zotero://attachment, which is what Standalone does anyway. Ancient HTML
annotations and highlights won't be displayed anymore, but I'm not sure
they worked anyway, and it hasn't been possible to create them in years.
We might be able to write out existing annotations to notes.
iframes are skipped during saving, in an attempt to reduce the number of
junk ad files. JS can still cause problems with viewing, so we might
still want to either disable scripts or force the viewed page offline
(if such a thing is possible).
There might be issues with auxiliary filename length/characters during
cross-platform file syncing. (We modified the WPD code to shorten/clean
them.)
- Add test for recogning within a collection (follow-up from #1015)
- Update/remove some outdated code
These tests are still skipped by default, since we don't want to actually do
lookups on every test run.
If the API returns a modified item after an upload (e.g., to strip invalid
characters), don't update the Date Modified field when saving those changes to
the local version (though it would still be good to avoid API-side changes as
much as possible).
And omit in ZFS file sync requests
The API previously didn't allow these properties to be set for group items,
because they were set atomically during the file upload process, but 1) that's
not really necessary (makes a little sense for 'filename', but not really a big
deal if an old file is renamed on another computer before the new file is
synced down) and 2) skipping them results in the properties getting erased
after items are uploaded and the empty values returned by the server overwrite
the local values.
Before 5.0 we performed a regexp on new item data values to determine if
they were integers and saved them natively in SQLite if so. We no longer
do that, but setField() used strict equality when checking for changes,
so an item could be marked as changed when comparing to a new string
value (e.g., from a write response from the API, which always returns
strings). To avoid that, this converts all old values in the DB to
strings and saves all incoming values as strings automatically. (This
should also help with searching and some other things.)
Until we have a consistent way of sanitizing HTML on client and server, account
for differences manually. More differences between HTMLPurifier and TinyMCE
should be added as necessary.
- Fix upgrading of Mozilla-style attachments/storage file paths on upgrade
(requires re-upgrade)
- Save relative paths using forward slashes for consistency, and convert
to platform-appropriate slashes on use
- Fixes#994, 5.0: "+" doesn't expand all collections within a library
- If a container (library, collection) is closed directly, the open state of
all containers below it are now restored when it's reopened. Previously all
collections would be closed on a manual reopen (though they might have been
restored on the next Zotero restart).
- If "-" is pressed, all containers are closed, and reopening the library will
show only top-level collections.
- Use custom exception for user-initiated sync cancellations, which can bubble
up to the sync runner -- this should help with a sync stop button (#915)
- Separate out deletions-downloading code
- Refactor delay generator handling on library version mismatch
- Clearer variable names
Return false for single ids or skip for multiple ids. This is the original
behavior, but at some point it started throwing an UnloadedDataException. IDs
are always loaded at initialization, though, so we know whether the objects
actually exist.
This is necessary to get a library version after the write instead of an
item version. Otherwise after a full-text write, the main library
version is behind, so the next sync checks all object types for that
library instead of getting a 304.
Full text is batched up to 500K characters or 10 items, whichever is
less.
This also switches to using ?format=versions for /fulltext requests,
which isn't currently necessary but reflects what it's actually doing.
If a version is returned for an item's full-text content but a 404 is returned
for the content itself (because it's missing in Elasticsearch for some reason),
don't throw an error.
Also remove legacy array comprehensions in fulltext and syncFullTextEngine test
files, which apparently weren't being run.
1c19fe8d81 isn't sufficient for local files, because detection is run
twice, so a translator may not be available when the detect callback is
run. This changes the test to poll for the translate icon, which is a
bit of a hack but does the job.
Unfortunately this isn't perfect either, because it seems the RIS
detection sometimes just isn't run, which means that the icon never
changes and the test times out. Maybe @simonster has an idea why that's
happening.
This is necessary because you can copy a database synced with a
different account into the data directory without affecting the stored
pref.
Also tweak the text to use proper quotes and remove quaint references to
"the server".
A queue can be created and passed as an option to data layer methods, which
will then queue events on that queue instead of the main internal queue. A
queue or an array of queues can then be passed to Zotero.Notifier.commit() to
commit those events.
Some auxiliary functions don't yet take a queue, so those events will still get
run on DB transaction commit.
Sync data processing now processes notifier events in batches to reduce
repaints, even though individual objects are processed within their own
transactions (so that failures don't roll back other objects' data).
Also remove some unused notifier code
Otherwise the test could run scrapeThisPage() before translators were ready. It
would be good to make scrapeThisPage wait for detection to complete so that an
early press still uses a translator for saving, but this way tests can also
test for the proper icon (though they don't now).
Previously, objects were first downloaded and saved to the sync cache,
which was then processed separately to create/update local objects. This
meant that a server bug could result in invalid data in the sync cache
that would never be processed. Now, objects are saved as they're
downloaded and only added to the sync cache after being successfully
saved. The keys of objects that fail are added to a queue, and those
objects are refetched and retried on a backoff schedule or when a new
client version is installed (in case of a client bug or a client with
outdated data model support).
An alternative would be to save to the sync cache first and evict
objects that fail and add them to the queue, but that requires more
complicated logic, and it probably makes more sense just to buffer a few
downloads ahead so that processing is never waiting for downloads to
finish.
- Hide notes, tags and related for feed items in itembox
- Add feed support for <enclosure> elements
- Add feed syncing methods for synced settings (additional work is
needed on the sync architecture to download synced settings from the
server)
- Change feed item clear policy to be less aggressive
- Adjust for deasyncification
- Disable translate-on-select
- Closeadomasven/zotero#7, Remove context menu items from feeds
This reverts Zotero.Translate.ItemGetter.prototype.nextItem() to being
synchronous post-deasyncification. This will need to be made to work
asynchronously in the future if _attachmentToArray(), which is called by
nextItem, is changed to use async file access (which might be required
at some point).
Addresses #734, [Async DB] Import/export fails
This allows Zotero.Relations.getByPredicateAndObject()/getByObject() and
Zotero.Item::getLinkedItem()/Zotero.Collection::getLinkedCollection() to
be synchronous, which is necessary for word processor integration.
While trying to get translation and citing working with asynchronously
generated data, we realized that drag-and-drop support was going to
be...problematic. Firefox only supports synchronous methods for
providing drag data (unlike, it seems, the DataTransferItem interface
supported by Chrome), which means that we'd need to preload all relevant
data on item selection (bounded by export.quickCopy.dragLimit) and keep
the translate/cite methods synchronous (or maintain two separate
versions).
What we're trying instead is doing what I said in #518 we weren't going
to do: loading most object data on startup and leaving many more
functions synchronous. Essentially, this takes the various load*()
methods described in #518, moves them to startup, and makes them operate
on entire libraries rather than individual objects.
The obvious downside here (other than undoing much of the work of the
last many months) is that it increases startup time, potentially quite a
lot for larger libraries. On my laptop, with a 3,000-item library, this
adds about 3 seconds to startup time. I haven't yet tested with larger
libraries. But I'm hoping that we can optimize this further to reduce
that delay. Among other things, this is loading data for all libraries,
when it should be able to load data only for the library being viewed.
But this is also fundamentally just doing some SELECT queries and
storing the results, so it really shouldn't need to be that slow (though
performance may be bounded a bit here by XPCOM overhead).
If we can make this fast enough, it means that third-party plugins
should be able to remain much closer to their current designs. (Some
things, including saving, will still need to be made asynchronous.)
Unfortunately this will need to be partly redone, since retrieveItem(), and
therefore itemToCSLJSON(), and therefore itemToExportFormat(), need to be
synchronous. The item data load statements in itemToExportFormat() will
probably need to be performed earlier, when they can be async, and made
available to the session for retrieval by retrieveItem(), but I'll let someone
more familiar with the citation infrastructure do that.
This restores some code in retrieveItem() that may have been accidentally
removed in a merge, though it probably won't be useful anymore anyway.
Addresses #529
Now that 500 errors are retried in file downloads (ec28c5a3), we have to
override the default backoff schedule in order to get expected failures.
This also fixes an error that occurred on a retried download.
In Fx44, SQL queries must use '?' with LIKE and cannot concatenate a
placeholder string (e.g., 'foo%'). This is for Sqlite.jsm only, so it
doesn't affect 4.0.
For some reason the confirmEx prompt behaves differently (as a modal
window?) on Ubuntu if it's the only window open, preventing the
setTimeout() callback within support.js::waitForWindow() from being run.
(Passing -c to the test runner fixed it.) This removes a win.close() in
the middle of the file, which was probably a mistake anyway, so that the
pref window stays open until the end of the file.
Tests should make no assumptions about the presence of bundled files and
should do a full resetDB() if they need them. But most tests don't need
them, and they're very slow to install. We can reconsider this if we
drastically speed up DB resetting in tests (e.g., by caching a pristine
data directory).
Also:
- Remove last-sync-time mechanism for both WebDAV and ZFS, since it can
be determined by storage properties (mtime/md5) in data sync
- Add option to include synced storage properties in item toJSON()
instead of local file properties
- Set "Fake-Server-Match" header in setHTTPResponse() test support
function, which can be used for request count assertions -- see
resetRequestCount() and assertRequestCount() in webdavTest.js
- Allow string (e.g., 'to_download') instead of constant in
Zotero.Sync.Data.Local.setSyncState()
- Misc storage tweaks
Improves UX of sync authentication.
The account is now linked and unlinked and an API key related to
the client is generated transparently in the background.
The API key is deleted on unlinking.
No sync options are allowed before linking an account.
Closes#864
This adds a 'channel' property to Zotero.HTTP.UnexpectedStatusException,
because the 'channel' property of the XHR can be garbage-collected
before handling, and the channel's 'securityInfo' property is necessary
to detect certificate errors.
This changes Zotero.Translate.Base.translate() to take an options object (in
order to take a 'collections' parameter, which is passed to the
Zotero.Translate.ItemSaver constructor). The old parameters are still supported
with a deprecation warning, and there may be other places that still need to be
updated.
This mostly gets ZFS file syncing and file conflict resolution working
with the API sync process. WebDAV will need to be updated separately.
Known issues:
- File sync progress is temporarily gone
- File uploads can result in an unnecessary 412 loop on the next data
sync
- This causes Firefox to crash on one of my computers during tests,
which would be easier to debug if it produced a crash log.
Also:
- Adds httpd.js for use in tests when FakeXMLHttpRequest can't be used
(e.g., saveURI()).
- Adds some additional test data files for attachment tests
- Different constructor parameters
- id property for logging
- fcall() -> start()
- add() to enqueue without starting
- runAll() to run down queue and return promises for all current tasks
- wait() to wait for all running tasks to finish
