This mostly gets ZFS file syncing and file conflict resolution working
with the API sync process. WebDAV will need to be updated separately.
Known issues:
- File sync progress is temporarily gone
- File uploads can result in an unnecessary 412 loop on the next data
sync
- This causes Firefox to crash on one of my computers during tests,
which would be easier to debug if it produced a crash log.
Also:
- Adds httpd.js for use in tests when FakeXMLHttpRequest can't be used
(e.g., saveURI()).
- Adds some additional test data files for attachment tests
When called on an identified object (i.e., one with an id or
library/key), loadAllData() must be called first. When called on a new
object (which is more common anyway), fromJSON() can be called
immediately.
If Standalone is closed in full-screen mode in 10.11, 'sizemode' on the main
window is persisted as 'fullscreen'. On reopen, the window doesn't go into
full-screen mode, but it still lacks menubar or titlebar. This patch forces the
window into 'normal' mode if it was left in 'fullscreen'.
(Firefox appears to ignore 'fullscreen' without actually changing the value,
but I didn't find the code that handles that.)
Absolute paths have been stored as strings on all platforms for a while,
but old Mac persistent descriptors (Base64-encoded opaque alias records)
could still exist in the DB. Additionally, relative paths for stored
files were stored as Mozilla-specific opaque strings rather than UTF-8
strings.
This adds a schema step to convert those to strings paths in the DB.
Since Mac persistent descriptors aren't converted if the file isn't
found, we still handle and (convert) old-style persistent descriptors if
necessary when reading paths from the DB.
This also moves path string handling -- converting a path to a prefixed
string for stored or base-dir-relative files -- to the
Zotero.Item#attachmentPath setter instead of save() so that reading it
back immediately returns the correct value. One consequence is that the
attachment link mode must now be set before setting the path.
Zotero.Item#getFile() is now deprecated in favor of getFilePath() and
getFilePathAsync() (which checks file existence).
Zotero.File.directoryContains() now takes string paths instead of files.
In Firefox 41, file: URIs can no longer be loaded (at least via Image)
from the hidden window on Windows/Linux, but chrome: URIs still work.
Not sure why I was using a file: URI to begin with.
If multiple collections are highlighted and none are in view, scroll to
the first one in the list.
The logic could be improved here a little more to scroll to the closest
collection instead of the first one, and also to scroll to a few rows
above or below the target.
This also fixes what was probably an incorrect highlight if there were
multiple collections and some had to be expanded first.
Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
Firefox no longer supports transparent windows because it made them do
extra preprocessing on a file to restore the drop shadow when the
Developer Edition theme was active:
https://bugzilla.mozilla.org/show_bug.cgi?id=1162649
This will appear much less frequently, since non-conflicting field changes on
both sides can be resolved automatically, but genuine field conflicts still
require manual conflict resolution.
The merge pane is no longer editable, since the itembox code to do that is
async and can't run in a modal window, but it's not really necessary,
particularly with conflicts happening less frequently.
TODO:
- Remote item deletions
- File conflicts
- Maybe handle some edge cases where the conflicted items fail to save
Save uploaded data to cache, and update local object if necessary (which
it mostly shouldn't be except for invalid characters and HTML filtering
in notes)
Also add some upload and JSON tests
If a file existed locally but somehow ended up marked as to-download without
existing on the server, it was never uploaded. I'm not sure when this can
happen, but I saw it while messing around. Maybe switching between ZFS and
WebDAV?
This will still only check and upload if there's another computer syncing files
to the same library, but we'll check all files in 5.0.
There's a lot more to do, and this isn't ready for actual usage, but the
basic functionality is mostly in place and has decent test coverage. It
can successfully upgrade a library last used with classic syncing and
pull down changes via the API. Uploading mostly works but is currently
disabled for safety until it has better test coverage.
Downloaded JSON is first saved to a cache table, which is then used to
populate other tables and later for generating PATCH requests and
automatically resolving conflicts (since it shows what was changed
locally and what was changed remotely). Objects with unmet dependencies
or unknown fields are skipped for now but don't block the rest of the
sync.
Some of the bigger remaining to-dos:
- Tests for uploading
- Re-do the preferences to get an API key
- File sync integration
- Full-text syncing integration
- Manual conflict resolution (though this already includes much smarter
conflict handling that automatically resolves many conflicts)
Previously, if .synced was already true, setting it to true and saving
would result in .synced == false unless skipSyncedUpdate was passed. Now
the value assigned to .synced is always used on the next save. If the
value hasn't changed and no other values have changed, a save will be a
no-op.
The default items cause problems with conflict resolution for existing
users (and not syncing them or ignoring conflicts for them is kind of
weird), and they require remote changes for new databases. I do like
there not being a completely empty library, but I think it's probably
better just to display a virtual welcome message with a link to the
Quick Start Guide somewhere else, such as in the right-hand pane. (A new
installation also opens the start page on zotero.org.)
Allow saving of CSS links over chrome://. We can scope this to
chrome://global/skin/aboutReader.css if anyone thinks of a reason why
this was disabled to begin with, but I'm not sure in what other
situations CSS it would apply.
- Restore About Zotero item
- Fix Quit item
- Close Standalone when Standalone window is closed (we did this before,
but apparently having our own hidden window breaks that mechanism)
Technically rather than waiting for the schema update we should wait for
translator initialization, which should wait for the schema update
itself, but the schema update also needs to initialize the translators,
so avoiding a hang is tricky, particularly with the use of Zotero.lazy()
for Zotero.Translators.init(). For now, just wait for the schema update.
Clicking the separate buttons would trigger actions in the first window
opened, and other things might not have been updating properly across
multiple windows.
Previously, clicking directly on a noautohide guidance panel closed it, even if
there was a forward nav button. (In the case of the main and save button
guidance panels, the save button would appear on the next Firefox restart, but
this ensures that the save guidance panel will be shown to new users even if
they click on the panel instead of the not-particularly-noticeable forward
button.)
In practice, this appears to be both unnecessary and harmful. Although
the first write happens effectively instantaneously, we can take
several hundred milliseconds to read the second write, and Standalone
can decide we are dead.
This apparently fixes#783 for @dstillman, although I was never able to
reproduce locally.
(I don't see any obvious reason I can't do this. We should probably clean this
up later, not call the endpoint "saveSnapshot", etc. Also, is there a reason it
doesn't follow the snapshot pref by default?)
And fix save button hover height on Windows, which somehow got out of
whack again.
Might need further tweaking on Linux
Need to find a better way to do these buttons...
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
For new installations, a panel will show for the Z icon with a ">"
button to go to a second panel for the save icon. For existing
installations, a panel will show just for the save icon. In both cases,
the panels are non-auto-hiding but are dismissed if clicked on or when a
new content page is loaded.
There's also a new close button, but it's not enabled currently. Might
be useful for panels with embedded links where an accidental click on
the panel alone shouldn't close the panel (like the Firefox customize
mode wizard).
- When relinking a missing stored file, copy it into the attachment's
storage directory automatically
- Previously, selecting a file outside the attachment subdir would
just result in a missing attachment, since it only looks for stored
files within the subdir
- Display an error message if a Windows shortcut (.lnk) is added via
drag-and-drop or via a file dialog on non-Windows systems, until we
can figure out how to determine the original file
- Shortcuts can cause errors during syncing, for unclear reasons
- Neither nsIFile::copyToFollowingLinks() nor nsIFile::target work for
me to get the original file, even when nsIFile::isSymlink() returns
true
- Windows file dialogs seem to automatically resolve shortcuts, so
it's only an issue there for drag-and-drop
- Disallow hidden files from being selected in relink dialog
- I think some people on Windows with hidden files shown relink the
.zotero* files that show up when they click Locate, which causes
file sync errors. Which brings us to...
- Fix file sync errors for *.lnk and .zotero* files
- Ignore existing .zotero* attachment files, treating the files as
missing instead to encourage relinking
- Strip leading period in getValidFileName() to prevent added files from
being hidden
- This allows hidden files to be added explicitly; they just won't
stay that way in the storage directory
(These things should have tests, but that will have to happen on the 5.0
branch.)
- Simplified schema
- Tags are now added without reloading entire tag selector
- On my system, adding 400 tags to an item (separately, with the tag
selector updating each time) went from 59 seconds to 42. (Given that
it takes only 13 seconds with the tag selector closed, though,
there's clearly more work to be done.)
- Tag selector now uses HTML flexbox (in identical fashion, for now, but
with the possibility of fancier changes later, and with streamlined
logic thanks to the flexbox 'order' property)
- Various async fixes
- Tests
To be used by updated deletion listener in new sync code
Also adds explicit Zotero.SyncedSettings.clear() in place of
Zotero.SyncSettings.set() without a value
Still need to make the progress indicator work again. Also there may be
some performance to be gained by pooling item saves into a transaction
if one is already open.
Restore prepopulated charset table, but this time with just the
encodings from the WHATWG Encoding Standard. Assigning a charset to
Zotero.Item::attachmentCharset runs the value through
Zotero.CharacterSets.toCanonical() automatically.
This migrates attachment charsets to the new canonical values, clearing any
that are unsupported.
Other legacy mappings could still be added back, as disussed in #760.
This output is basically only useful when something hangs, so it can
stay off all other times.
For now, 6 can be the new only-use-when-something-is-actively-broken
level. At some point we may want to move DB activity to 4 and make this
sort of thing 5, because we don't have much that's 4 right now.
This uses ISO 8601 dates for generateAllTypesAndFieldsData (and
changes populateDBWithSampleData to use Item#fromJSON), and makes
translators expect ISO 8601 accessDates, although SQL accessDates are
still supported with a deprecation warning. Canonicalization happens in
Zotero.Translate, so I need to remember to update connectors as well.
And add group.fromJSON(json, userID), which sets editable and
filesEditable properties based on the group JSON (libraryReading, role
lists, etc.) and the given user
sinon.useFakeXMLHttpRequest() doesn't work in our case, but if
Zotero.HTTP.mock is set, Zotero.HTTP.request() will create new instances of
that object instead of the built-in XMLHttpRequest, so it can be set to
FakeXMLHttpRequest:
Zotero.HTTP.mock = sinon.FakeXMLHttpRequest;
var server = sinon.fakeServer.create();
server.autoRespond = true;
server.respondWith("GET", "/users/1",
[200, {"Content-Type": "application/json"}, '{"userID": 1}']);
var userInfo = yield getUser();
Zotero.HTTP.mock = null;
server.restore(); // probably not necessary
Also adds Zotero.DataObjects::getLoaded(), which returns an array of all
loaded objects of the given type. This is useful for selective
reloading, for example with item.reload(['relations'], true).
In particular, 0 is kept as a value, and passing undefined to setField
now throws an error.
I'm not sure if we actually want to return an empty string in all cases
for missing/invalid fields, but that's what we do currently.
And simplify tree view load event handling, which may or may not have
been contributing to intermittent test failures, but is cleaner this way
regardless.
Relations are now properties of collections and items rather than
first-class objects, stored in separate collectionRelations and
itemRelations tables with ids for subjects, with foreign keys to the
associated data objects.
Related items now use dc:relation relations rather than a separate table
(among other reasons, because API syncing won't necessarily sync both
items at the same time, so they can't be stored by id).
The UI assigns related-item relations bidirectionally, and checks for
related-item and linked-object relations are done unidirectionally by
default.
dc:isReplacedBy is now dc:replaces, so that the subject is an existing
object, and the predicate is now named
Zotero.Attachments.replacedItemPredicate.
Some additional work is still needed, notably around following
replaced-item relations, and migration needs to be tested more fully,
but this seems to mostly work.
The test runner now downloads and caches the PDF tools for the current
platform within the test data directory and only redownloads them when
out of date, and it updates the download URL so that the full-text code
pulls from the cache directory via a file:// URL.
The installPDFTools() support function now installs the files directly
instead of going through the prefs, and a new uninstallPDFTools()
function removes the tools. Since the presence of the PDF tools can
affect other tests, tests that need the tools should install them in a
before() and uninstall them in an after(), leaving most tests to run
without PDF indexing.
This also adds a callback to the waitForWindow() support function. If a
modal dialog is opened, it blocks the next promise handler from running,
so a callback has to be used to interact with and close the dialog
immediately.
Character sets are now populated on demand, so they can't be run through
Zotero.CharacterSets.getName() in getContentsAsync(), since they might
not exist. (I'm also not sure why this was being done anyway.)
For now, this just calls Zotero.Utilities.generateObjectKey(), but this
function makes more sense in DataObjectUtilities. It does need to be
accessible to the connectors, but if it's possible to add an alias in
Zotero.Utilities just for the connectors, it'd probably be better to do
that and use Zotero.DataObjectUtilities.generateKey() elsewhere.
These previously returned an itemID, but now that new saved items can be edited
without a refetch, they should just return the new item.
(Possible I missed a few spots where these are called.)
Notifier.trigger() needs to be async, since if it actually runs it waits for
promises returned from observers. But the vast majority of trigger() calls are
in transactions where they just queue and can therefore be synchronous. This
replaces all such calls with Notifier.queue().
This should fix a race condition that was causing the emptyTrash() test to fail
intermittently.
citeproc-js relies on this in several locations. Seems that Zotero passes these IDs to citeproc from the item picker. We also need to consider existing embedded items in Word/LO documents, but they do have embedded URIs, so it shouldn't be a problem.
CC @fbennett
* Enable legacy mode for export translators compatible with pre-4.0.27:
* Add compatibility mappings, so that current translators don't break if they specify minVersion lower than 4.0.27. This does introduce non-compatible changes, specifically, "version" field in legacy mode is "versionNumber" in the new format. "version" in the new format corresponds to the "version" as specified for Zotero API JSON format. New translators should expect Zotero web API JSON format and should specify minVersion 4.0.27.
* Update CSL mappings to comply with new itemToExportFormat
* CSL JSON export translator needs to be updated to be compatible with 4.0.27 to export correct CSL JSON
* Use item URI for id in CSL JSON instead of item ID
* Fix note and attachment handling in itemToCSLJSON
After saving a new object and reloading primary data and any changed
data (which we can maybe reconsider at some point), mark all other data
types as loaded, since there's no other data we don't have. For example,
this allows for item.save() to be followed by item.setField() without
needing to call item.loadItemData() first.
updateBundledStyles() already has the contents of a new/updated
translator when it copies it into the data dir, so there's no need for
Zotero.Translators.reinit() to read it again from the just-copied file.
This passes the metadata from updateBundledStyles() to reinit() to avoid
the extra file read.
(Alas, this appears to make essentially zero difference on an OS X
system with an SSD, but maybe it will help elsewhere.)
Get rid of data_access.js, at long last. Existing calls to
Zotero.getCollections() will need to be replaced with
Zotero.Collections.getByLibrary() or .getByParent().
Also removes Zotero.Collection::getCollections(), which is redundant
with Zotero.Collections.getByLibrary(), and Zotero.Collections.add().
The latter didn't didn't include a libraryID anyway, so code might as
well just use 'new Zotero.Collection' instead.
If a view or other resources are destroyed while a promise is being
resolved, subsequent code can fail. This is generally harmless, but it
results in unnecessary errors being logged to the console.
To address this, promises can use a new function,
Zotero.Promise.check(), to test whether a value is truthy or 0 and
automatically throw a specific error that's ignored by the unhandled
rejection handler if not.
Example usage:
getAsync().tap(() => Zotero.Promise.check(this.win));
If this.win is cleaned up while getAsync() is being resolved, subsequent
lines won't be run, and nothing will be logged to the console.
Change all attachment functions to take parameter objects, including a
'collections' property to assign collections. (Previously, calling code
assigned collections separately, which required a nested transaction,
which is no longer possible.)
Fixes#723, Can't attach files by dragging
Wait for the pane's collectionSelected() to finish before returning from
collectionTreeView select methods (e.g., selectLibrary()), and wait for
previous items view to finish loading before creating a new one in
collectionSelected(). This ensures that the items view has been created (though
not loaded) before returning from a select. The tree can still get a bit
confused switching between collections, but I think we're getting closer to
fixing that.
Also switch the items tree to use the same pattern.
This also fixes dragging items to collections (#731).
Groups were already being loaded for the collections list, so we might
as well just store them initially and let Zotero.Libraries.getName() be
a synchronous call.
- Moved ::_get() and _set() from Collection/Search into DataObject, and
disabled in Item
- Don't disable new items after save. We now put new objects into the
DataObjects cache from save() so that changes made post-save are
picked up by other code using .get().
- Added 'skipCache' save() option to avoid reloading data on new objects
and adding them to the cache. (This will be used in syncing, where
objects might be in another library where they're not needed right
away.) Objects created with this option are instead disabled to
prevent reuse.
- Modified some tests to try to make sure we're reloading everything properly
after a save.
- Documented save() options
- Fixes some saving and erasing issues with collections and searches
- Adds Zotero.DataObject::eraseTx() to automatically start transaction,
and have .erase() log a warning like .save()
- Adds createUnsavedDataObject() and createDataObject() helper functions
for tests
The sync cache will have pristine copies of the existing versions of
local objects for better conflict resolution, but downloads will get
saved to the sync cache first before processing, so the cache needs to
be able to hold more than one version.
Replace Z.DataObjects::diff() with Z.DataObjectUtilities.diff(). Instead
of just returning two objects with the differing fields, the new diff()
generates a changeset with operations to apply with applyChanges(),
including at the array member level for collections and tags. This,
combined with cached pristine copies of objects, will allow for vastly
better conflict resolution, with automatic merging of non-conflicting
changes.
Creators currently don't show granular changes, and ordering might make
it too tough to do so. Relations diffing isn't yet implemented.
