Improves UX of sync authentication.
The account is now linked and unlinked and an API key related to
the client is generated transparently in the background.
The API key is deleted on unlinking.
No sync options are allowed before linking an account.
Display a warning when choosing data directory
Or on opening Zotero Pane for existing users with data dir in dropbox
Also:
Fix a bug where it won't use custom path if "Choose..." button is
pressed instead of radio button.
Change filepicker to show current data directory on display
There's a lot more to do, and this isn't ready for actual usage, but the
basic functionality is mostly in place and has decent test coverage. It
can successfully upgrade a library last used with classic syncing and
pull down changes via the API. Uploading mostly works but is currently
disabled for safety until it has better test coverage.
Downloaded JSON is first saved to a cache table, which is then used to
populate other tables and later for generating PATCH requests and
automatically resolving conflicts (since it shows what was changed
locally and what was changed remotely). Objects with unmet dependencies
or unknown fields are skipped for now but don't block the rest of the
sync.
Some of the bigger remaining to-dos:
- Tests for uploading
- Re-do the preferences to get an API key
- File sync integration
- Full-text syncing integration
- Manual conflict resolution (though this already includes much smarter
conflict handling that automatically resolves many conflicts)
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
The test runner now downloads and caches the PDF tools for the current
platform within the test data directory and only redownloads them when
out of date, and it updates the download URL so that the full-text code
pulls from the cache directory via a file:// URL.
The installPDFTools() support function now installs the files directly
instead of going through the prefs, and a new uninstallPDFTools()
function removes the tools. Since the presence of the PDF tools can
affect other tests, tests that need the tools should install them in a
before() and uninstall them in an after(), leaving most tests to run
without PDF indexing.
This also adds a callback to the waitForWindow() support function. If a
modal dialog is opened, it blocks the next promise handler from running,
so a callback has to be used to interact with and close the dialog
immediately.
To deal with issues running .vbs scripts on some systems. If the 3.02
binaries haven't been reinstalled since this version, reinstall them as
3.02a automatically or on manual upgrade to fix corrupted binaries from
previous gzip issue.
Other platforms unchanged
Use stdout redirection scripts for pdfinfo and, on Windows, a script to
run pdftotext hidden, which together allow for all unmodified binaries
(including, probably, symlinked system ones, though I didn't test that).
On Windows, using a .vbs does cause a brief wait cursor. The stock
pdfinfo needs the redirection script anyway, so that's unavoidable, but
on the async branch I think we'll be able to switch to pdf.js for the
page count, at which point maybe I'll try to remember how I modified the
Windows binaries to be hidden and use a modified version of pdftotext to
avoid VBScript. (We use the stock pdftotext elsewhere already.)
If automatic translator/style updates are enabled, at least one of the
PDF tools is installed, and the repo returns a more recent version
number than what's installed, automatically upgrade the tools. (Version
3.02 counts as lower, since Poppler's version numbers are lower.)
If an error occurs, wait increasing amounts of time to try the downloads
again, up to one week.
- Use async DB and OS.File for bundled file updates
- Remove support for translator/style ZIP files -- the two options are
now non-unpacked XPIs with subfolders or unpacked source installations
- Now that we have async file access, don't store translator code in
database cache -- just store metadata so that it's available without
reading each translator file
- Change the (previously partially asyncified) Zotero.Styles/Translators
APIs a bit -- while the getAll/getVisible methods are asynchronous and
will wait for loading, the get() methods are synchronous and require
styles/translators to be initialized before they're called. Most
places that end up calling get() probably call getAll/getVisible first
and should therefore be async, but if there's any way to trigger a
get() first, that will need to be adjusted.
- Asyncify various other style/translator-related code
XPI support is untested, as is style/translator usage, so there are
almost certainly bugs. The latter depends on updated export format
support (#659), since toArray() no longer exists on this branch.
Addresses #529 and #520
Note that this loses conflicting changes to translate_item.js from 849803473a,
so those will need to be reapplied if applicable. /cc @aurimasv, @mtd91429
[and delete unused "#zotero-prefpane-keys checkbox" (there used to be a checkbox to "Try to override conflicting shortcuts", which has been definely removed by 5b34dce40f )]
This required doing additional caching at startup (e.g., item types and fields)
so that various methods can remain synchronous.
This lets us switch back to using the current Sqlite.jsm. Previously we were
bundling the Fx24 version, which avoided freezes with locking_mode=EXCLUSIVE
with both sync and async queries.
Known broken things:
- Autocomplete
- Database backup
- UDFs (e.g., REGEXP function used in Zotero.DB.getNextName())
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
"height: auto" (from another extension's fix for this) actually caused
the bottoms of tree rows to be cut off on Windows (and at least on one
Linux installation with Chinese characters).
FT sync is enabled by default for new users and configurable in the Sync
prefs.
Also disable downgrades once full-text sync is enabled, since otherwise
someone switching back and forth between old and new versions could miss
full-text content updates.
Adds a 'foreground' flag to Zotero.HTTP.promise() options
Also, can now pass successCodes: false to always resolve the promise and never
throw UnexpectedStatusException
Dual boot and VM setups worked, but only absolute paths were synced,
because syncing uses attachmentPath, which was resolving relative paths
itself (#51). attachmentPath now returns what's in the database (as it
did originally), and getFile() does the resolving instead.
This means that anything relying on attachmentPath directly needs to be
aware that it might get a placeholder-prefixed relative path.
And probably other locales.
(Also, there must be a better way to deal with bottom elements getting
cut off than these extra <separator/> elements that result in extra
space when they're not needed.)
Deleted files are purged at the end of every sync, without any delay.
(If there's a conflict, it will be resolved before the file is deleted.)
Orphaned files are deleted once every 10 days, since it's a potentially
expensive operation for the server.
Prefpanes are now in separate overlays, which fixes the age-old resizing
bug when switching between panes (at least on OS X) that varied
depending on the initial pane. Code has also been moved into
pane-specific files and objects, with strict mode enabled. When calling
code from another pane (e.g., for a UI update in another pane), first
check whether the Zotero_Preferences.[Pane] object exists--if it doesn't
then nothing needs to be called.
This change breaks the word integration plugin pref overlays, which
hopefully can be rewritten to work with either overlay format.
There is a good chance that this breaks some other things in the
preferences too.
Fixes#243
- If attachment exists at same relative path in new base directory,
leave it alone so that it continues to work.
- If attachment doesn't exist in new base directory, revert it to an
absolute path.
- If new base directory is an ancestor or descendant of the previous
base directory, adjust relative paths below the new directory so that
they keep working.
- More dialog changes
- Select current base directory in file picker when changing directory
- Always use .persistentDescriptor instead of initWithPath(), though it
probably doesn't matter, and wrap in try/catch in case the old setting
is broken in some way.
- New function Zotero.File.directoryContains(dir, file), since
nsIFile.contains() isn't recursive
- Don't use a private Zotero.Item property from outside to force path changes.
- Changed placeholder to just "attachments:" for consistency with "storage:"
- Reworked dialog text
- Use a fancier (and undocumented) filefield XUL element for path
- A few small code tweaks
Translator updates and broken translator reporting both make automatic
requests to zotero.org, but the version pref is just adding a header
with the Zotero version on existing requests.
Can be moved/removed via Customize as well as via Zotero prefs. The icon
can be in the toolbar or in the add-on bar, but not both, because
it's now the same button with different styling depending on where it is.
If the icon does not exist in the UI and the pref is set to show (as it
is by default), the icon is added to the add-on bar and the add-on bar is
persisted open. This has the effect of showing the icon/bar on
first run. If the icon is removed or the bar is hidden, they remain that
way.
Can choose to download files "at sync time" or "as needed"
On-demand defaults to on, but remains off for existing users
To-do:
- Handling of local and remote file changes on on-demand download
(currently if a file exists it isn't downloaded, which means a
remotely modified file won't be redownloaded in on-demand mode)
- Additional control over file downloading and retention
Other changes:
- Overhauled entire file syncing architecture
- Replaced numAttachments column with Note and Attachment columns with
dynamic icons to indicate status
- Double-clicking a parent with a missing best attachment and on-demand
downloading off no longer loads the parent URL
- Bugs
- Zotero.Utilities is now a singleton
- Zotero.Utilities.HTTP is now Zotero.HTTP
- Zotero.Utilities.md5 and Zotero.Utilities.Base64 are now located under Zotero.Utilities.Internal
- Zotero.Utilities.AutoComplete has been eliminated
This needs testing to make sure there is no associated breakage.
also adds a parameter to disable proxy redirection by domain (although if I had known how much work would be involved in implementing this before I started, I wouldn't have bothered)
- Group file sync via Zotero File Storage
- Split file syncing into separate modules for ZFS and WebDAV
- Dragging items between libraries copies child notes, snapshots/files, and links based on checkboxes for each (enabled by default) in the Zotero preferences
- Sync errors now trigger an exclamation/error icon separate from the sync icon, with a popup window displaying the error and an option to report it
- Various errors that could cause perpetual sync icon spinning now stop the sync properly
- Zotero.Utilities.md5(str) is now md5(strOrFile, base64)
- doPost(), doHead(), and retrieveSource() now takes a headers parameter instead of requestContentType
- doHead() can now accept an nsIURI (with login credentials), is a background request, and isn't cached
- When library access or file writing access is denied during sync, display a warning and then reset local group to server version
- Perform additional steps (e.g., removing local groups) when switching sync users to prevent errors
- Compare hash as well as mod time when checking for modified local files
- Don't trigger notifications when removing groups from the client
- Clear relation links to items in removed groups
- Zotero.Item.attachmentHash property to get file MD5
- importFromFile() now takes libraryID as a third parameter
- Zotero.Attachments.getNumFiles() returns the number of files in the attachment directory
- Zotero.Attachments.copyAttachmentToLibrary() copies an attachment item, including files, to another library
- Removed Zotero.File.getFileHash() in favor of updated Zotero.Utilities.md5()
- Zotero.File.copyDirectory(dir, newDir) copies all files from dir into newDir
- Preferences shuffling: OpenURL to Advanced, import/export character set options to Export, "Include URLs of paper articles in references" to Styles
- Other stuff I don't remember
Suffice it to say, this could use testing.
