This changes the way item types, item fields, creator types, and CSL
mappings are defined and handled, in preparation for updated types and
fields.
Instead of being predefined in SQL files or code, type/field info is
read from a bundled JSON file shared with other parts of the Zotero
ecosystem [1], referred to as the "global schema". Updates to the
bundled schema file are automatically applied to the database at first
run, allowing changes to be made consistently across apps.
When syncing, invalid JSON properties are now rejected instead of being
ignored and processed later, which will allow for schema changes to be
made without causing problems in existing clients. We considered many
alternative approaches, but this approach is by far the simplest,
safest, and most transparent to the user.
For now, there are no actual changes to types and fields, since we'll
first need to do a sync cut-off for earlier versions that don't reject
invalid properties.
For third-party code, the main change is that type and field IDs should
no longer be hard-coded, since they may not be consistent in new
installs. For example, code should use `Zotero.ItemTypes.getID('note')`
instead of hard-coding `1`.
[1] https://github.com/zotero/zotero-schema
Separate flags for hiding the retraction altogether and for hiding
citation warnings for it
New functions:
Zotero.Retractions.hideRetraction(item)
Zotero.Retractions.shouldShowCitationWarning(item)
Zotero.Retractions.disableCitationWarningsForItem(item)
Addresses #1710
This shouldn't be possible, but there've been a couple reports of people
ending up on version 103 without the table, so create it again with IF
NOT EXISTS. This is obviously a bad fix, but until we know how this
happened it's the best we can do.
- Check for retracted items using data from Retraction Watch
- Show an X next to retracted items in the items list, and show a
scary message at the top of the item pane with more info and links.
- Lookup is done in a privacy-preserving manner using k-anonymity --
the server is unable to determine the specific items that exist in
the client, so people who don't sync don't need to share any library
data (though the server doesn't log the lookups anyway).
TODO:
- Pop up an alert when new items are found
- Show a confirmation prompt when citing a retracted item
- Support items without DOIs or PMIDs
- Add a proper PMID field and expand DOI to more item types so these
values don't need to be parsed out of Extra
- Clear the banner immediately when all possible fields are cleared
instead of waiting a few seconds
A submitted database had a text userID with a trailing "A0. Not sure how
that happened -- it doesn't appear to be possible in current code -- but
it caused group permissions not to be properly synced.
If somebody switched accounts in a previous version, it was apparently
possible for related items to end up pointing at an item URI with the
old userID, which could cause a 403 on sync.
https://forums.zotero.org/discussion/70989/report-id-477331252
(5.0 deletes data when switching accounts to avoid exactly this sort of
bug.)
These should've all been replaced with dc:replaces relations in a schema
update step, so any that exist were likely synced down from the API
(since fixed) and should be obsolete/redundant.
This also adds a mechanism for indicating that DB userdata upgrades
after a certain version are minor and shouldn't show "Upgrading
database…" or create a pre-upgrade backup.
Instead of My Publications being a separate library, have it be a
special collection inside My Library. Top-level items can be dragged
into it as before, and child items can be toggled off and on with a
button in the item pane. Newly added child items won't be shown by
default.
For upgraders, items in the My Publications library will be moved into
My Library, which might result in their being duplicated if the items
weren't removed from My Library. The client will then upload those new
items into My Library.
The API endpoint will continue to show items in the separate My
Publications library until My Publications items are added to My
Library, so the profile page will continue to show them.
* Mark feedItems read in a single batch SQL update
* Automatically remove old feed items
* User-facing preference globally and per-feed for feed item expiration
Before 5.0 we performed a regexp on new item data values to determine if
they were integers and saved them natively in SQLite if so. We no longer
do that, but setField() used strict equality when checking for changes,
so an item could be marked as changed when comparing to a new string
value (e.g., from a write response from the API, which always returns
strings). To avoid that, this converts all old values in the DB to
strings and saves all incoming values as strings automatically. (This
should also help with searching and some other things.)
Previously, objects were first downloaded and saved to the sync cache,
which was then processed separately to create/update local objects. This
meant that a server bug could result in invalid data in the sync cache
that would never be processed. Now, objects are saved as they're
downloaded and only added to the sync cache after being successfully
saved. The keys of objects that fail are added to a queue, and those
objects are refetched and retried on a backoff schedule or when a new
client version is installed (in case of a client bug or a client with
outdated data model support).
An alternative would be to save to the sync cache first and evict
objects that fail and add them to the queue, but that requires more
complicated logic, and it probably makes more sense just to buffer a few
downloads ahead so that processing is never waiting for downloads to
finish.
- Hide notes, tags and related for feed items in itembox
- Add feed support for <enclosure> elements
- Add feed syncing methods for synced settings (additional work is
needed on the sync architecture to download synced settings from the
server)
- Change feed item clear policy to be less aggressive
- Adjust for deasyncification
- Disable translate-on-select
- Closeadomasven/zotero#7, Remove context menu items from feeds
Also:
- Remove last-sync-time mechanism for both WebDAV and ZFS, since it can
be determined by storage properties (mtime/md5) in data sync
- Add option to include synced storage properties in item toJSON()
instead of local file properties
- Set "Fake-Server-Match" header in setHTTPResponse() test support
function, which can be used for request count assertions -- see
resetRequestCount() and assertRequestCount() in webdavTest.js
- Allow string (e.g., 'to_download') instead of constant in
Zotero.Sync.Data.Local.setSyncState()
- Misc storage tweaks
Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
This will appear much less frequently, since non-conflicting field changes on
both sides can be resolved automatically, but genuine field conflicts still
require manual conflict resolution.
The merge pane is no longer editable, since the itembox code to do that is
async and can't run in a modal window, but it's not really necessary,
particularly with conflicts happening less frequently.
TODO:
- Remote item deletions
- File conflicts
- Maybe handle some edge cases where the conflicted items fail to save
- Simplified schema
- Tags are now added without reloading entire tag selector
- On my system, adding 400 tags to an item (separately, with the tag
selector updating each time) went from 59 seconds to 42. (Given that
it takes only 13 seconds with the tag selector closed, though,
there's clearly more work to be done.)
- Tag selector now uses HTML flexbox (in identical fashion, for now, but
with the possibility of fancier changes later, and with streamlined
logic thanks to the flexbox 'order' property)
- Various async fixes
- Tests
Restore prepopulated charset table, but this time with just the
encodings from the WHATWG Encoding Standard. Assigning a charset to
Zotero.Item::attachmentCharset runs the value through
Zotero.CharacterSets.toCanonical() automatically.
This migrates attachment charsets to the new canonical values, clearing any
that are unsupported.
Other legacy mappings could still be added back, as disussed in #760.
And add group.fromJSON(json, userID), which sets editable and
filesEditable properties based on the group JSON (libraryReading, role
lists, etc.) and the given user
Relations are now properties of collections and items rather than
first-class objects, stored in separate collectionRelations and
itemRelations tables with ids for subjects, with foreign keys to the
associated data objects.
Related items now use dc:relation relations rather than a separate table
(among other reasons, because API syncing won't necessarily sync both
items at the same time, so they can't be stored by id).
The UI assigns related-item relations bidirectionally, and checks for
related-item and linked-object relations are done unidirectionally by
default.
dc:isReplacedBy is now dc:replaces, so that the subject is an existing
object, and the predicate is now named
Zotero.Attachments.replacedItemPredicate.
Some additional work is still needed, notably around following
replaced-item relations, and migration needs to be tested more fully,
but this seems to mostly work.
The sync cache will have pristine copies of the existing versions of
local objects for better conflict resolution, but downloads will get
saved to the sync cache first before processing, so the cache needs to
be able to hold more than one version.
- Use async DB and OS.File for bundled file updates
- Remove support for translator/style ZIP files -- the two options are
now non-unpacked XPIs with subfolders or unpacked source installations
- Now that we have async file access, don't store translator code in
database cache -- just store metadata so that it's available without
reading each translator file
- Change the (previously partially asyncified) Zotero.Styles/Translators
APIs a bit -- while the getAll/getVisible methods are asynchronous and
will wait for loading, the get() methods are synchronous and require
styles/translators to be initialized before they're called. Most
places that end up calling get() probably call getAll/getVisible first
and should therefore be async, but if there's any way to trigger a
get() first, that will need to be adjusted.
- Asyncify various other style/translator-related code
XPI support is untested, as is style/translator usage, so there are
almost certainly bugs. The latter depends on updated export format
support (#659), since toArray() no longer exists on this branch.
Addresses #529 and #520
Instead of limiting charsets to a fixed list, dynamically populate it
with any charset name of less than 50 ASCII characters. Previously,
unknown charsets were discarded.
Zotero.Item.prototype.attachmentCharset now always returns a charset
name. It can be set with either a name or a charsetID.
Also:
- Remove the unused 'originalPath' column in itemAttachments
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
Full-text content is now synced to the server and other clients. Up to
~500K of content is transferred with each sync, so multiple syncs may be
necessary. Downloaded content is written to temporary files and
processed on idle (currently 5 seconds), so synced content may not be
immediately available.
Performance will improve with API syncing and async DB in 4.1.
This also includes some other full-text tweaks, such as storing
text converted from HTML in cache files, which may speed up phrase
searching.
This change should improve Firefox startup time. If the Zotero pane is
opened before Zotero has initialized, the pane will be blocked with a
progress bar until it's done. Standalone currently does the same, but it
should be changed to just delay opening the window if possible.
The upgrade wizard has been disabled for schema upgrades, in the hope
that upgrades can be done in a way that won't break compatibility with
earlier versions (or that can at least be done in a way such that we can
put out point releases of the last major version that provide
compatibility during beta/post-upgrade periods).
This patch likely breaks many things, and definitely breaks connector
mode.
This change also removes upgrade steps for databases from Zotero 2.1b2
and earlier. Users with such databases will need to upgrade via Zotero
4.0.x first or delete their data directories and start anew. This should
only affect users who haven't opened Zotero since Nov. 2010.
- New tag colors support, with the ability to assign colors to up to 6
tags per library. Tags with colors assigned will show up at the top of
the tag selector and can be added to (and removed from) selected items
by pressing the 1-6 keys on the keyboard. The tags will show up as
color swatches before an item's title in the items list.
- Synced settings, with Notifier triggers when they change and
accessible via the API (currently restricted on the server to
'tagColors', but available for other things upon request)
- Silent DB upgrades for backwards-compatible changes. We'll do
something fancier with async DB queries in 4.0, but this will work for
changes that can be made without breaking compatibility with older
clients, like the creation of new tables. The 'userdata' value is
capped at 76, while further increments go to 'userdata2'.
TODO:
- Try to avoid jitter when redrawing swatches
- Optimize tag color images for retina displays
- Redo attachment dots in flat style?
- Clear all colors from an item with 0 (as in Thunderbird), but I don't
think we can do this without undo
