Previously only individual objects from successful requests that
couldn't be processed for some reason would be added to the queue.
`Sync.APIClient.downloadObjects()` now returns clearer and more
consistent results. It now returns an array of promises for objects with
a `keys` array of requested keys and either a `json` array of returned
API JSON objects or an `error` Error, depending on whether the request
succeeded or failed. This makes it easier to detect remotely missing
objects and request failures.
This includes the `Zotero.RDF.getResourceURI()` fix in
zotero/translate@85b39a5be.
This position change here for `this._handleRelated()` for attachments
isn't necessary, since the attachment is already saved and already has
an id, but it's done for consistency with the call for notes (where the
note previously didn't yet have an id).
Extended Mendeley Import test to include a scenario where other users
attached an annotation to an item in a group library that also exists
in user's library.
This prevents metadata changes from clearing the isRead status of feed
items as long as GUIDs remain constant. Previously, feed items with
randomized properties (like URLs generated dynamically each time the
feed is served) would be incorrectly marked as unread on every update.
This test covers couple of scenarios downloading a 16MB file from a
local http server (simple and concurrently) using Zotero.File.download.
When running current implementation, this seems to fail persistently.
In real-world usage, current implementation seems to work
intermittently, failing more often for larger files.
This intermittent behaviour is most likely cause of #2216.
- Messages are shown once a day by default (within the same session for
id-less messages)
- Messages with an `id` attribute include a checkbox to not show again
for 30 days
- If an `infoURL` attribute is provided, a "More Information" button is
shown that launches that URL
- If `title` is provided, it's used for the dialog title. Otherwise
"Warning" is shown.
- Fixes selection events always being debounced
- Fixes some failing tests
- Ensures Select All command selects search matching children of
collapsed parents. Adds tests for this case
- Just a single huge commit. This has been developed over too long a
time, required many tiny changes across too many files and has seen too
many iterations to be separated into separate commits.
The original branch with all the messy commits will be kept around for
posterity
bb220ad0f2...adomasven:feature/react-item-tree
- Replaces XUL <tree> element across the whole zotero client codebase
with a custom supermegafast virtualized-table inspired by
react-virtualized yet mimicking old XUL treeview API. The
virtualized-table sits on top on a raw-to-the-metal,
interpreted-at-runtime JS based windowing solution inspired by
react-window. React-based solutions could not be used because they were
slow and Zotero UI needs to be responsive and be able to
display thousands of rows in a treeview without any slowdowns.
- Attempts were made at making this screen-reader friendly, but yet to
be tested with something like JAWS
- RTL-friendly
- Styling and behaviour across all platforms was copied as closely as
possible to the original XUL tree
- Instead of row-based scroll snapping this has smooth-scrolling. If
you're using arrow keys to browse through the tree then it effectively
snap-scrolls. Current CSS snap scroll attributes do not seem to work in
the way we would require even on up-to-date browsers, yet alone the ESR
version of FX that Zotero is on. JS solutions are either terrible for
performance or produce inexcusable jitter.
- When dragging-and-dropping items the initial drag freezes the UI for
a fairly jarring amount of time. Does not seem to be fixable due to
the synchronous code that needs to be run in the dragstart handler.
Used to be possible to run that code async with the XUL tree.
- Item tree column picker no longer has a dedicated button. Just
right-click the columns. The column preferences (width, order, etc) are
no longer handled by XUL, which required a custom serialization and
storage solution that throws warnings in the developer console due to
the amount of data being stored. Might cause temporary freezing on HDDs
upon column resize/reorder/visibility toggling.
- Context menu handling code basically unchanged, but any UI changes
that plugins may have wanted to do (including adding new columns) will
have to be redone by them. No serious thought has gone into how plugin
developers would achieve that yet.
- Opens up the possibility for awesome alternative ways to render the
tree items, including things like multiple-row view for the item tree,
which has been requested for a long while especially by users switching
from other referencing software
- Delete unused embedded images when note is closed.
- Load images as soon as they are downloaded.
- Introduce new notification for download event, and a test for it.
- Prevent simultaneous downloads of the same attachment.
Just don't consider items in the trash to be linked items in
item.getLinkedItem(). I'm not totally sure why we didn't do this many
years ago, since it's one of the biggest sources of confusion in Zotero.
This addresses #1648 and closes#1610, though not by undeleting and
overwriting the item in the trash. When deleting an item and then
re-dragging it from another library, I think most people would expect
the item in the trash to still exist (possibly with notes, etc.), rather
than having been automatically restored and overwritten with new data.
And since `Zotero.Libraries.isGroupLibrary()` just checked that
property, that hasn't worked either since `isGroup` was added in 2015.
There's no test for `.isGroup`, and the test for `isGroupLibrary()` used
`if()` instead of `it()`, so it never actually ran.
Also:
- Remove old code block in search.js that called `isGroupLibrary()`.
Since `isGroupLibrary()` didn't work, this block was unused, and its
logic was previously added elsewhere.
Among other things, if the name of a translator was changed, until
restart Zotero.Translators.getAllForType() would return the old cache
entry pointing to a file that no longer existed.
- Make a copy of the database after first initialization that can be
swapped in when reinitializing in resetDB()
- Avoid unnecessary one-second delay on every reset
Probably more that can be done, but this should take minutes off the
test runs
- Add/Remove Dictionaries window
- Better account for the (unlikely) possibility that a dictionary could
be replaced by another more popular dictionary provided by a different
extension id (tested)
- Better account for the (very unlikely) possibility that an extension
could bundle multiple dictionaries (untested)
- Use toolkit version comparator for proper extension version
comparisons
- Localize strings
- Add tests for updating
This connects the new feed processor to Zotero's existing feed reader. The new
feed processor assumes a content window environment, so a sandbox in a parent
window is used to load it.
This adds extra tests to check parsing behavior such as entities, tag handling,
CDATA, etc. This will help ensure the new feed processor matches the previous
behavior.
Filesystems care about byte length, not character length, so treat
maxLength as the byte length limit and truncate accordingly.
This will also now remove entire emoji characters without corrupting
them.
An extra sync loop would be performed for every object downloaded, so a
download to an empty database could result in a huge number of
unnecessary loops. This was a regression from 52932b6eb, which started
queuing auto-syncs while a sync was in progress. The fix here is to skip
auto-sync for all objects saved from a sync download.
There are two new mechanisms involved:
- Event-level notifier options that get passed to passed to notify() at
the top level of extraData rather than being included with every
object (e.g., because `skipAutoSync` should apply to an entire save
transaction)
- The ability to pass event-level notifier options when initializing
a Zotero.Notifier.Queue, such as the one used for sync downloads
Reduce timeout after an object change from 15 seconds to 3 seconds,
ecxept for individual notes. We should probably kick off an auto-sync
3 seconds after a note is blurred, but that's not implemented.
We also now skip file syncing and full-text content syncing altogether
unless an attachment has changed.
Reindexing an item now kicks off auto-sync, whereas before it didn't.
We weren't making actual upload requests after a quota error if the file
would exceed the quota, but we were still going through all attachments
to upload, which in some cases involves making stat() calls. We now just
stop the queue immediately after a quota error or when starting a new
background sync after a previous quota error.
Closes#1255
This adds a very basic reimplementation of the annotation widget for use
in the CR window. It's not pixel-perfect, but it's close enough that
most people probably won't notice. We'll want to replace this with a
real version that's shared between the PDF reader and the rest of the
codebase.
Image annotations currently show an "[image not shown]" placeholder.
Showing images is tricky: we don't have the remote cache image, so if
the remote position has changed, we could only show an image by
rendering it from the file, and only if the file itself hasn't changed.
Just for a better user experience, we could use the local image as long
as the position and file are both the same, but that would take some
rejiggering of the CR window.
Tags aren't shown because they're not shown for CR at all, though that
could be changed in the future.
The page index needs to be per-person in group libraries, and it should
still work in read-only libraries, so it doesn't make sense to store it
on the item. This uses a synced setting in the user's library instead.
This changes the attachment saveFile() function in translators to be
async. In order for errors to be properly caught, translators will need
to be changed to make doExport() async and await on saveFile() calls.
(The translation architecture theoretically already allows doExport() to
be async.)
Returning ids mirrors getAttachments() and getNotes(), but I think we
want to move towards a world where those return actual objects, even if
we need async load calls on the returned objects.
E.g., embedded attachment notes with no note don't have an itemNotes row
and don't output noteSchemaVersion in their JSON, but they shouldn't
trigger a conflict
Added Zotero.DataObjects.sortByParent() to sort child items immediately
after their parent items. Zotero.DataObjects.sortByLevel(), which is
used for collections, sorts each level together, but that's less
appropriate for items where, e.g., an embedded-image attachment should
immediately follow the note that depends on it.
```
var noteContents = item.note; // was item.getNote()
var schemaVersion = item.noteSchemaVersion;
item.setNote(contents) // default to Zotero.Notes.schemaVersion
item.setNote(contents, schemaVersion) - explicit version
```
Generally, don't clear the change state for primaryData when calling
loadPrimaryData() on an item that doesn't exist. Specifically, this
fixes an issue where calling loadPrimaryData() on a nonexistent item
after setting a libraryID and key would also clear any item type set in
the constructor. The same would apply for props set in the
Collection/Search constructors.
Instead of showing warning button on toolbar
Otherwise if one person in a group upgrades to a beta with new fields,
everyone in the group will get a warning, even if they're already on the
latest release version. In a personal library, the user can upgrade to
the same version.
E.g., if a local item has been modified in a way that conflicts with a
remote item that also has a new, unknown field, don't show the CR
window -- just add the item to the sync queue and show the
some-data-could-not-be-downloaded error.
While we need to pass all items from descendant collections to
translators, only the immediate child collections of the selected
library or collection should be returned from Zotero.nextCollection().
I'm not sure how long we've been doing this wrong, but it resulted in
duplicated subcollections when round-tripping Zotero RDF. (TEI is the
only other translator that uses nextCollection(), so its exports were
probably similarly incorrect.)
https://forums.zotero.org/discussion/87135/export-import-creates-numerous-duplicate-subcollections
But skip it at startup, even if flagged on, if there are schema update
steps to perform, to avoid creating tables that aren't expected to exist
yet.
Originally added in 5b9e6497a but disabled in c4cc44528 and 7a434df53
This lays the groundwork for moving collections and searches to the
trash instead of deleting them outright. We're not doing that yet, so
the `deleted` property will never be set (except for items), but this
will allow clients from this point forward to sync collections and
searches with that property for when it's used in the future. For now,
such objects will just be hidden from the collections pane as if they
had been deleted.
Originally added in 5b9e6497a. We stopped running the integrity check
before userdata upgrades in c4cc44528 because this new behavior was
breaking upgrades, but it could still be run when coming from the DB
Repair Tool, which could cause problems if, say, you recovered a
database from a computer that had an older version of Zotero and ran the
DB Repair Tool on it. Disabling this for now until we have a better
solution.
ConcurrentCaller wasn't waiting properly if start() was called again
while it was pausing, so 429 caused an immediate retry, which is pretty
much exactly what you don't want a 429 to do.
After a local item change, Zotero uploads the JSON and then applies the
saved JSON returned from the API to the local object using `fromJSON()`,
the same as it would apply any other remote change.
`fromJSON()` is meant to migrate Extra lines into real types and fields
after future item type/field changes. It calls
`Z.Utilities.Internal.extractExtraFields()`, which looks for valid item
type or CSL type values in Type lines in Extra, handles the rest of
parsing accordingly, and passes back the parsed item type. `fromJSON()`
wasn't handling `itemType` in the response object, so the item type
didn't get applied and the Type line was stripped. This fixes that.
Since valid type values are now parsed, if you have a Journal Article
item with a Pages field and enter "Type: song" into Extra and sync, the
item will be converted to Audio Recording and `Pages: 123` will be
placed in Extra.
https://forums.zotero.org/discussion/comment/369221/#Comment_369221
- Create userdata tables and indexes that are missing
- Delete tables and triggers that should no longer exist
- Run schema integrity check before user data migration
- Run schema integrity check after restart error
This is meant to address two problems:
1) Database damage, and subsequent use of the DB Repair Tool, that
results in missing tables
2) A small number of cases of schema update steps somehow not being
reflected in users' databases despite their having updated userdata
numbers, which are set within the same transaction. Until we figure
out how that's happening, we should start adding conditional versions
of schema update steps to the integrity check.
This is currently only running the update check after a restart error,
which might not occur for all missed schema update steps, so we might
want other triggers for calling setIntegrityCheckRequired().
- Using `sandboxPrototype` properly uses window as prototype
- This commit removes the need for our patch in babel-worker.js:
3d0bc4cf9f
- We properly inject into frames in the client if we ever include frames
Our SingleFileZ integration would save images inside directories following the
SingleFileZ format. However, Zotero does not support syncing sub-directories of
attachments. This commit switch back to a single HTML file with base64 encoded
resources. We think that the 33% increase in resources will be offset by the
compression of HTML and removal of JavaScript and unused CSS.
This commit does not fix past snapshots that were saved using SingleFileZ.
Possibly caused by a third-party client uploading mtimes that then
aren't synced, or that differ from what get synced. When we detect this,
try to correct it by updating mtimes on WebDAV and the API to match the
local file.
https://forums.zotero.org/discussion/83554/zotero-loop-syncs-2000-items
Zotero.File.move() now forces `overwrite` if the old and new filenames
differ only by case, since otherwise on a case-insensitive filesystem
OS.File.move() does an existence check and thinks that the target file
exists.
Previously, if an object was uploaded but the API returned 'unchanged',
the uploaded data would be written to the sync cache, which, given that
most requests are patch requests, could result in an empty or mostly
empty object being saved to the sync cache. That would cause the next
sync to treat most/all local fields as changed and either upload them
unnecessarily or trigger a conflict instead of merging changes
automatically.
When non-conflicting changes were automatically merged, the local object
would be correctly marked as unsynced, but the merged object rather than
the remote object would be saved to the sync cache. When the object was
then uploaded, it matched the cache version exactly, so an empty patch
object (other than an unchanged dateModified, which is always included)
would be uploaded and the local change wouldn't make it to the server.
The empty patch would result in an 'unchanged' response, which would
cause the empty patch object to be saved to the sync cache (which is a
bug that I'll fix separately). If the local object was modified again,
the patch would include all fields (since the cache object was empty)
and the local change would be uploaded, but there could also be
unnecessary conflicts due to it looking like all local fields had been
modified.
This patch causes the remote object to be saved to the sync cache
instead, so the local change looks like a local edit and is correctly
uploaded.
Don't include linked-object or replaced-item relations. Previously, if
you duplicated an item, modified it to represent a different source, and
dragged it to another library where you had already copied the original
item, the new item wouldn't be transferred.
https://forums.zotero.org/discussion/comment/353246/#Comment_353246
- When changing type based on 'type:' line, move existing fields that
are no longer valid to Extra
- Remove 'type:' line with CSL type if the item's existing type is one
of the types mapped to it
It shouldn't be possible to nest two collections inside each other, but
if it happens, fix it in the integrity check.
Also detect it from CollectionTreeView::expandToCollection() (used when
showing the collections containing an item) and crash Zotero with a flag
to run an integrity check after restart. Previously, this would result
in an infinite loop.
This may be the cause of some of the collection disappearances people
have reported. If parentCollectionID never leads to a null, the
collection won't appear anywhere in the tree.
TODO:
- Figure out how this is happening
- Detect and fix it automatically for people it's happened to
If you'd never set a Quick Copy locale, the option would show as using
the current locale, but Quick Copy itself would use the last locale from
"Create Bibliography from Items". That was a side effect of behavior we
put in place in 2015 so that documents created before 4.0.27 that relied
on the removed bibliographyLocale pref would continue using the migrated
locale, but now that we've had an explicit locale option for years in
the document preferences I think we can stop doing that.
Previously, files updated remotely wouldn't be downloaded in "as needed"
mode if a copy of the file already existed locally and could only be
re-downloaded by deleting the file via Show File.
This causes remotely modified files that exist locally to be downloaded
at sync time, even in "as needed" mode, by marking them as
"force_download". While this might not be ideal for people who use "as
needed" to limit data transfer, it's better for people who use it simply
to limit local storage, and ending up with an outdated file while
offline seems worse than a little bit of extra data transfer.
In the future, we'll likely also provide ways to explicitly download and
remove files, so keeping chosen files in sync makes sense.
Files modified remotely before this change (which were marked as
"to_download" instead of "force_download") won't be downloaded as sync
time in "as needed" mode, but they'll now be re-downloaded on open.
Fixes#1322
- Use full-text cache file from syncing if available when reindexing via
info pane or Rebuild Index → Index Unindexed Items. Only discard it for
full index rebuild. This allows Index Unindexed Items to be used to
force immediate processing of queued content from syncing and avoids
unnecessary syncing back of identical content. Previously, the cache
file was used for a manual index only when the local file didn't exist.
- When rebuilding index, don't clear indexed items with missing local
file that are missing stats due to a pre-411180ef bug.
- indexItems() now takes an 'options' object as its second parameter
- Minor code cleanup
- Don't clear item's index stats (and show "Unknown") when an item is
reindexed remotely and the content matches the local content
- Always update an item's state and its stats in the same query, to
avoid incorrect feedback immediately after indexing
- Clean up `setItemContent()` tests
This is a prerequisite for starting to use new fields in translators,
since otherwise switching from, say, storing originalDate in Extra to
using an originalDate field would cause the value to be lost in clients
without the newer schema.
Closes#1504
Follow-up to df40ee7216
- Restore opening of non-HTTP URL schemes in link attachments
- Remove scheme whitelist for link attachments, since it's not enforced
via the API anyway and we prompt before an external application is
opened. Instead, just block a few schemes (e.g., 'javascript') from
launching.
TODO:
- Provide some way to change/reset an application association if the
user checks the box to automatically open that scheme.
- Show an error message if a link attachment with an invalid URL is
double-clicked
- Support launching plausible HTTP URLs without schemes from
Zotero.loadURI(), and use that when launching from URL field
- Show correct cursor feedback on URL label -- only show pointer if the
URL is launchable
- Don't launch non-HTTP URLs (e.g., zotero://) from URL field
- Don't open HTTP URLs in viewer when using ZoteroPane.launchURI()
It shouldn't be possible for collections to be nested this way, if it
happens, it shouldn't result in an infinite loop.
This removes one of the parent assignments at sync time.
Improvements:
- Fixes autocomplete text remaining in field after selection in Fx60
- No more text or icon shifting on select (tested on macOS)
Changes:
- Tags are now selected on mousedown with no active state, as in web
library
Regressions:
- Tooltip with tag type doesn't appear when hovering over icon
- Pressing Tab after modifying a tag loses focus
- Right-click in textbox shows custom menu instead of default text
editing context menu (Cut/Copy/Paste)
To-do:
- Switch to this version for note tags box
- Style colored tags in autocomplete drop-down? Sort to top?
- Only show delete button on row hover, as in web library?
This was a regression from 4b60c6ca27. The original plan for introducing
new fields was to have them save to the sync cache even if they weren't
supported by the current Zotero version and process them on upgrade, and
so I changed `DataObjectUtilities.patch()` to omit fields that didn't
exist locally when patching the sync cache JSON so they wouldn't be
wiped on the server. That caused this bug where locally deleted fields
were restored on every sync. It's also no longer necessary now that
we decided to just reject unknown fields from saving, so we can just
revert to the previous behavior of blanking out locally missing fields
(with the tweak that fields that are already false or empty in the base
version can be omitted).
And fix a couple things for if we turn it back on
This code came along with the type/field handling overhaul, but I think
it was originally intended for handling unknown fields during sync
before we decided on strict mode, so it wasn't finished and causes
various problems [1]. It could still be useful for preserving fields
from translators before they're available on items, but the better fix
there is just to add the missing fields, so I'm not sure if we'll end up
needing it.
[1] https://groups.google.com/d/msg/zotero-dev/a1IPUJ2m_3s/hfmdK2P3BwAJ
Say "Use the [local|remote] version for all remaining conflicts" for
everything instead of saying "Use [local|remote] fields for all
remaining conflicts" for some conflicts.
This also fixes a test failure after 54343c49fb.
This changes the way item types, item fields, creator types, and CSL
mappings are defined and handled, in preparation for updated types and
fields.
Instead of being predefined in SQL files or code, type/field info is
read from a bundled JSON file shared with other parts of the Zotero
ecosystem [1], referred to as the "global schema". Updates to the
bundled schema file are automatically applied to the database at first
run, allowing changes to be made consistently across apps.
When syncing, invalid JSON properties are now rejected instead of being
ignored and processed later, which will allow for schema changes to be
made without causing problems in existing clients. We considered many
alternative approaches, but this approach is by far the simplest,
safest, and most transparent to the user.
For now, there are no actual changes to types and fields, since we'll
first need to do a sync cut-off for earlier versions that don't reject
invalid properties.
For third-party code, the main change is that type and field IDs should
no longer be hard-coded, since they may not be consistent in new
installs. For example, code should use `Zotero.ItemTypes.getID('note')`
instead of hard-coding `1`.
[1] https://github.com/zotero/zotero-schema
- Move 5xx retries and connection checking out of the sync API client
and into HTTP.request() so that they apply to all requests. 429 handling
remains in the API client, since not all callers necessarily want to
handle that the same way. Callers can still handle 5xx themselves by
including the relevant 5xx status codes in `successCodes` or by passing
`errorDelayMax: 0`.
- Add `cancellerReceiver` option, which is a callback that receives a
function that will cancel the request, whether it's an active request
or an automatic delay before a 5xx retry.
This also updates Sinon to 7.3.2.
`nsIFilePicker::show()` is removed in Firefox 60 in favor of `open()`,
which takes a callback (and apparently has been preferred for a long
time).
There's no point switching to that, so this module is a version of
nsIFilePicker with an async `show()` that returns a promise and some
XPCOM-isms replaced (e.g., string paths instead of nsIFile).
nsIURL doesn't seem to work anymore, so add Zotero.Utilities.parseURL(),
which uses the `url` package from NPM and adds fileName, fileExtension,
and fileBaseName.
Zotero.Translate::setTranslatorProviderMethods(methods) can be used to
provide custom 'get' and 'getAllForType' methods that override the
default Zotero.Translators methods.
