We switched file downloads to use XHR to fix downloads failing via
authenticated proxies in fx115+, but that made them subject to our
default 30-second timeout in `Zotero.HTTP.request()`. That timeout uses
the XHR `timeout` property, which applies to the whole request, even if
data is actively being downloaded. As a result, file downloads would
fail for people downloading large files or on very slow connections.
This implements manual connect and inactivity timeouts when using
`Zotero.HTTP.download()`. Currently, these both use `options.timeout`,
or the default 30 seconds, but we could probably take separate options
and have lower defaults now that they no longer apply to the whole
request.
When an item is erased (removed from the trash or cleaned up from a
feed), we set a flag to purge values in `itemDataValues` on the next
startup, with this query:
DELETE FROM itemDataValues WHERE valueID NOT IN (SELECT valueID FROM itemData);
For some people, that query was incredibly slow and would result in
Zotero intermittently hanging on "Loading items…" for a long time at
startup. It's possible this is mostly limited to people who subscribe to
high-volume feeds and have a lot of item churn. One affected person had
>900K values in `itemDataValues` despite having only 20K items.
It turns out the slow query is due to the foreign-key constraint on
`itemData(valueID)` that references`itemDataValues(valueID)`. SQLite is
checking every row being deleted from `itemDataValues` against
`itemData`, even though the query is specifically removing rows that
don't exist in `itemData`! For the 900K-value DB, disabling foreign-key
checks causes the `DELETE` query to take 25 seconds instead of...some
much longer time that I didn't wait for.
We already had an `executeTransaction()` flag, `disableForeignKeys`, to
temporarily disable foreign-key checks, but it didn't do so in a way
that was safe for post-initialization usage -- a write query outside of
a transaction could've run between the transaction commit and
foreign-key checks being re-enabled. This commit changes it to properly
block all other queries unless they include an `ignoreDBLock` option,
meaning that queries within the function passed to the transaction need
to include that option. (And since that's not realistic for the couple
other uses of `disableForeignKeys` -- one for a test and one in code
that almost certainly hasn't been run by anyone in 15 years -- those now
just run `PRAGMA foreign_keys=OFF|ON` explicitly, leaving this as the
only current use.)
Adds a new function, Zotero.HTTP.download(), that uses
Zotero.HTTP.request(). This fixes downloads via authenticated proxies in
Zotero 7 and gives us other request() functionality (e.g., 5xx retrying)
for free.
The downside is that this is probably less efficient, potentially
loading large downloads in memory. We should create a replacement for
request() based on fetch() that supports getting the body as a
ReadableStream.
Fixes#5062
`firstInSession` wasn't being properly set, so previously failed data
objects weren't being retried until a manual sync. Local files also
weren't being scanned for changes for three hours, but that's probably
better in the age of DB-stored annotations, so leave that off for now
and address properly in #5025.
Possible we'll get complaints about no longer being able to open
multiple copies of at least runJS and csledit, and we can reconsider if
so, but reopening the existing window is certainly the intended
behavior.
And enable migration of Extra rows to valid fields in non-strict mode,
but since that's only used by translators now, that probably won't do
anything
Fixes#3422
Fix error if local note or attachment is added to a parent item while
the remote version was added to a collection. Now, the new local parent
item will be added to any collections the remote child item was added
to.
Fixes#2934
