This changes `createDataObject()` to pass `skipSelect: true` for objects
other than items. If a test is creating a bunch of collections, there's
no reason for each one to be selected and for an items list to start to
load. If a test does need a new collection or search to be selected, it
can call the new convenience function `await select(win, obj)`, which
will select the passed object in the collection tree and wait for its
items list to load. I'm hoping this reduces random test failures due to
items list churn.
- Currently enabled only for ScienceDirect. Can be enabled via a whitelist
- Matches the HiddenBrowser loaded HTML page for a captcha element. If
the captcha element class changes, this will break (but the
alternative is potentially displaying a captcha clearing window when
something else that is not a captcha guard is loaded).
- Captcha clear timeout for 60s.
- Doesn't automatically switch focus back to the browser which intiated
the item save via the Connector.
- Stores the cookies used to clear the captcha for future saves from the
same domain. Discards Connector supplied User Agent, since CF bot
detector checks UA header against actual UA behavior like TLS handshake
and if the UA acts different to what it's supposed to, the bot
challenge is not cleared.
Other changes:
- Adjusted the cookie sandbox to allow multiple cookie sandboxes to be
active (and simplified some legacy code that was meant to cover a bug
in old FX codebase).
- HiddenBrowser API changed to be Object oriented, translator tester
in the translate repo will need to be updated after a merge (have the
change ready).
- Improved Connector Server attachment progress handling
- Just a single huge commit. This has been developed over too long a
time, required many tiny changes across too many files and has seen too
many iterations to be separated into separate commits.
The original branch with all the messy commits will be kept around for
posterity
bb220ad0f2...adomasven:feature/react-item-tree
- Replaces XUL <tree> element across the whole zotero client codebase
with a custom supermegafast virtualized-table inspired by
react-virtualized yet mimicking old XUL treeview API. The
virtualized-table sits on top on a raw-to-the-metal,
interpreted-at-runtime JS based windowing solution inspired by
react-window. React-based solutions could not be used because they were
slow and Zotero UI needs to be responsive and be able to
display thousands of rows in a treeview without any slowdowns.
- Attempts were made at making this screen-reader friendly, but yet to
be tested with something like JAWS
- RTL-friendly
- Styling and behaviour across all platforms was copied as closely as
possible to the original XUL tree
- Instead of row-based scroll snapping this has smooth-scrolling. If
you're using arrow keys to browse through the tree then it effectively
snap-scrolls. Current CSS snap scroll attributes do not seem to work in
the way we would require even on up-to-date browsers, yet alone the ESR
version of FX that Zotero is on. JS solutions are either terrible for
performance or produce inexcusable jitter.
- When dragging-and-dropping items the initial drag freezes the UI for
a fairly jarring amount of time. Does not seem to be fixable due to
the synchronous code that needs to be run in the dragstart handler.
Used to be possible to run that code async with the XUL tree.
- Item tree column picker no longer has a dedicated button. Just
right-click the columns. The column preferences (width, order, etc) are
no longer handled by XUL, which required a custom serialization and
storage solution that throws warnings in the developer console due to
the amount of data being stored. Might cause temporary freezing on HDDs
upon column resize/reorder/visibility toggling.
- Context menu handling code basically unchanged, but any UI changes
that plugins may have wanted to do (including adding new columns) will
have to be redone by them. No serious thought has gone into how plugin
developers would achieve that yet.
- Opens up the possibility for awesome alternative ways to render the
tree items, including things like multiple-row view for the item tree,
which has been requested for a long while especially by users switching
from other referencing software
Our SingleFileZ integration would save images inside directories following the
SingleFileZ format. However, Zotero does not support syncing sub-directories of
attachments. This commit switch back to a single HTML file with base64 encoded
resources. We think that the 33% increase in resources will be offset by the
compression of HTML and removal of JavaScript and unused CSS.
This commit does not fix past snapshots that were saved using SingleFileZ.
The test incorrectly contained an item wait for an attachment that
doesn't get saved to the group library, but the test was passing because
the main item was being moved to the group twice. 94ccba45b somehow
fixed that, and since the behavior during the test is now correct, I'm
not going to worry about this unless we notice a problem.
If there's no translated PDF or the translated PDF fails and the item
has a DOI, check Zotero's Unpaywall mirror for possible sources and try
to download one of those.
Unlike with "Add Item by Identifier" and "Find Available PDF" in the
item context menu, this does not try the DOI/URL page, since it would
result in more data leakage and most of the time you'd be saving from
the DOI page already. We could consider offering it as an option, but
for it to be useful, you'd have to have an institutional subscription,
be on-campus or connected via VPN (for now), and be saving from
somewhere other than the main page.
A new connector endpoint, sessionProgress, takes the place of
attachmentProgress. Unlike attachmentProgress, sessionProgress can show
new attachments that have been added to the save, and with a little more
work should also be able to show when a parent item has been recognized
for a directly saved PDF.
This also adds support for custom PDF resolvers, available to all PDF
retrieval methods. I'll document those separately.
Closes#1542
Add newly added attachments to a queue, start processing it after five
seconds have passed since the last attachment was added, and process
another every half second after that unless another is added.
This queue won't survive a restart, so the queue should really be in the
DB, but this should avoid problems when adding multiple attachments at
once.
Addresses #1284
Applies to dragging to the collections pane or the items pane, adding
via New Item menu, or saving via the connector server
If the renaming pref is enabled, the PDF is renamed after recognition.
Can be disabled in the preferences
Closes#917
- Updates /saveItems and /saveSnapshot to take a sessionID
- Provides a list of editable collections in the current library
- Adds an /updateSession method that takes a sessionID and updates the
collection and tags of any items saved via that operation (and changes
the currently selected collection)
Cross-library changes are not yet supported
Return a 500 for read-only libraries for all save modes. Read-only views
within editable libraries will save to the library root.
Addresses #185, RIS/BibTeX interception to read-only view behaves
differently from save button
Attachments are now saved before the connector server responds, because they're
no longer started out-of-band in saveItems(). This is necessary to prevent
transaction badness during imports, but it may not be what we want for the
connector, so we may want to revisit this after further testing.
Improves proxy support
- Automatically detect and dehyphenise https proxies which use EZProxy
HttpsHyphens
- Web translators now pass around Zotero.Proxy instances which can
proxify/deproxify urls passed to `translate.setLocation()` before calling
`translate.getTranslators()`/ translate.detect()`. The proxy passing is
done within connector background/injected processes and between
standalone and connectors.
- Proxy protocol unified with connectors. Connectors can now pass
proxies to `/connector/save_items`. The proxies will be used to resolve
true item and attachment urls when saving.
Closeszotero/zotero#578, zotero/zotero#721
Relevant zotero/zotero#34, zotero/zotero#556
server_connectorTest - alternating port prevents
exceptions not catchable in JS for httpd server when the socket
and the port remains open after httpd.stop() callback
support - changes the window on which `setTimeout()` is ran in dialogs.
If the timeout is ran on the main window the `dialog` object appears to
lose certain properties and not respond to interactions completely.
Indexing currently happens a second after the 201 is returned to the
connector, so we have to wait for that before continuing tests, or else
a DB clear in a later test (e.g., storageLocal) will cause an error to
be logged when the indexing kicks off.
