The 'application/pdf' Content-Type shouldn't have any parameters, but a
site was returning 'application/pdf;charset=ISO-8859-1', so instead just
look for the 'application/pdf' prefix.
Also:
- Actually use the cookie sandbox passed to processDocuments()
For zotero/translation-server#16, we want to include Accept-Language
(and maybe other headers) from the client request in upstream requests,
which requires passing it to both non-translate processDocuments().
translation-server's non-translate processDocuments() is defined in that
repo, but it's called by the cross-repo translate processDocuments() in
utilities_translate.js, so the signature needs to be changed in both
repos.
We also apparently weren't using the cookieSandbox in client
processDocuments() calls, though I think that only would've affected
translator testing.
This fixes direct and VPN-based retrieval of PDFs for Elsevier (e.g.,
ScienceDirect) items that have a DOI but no URL, since Elsevier resolves
DOIs through an intermediate page.
This reverts commit e526a8fa89.
This is a repeat of a0ca67d879. The reverted changes break Scaffold.
This will need to be fixed in a way that allows Scaffold to continue to
work.
In 2652fac24 we started failing the translator if doGet/doPost returned
a >=400 status code, but at least one translator (Primo) relied on doGet
continuing after a 404. This allows translators to specify the status
codes that are allowed, similar to the same argument in
Zotero.HTTP.request().
(We'll clean up the signature at some point or just get rid of these
functions, but this is a quick fix for Primo.)
I didn't look into the details, but `this` isn't defined when this file
is loaded within Scaffold, so this line caused an error and the
translator tester was broken.
Delay requests to the same domain by 1 second, respect a Retry-After
header if present for 429 and 503, and delay for 10 seconds on 429 or
5xx otherwise.
Currently only .status and .getResponseHeader() (for getting 'Location')
are available in the returned object, but we could make the body
available if necessary.
If there's no translated PDF or the translated PDF fails and the item
has a DOI, check Zotero's Unpaywall mirror for possible sources and try
to download one of those.
Unlike with "Add Item by Identifier" and "Find Available PDF" in the
item context menu, this does not try the DOI/URL page, since it would
result in more data leakage and most of the time you'd be saving from
the DOI page already. We could consider offering it as an option, but
for it to be useful, you'd have to have an institutional subscription,
be on-campus or connected via VPN (for now), and be saving from
somewhere other than the main page.
A new connector endpoint, sessionProgress, takes the place of
attachmentProgress. Unlike attachmentProgress, sessionProgress can show
new attachments that have been added to the save, and with a little more
work should also be able to show when a parent item has been recognized
for a directly saved PDF.
This also adds support for custom PDF resolvers, available to all PDF
retrieval methods. I'll document those separately.
Closes#1542
If you had Quick Copy set to a missing translator, had a site-specific
Quick Copy setting, and loaded a page in the browser, the next click on
an item would result in a restart error.
- Show a clearer message on a certificate error that includes the
underlying error, which should make debugging much simpler. (No more
checking in a browser and hoping it's the same connection.)
- Mention proxy server in message on startup proxy-check failure
- Include link to connection-error KB page on sync connection failure
Closes#1191Closes#1513
If a proxy is required and we can't connect to S3, show the (sync) error
icon. For SSL certificate errors, which are the most common cause of
this, the panel includes a link to the SSL cert troubleshooting page.
A submitted database had a text userID with a trailing "A0. Not sure how
that happened -- it doesn't appear to be possible in current code -- but
it caused group permissions not to be properly synced.
- Add the ability to extract a PDF URL from a given webpage using the
translation framework
- Add the ability to get open-access PDFs from landing pages from
Unpaywall data in addition to direct PDF URLs
- Use the above functionality to improve PDF retrieval for "Add Item by
Identifier"
- Add "Find Available PDFs" option to the item context menu to retrieve
PDFs for existing items from the DOI or URL page or using Unpaywall
data. The option appears for single items with a DOI or URL and no PDF,
and it always appears when selecting multiple top-level items (but
skips ineligible items).
PDF extraction from DOI/URL pages will currently only work with
unauthenticated access (i.e., on-campus or VPN, but not via a web-based
proxy).
Supersedes and closes#948
Add Z.DataDirectory.getSubdirectory which, optionally, creates
the directory.
Add async Z.DataDirectory.removeSubdirectory and use it for
Z.removeTempDirectory (was sync call before!).
Move `Zotero.getString()` and intl init code to `Zotero.intl` to make
it easier to re-use.
Link `Zotero.getString()` to `Zotero.intl.getString()`.
Do not expose `getStringFromBundle`, `pluralFormGet`, and
`pluralFormNumForms` because they are not used.
Handle an array of objects with 'url' and 'version' rather than just an
array of URLs.
Also:
- Don't throw an error from addOpenAccessPDF() if there's an error from
getOpenAccessPDFURLs()
- Make addPDFFromURLs() a separate function so URL lookup can be done
separately from download
Previously the handler would be called even on error pages, which often
meant that an import translator (e.g., BibTeX) would fail to find
anything on the page and the save popup would just close silently. The
popup will now show an error message as soon as the error occurs.
Use `getResource` in Zotero.Date.init (this turns it into a
synchronous function). Zotero.File.getResource makes it easier
to load local files on platforms that do not support the
`resource://` URLs.
At some point Mendeley seems to have changed the default path to the
data directory on Windows to remove the period, and for people with the
old directory we were linking rather than storing attachment files from
"Downloaded".
E.g., if a .pdf is really an HTML file, we try to load it in a hidden
browser (because we properly detect the content type), but then the .pdf
extension causes the hidden browser to launch it via the OS and the
hidden browser never finishes loading it. This adds a 5-second timeout
to abort the process.
When the associated-files pref is enabled, Add Item by Identifier uses a
Zotero Unpaywall mirror to find available open-access PDFs. No details
about the contents of searches are logged.
For each PDF with an associated URL in the Downloaded directory, we were
copying all files in the directory (!) to the attachment's storage
directory. (Zotero imports always have files in separate directories,
and this was a function used to save both single files and HTML
snapshots.)
We'll clean up the extra files in a separate step.
If "Place imported collections and items into new collection" is
unchecked, previously imported collections will be reused when they're
in the right place in the hierarchy rather than creating new ones.
Use the new wizard for all imports (even if no Mendeley DB), and add a
page with a "Place imported collections and items into new collection"
option. If deselected, collections are added to the library root.
If there's a single PDF file and a single PDF URL and the file exists,
make an imported_url attachment instead of separate file and linked_url
attachments.
Addresses #1451
Accept Mendeley SQLite databases via File → Import… and perform a
direct import, including collections, timestamps, notes, attachments,
and extracted annotations.
When a Mendeley database is present, File → Import… shows a wizard that
lets you choose between a file and Mendeley for the source, and choosing
the latter shows a list of available databases in the Mendeley data
directory.
Known fields that aren't valid for a type are stored in Extra.
Files in the Mendeley 'Downloaded' folder are stored. Files elsewhere
are linked.
Previously you could use Zotero.DBConnection to open another database in
the data directory, but not one stored elsewhere in the filesystem. This
allows an absolute path to be passed instead. Various operations
(backups, corrupt DB recovery, pragma commands) are disabled for
external databases.
This previously returned false if a non-base-mapped field was passed,
even if the field was valid for the given item type. It now returns the
passed field as long as the field is valid for the type, which matches
the behavior for base fields.
1) text() wasn't handling the index property.
2) This removes the warning that attr()/text() no longer no require a
document as the first argument, because there's no reason to prevent
translators from being able to pass an element. It would require
rewriting various translators unnecessarily and make certain patterns
more verbose (because you'd need to match based on global scope in each
selector).
It won't be necessary to pass a Document once we remove 4.0 support and
the global attr()/text() are always available, so we can add a warning
for that then.
Fixeszotero/translators#1647
This is loosely based on the same functionality in ZotFile, but it tries
to do the right thing based on existing Zotero settings: either the new
PDF handler setting in the prefs or the system-default app. The latter
can only reliably be determined on Windows (and this uses ZotFile's
function to read that from the registry), but this tries to figure it
out on macOS and Linux too using the Mozilla handler service. (The
handler service only gets you an app name, not a path, so on Linux we
can try reading mimetypes.list and the like in case someone is using a
system-default okular or evince not in /usr/bin, but that's not yet
implemented.)
This uses the new 5.0 URL format, and a 'page' query parameter instead
of a path component:
zotero://open-pdf/library/items/[itemKey]?page=[page]
zotero://open-pdf/groups/[groupID]/items/[itemKey]?page=[page]
It also accepts ZotFile-style URLs, though, so if you uninstall ZotFile
you should still be able to open those links. ZotFile will need to
accept the new format for new links to work when ZotFile is installed,
since it will override this handler.
This functionality will be necessary for annotation extraction (#1018)
and for imported annotations from Mendeley (#1451).
Shows a prompt once per restart or once per day, whichever is longer,
with an option to not show again for a month. Can be disabled completely
by setting extensions.zotero.showConnectorVersionWarning to false.
Currently prompts for connector versions below 5.0.35.
This is a temporary solution for #1489 until the connector checks and
warns on its own when it's outdated and most people are on a version
that does that (particularly Safari users, who don't auto-upgrade).
The progress percentage is based on the most recent transaction
(or undeterminate if this is the first session transaction)
Fix undefined function call error
Local object versions can be higher than remote versions, because we
upload in batches and only record the version from the last batch.
This could cause trouble if an object failed to upload during a Restore
to Online Library, causing it to be retried later with version 0 (unlike
during a restore when the version is omitted), causing the library to be
reset, causing any local objects with higher local versions to be
redownloaded.
Set 'control' to the id of either the value label or the textbox,
depending on whether the field is being edited. This causes NVDA to read
the label associated with the textbox, but seemingly only the first time
it's selected.
Unfortunately NVDA also does some other unpleasant things, like reading
a description from the duplicate-merging pane, which isn't the active
element in the deck, and reading the entire text, including all field
labels, of the item box) I'm not sure how much we can improve this until
we're on Electron.
VoiceOver doesn't read the textbox's label either, even though it does
so in the bookmarks window in Firefox. Maybe things have improved since
Firefox 52, so we can text again after upgrading to Firefox 60.
Addresses #1411
Moving a translated item from a non-filesEditable library to a
filesEditable library would fail (because the endpoint handler was named
"SaveItem" instead of "SaveItems").
Fixeszotero/zotero-connectors#235
E.g., if you have a main profile using ~/Zotero and create a second
"Work" profile, a "~/Zotero Work" data directory will be created
automatically and set as a custom data directory
If you entered a date that was in DST (including the current date at
00:00:00) but you were in standard time, the previous day was displayed.
Since access dates for saved items include explicit timestamps and most
people don't work between the hours of midnight and 3 a.m., this didn't
come up very often, and it was easy to miss when it did, since you'd be
unlikely to notice it for previously entered dates. (Someone noticed it
today because DST ended in Australia.)
This only affected the item pane, so citations were unaffected.
If switching from a filesEditable library to a non-filesEditable
library, files are removed. If going the other direction (including if
the original save was to a non-filesEditable library), the save is
performed again from the beginning in order to include attachments. If
switching between two filesEditable libraries, the storage directory is
just moved.
Addresses zotero/zotero-connectors#220
Move an item and its attachments to another library. Attachments are
removed as necessary if linked files or all files aren't supported in
the target library.
