Commit graph

741 commits

Author SHA1 Message Date
Dan Stillman
ce5be0bc75 Automatically download open-access PDFs when saving via the connector
If there's no translated PDF or the translated PDF fails and the item
has a DOI, check Zotero's Unpaywall mirror for possible sources and try
to download one of those.

Unlike with "Add Item by Identifier" and "Find Available PDF" in the
item context menu, this does not try the DOI/URL page, since it would
result in more data leakage and most of the time you'd be saving from
the DOI page already. We could consider offering it as an option, but
for it to be useful, you'd have to have an institutional subscription,
be on-campus or connected via VPN (for now), and be saving from
somewhere other than the main page.

A new connector endpoint, sessionProgress, takes the place of
attachmentProgress. Unlike attachmentProgress, sessionProgress can show
new attachments that have been added to the save, and with a little more
work should also be able to show when a parent item has been recognized
for a directly saved PDF.

This also adds support for custom PDF resolvers, available to all PDF
retrieval methods. I'll document those separately.

Closes #1542
2018-08-16 00:57:22 -04:00
Dan Stillman
bcf94942f9 Don't include "Zotero" in User-Agent sent to non-Zotero sites 2018-08-16 00:57:22 -04:00
Adomas Venčkauskas
01f3159b2f Ensure all failing integration tests are fixed (8d8410ff). Closes #1546 2018-08-14 11:40:19 +03:00
Dan Stillman
11f8bd20d6 Fix title display when installing style via connector (broken in 0f1687f1c7) 2018-08-13 18:17:35 -04:00
Adomas Venčkauskas
8d8410ffa0 Fix failing integration test (regr 0f1687f). Closes #1546 2018-08-13 14:22:15 +03:00
Dan Stillman
6478a375e3 Fix WebDAV tests after 276bb99fc9 2018-08-09 18:20:02 -04:00
Dan Stillman
679a6d5cc7 PDF retrieval improvements
- Add the ability to extract a PDF URL from a given webpage using the
  translation framework
- Add the ability to get open-access PDFs from landing pages from
  Unpaywall data in addition to direct PDF URLs
- Use the above functionality to improve PDF retrieval for "Add Item by
  Identifier"
- Add "Find Available PDFs" option to the item context menu to retrieve
  PDFs for existing items from the DOI or URL page or using Unpaywall
  data. The option appears for single items with a DOI or URL and no PDF,
  and it always appears when selecting multiple top-level items (but
  skips ineligible items).

PDF extraction from DOI/URL pages will currently only work with
unauthenticated access (i.e., on-campus or VPN, but not via a web-based
proxy).

Supersedes and closes #948
2018-08-07 04:58:15 -04:00
Dan Stillman
1f8cf22a35 Delete group files when deleting group
Fixes #1532
2018-07-15 23:09:34 -04:00
Dan Stillman
99584dc918 Import base-directory-relative linked files
Zotero RDF contained 'attachments:' paths when files weren't included
but they weren't imported properly
2018-06-30 09:19:09 +02:00
Dan Stillman
3c82263082 Fix Add Item by Identifier test failure due to capitalization change 2018-06-27 16:28:38 +02:00
Dan Stillman
2a7f31813e Disable JS in hidden browser when indexing HTML files without a charset
This could cause imports that linked to HTML files to hang, possibly
from network requests that failed.
2018-06-18 20:19:02 -04:00
Dan Stillman
4554998798 Add Zotero.File.moveToUnique(), replacing Zotero.moveToUnique()
New function takes string paths instead of nsIFile instances
2018-06-17 18:08:03 -04:00
Dan Stillman
f3a62f5a63 Automatically download open-access PDFs via Add Item by Identifier
When the associated-files pref is enabled, Add Item by Identifier uses a
Zotero Unpaywall mirror to find available open-access PDFs. No details
about the contents of searches are logged.
2018-06-16 14:35:44 -04:00
Dan Stillman
5ddbe433b9 Fix Backoff and Retry-After header parsing 2018-06-12 15:17:46 -04:00
Dan Stillman
3241faf503 Fix tag saving for intercepted imports from connector
Save tags in intercepted files as automatic tags
2018-05-23 21:16:18 -04:00
Dan Stillman
4acd178819 Fix user-agent handling for faked UAs from connector 2018-05-15 20:18:41 -04:00
Martynas Bagdonas
9165a0247f Fix PMID and arXiv identifiers extraction (#1498) 2018-05-07 06:04:11 -04:00
Martynas Bagdonas
c0a4fa43f0 Add a test for PDF recognition by DOI (#1496) 2018-05-04 03:14:26 -04:00
Dan Stillman
8f39e9cb36 Rename PDF recognizer tests to reflect arXiv ID lookup
Addresses #1494 and #1495
2018-05-04 01:16:04 -04:00
Dan Stillman
d0e42a1186 Automatically switch to My Library on connector save in noneditable view 2018-04-27 18:27:06 -04:00
Dan Stillman
d573a5b639 Merge pull request #1476 from adomasven:feature/connector-doc-integration
Doc Integration Endpoint for the Connector
2018-04-24 01:37:48 -04:00
Adomas Venčkauskas
593153eebe Adds a progress bar for non quick-format integration actions
The progress percentage is based on the most recent transaction
(or undeterminate if this is the first session transaction)

Fix undefined function call error
2018-04-24 01:37:10 -04:00
Dan Stillman
33f8fcfafb Fix syncing of saved search changes
Condition changes were only uploaded after every other change + sync
2018-04-22 17:27:33 -04:00
Adomas Venčkauskas
5e5b567782 Add a connector document integration endpoint
Specifically for google docs via the connector, but could potentially be
used for any integration via HTTP or connector.
2018-04-20 13:08:34 +03:00
Martynas Bagdonas
fa0576a4dd Add arXiv identifier support (#1486) 2018-04-18 13:03:10 -04:00
Dan Stillman
3835bc9110 Assign merge-tracking relations properly when merging >2 items
https://forums.zotero.org/discussion/71459/incorrect-document-refresh-after-three-item-merge
2018-04-15 17:05:55 -04:00
Frank Bennett
94271325d7 Asyncify Zotero.Relations.getByObject(), called only from item merge. (#1480)
Use getAsync() to acquire subject in Zotero.Relations.getByObject()
2018-04-14 10:26:16 -04:00
Dan Stillman
6f1833f936 Remove items from trash and My Publications when removed via sync
Zotero.Item::fromJSON() wasn't properly accounting for missing 'deleted'
or 'inPublications' properties.
2018-04-13 23:36:59 -04:00
Dan Stillman
9a304b6699 Better handling of remotely changed items in locally missing collections 2018-04-07 17:04:35 -04:00
Dan Stillman
ac4abf0ebb Avoid race conditions in conflict resolution tests 2018-04-07 17:04:00 -04:00
Dan Stillman
4f9847da04 Save parent item to correct library when recognizing PDF without DOI 2018-04-02 15:34:22 -04:00
Dan Stillman
9e955bde99 Add Zotero.Item.prototype.moveToLibrary()
Move an item and its attachments to another library. Attachments are
removed as necessary if linked files or all files aren't supported in
the target library.
2018-03-31 08:26:36 -04:00
Dan Stillman
bc141ce36b Add .allowsLinkedFiles property to Zotero.Library objects 2018-03-31 08:26:36 -04:00
Adomas Venčkauskas
bff7cee374 Add a test case for copied citations (for 9c7271c6) 2018-03-28 14:55:46 +03:00
Adomas Venčkauskas
3445519714 Fix citationByIndex[i].sortedItem errors during citation insertion
Caused by inproper handling of copy-pasted citations in documents
2018-03-26 15:31:26 +03:00
Dan Stillman
1f320e1f5d Be more lenient about Extra field values than citeproc-js
Allow fields like "Original Date: 2018" and convert them to
"original-date: 2018" when sending to citeproc-js.

For reference:

http://citeproc-js.readthedocs.io/en/latest/csl-json/markup.html#cheater-syntax-for-odd-fields
2018-03-01 17:05:24 -05:00
Dan Stillman
e1e5178869 Shorten auto-renaming pref
So we now have autoRecognizeFiles and autoRenameFiles.
2018-03-01 04:09:17 -05:00
Dan Stillman
1cfc90186f Serialize attachment indexing
Add newly added attachments to a queue, start processing it after five
seconds have passed since the last attachment was added, and process
another every half second after that unless another is added.

This queue won't survive a restart, so the queue should really be in the
DB, but this should avoid problems when adding multiple attachments at
once.

Addresses #1284
2018-03-01 01:28:28 -05:00
Dan Stillman
97661539dc Automatically retrieve metadata when saving PDFs
Applies to dragging to the collections pane or the items pane, adding
via New Item menu, or saving via the connector server

If the renaming pref is enabled, the PDF is renamed after recognition.

Can be disabled in the preferences

Closes #917
2018-03-01 01:28:28 -05:00
Dan Stillman
3cc92fe1e9 Fix PDF recognition within collection (broken by f8b41c971c) 2018-03-01 01:20:29 -05:00
Dan Stillman
f99038fd37 Fix recognizer dialog with multiple tests
Do cleanup on 'unload' rather than 'close' (which is limited to a click
on the close button and doesn't get called for win.close()) and clear
the queue after each test.
2018-03-01 01:20:29 -05:00
Dan Stillman
f8b41c971c Automatic attachment file renaming changes
Automatic renaming is now done for dragging of an external file onto an
item (as added in 7cb95f41) as well as dragging as a linked file,
dragging a non-native (not text or HTML) URL, "Attach Link to File…",
"Attach Stored Copy of File…", and "Retrieve Metadata for PDF". It only
applies if a single file is being added and if the parent item has no
non-HTML attachments. By default, the renaming only applies to PDFs, but
this can be changed with the renameAttachmentFiles.automatic.fileTypes
hidden pref.

A new General pref, "Automatically rename attachment files using parent
metadata", controls whether the renaming happens.

Files saved via web translators are renamed regardless of this pref,
because they would often be gibberish otherwise.

Closes #113
2018-02-27 18:45:03 -05:00
Dan Stillman
f5b1ee44f3 Add Zotero.File.rename() (extracted from Zotero.Item::renameAttachmentFile()) 2018-02-27 14:24:26 -05:00
Dan Stillman
c0b63e5928 Better handling of 403 for attachment metadata upload
Check file-editing access for the group from the API before offering to
reset, update the filesEditable setting properly, and restart the sync
automatically after resetting.
2018-02-23 17:59:32 -05:00
Martynas Bagdonas
991a50d090 New PDF recognizer 2018-02-20 09:32:06 +02:00
Martynas Bagdonas
d810deaa9c Update PDF tools version and fix fulltext test 2018-02-14 04:00:02 -05:00
Dan Stillman
3f6ecc0021 Fix "Can't queue event outside of a transaction"
If a transaction took over 30 seconds and another transaction timed out
waiting for it, the second transaction would reset the notifier queue,
but if the first transaction then tried to queue an event, it would fail
with this error and roll back. (It would be nice to figure out why
transactions are taking over 30 seconds, though.)
2018-02-08 02:07:44 -05:00
Dan Stillman
c8cf9b9e6f Support for connector-based save target selection
- Updates /saveItems and /saveSnapshot to take a sessionID
- Provides a list of editable collections in the current library
- Adds an /updateSession method that takes a sessionID and updates the
  collection and tags of any items saved via that operation (and changes
  the currently selected collection)

Cross-library changes are not yet supported
2018-02-07 04:04:37 -05:00
Dan Stillman
45ddf9827c Reset PDF tools path for tests in resetDB()
And include path on error when running PDF tool
2018-02-07 04:04:37 -05:00
Dan Stillman
38411fb56c Allow dragging parent items to collection if children are selected
This is a simplified version of the fix from #872. Unlike the proposal
in #36, this doesn't require all child items to be selected, since in a
search some children might be grayed out. If the child of an unselected
parent item is included, the drag isn't allowed.

Closes #36
2018-02-03 04:15:09 -05:00