Commit graph

1059 commits

Author SHA1 Message Date
Dan Stillman
806ba2eb72 Fix test failure after 40fe85b274 2020-04-29 16:29:22 -04:00
Dan Stillman
2c046a227b Fix sync error on collection/search conflict
> Invalid data error for collection [key] in [library]: Unknown
> collection property 'dateAdded'"

Regression from 4b60c6ca27, probably
2020-04-23 03:56:14 -04:00
Dan Stillman
81e2bee452 Fix 'false' filename after case-only rename on Macs
Zotero.File.move() now forces `overwrite` if the old and new filenames
differ only by case, since otherwise on a case-insensitive filesystem
OS.File.move() does an existence check and thinks that the target file
exists.
2020-04-22 00:01:28 -04:00
Dan Stillman
ad13313924 Don't change existing sync cache object on 'unchanged' response
Previously, if an object was uploaded but the API returned 'unchanged',
the uploaded data would be written to the sync cache, which, given that
most requests are patch requests, could result in an empty or mostly
empty object being saved to the sync cache. That would cause the next
sync to treat most/all local fields as changed and either upload them
unnecessarily or trigger a conflict instead of merging changes
automatically.
2020-04-14 05:07:50 -04:00
Dan Stillman
083588e211 Save correct data to cache when merging non-conflicting sync changes
When non-conflicting changes were automatically merged, the local object
would be correctly marked as unsynced, but the merged object rather than
the remote object would be saved to the sync cache. When the object was
then uploaded, it matched the cache version exactly, so an empty patch
object (other than an unchanged dateModified, which is always included)
would be uploaded and the local change wouldn't make it to the server.
The empty patch would result in an 'unchanged' response, which would
cause the empty patch object to be saved to the sync cache (which is a
bug that I'll fix separately). If the local object was modified again,
the patch would include all fields (since the cache object was empty)
and the local change would be uploaded, but there could also be
unnecessary conflicts due to it looking like all local fields had been
modified.

This patch causes the remote object to be saved to the sync cache
instead, so the local change looks like a local edit and is correctly
uploaded.
2020-04-13 03:42:28 -04:00
Dan Stillman
617564982c Include only related-item relations when duplicating items
Don't include linked-object or replaced-item relations. Previously, if
you duplicated an item, modified it to represent a different source, and
dragged it to another library where you had already copied the original
item, the new item wouldn't be transferred.

https://forums.zotero.org/discussion/comment/353246/#Comment_353246
2020-04-13 01:31:21 -04:00
Dan Stillman
e9ea9ae171 Leave Event Place and Publisher Place in Extra on import
These CSL variables are both currently mapped to the Zotero Place field,
but that's imprecise and will change:

https://github.com/citation-style-language/zotero-bits/issues/6

So for now, don't convert these to a Place line or make them eligible
for migration to a real field (once we turn that on).

https://groups.google.com/d/msgid/zotero-dev/806a22e3-3d6a-4d86-8747-10c787291a93%40googlegroups.com
2020-04-04 04:17:51 -04:00
Dan Stillman
0e3d707576 Remove a couple redundant lines in Extra on RDF import
`publicationTitle`/`reporter` (and fields mapped to `publicationTitle`)
and `meetingName`/`conferenceName`

I assume these should just be base-field mappings, but since they're
not, they're not automatically deduplicated in `fromJSON()` and need to
be handled separately.

https://groups.google.com/d/msgid/zotero-dev/806a22e3-3d6a-4d86-8747-10c787291a93%40googlegroups.com
2020-04-04 04:17:27 -04:00
Dan Stillman
91ca6d2ba6 extraToCSL(): Fix "zoteroField[1] is undefined" on single-character field
E.g., "a:" on a line

https://forums.zotero.org/discussion/82286/new-line-in-extra-field-induces-error-during-bibliography-generation
2020-04-03 18:13:10 -04:00
Dan Stillman
8b0dbc7708 Update fileInterface test after 0f212bdd76 2020-03-24 17:53:13 -04:00
Dan Stillman
679148d8d6 Don't store "Version Number" in Extra for books coming from RDF
The RDF translator assigns 'versionNumber' in addition to 'edition'.
2020-03-24 14:48:09 -04:00
Dan Stillman
d3601bba24 Remove incorrect test for Extra 'type:' extraction 2020-03-23 19:09:59 -04:00
Dan Stillman
52d5b68564 Update 'type:' handling when migrating Extra lines
- When changing type based on 'type:' line, move existing fields that
  are no longer valid to Extra
- Remove 'type:' line with CSL type if the item's existing type is one
  of the types mapped to it
2020-03-22 15:19:24 -04:00
Dan Stillman
02b43cbfac Refactor migrateExtraFields() tests 2020-03-21 00:56:55 -04:00
Dan Stillman
592188bfc5 Disabled test for keeping unfiled item selected on add to collection
A fix will need to wait for the Reactified items tree

Addresses #1799
2020-03-19 13:08:23 -04:00
Dan Stillman
f1816c12ad Fix test from 28ea77dae 2020-03-17 14:17:02 -04:00
Dan Stillman
28ea77dae1 Add test for ignoring invalid item type in Extra 2020-03-17 13:51:19 -04:00
Dan Stillman
3f2998bb6b Don't extract 'type: note' or 'type: attachment' from Extra 2020-03-17 13:50:45 -04:00
Dan Stillman
d8d07362ba Add additional test for creator migration from Extra 2020-03-17 10:37:16 -04:00
Dan Stillman
dd9ec54c3b Don't try to migrate Extra fields in read-only libraries 2020-03-15 19:52:56 -04:00
Dan Stillman
23c936f016 Fix creator migration from Extra 2020-03-15 13:51:04 -04:00
Dan Stillman
0679809735 Fix invalid collection nesting in DB integrity check
It shouldn't be possible to nest two collections inside each other, but
if it happens, fix it in the integrity check.

Also detect it from CollectionTreeView::expandToCollection() (used when
showing the collections containing an item) and crash Zotero with a flag
to run an integrity check after restart. Previously, this would result
in an infinite loop.

This may be the cause of some of the collection disappearances people
have reported. If parentCollectionID never leads to a null, the
collection won't appear anywhere in the tree.

TODO:

- Figure out how this is happening
- Detect and fix it automatically for people it's happened to
2020-03-13 18:07:20 -04:00
Dan Stillman
9175f9ade8 Don't use "Create Bib" locale for Quick Copy when no explicit locale set
If you'd never set a Quick Copy locale, the option would show as using
the current locale, but Quick Copy itself would use the last locale from
"Create Bibliography from Items". That was a side effect of behavior we
put in place in 2015 so that documents created before 4.0.27 that relied
on the removed bibliographyLocale pref would continue using the migrated
locale, but now that we've had an explicit locale option for years in
the document preferences I think we can stop doing that.
2020-03-13 17:06:29 -04:00
Dan Stillman
d389a71280 Download remotely updated files in "as needed" file sync mode
Previously, files updated remotely wouldn't be downloaded in "as needed"
mode if a copy of the file already existed locally and could only be
re-downloaded by deleting the file via Show File.

This causes remotely modified files that exist locally to be downloaded
at sync time, even in "as needed" mode, by marking them as
"force_download". While this might not be ideal for people who use "as
needed" to limit data transfer, it's better for people who use it simply
to limit local storage, and ending up with an outdated file while
offline seems worse than a little bit of extra data transfer.

In the future, we'll likely also provide ways to explicitly download and
remove files, so keeping chosen files in sync makes sense.

Files modified remotely before this change (which were marked as
"to_download" instead of "force_download") won't be downloaded as sync
time in "as needed" mode, but they'll now be re-downloaded on open.

Fixes #1322
2020-03-09 01:19:52 -04:00
Dan Stillman
76a1535a60 Full-text indexing improvements
- Use full-text cache file from syncing if available when reindexing via
  info pane or Rebuild Index → Index Unindexed Items. Only discard it for
  full index rebuild. This allows Index Unindexed Items to be used to
  force immediate processing of queued content from syncing and avoids
  unnecessary syncing back of identical content. Previously, the cache
  file was used for a manual index only when the local file didn't exist.
- When rebuilding index, don't clear indexed items with missing local
  file that are missing stats due to a pre-411180ef bug.
- indexItems() now takes an 'options' object as its second parameter
- Minor code cleanup
2020-03-09 01:19:52 -04:00
Dan Stillman
411180ef83 Full-text indexing fixes
- Don't clear item's index stats (and show "Unknown") when an item is
  reindexed remotely and the content matches the local content
- Always update an item's state and its stats in the same query, to
  avoid incorrect feedback immediately after indexing
- Clean up `setItemContent()` tests
2020-03-06 03:11:16 -05:00
Dan Stillman
da5e8c549e Convert Zotero fields in Extra to CSL fields for citeproc-js 2020-03-02 01:38:10 -05:00
Dan Stillman
83cc65eea6 Fix parsing of CSL date fields in extractExtraField() 2020-03-02 01:34:27 -05:00
Dan Stillman
a9dee2f487 Item::fromJSON(): Remove invalid-for-type Type-mapped fields
"Type: [val]` in Extra means Item Type to citeproc-js, and Type values
from translators mostly aren't going to be useful if the item type
doesn't have a Type-mapped field.

https://forums.zotero.org/discussion/comment/348864/#Comment_348864
2020-02-17 11:39:35 -05:00
Dan Stillman
b2bf60e1d7 Item.fromJSON(): Dedupe invalid-for-type fields when storing in Extra
3de54455f6 removed redundant base-mapped fields when a valid-for-type
field was also set, but that still left duplicate fields in Extra when a
valid field wasn't set. This will happen until translators (most notably
Embedded Metadata) are fixed to stop setting redundant fields.

https://forums.zotero.org/discussion/81262/translator-error-sage-lots-of-extra-data-in-extra
2020-02-17 00:22:34 -05:00
Dan Stillman
11caa1b719 Group strict-mode and non-strict-mode tests for Item::fromJSON() 2020-02-16 18:05:45 -05:00
Dan Stillman
b41734924d Further fixing of "Too many sync requests" error
Follow-up to 804a898c98

Addresses #1788
2020-02-16 18:05:45 -05:00
Dan Stillman
804a898c98 Hopefully fix "Too many sync requests" after file upload 412
Addresses #1788
2020-02-16 13:06:49 -05:00
Dan Stillman
a53f363b8d Additional fix for search crash with includeParentsAndChildren
Follow-up to 76081ab05
2020-02-11 13:09:54 -05:00
Dan Stillman
76081ab05f Fix crash when search uses no-op condition and includeParentsAndChildren
E.g., a nonexistent saved search
2020-02-11 00:23:45 -05:00
Dan Stillman
86a5c46b1e Find Available PDF: Don't mark URLs that redirect as tried
https://forums.zotero.org/discussion/81182
2020-02-02 23:47:40 -05:00
Dan Stillman
4e11c7927d Don't skip creator lines in Extra in fromJSON
Regression in 3de54455f6
2020-01-27 22:01:25 -05:00
Dan Stillman
3de54455f6 Automatically save unknown/invalid fields to Extra in non-strict mode
This is a prerequisite for starting to use new fields in translators,
since otherwise switching from, say, storing originalDate in Extra to
using an originalDate field would cause the value to be lost in clients
without the newer schema.

Closes #1504
2020-01-26 03:29:23 -05:00
Dan Stillman
3d2afa9c7f Additional adjustments to URL handling
Follow-up to df40ee7216

- Restore opening of non-HTTP URL schemes in link attachments
- Remove scheme whitelist for link attachments, since it's not enforced
  via the API anyway and we prompt before an external application is
  opened. Instead, just block a few schemes (e.g., 'javascript') from
  launching.

TODO:

- Provide some way to change/reset an application association if the
  user checks the box to automatically open that scheme.
- Show an error message if a link attachment with an invalid URL is
  double-clicked
2020-01-04 02:30:12 -05:00
Dan Stillman
6070743ff0 Fix Nextcloud WebDAV syncing
Add Zotero.HTTP.CookieBlocker mechanism to block storing and sending
of all cookies for a given URL prefix, and use that for the configured
WebDAV URL.

https://forums.zotero.org/discussion/80429/sync-error-in-5-0-80
2020-01-04 02:29:52 -05:00
Dan Stillman
12ccbf45c8 Make HTTP tests more reliable 2019-12-23 04:31:24 -05:00
Dan Stillman
9c380c362a Hopefully fix 503 from NextCloud (since 5.0.78)
https://forums.zotero.org/discussion/comment/344790/#Comment_344790
2019-12-22 16:32:36 -05:00
Dan Stillman
b7dc0d8b0a Fix parsing of pre-1000 years
If a leading zero (e.g., '068'), parse as actual year instead of
19xx/20xx.
2019-12-19 02:15:36 -05:00
Dan Stillman
df40ee7216 Adjust URL launching behavior
- Support launching plausible HTTP URLs without schemes from
  Zotero.loadURI(), and use that when launching from URL field
- Show correct cursor feedback on URL label -- only show pointer if the
  URL is launchable
- Don't launch non-HTTP URLs (e.g., zotero://) from URL field
- Don't open HTTP URLs in viewer when using ZoteroPane.launchURI()
2019-12-16 14:56:06 -05:00
Dan Stillman
bbbd02444b Restore 'yesterday'/'today'/'tomorrow' parsing for dates in searches
Follow-up to a549a64de9, which removed it from strToDate()
2019-12-06 03:12:48 -07:00
Dan Stillman
61cebbd8f7 Fix display of a couple WebDAV verification errors
"spec is undefined"

Fixes #1745
2019-12-01 03:39:54 -07:00
Dan Stillman
fbb54a7621 Fix "string is undefined" export error
Regression from a549a64de9
2019-11-29 01:50:43 -07:00
Dan Stillman
8c7677a009 Fix out-of-memory error syncing collections nested inside each other
It shouldn't be possible for collections to be nested this way, if it
happens, it shouldn't result in an infinite loop.

This removes one of the parent assignments at sync time.
2019-11-26 15:30:43 -07:00
Dan Stillman
5791ffeb16 Reactify item tags box
Improvements:

- Fixes autocomplete text remaining in field after selection in Fx60
- No more text or icon shifting on select (tested on macOS)

Changes:

- Tags are now selected on mousedown with no active state, as in web
  library

Regressions:

- Tooltip with tag type doesn't appear when hovering over icon
- Pressing Tab after modifying a tag loses focus
- Right-click in textbox shows custom menu instead of default text
  editing context menu (Cut/Copy/Paste)

To-do:

- Switch to this version for note tags box
- Style colored tags in autocomplete drop-down? Sort to top?
- Only show delete button on row hover, as in web library?
2019-11-08 06:41:06 -05:00
Dan Stillman
cbfa4be437 Add 'Type:' to extraToCSL() test
Follow-up to #1744
2019-10-26 17:05:45 -04:00
Dan Stillman
c25fbe7c1c Remove all colored tags on selected items if 0 is pressed
Like Thunderbird
2019-10-26 16:45:04 -04:00
Dan Stillman
52fd91950d Fix deleted item fields reappearing on sync
This was a regression from 4b60c6ca27. The original plan for introducing
new fields was to have them save to the sync cache even if they weren't
supported by the current Zotero version and process them on upgrade, and
so I changed `DataObjectUtilities.patch()` to omit fields that didn't
exist locally when patching the sync cache JSON so they wouldn't be
wiped on the server. That caused this bug where locally deleted fields
were restored on every sync. It's also no longer necessary now that
we decided to just reject unknown fields from saving, so we can just
revert to the previous behavior of blanking out locally missing fields
(with the tweak that fields that are already false or empty in the base
version can be omitted).
2019-10-24 01:22:00 -04:00
Dan Stillman
513f7d6555 Fx60: Fix localization
Also fixes #1690 (at least on Catalina), and possibly other things

general.useragent.locale and intl.locale.matchOS are no longer used.
2019-10-23 19:08:13 -04:00
Dan Stillman
d4f682aa88 Merge branch 'fx60' 2019-10-21 21:44:01 -04:00
Dan Stillman
956813ac1f Fix "err is undefined" on 200 response for nonexistent WebDAV file
Fixes #1741
2019-10-20 15:03:18 -04:00
Dan Stillman
1710eb1c4b Don't store unknown/invalid fields in Extra in non-strict mode
And fix a couple things for if we turn it back on

This code came along with the type/field handling overhaul, but I think
it was originally intended for handling unknown fields during sync
before we decided on strict mode, so it wasn't finished and causes
various problems [1]. It could still be useful for preserving fields
from translators before they're available on items, but the better fix
there is just to add the missing fields, so I'm not sure if we'll end up
needing it.

[1] https://groups.google.com/d/msg/zotero-dev/a1IPUJ2m_3s/hfmdK2P3BwAJ
2019-10-18 03:37:24 -04:00
Dan Stillman
89672ed0a4 Fix "c1 is undefined" sync CR error 2019-10-09 18:29:04 -04:00
Dan Stillman
44d4586242 Fix test after 8aeb6f7fe4 2019-09-28 07:40:01 -04:00
Dan Stillman
5723683b3b Fix "attachmentSyncedModificationTime must be a number" sync error
https://forums.zotero.org/discussion/79011/zotero-error-report
2019-09-21 01:59:09 -04:00
Dan Stillman
4b7cdddb4a Change apply-to-all checkbox text for some conflicts
Say "Use the [local|remote] version for all remaining conflicts" for
everything instead of saying "Use [local|remote] fields for all
remaining conflicts" for some conflicts.

This also fixes a test failure after 54343c49fb.
2019-09-21 01:35:36 -04:00
Dan Stillman
4b60c6ca27 Type/field handling overhaul
This changes the way item types, item fields, creator types, and CSL
mappings are defined and handled, in preparation for updated types and
fields.

Instead of being predefined in SQL files or code, type/field info is
read from a bundled JSON file shared with other parts of the Zotero
ecosystem [1], referred to as the "global schema". Updates to the
bundled schema file are automatically applied to the database at first
run, allowing changes to be made consistently across apps.

When syncing, invalid JSON properties are now rejected instead of being
ignored and processed later, which will allow for schema changes to be
made without causing problems in existing clients. We considered many
alternative approaches, but this approach is by far the simplest,
safest, and most transparent to the user.

For now, there are no actual changes to types and fields, since we'll
first need to do a sync cut-off for earlier versions that don't reject
invalid properties.

For third-party code, the main change is that type and field IDs should
no longer be hard-coded, since they may not be consistent in new
installs. For example, code should use `Zotero.ItemTypes.getID('note')`
instead of hard-coding `1`.

[1] https://github.com/zotero/zotero-schema
2019-09-16 02:27:22 -04:00
Dan Stillman
20d7ef6c0b Add separate .eslintrc for tests 2019-09-16 01:28:52 -04:00
Dan Stillman
e8dd1f7824 Test updates for newer Sinon 2019-09-16 01:27:30 -04:00
Dan Stillman
25e34b7b99 Update WebDAV request timeouts for HTTP.request() auto 5xx retry 2019-09-16 01:27:18 -04:00
Dan Stillman
dc60e5f840 HTTP.request() improvements
- Move 5xx retries and connection checking out of the sync API client
  and into HTTP.request() so that they apply to all requests. 429 handling
  remains in the API client, since not all callers necessarily want to
  handle that the same way. Callers can still handle 5xx themselves by
  including the relevant 5xx status codes in `successCodes` or by passing
  `errorDelayMax: 0`.
- Add `cancellerReceiver` option, which is a callback that receives a
  function that will cancel the request, whether it's an active request
  or an automatic delay before a 5xx retry.

This also updates Sinon to 7.3.2.
2019-09-16 01:26:34 -04:00
Dan Stillman
f04f186396 Add missing yield in sync code 2019-09-16 00:47:45 -04:00
Dan Stillman
7ed0d8a408 Fix recognizePDF test after DOI translator change 2019-09-16 00:46:49 -04:00
Dan Stillman
54343c49fb Fix "getNote() can only be called on notes and attachments" CR error
This bug may be as old as the sync system itself. It could occur if
there were conflicts for both a note and a regular item in the same
batch.
2019-09-16 00:39:06 -04:00
Dan Stillman
b08bd6849e Fx60: Update DB query onRow() behavior
onRow() handlers now get passed a cancellation function as a second
argument
2019-08-27 06:00:35 -04:00
Dan Stillman
433794916a Fx60: Remove instances of nsILocalFile
nsILocalFile was merged into nsIFile, so either replace it with that or
switch to Zotero.File.pathToFile().
2019-08-27 00:47:39 -04:00
Dan Stillman
6f965251ed Add FilePicker module to replace nsIFilePicker
`nsIFilePicker::show()` is removed in Firefox 60 in favor of `open()`,
which takes a callback (and apparently has been preferred for a long
time).

There's no point switching to that, so this module is a version of
nsIFilePicker with an async `show()` that returns a promise and some
XPCOM-isms replaced (e.g., string paths instead of nsIFile).
2019-08-27 00:47:39 -04:00
Dan Stillman
75daaa3b7c Fix incorrect 'return' in test 2019-08-27 00:47:39 -04:00
Dan Stillman
8c59df435f Fx60: Fix snapshot filenames
nsIURL doesn't seem to work anymore, so add Zotero.Utilities.parseURL(),
which uses the `url` package from NPM and adds fileName, fileExtension,
and fileBaseName.
2019-08-27 00:47:39 -04:00
Dan Stillman
bb59429664 Add "Convert Linked Files to Stored Files…" menu option
In new File → Manage Attachments submenu

Closes #1637
2019-08-19 05:00:32 -04:00
Dan Stillman
c2e79c0717 Close window after protocol handler tests
Regression from fda002ec34
2019-08-03 02:48:01 -04:00
Dan Stillman
fda002ec34 Beginning of tests for zotero://select 2019-08-02 02:29:08 -04:00
Dan Stillman
9becefc3fd Fix test for AMA subtitle capitalization 2019-08-01 17:32:56 -04:00
Dan Stillman
9b82373f70 Add ability for Scaffold to provide alternative translators
Zotero.Translate::setTranslatorProviderMethods(methods) can be used to
provide custom 'get' and 'getAllForType' methods that override the
default Zotero.Translators methods.
2019-07-24 04:31:11 -04:00
Dan Stillman
2603373b86 Additional protections for HTTP endpoints
Reject browser-based requests that don't require a CORS preflight
request [1] if they don't come from the connector or include
Zotero-Allowed-Request: 1

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#Simple_requests
2019-07-15 07:30:12 -04:00
Dan Stillman
708b8798c2 Check for X-Zotero-Connector-API-Version header in /connector/import 2019-07-12 02:08:55 -04:00
Adomas Venčkauskas
c4f40c38e9 Unsuppress accidental test suppression from fe01e17 2019-07-04 15:46:24 +03:00
Adomas Venčkauskas
fe01e17e93 Fix integration test breakage (regression cd14a53) 2019-07-04 15:38:46 +03:00
Adomas Venčkauskas
cd14a536dc Add retracted citation warning tests. Fix a bug with embedded prompts 2019-07-04 15:16:26 +03:00
Dan Stillman
82e7f67df8 Fix retraction test 2019-07-04 07:58:56 -04:00
Dan Stillman
0beddb9680 Add flags to disable retraction warnings
Separate flags for hiding the retraction altogether and for hiding
citation warnings for it

New functions:

Zotero.Retractions.hideRetraction(item)
Zotero.Retractions.shouldShowCitationWarning(item)
Zotero.Retractions.disableCitationWarningsForItem(item)

Addresses #1710
2019-07-03 01:23:02 -04:00
Dan Stillman
f49d5805cd Apply subtitle capitalization to additional styles
This also fixes a style initialization error when the style id is
invalid, which was a regression in a0be2da42c.
2019-07-01 22:39:12 -04:00
Dan Stillman
ead93b6ccc Stop uploading files on quota error until next manual sync or restart 2019-06-22 05:29:47 -04:00
Dan Stillman
50a4308514 Remove retraction flag from items that no longer match 2019-06-19 06:41:33 -04:00
Dan Stillman
b4b19c4811 Add Zotero.Retractions.getRetractionsFromJSON()
Addresses #1703
2019-06-12 01:05:49 -04:00
Dan Stillman
d353439980 Add retractions.enabled hidden pref to disable retraction checking 2019-06-11 21:24:17 -04:00
Dan Stillman
5c03813d81 Add Retracted Items virtual collection
Shown automatically when retracted items are detected
2019-06-10 02:38:42 -04:00
Dan Stillman
79a8917bdc Fix test failure after 3727d0f559 2019-06-07 03:17:26 -04:00
Dan Stillman
7f4f2770ba Add notification banner when retracted items are found
And other retraction tweaks
2019-06-07 01:15:17 -04:00
Dan Stillman
985f10d5ed Fix WebDAV test after 1f1b2e9b2 2019-06-03 03:35:32 -04:00
Dan Stillman
c0e2aa03d3 Fix test for delay after second library version conflict 2019-05-31 02:44:41 -04:00
Dan Stillman
e33a2d730d Fix tag type handling when merging items
Most importantly, don't change all tags to manual on the merged item.
2019-05-29 05:43:33 -04:00
Adomas Ven
48778f2847 Document export-import UI and integration code (#1501) 2019-05-14 21:06:18 -04:00
Dan Stillman
3fbb17a2e6 Properly upload local changes after automatic conflict resolution
If an object changed on both sides and the changes were either
non-conflicting or identical but there were other local changes, the
local object was incorrectly being marked as synced, causing it not to
be uploaded until it was next modified locally.
2019-05-08 04:34:21 -04:00
Dan Stillman
ec0b993a17 Fix erroneous sync conflict if creators are changed in the same way 2019-05-03 01:15:27 -04:00
Adomas Venčkauskas
78c3d5808b Fix citeproc errors due to improper citeproc state updates
The error is triggered upon initial interaction with a doc after Zotero
restart or if new external citations (copied into the document) are
peresnt and `session.updateSession()` is called without a subsequent
`session.updateDocument()` call. `session.updateSession()` is called
without a subsequent `session.updateDocument()` call every time the
user cancels a citation insert.

More specifically, `session.updateSession()` is called every time a
citation dialog is invoked. It retrieves all citations and writes them
into a local `session.citationsByIndex` object. Moreover, it marks
each citation that hasn't seen before in a `session.newIndices` object.
`session.newIndices` is there to ensure that we load every new citation
into citeproc upon document update. This object is built by marking any
citation that does not appear in the previous invocation's list of
citations as new. However, if the document is never updated (because the
user cancels the insertion) then the new indices are not loaded
into citeproc. This commit fixes that, by excluding citeproc unloaded
items from the previous invocation's citation list.
2019-04-08 17:43:52 +03:00
Frank Bennett
e618410eb2 Export CSL JSON with title-short rather than shortTitle 2019-04-04 00:45:32 +09:00
Dan Stillman
9ed48f3837 Keep earliest Date Added when merging items
Closes #1669
2019-04-02 14:01:25 -04:00
Dan Stillman
d9cee322cd Tag selector performance overhaul
- Use react-virtualized to render tags on demand, reducing the number
  of DOM elements from potentially tens of thousands to <100. This
  requires tags to be absolutely positioned, so sizing and
  positioning need to be precomputed rather than relying on CSS.
- Avoid unnecessary refreshes, speed up tag retrieval, and optimize
  sorting
- Debounce reflowing when resizing tag selector

Also:

- Scroll to top when changing collections
- Allow tags to take up full width of tag selector without truncation

Closes #1649
Closes #281
2019-03-28 06:28:34 -04:00
Dan Stillman
71f9420cff Fix linked-file rename pref when retrieving metadata
Due to a typo in d0f7fd6df7, linked files were still being renamed even
with the pref off if metadata was found for the file. The test I added
was only for adding a file to an existing item, which didn't trigger
metadata retrieval.

This also adds a hook for stubbing the actual PDF recognition process so
we can test certain behaviors without making HTTP requests.
2019-03-27 06:43:07 -04:00
Dan Stillman
49226edd07 Temporarily ignore collection-title for podcast in itemFromCSLJSON test
See #1667
2019-03-21 02:22:07 -04:00
Dan Stillman
8f1f1f1fba Update itemFromCSLJSON test for podcast broadcast mapping
Follow up to bf4deeff8f
2019-03-21 02:20:38 -04:00
Dan Stillman
03941dafe0 Fix tests after 42667e7090
It seems like the Zotero.Utilities.debounce() on handleSearch() in
tagSelector.jsx was somehow causing the function to be run without being
triggered from the onSearch events, resulting in an extra render. I'm
not sure why that was happening, but it's fixed now that there's no
longer a debounce() there.
2019-03-20 08:54:48 -04:00
Dan Stillman
a92b29cebf Fix race conditions in tag selector tests
And take an optional second parameter in waitForTagSelector() to
indicate how many updates to wait for, since certain operations trigger
two updates, one from notify() and the other from onItemViewChanged().
2019-03-18 04:58:50 -04:00
Dan Stillman
b30d31c4ee Fix tag selector test 2019-03-15 21:57:38 -04:00
Dan Stillman
7717df637e Fix hang in server_connector test after 94ccba45b
The test incorrectly contained an item wait for an attachment that
doesn't get saved to the group library, but the test was passing because
the main item was being moved to the group twice. 94ccba45b somehow
fixed that, and since the behavior during the test is now correct, I'm
not going to worry about this unless we notice a problem.
2019-03-15 21:01:12 -04:00
Dan Stillman
94ccba45b9 Avoid unnecessary tag queries (regression from React tag selector) 2019-03-15 12:46:27 -04:00
Adomas Venčkauskas
72fb67d15b Restore "Delete All Automatic Tags" menu option for tag selector. Closes #1660 2019-03-06 16:22:05 +02:00
Dan Stillman
b20bf345e6 Address race condition in tag selector tests (#1659) 2019-03-06 07:27:11 -05:00
Adomas Venčkauskas
e8f186479f Address tag selector tests race conditions. Addresses #1659 2019-03-06 13:13:40 +02:00
Dan Stillman
d7dc5670d5 When clearing automatic tags, don't delete manual tags with same name 2019-03-05 07:55:00 -05:00
Dan Stillman
fc43514ff0 Deselect tags when deleted (regression from Reactification) 2019-03-05 07:55:00 -05:00
Dan Stillman
04779d8d1c Add import option for storing or linking files
This allows files in Mendeley imports to be stored and files in
RIS/BibTeX/etc. to be linked.

Closes #329
2019-03-02 06:24:33 -05:00
Dan Stillman
d0f7fd6df7 Add separate "Rename linked files" pref for auto-renaming
Defaulting to false

People who've been relying on auto-renaming of linked files for the last
year will have to turn this back on.
2019-02-26 22:23:43 -05:00
Dan Stillman
945c413c42 Closes #1644, Allow selecting multiple items with zotero://select
This adds selectItems() to ZoteroPane and collectionTreeView and removes
the ancient, unused 'expand' argument to selectItem(), which didn't
really make sense there. It also includes a new
itemTreeView::ensureRowsAreVisible() that tries to scroll to an
appropriate place (or, better yet, not scroll at all) given the
specified rows and page size.
2019-02-23 17:35:09 -05:00
Dan Stillman
4d3625f101 Fix tests after removing RECOGNIZE_URL 2019-02-19 04:37:22 -05:00
Dan Stillman
1061893998 "Attachment Content" search improvements
- Fix incorrect results for ANY search with multiple "Attachment
  Content" conditions and no other conditions
- Dramatically speed up single-word searches by avoiding unnecessary
  text scans (which probably addresses #1595)
- Clean up code
2019-02-19 04:10:25 -05:00
Frank Bennett
2c136ffbb0 Add test of multiple new insertions with implicit update on the first 2019-02-13 10:18:05 +02:00
Frank Bennett
bffbe00399 Test of implicit citation update 2019-02-13 10:17:58 +02:00
Frank Bennett
0e18c5f1de Test for bibliography sort update on change to item 2019-02-10 11:07:10 +09:00
Dan Stillman
3942b0e0d5 Fix a tag selector test after b1fad505d7 2019-01-30 04:44:16 -05:00
Dan Stillman
b1fad505d7 Fix sorting of colored tags after Reactification
Regressed to alphabetic sorting instead of number-key sorting
2019-01-29 22:18:49 -05:00
Dan Stillman
2d71b13ce0 Fix some spurious failures in PDF retrieval test 2019-01-29 07:35:39 -05:00
Dan Stillman
c2a637472e Fix foreign key constraint error for collections with all-numeric keys
This was like a lottery where instead of winning something your library
stopped syncing.
2019-01-29 05:09:00 -05:00
Dan Stillman
8b4bb62efc Don't load linked URLs during import
Regression from 5a6a772ca2, I think
2019-01-26 00:20:26 -05:00
Adomas Venčkauskas
f59f3c5f7e Fix a failing /connector/updateSession test (regr 90a70f7) 2019-01-22 12:25:15 +02:00
Dan Stillman
eb5d28b626
Merge pull request #1606 from adomasven/feature/reactify
Reactified Tag Selector
2019-01-21 04:07:28 -05:00
Adomas Venčkauskas
a24cada451 React Tag Selector polish, i18n and tests
- Added icon-button UI code for the menubutton
- Upgrade to React 16 to allow non-standard attrs, such as `tooltiptext`
to support XUL tooltips
- Add i18n support for React UI elements
- Update tests for reactified tag selector
2019-01-21 11:01:27 +02:00
Philipp Zumstein
97e53c5230 Fix some typos found by codespell (#1570)
$ codespell --skip="./.git,./styles,./translators,./chrome/locale,./resource,./chrome/content/zotero/xpcom/xregexp,./chrome/content/zotero/xpcom/rdf,./chrome/content/zotero/xpcom/citeproc.js,./test/resource,./node_modules" -q 2 -i 3 -w
2019-01-20 20:39:27 -05:00
Dan Stillman
9c2d0d7272 Add skipped test for importing related items from Zotero RDF
This is hard to do currently because the natural place to do it (and
where the previous seeAlso stuff was done) is translate_item.js, but
with async import translators that now only gets one item at a time,
whereas saving item relations requires all items to be saved. So this
would probably need to be done in the import code in translate.js.

It might also require undoing
https://github.com/zotero/zotero/pull/453 so that getResourceURI() works
on notes and figuring out another solution for the problem that was
trying to solve.
2019-01-14 02:36:59 -05:00
Dan Stillman
c8fb2ecd1f Add test for exporting related items in Zotero RDF
These aren't currently imported, but better to at least include them.
2019-01-14 02:29:09 -05:00
Dan Stillman
0d5abb018a Ignore invalid paths during export
Invalid paths, including Windows UNC paths on other OSes, caused exports
to fail. Now they're ignored, which is what we do for other missing
attachment files.

Fixes #1622
2019-01-12 02:34:31 -05:00
Dan Stillman
9c0f5998a3 Use correct time-zone offset for access dates in bibliographies
Fixes #1069
2019-01-10 03:14:28 -05:00
Dan Stillman
5a6a772ca2 Fix linked-URL attachments not being saved to groups without files
It looks like this may have been broken for years.
2019-01-08 03:45:26 -05:00
Dan Stillman
506ed313da Fix moving item via connector to library with files from one without
Broken in ce5be0bc75 for non-webpage items
2019-01-07 05:16:17 -05:00
Dan Stillman
7575cd8b29 Don't accept keyboard input before new-collection prompt appears
Fixes #1613
2018-12-27 07:11:15 -05:00
Adomas Venčkauskas
4072d444e7 Ensure Test Import Translator.js #doImport() does not rely on #detectImport() 2018-12-21 15:16:27 +02:00
Dan Stillman
37c51242d2 Add test for copying collection to another library 2018-12-16 02:24:21 -05:00
Dan Stillman
80f6b857f6 Add Utilities.Internal.getNextName(name, existingNames)
Function to get the next available name when duplicating something
2018-12-16 02:24:21 -05:00
Dan Stillman
223f582aa7 Fix search error on nonexistent collection in recursive mode
And don't return results for a nonexistent parent search
2018-11-28 15:31:57 -07:00
Dan Stillman
6137aeddb8 Follow meta redirects for Find Available PDF
This fixes direct and VPN-based retrieval of PDFs for Elsevier (e.g.,
ScienceDirect) items that have a DOI but no URL, since Elsevier resolves
DOIs through an intermediate page.
2018-11-26 00:57:48 -07:00
Dan Stillman
4b81e03f28 Improve reliability of PDF retrieval delay tests 2018-10-09 19:14:59 -04:00
Dan Stillman
477c28b1e3 Fix another PDF retrieval test after be8db4fc50 2018-10-09 19:03:06 -04:00
Dan Stillman
f3aef41c90 Update tests for auto-recognizing dragged PDFs after be8db4fc50 2018-10-09 18:25:30 -04:00
Dan Stillman
1b9811c31d Fix test failures after 18f79f9796 2018-10-06 01:38:32 -04:00
Martynas Bagdonas
ae7ef109b7 Add universal progress queues (#1573) 2018-10-05 01:56:46 -04:00
Martynas Bagdonas
ec5424d5ae Follow meta redirects and use the final URL when processing documents (#1568) 2018-10-03 22:24:39 -04:00
Dan Stillman
d899134e7c Automatically delay between PDF retrieval requests to the same domain
Delay requests to the same domain by 1 second, respect a Retry-After
header if present for 429 and 503, and delay for 10 seconds on 429 or
5xx otherwise.
2018-09-22 04:03:25 -04:00
Dan Stillman
b782120840 Add followRedirects: false option to Zotero.HTTP.request()
Currently only .status and .getResponseHeader() (for getting 'Location')
are available in the returned object, but we could make the body
available if necessary.
2018-09-22 04:03:25 -04:00
Dan Stillman
7cf466a0b6 Save OA PDFs when the DOI resolves directly to the file 2018-09-06 16:44:11 -04:00
Dan Stillman
18821984e0 Accept a Blob as input in Zotero.File.putContentsAsync() 2018-09-06 16:44:11 -04:00
Dan Stillman
05d8e7a8a3 Check Extra field for DOIs for PDF retrieval
E.g., a book with a DOI in Extra

Closes #1551
2018-08-30 16:52:24 -04:00
Dan Stillman
e7e681c113 Set retrieved OA PDF title to reflect version
"Full Text", "Accepted Version", or "Submitted Version"
2018-08-25 16:48:17 -04:00
Dan Stillman
d5e7bae254 Fix broken test after ec76575645 2018-08-19 04:31:40 -04:00
Dan Stillman
e48a1a2abb Remove lots of old Zotero for Firefox code
We still run tests in Firefox, so not everything can be removed.
2018-08-17 03:17:08 -04:00
Dan Stillman
c2ca70055c Merge branch 'electron' 2018-08-16 19:16:46 -04:00
Dan Stillman
ce5be0bc75 Automatically download open-access PDFs when saving via the connector
If there's no translated PDF or the translated PDF fails and the item
has a DOI, check Zotero's Unpaywall mirror for possible sources and try
to download one of those.

Unlike with "Add Item by Identifier" and "Find Available PDF" in the
item context menu, this does not try the DOI/URL page, since it would
result in more data leakage and most of the time you'd be saving from
the DOI page already. We could consider offering it as an option, but
for it to be useful, you'd have to have an institutional subscription,
be on-campus or connected via VPN (for now), and be saving from
somewhere other than the main page.

A new connector endpoint, sessionProgress, takes the place of
attachmentProgress. Unlike attachmentProgress, sessionProgress can show
new attachments that have been added to the save, and with a little more
work should also be able to show when a parent item has been recognized
for a directly saved PDF.

This also adds support for custom PDF resolvers, available to all PDF
retrieval methods. I'll document those separately.

Closes #1542
2018-08-16 00:57:22 -04:00
Dan Stillman
bcf94942f9 Don't include "Zotero" in User-Agent sent to non-Zotero sites 2018-08-16 00:57:22 -04:00
Adomas Venčkauskas
01f3159b2f Ensure all failing integration tests are fixed (8d8410ff). Closes #1546 2018-08-14 11:40:19 +03:00
Dan Stillman
11f8bd20d6 Fix title display when installing style via connector (broken in 0f1687f1c7) 2018-08-13 18:17:35 -04:00
Adomas Venčkauskas
8d8410ffa0 Fix failing integration test (regr 0f1687f). Closes #1546 2018-08-13 14:22:15 +03:00
Dan Stillman
6478a375e3 Fix WebDAV tests after 276bb99fc9 2018-08-09 18:20:02 -04:00
Dan Stillman
679a6d5cc7 PDF retrieval improvements
- Add the ability to extract a PDF URL from a given webpage using the
  translation framework
- Add the ability to get open-access PDFs from landing pages from
  Unpaywall data in addition to direct PDF URLs
- Use the above functionality to improve PDF retrieval for "Add Item by
  Identifier"
- Add "Find Available PDFs" option to the item context menu to retrieve
  PDFs for existing items from the DOI or URL page or using Unpaywall
  data. The option appears for single items with a DOI or URL and no PDF,
  and it always appears when selecting multiple top-level items (but
  skips ineligible items).

PDF extraction from DOI/URL pages will currently only work with
unauthenticated access (i.e., on-campus or VPN, but not via a web-based
proxy).

Supersedes and closes #948
2018-08-07 04:58:15 -04:00
Sylvester Keil
aa908516eb
Use spy.resetHistory instead of reset
Reset was deprecated in 4.1.4 and removed in 5.0.
2018-08-04 22:07:27 +02:00
Sylvester Keil
37412c68ff
Fix style download stub 2018-08-03 13:42:15 +02:00
Sylvester Keil
aa94a62a74
Make test more independent of db state
If there are library ids above 9 string sorting may not put the
highest id last in the array.
2018-08-03 11:53:54 +02:00
Sylvester Keil
824e9c78d6
Move localeCompare tests to Zotero.Intl tests 2018-08-01 15:41:54 +02:00
Sylvester Keil
8e6896c27d
Move Zotero.getString() test to Zotero.Intl tests 2018-07-30 10:20:05 +02:00
Sylvester Keil
143fdd5f2b
Revert "Remove legacy iterator protocol use in Zotero.DB"
Reverting this until we can use Fx60 for testing.

This reverts commit dfe412d448.
2018-07-30 10:18:11 +02:00
Sylvester Keil
dfe412d448
Remove legacy iterator protocol use in Zotero.DB 2018-07-19 12:21:01 +02:00
Dan Stillman
1f8cf22a35 Delete group files when deleting group
Fixes #1532
2018-07-15 23:09:34 -04:00
Sylvester Keil
53522c2cbe Add Zotero.File.getResource for local resources
Use `getResource` in Zotero.Date.init (this turns it into a
synchronous function). Zotero.File.getResource makes it easier
to load local files on platforms that do not support the
`resource://` URLs.
2018-07-11 14:08:45 +03:00
Dan Stillman
99584dc918 Import base-directory-relative linked files
Zotero RDF contained 'attachments:' paths when files weren't included
but they weren't imported properly
2018-06-30 09:19:09 +02:00
Dan Stillman
3c82263082 Fix Add Item by Identifier test failure due to capitalization change 2018-06-27 16:28:38 +02:00
Dan Stillman
2a7f31813e Disable JS in hidden browser when indexing HTML files without a charset
This could cause imports that linked to HTML files to hang, possibly
from network requests that failed.
2018-06-18 20:19:02 -04:00
Dan Stillman
4554998798 Add Zotero.File.moveToUnique(), replacing Zotero.moveToUnique()
New function takes string paths instead of nsIFile instances
2018-06-17 18:08:03 -04:00
Dan Stillman
f3a62f5a63 Automatically download open-access PDFs via Add Item by Identifier
When the associated-files pref is enabled, Add Item by Identifier uses a
Zotero Unpaywall mirror to find available open-access PDFs. No details
about the contents of searches are logged.
2018-06-16 14:35:44 -04:00
Dan Stillman
5ddbe433b9 Fix Backoff and Retry-After header parsing 2018-06-12 15:17:46 -04:00
Dan Stillman
3241faf503 Fix tag saving for intercepted imports from connector
Save tags in intercepted files as automatic tags
2018-05-23 21:16:18 -04:00
Dan Stillman
4acd178819 Fix user-agent handling for faked UAs from connector 2018-05-15 20:18:41 -04:00
Martynas Bagdonas
9165a0247f Fix PMID and arXiv identifiers extraction (#1498) 2018-05-07 06:04:11 -04:00
Martynas Bagdonas
c0a4fa43f0 Add a test for PDF recognition by DOI (#1496) 2018-05-04 03:14:26 -04:00
Dan Stillman
8f39e9cb36 Rename PDF recognizer tests to reflect arXiv ID lookup
Addresses #1494 and #1495
2018-05-04 01:16:04 -04:00
Dan Stillman
d0e42a1186 Automatically switch to My Library on connector save in noneditable view 2018-04-27 18:27:06 -04:00
Dan Stillman
d573a5b639 Merge pull request #1476 from adomasven:feature/connector-doc-integration
Doc Integration Endpoint for the Connector
2018-04-24 01:37:48 -04:00
Adomas Venčkauskas
593153eebe Adds a progress bar for non quick-format integration actions
The progress percentage is based on the most recent transaction
(or undeterminate if this is the first session transaction)

Fix undefined function call error
2018-04-24 01:37:10 -04:00
Dan Stillman
33f8fcfafb Fix syncing of saved search changes
Condition changes were only uploaded after every other change + sync
2018-04-22 17:27:33 -04:00
Adomas Venčkauskas
5e5b567782 Add a connector document integration endpoint
Specifically for google docs via the connector, but could potentially be
used for any integration via HTTP or connector.
2018-04-20 13:08:34 +03:00
Martynas Bagdonas
fa0576a4dd Add arXiv identifier support (#1486) 2018-04-18 13:03:10 -04:00
Dan Stillman
3835bc9110 Assign merge-tracking relations properly when merging >2 items
https://forums.zotero.org/discussion/71459/incorrect-document-refresh-after-three-item-merge
2018-04-15 17:05:55 -04:00
Frank Bennett
94271325d7 Asyncify Zotero.Relations.getByObject(), called only from item merge. (#1480)
Use getAsync() to acquire subject in Zotero.Relations.getByObject()
2018-04-14 10:26:16 -04:00
Dan Stillman
6f1833f936 Remove items from trash and My Publications when removed via sync
Zotero.Item::fromJSON() wasn't properly accounting for missing 'deleted'
or 'inPublications' properties.
2018-04-13 23:36:59 -04:00
Dan Stillman
9a304b6699 Better handling of remotely changed items in locally missing collections 2018-04-07 17:04:35 -04:00
Dan Stillman
ac4abf0ebb Avoid race conditions in conflict resolution tests 2018-04-07 17:04:00 -04:00
Dan Stillman
4f9847da04 Save parent item to correct library when recognizing PDF without DOI 2018-04-02 15:34:22 -04:00
Dan Stillman
9e955bde99 Add Zotero.Item.prototype.moveToLibrary()
Move an item and its attachments to another library. Attachments are
removed as necessary if linked files or all files aren't supported in
the target library.
2018-03-31 08:26:36 -04:00