Commit graph

886 commits

Author SHA1 Message Date
Dan Stillman
d389a71280 Download remotely updated files in "as needed" file sync mode
Previously, files updated remotely wouldn't be downloaded in "as needed"
mode if a copy of the file already existed locally and could only be
re-downloaded by deleting the file via Show File.

This causes remotely modified files that exist locally to be downloaded
at sync time, even in "as needed" mode, by marking them as
"force_download". While this might not be ideal for people who use "as
needed" to limit data transfer, it's better for people who use it simply
to limit local storage, and ending up with an outdated file while
offline seems worse than a little bit of extra data transfer.

In the future, we'll likely also provide ways to explicitly download and
remove files, so keeping chosen files in sync makes sense.

Files modified remotely before this change (which were marked as
"to_download" instead of "force_download") won't be downloaded as sync
time in "as needed" mode, but they'll now be re-downloaded on open.

Fixes #1322
2020-03-09 01:19:52 -04:00
Dan Stillman
76a1535a60 Full-text indexing improvements
- Use full-text cache file from syncing if available when reindexing via
  info pane or Rebuild Index → Index Unindexed Items. Only discard it for
  full index rebuild. This allows Index Unindexed Items to be used to
  force immediate processing of queued content from syncing and avoids
  unnecessary syncing back of identical content. Previously, the cache
  file was used for a manual index only when the local file didn't exist.
- When rebuilding index, don't clear indexed items with missing local
  file that are missing stats due to a pre-411180ef bug.
- indexItems() now takes an 'options' object as its second parameter
- Minor code cleanup
2020-03-09 01:19:52 -04:00
Dan Stillman
411180ef83 Full-text indexing fixes
- Don't clear item's index stats (and show "Unknown") when an item is
  reindexed remotely and the content matches the local content
- Always update an item's state and its stats in the same query, to
  avoid incorrect feedback immediately after indexing
- Clean up `setItemContent()` tests
2020-03-06 03:11:16 -05:00
Dan Stillman
da5e8c549e Convert Zotero fields in Extra to CSL fields for citeproc-js 2020-03-02 01:38:10 -05:00
Dan Stillman
83cc65eea6 Fix parsing of CSL date fields in extractExtraField() 2020-03-02 01:34:27 -05:00
Dan Stillman
a9dee2f487 Item::fromJSON(): Remove invalid-for-type Type-mapped fields
"Type: [val]` in Extra means Item Type to citeproc-js, and Type values
from translators mostly aren't going to be useful if the item type
doesn't have a Type-mapped field.

https://forums.zotero.org/discussion/comment/348864/#Comment_348864
2020-02-17 11:39:35 -05:00
Dan Stillman
b2bf60e1d7 Item.fromJSON(): Dedupe invalid-for-type fields when storing in Extra
3de54455f6 removed redundant base-mapped fields when a valid-for-type
field was also set, but that still left duplicate fields in Extra when a
valid field wasn't set. This will happen until translators (most notably
Embedded Metadata) are fixed to stop setting redundant fields.

https://forums.zotero.org/discussion/81262/translator-error-sage-lots-of-extra-data-in-extra
2020-02-17 00:22:34 -05:00
Dan Stillman
11caa1b719 Group strict-mode and non-strict-mode tests for Item::fromJSON() 2020-02-16 18:05:45 -05:00
Dan Stillman
b41734924d Further fixing of "Too many sync requests" error
Follow-up to 804a898c98

Addresses #1788
2020-02-16 18:05:45 -05:00
Dan Stillman
804a898c98 Hopefully fix "Too many sync requests" after file upload 412
Addresses #1788
2020-02-16 13:06:49 -05:00
Dan Stillman
a53f363b8d Additional fix for search crash with includeParentsAndChildren
Follow-up to 76081ab05
2020-02-11 13:09:54 -05:00
Dan Stillman
76081ab05f Fix crash when search uses no-op condition and includeParentsAndChildren
E.g., a nonexistent saved search
2020-02-11 00:23:45 -05:00
Dan Stillman
86a5c46b1e Find Available PDF: Don't mark URLs that redirect as tried
https://forums.zotero.org/discussion/81182
2020-02-02 23:47:40 -05:00
Dan Stillman
4e11c7927d Don't skip creator lines in Extra in fromJSON
Regression in 3de54455f6
2020-01-27 22:01:25 -05:00
Dan Stillman
3de54455f6 Automatically save unknown/invalid fields to Extra in non-strict mode
This is a prerequisite for starting to use new fields in translators,
since otherwise switching from, say, storing originalDate in Extra to
using an originalDate field would cause the value to be lost in clients
without the newer schema.

Closes #1504
2020-01-26 03:29:23 -05:00
Dan Stillman
3d2afa9c7f Additional adjustments to URL handling
Follow-up to df40ee7216

- Restore opening of non-HTTP URL schemes in link attachments
- Remove scheme whitelist for link attachments, since it's not enforced
  via the API anyway and we prompt before an external application is
  opened. Instead, just block a few schemes (e.g., 'javascript') from
  launching.

TODO:

- Provide some way to change/reset an application association if the
  user checks the box to automatically open that scheme.
- Show an error message if a link attachment with an invalid URL is
  double-clicked
2020-01-04 02:30:12 -05:00
Dan Stillman
6070743ff0 Fix Nextcloud WebDAV syncing
Add Zotero.HTTP.CookieBlocker mechanism to block storing and sending
of all cookies for a given URL prefix, and use that for the configured
WebDAV URL.

https://forums.zotero.org/discussion/80429/sync-error-in-5-0-80
2020-01-04 02:29:52 -05:00
Dan Stillman
12ccbf45c8 Make HTTP tests more reliable 2019-12-23 04:31:24 -05:00
Dan Stillman
9c380c362a Hopefully fix 503 from NextCloud (since 5.0.78)
https://forums.zotero.org/discussion/comment/344790/#Comment_344790
2019-12-22 16:32:36 -05:00
Dan Stillman
b7dc0d8b0a Fix parsing of pre-1000 years
If a leading zero (e.g., '068'), parse as actual year instead of
19xx/20xx.
2019-12-19 02:15:36 -05:00
Dan Stillman
df40ee7216 Adjust URL launching behavior
- Support launching plausible HTTP URLs without schemes from
  Zotero.loadURI(), and use that when launching from URL field
- Show correct cursor feedback on URL label -- only show pointer if the
  URL is launchable
- Don't launch non-HTTP URLs (e.g., zotero://) from URL field
- Don't open HTTP URLs in viewer when using ZoteroPane.launchURI()
2019-12-16 14:56:06 -05:00
Dan Stillman
bbbd02444b Restore 'yesterday'/'today'/'tomorrow' parsing for dates in searches
Follow-up to a549a64de9, which removed it from strToDate()
2019-12-06 03:12:48 -07:00
Dan Stillman
61cebbd8f7 Fix display of a couple WebDAV verification errors
"spec is undefined"

Fixes #1745
2019-12-01 03:39:54 -07:00
Dan Stillman
fbb54a7621 Fix "string is undefined" export error
Regression from a549a64de9
2019-11-29 01:50:43 -07:00
Dan Stillman
8c7677a009 Fix out-of-memory error syncing collections nested inside each other
It shouldn't be possible for collections to be nested this way, if it
happens, it shouldn't result in an infinite loop.

This removes one of the parent assignments at sync time.
2019-11-26 15:30:43 -07:00
Dan Stillman
5791ffeb16 Reactify item tags box
Improvements:

- Fixes autocomplete text remaining in field after selection in Fx60
- No more text or icon shifting on select (tested on macOS)

Changes:

- Tags are now selected on mousedown with no active state, as in web
  library

Regressions:

- Tooltip with tag type doesn't appear when hovering over icon
- Pressing Tab after modifying a tag loses focus
- Right-click in textbox shows custom menu instead of default text
  editing context menu (Cut/Copy/Paste)

To-do:

- Switch to this version for note tags box
- Style colored tags in autocomplete drop-down? Sort to top?
- Only show delete button on row hover, as in web library?
2019-11-08 06:41:06 -05:00
Dan Stillman
cbfa4be437 Add 'Type:' to extraToCSL() test
Follow-up to #1744
2019-10-26 17:05:45 -04:00
Dan Stillman
c25fbe7c1c Remove all colored tags on selected items if 0 is pressed
Like Thunderbird
2019-10-26 16:45:04 -04:00
Dan Stillman
52fd91950d Fix deleted item fields reappearing on sync
This was a regression from 4b60c6ca27. The original plan for introducing
new fields was to have them save to the sync cache even if they weren't
supported by the current Zotero version and process them on upgrade, and
so I changed `DataObjectUtilities.patch()` to omit fields that didn't
exist locally when patching the sync cache JSON so they wouldn't be
wiped on the server. That caused this bug where locally deleted fields
were restored on every sync. It's also no longer necessary now that
we decided to just reject unknown fields from saving, so we can just
revert to the previous behavior of blanking out locally missing fields
(with the tweak that fields that are already false or empty in the base
version can be omitted).
2019-10-24 01:22:00 -04:00
Dan Stillman
513f7d6555 Fx60: Fix localization
Also fixes #1690 (at least on Catalina), and possibly other things

general.useragent.locale and intl.locale.matchOS are no longer used.
2019-10-23 19:08:13 -04:00
Dan Stillman
d4f682aa88 Merge branch 'fx60' 2019-10-21 21:44:01 -04:00
Dan Stillman
956813ac1f Fix "err is undefined" on 200 response for nonexistent WebDAV file
Fixes #1741
2019-10-20 15:03:18 -04:00
Dan Stillman
1710eb1c4b Don't store unknown/invalid fields in Extra in non-strict mode
And fix a couple things for if we turn it back on

This code came along with the type/field handling overhaul, but I think
it was originally intended for handling unknown fields during sync
before we decided on strict mode, so it wasn't finished and causes
various problems [1]. It could still be useful for preserving fields
from translators before they're available on items, but the better fix
there is just to add the missing fields, so I'm not sure if we'll end up
needing it.

[1] https://groups.google.com/d/msg/zotero-dev/a1IPUJ2m_3s/hfmdK2P3BwAJ
2019-10-18 03:37:24 -04:00
Dan Stillman
89672ed0a4 Fix "c1 is undefined" sync CR error 2019-10-09 18:29:04 -04:00
Dan Stillman
44d4586242 Fix test after 8aeb6f7fe4 2019-09-28 07:40:01 -04:00
Dan Stillman
5723683b3b Fix "attachmentSyncedModificationTime must be a number" sync error
https://forums.zotero.org/discussion/79011/zotero-error-report
2019-09-21 01:59:09 -04:00
Dan Stillman
4b7cdddb4a Change apply-to-all checkbox text for some conflicts
Say "Use the [local|remote] version for all remaining conflicts" for
everything instead of saying "Use [local|remote] fields for all
remaining conflicts" for some conflicts.

This also fixes a test failure after 54343c49fb.
2019-09-21 01:35:36 -04:00
Dan Stillman
4b60c6ca27 Type/field handling overhaul
This changes the way item types, item fields, creator types, and CSL
mappings are defined and handled, in preparation for updated types and
fields.

Instead of being predefined in SQL files or code, type/field info is
read from a bundled JSON file shared with other parts of the Zotero
ecosystem [1], referred to as the "global schema". Updates to the
bundled schema file are automatically applied to the database at first
run, allowing changes to be made consistently across apps.

When syncing, invalid JSON properties are now rejected instead of being
ignored and processed later, which will allow for schema changes to be
made without causing problems in existing clients. We considered many
alternative approaches, but this approach is by far the simplest,
safest, and most transparent to the user.

For now, there are no actual changes to types and fields, since we'll
first need to do a sync cut-off for earlier versions that don't reject
invalid properties.

For third-party code, the main change is that type and field IDs should
no longer be hard-coded, since they may not be consistent in new
installs. For example, code should use `Zotero.ItemTypes.getID('note')`
instead of hard-coding `1`.

[1] https://github.com/zotero/zotero-schema
2019-09-16 02:27:22 -04:00
Dan Stillman
20d7ef6c0b Add separate .eslintrc for tests 2019-09-16 01:28:52 -04:00
Dan Stillman
e8dd1f7824 Test updates for newer Sinon 2019-09-16 01:27:30 -04:00
Dan Stillman
25e34b7b99 Update WebDAV request timeouts for HTTP.request() auto 5xx retry 2019-09-16 01:27:18 -04:00
Dan Stillman
dc60e5f840 HTTP.request() improvements
- Move 5xx retries and connection checking out of the sync API client
  and into HTTP.request() so that they apply to all requests. 429 handling
  remains in the API client, since not all callers necessarily want to
  handle that the same way. Callers can still handle 5xx themselves by
  including the relevant 5xx status codes in `successCodes` or by passing
  `errorDelayMax: 0`.
- Add `cancellerReceiver` option, which is a callback that receives a
  function that will cancel the request, whether it's an active request
  or an automatic delay before a 5xx retry.

This also updates Sinon to 7.3.2.
2019-09-16 01:26:34 -04:00
Dan Stillman
f04f186396 Add missing yield in sync code 2019-09-16 00:47:45 -04:00
Dan Stillman
7ed0d8a408 Fix recognizePDF test after DOI translator change 2019-09-16 00:46:49 -04:00
Dan Stillman
54343c49fb Fix "getNote() can only be called on notes and attachments" CR error
This bug may be as old as the sync system itself. It could occur if
there were conflicts for both a note and a regular item in the same
batch.
2019-09-16 00:39:06 -04:00
Dan Stillman
b08bd6849e Fx60: Update DB query onRow() behavior
onRow() handlers now get passed a cancellation function as a second
argument
2019-08-27 06:00:35 -04:00
Dan Stillman
433794916a Fx60: Remove instances of nsILocalFile
nsILocalFile was merged into nsIFile, so either replace it with that or
switch to Zotero.File.pathToFile().
2019-08-27 00:47:39 -04:00
Dan Stillman
6f965251ed Add FilePicker module to replace nsIFilePicker
`nsIFilePicker::show()` is removed in Firefox 60 in favor of `open()`,
which takes a callback (and apparently has been preferred for a long
time).

There's no point switching to that, so this module is a version of
nsIFilePicker with an async `show()` that returns a promise and some
XPCOM-isms replaced (e.g., string paths instead of nsIFile).
2019-08-27 00:47:39 -04:00
Dan Stillman
75daaa3b7c Fix incorrect 'return' in test 2019-08-27 00:47:39 -04:00
Dan Stillman
8c59df435f Fx60: Fix snapshot filenames
nsIURL doesn't seem to work anymore, so add Zotero.Utilities.parseURL(),
which uses the `url` package from NPM and adds fileName, fileExtension,
and fileBaseName.
2019-08-27 00:47:39 -04:00