Commit graph

1065 commits

Author SHA1 Message Date
Dan Stillman
914f86839c Fix test failure in libraryTreeView after items test 2021-03-03 06:23:54 -05:00
Dan Stillman
79950ecfe5 Exclude annotations from library export
We'll want to support this eventually, but for now they would just break
translation.
2021-03-02 18:10:44 -05:00
Dan Stillman
2d2062a63d Block saving of annotations and embedded images in groups
Both should already be disabled via the UI, but just in case there's
some way they sneak through
2021-03-02 18:10:44 -05:00
Adomas Venčkauskas
0fc9037a54 Add a version of citation dialog for citing notes 2021-03-02 18:10:44 -05:00
Dan Stillman
73ba5f9ffe Auto-sync after setting change 2021-03-02 18:10:44 -05:00
Dan Stillman
77ea19277d Remove embedded-image attachment under annotation in test 2021-03-02 18:10:44 -05:00
Dan Stillman
651aabc4dc Prevent setting parent item to non-regular item
Except for embedded-image attachments under notes and annotations under
attachments
2021-03-02 18:10:44 -05:00
Dan Stillman
8b57d0000b Don't orphan embedded-image attachments in schema integrity check! 2021-03-02 18:10:44 -05:00
Dan Stillman
eb865e2625 setAttachmentPageIndex() → setAttachmentLastPageIndex() 2021-03-02 18:10:44 -05:00
Dan Stillman
c8ee3196cd Replace attachmentPageIndex with synced-setting-based mechanism
The page index needs to be per-person in group libraries, and it should
still work in read-only libraries, so it doesn't make sense to store it
on the item. This uses a synced setting in the user's library instead.
2021-03-02 18:10:44 -05:00
Dan Stillman
4a4a54da7b Clear annotation change state after save 2021-03-02 17:58:28 -05:00
Dan Stillman
98f9a99126 Fix annotation tests after adding isExternal to annotation JSON 2021-03-02 17:58:28 -05:00
Dan Stillman
9f954488c6 Delete cache image when image annotation position changes 2021-03-02 17:58:28 -05:00
Dan Stillman
89d9efdec7 Use seconds instead of ms for attachmentLastProcessedModificationTime 2021-03-02 17:58:28 -05:00
Dan Stillman
88f886cb30 Don't delete all annotations when updating attachment item 2021-03-02 17:58:28 -05:00
Dan Stillman
857b413df1 Don't add non-syncing items to sync delete log
Currently just external annotations
2021-03-02 17:58:28 -05:00
Dan Stillman
226fc90308 Switch to cache files for image annotations
Instead of embedded-image attachments

Create with Zotero.Annotations.saveCacheImage({ libraryID, key }, blob)
2021-03-02 17:58:28 -05:00
Dan Stillman
8c464a7a8b Add options argument for Zotero.File.createDirectoryIfMissingAsync()
To allow passing 'from' argument through to OS.File.makeDir()
2021-03-02 17:58:28 -05:00
Dan Stillman
ebc0ca2462 Add annotationIsExternal property to annotations 2021-03-02 17:58:28 -05:00
Martynas Bagdonas
5266734ac9 Fix note editor tests 2021-03-02 17:58:28 -05:00
Dan Stillman
1f2408a9c5 Fix ZFS tests after removing format=json from syncAPIClient 2021-03-02 17:58:28 -05:00
Dan Stillman
0fe6e3d055 Fix some test breakages related to .note/.getNote() changes 2021-03-02 17:58:28 -05:00
Dan Stillman
9997ece0d2 Fix annotation tests for recent changes 2021-03-02 17:58:28 -05:00
Dan Stillman
55c6de23ba Add "Include Annotations" checkbox to export options dialog
This changes the attachment saveFile() function in translators to be
async. In order for errors to be properly caught, translators will need
to be changed to make doExport() async and await on saveFile() calls.
(The translation architecture theoretically already allows doExport() to
be async.)
2021-03-02 17:58:28 -05:00
Dan Stillman
b1e324ddfb Fix marking of childItems as loaded for non-top-level items
Now that attachments and notes can have child items too
2021-03-02 17:58:28 -05:00
Dan Stillman
6ee3863e37 Fix clearing of annotation fields 2021-03-02 17:58:28 -05:00
Dan Stillman
69958d4356 Return Zotero.Item objects from getAnnotations() instead of ids
Returning ids mirrors getAttachments() and getNotes(), but I think we
want to move towards a world where those return actual objects, even if
we need async load calls on the returned objects.
2021-03-02 17:58:28 -05:00
Dan Stillman
60cb299f40 Fix uploading of embedded-image attachments 2021-03-02 17:58:28 -05:00
Dan Stillman
f425b8d8d9 Fix image annotation test after Item.annotationImageURL removal 2021-03-02 17:58:28 -05:00
Dan Stillman
9a41dc69fe Add PDF attachment properties to Zotero.Item
- .attachmentLastProcessedModificationTime
- .attachmentPageIndex
2021-03-02 17:58:27 -05:00
Adomas Venčkauskas
51db52a5ea Changes for note insertion in non-GoogleDocs 2021-03-02 17:43:10 -05:00
Adomas Venčkauskas
2e9e655479 Add ability to cite Zotero notes.
<span class=citation data-citation=serialized-citation-data/>
elements will automatically be converted to zotero citations
2021-03-02 17:43:09 -05:00
Adomas Venčkauskas
2b3669afd8 Consolidate Integration.Fields into Integration.Session 2021-03-02 17:43:09 -05:00
Dan Stillman
ead8c6bb45 Fix Everything search after annotations
And replace ancient 'annotation' search condition with
'annotationText'/'annotationComment'
2021-03-02 17:39:39 -05:00
Dan Stillman
199619f40e Remove .noteSchemaVersion
This leaves item.note in place, rather than reverting all the
`getNote()` → `.note` changes. We can consider which we want to keep.
2021-03-02 17:36:05 -05:00
Dan Stillman
602e4c1e1f More annotation 'position' fixes 2021-03-02 17:36:05 -05:00
Dan Stillman
2536edb6ab Fix annotation 'position' handling and update additional sortIndex tests
Item.position has to be a string. It still gets passed to/from the PDF
reader as an object.
2021-03-02 17:36:05 -05:00
Dan Stillman
151a14c0a8 Update test for shortened annotation sortIndex 2021-03-02 17:36:05 -05:00
Dan Stillman
a0fb7a8583 Ignore remote noteSchemaVersion: 0 for items with no local note
E.g., embedded attachment notes with no note don't have an itemNotes row
and don't output noteSchemaVersion in their JSON, but they shouldn't
trigger a conflict
2021-03-02 17:36:05 -05:00
Dan Stillman
b2aee30410 Fix fromJSON() not importing parentItem for annotations 2021-03-02 17:36:05 -05:00
Dan Stillman
b8eeb3abaa Drop decimal places from annotation sortIndex 2021-03-02 17:36:05 -05:00
Dan Stillman
99b959285e Send Zotero-Schema-Version header with API requests
Set to the current global schema version
2021-03-02 17:36:05 -05:00
Dan Stillman
e133aab530 Annotation support in Item::fromJSON()/toJSON()
And clean up embedded-image handling
2021-03-02 17:36:05 -05:00
Dan Stillman
a94323fc15 Sort multiple levels of items when generating API JSON
Added Zotero.DataObjects.sortByParent() to sort child items immediately
after their parent items. Zotero.DataObjects.sortByLevel(), which is
used for collections, sorts each level together, but that's less
appropriate for items where, e.g., an embedded-image attachment should
immediately follow the note that depends on it.
2021-03-02 17:36:05 -05:00
Dan Stillman
f662b58331 Update tests after sortIndex length change 2021-03-02 17:36:05 -05:00
Dan Stillman
427a227370 Rename 'area' annotations to 'image' annotations 2021-03-02 17:36:05 -05:00
Dan Stillman
078a18f7c0 Shorten sortIndex to 22 characters 2021-03-02 17:36:05 -05:00
Dan Stillman
4142f4b316 Replace occurrences of .getNote() with .note 2021-03-02 17:36:05 -05:00
Dan Stillman
ebc53a2bbc Add noteSchemaVersion, and replace item.getNote() with .note
```
var noteContents = item.note; // was item.getNote()
var schemaVersion = item.noteSchemaVersion;

item.setNote(contents) // default to Zotero.Notes.schemaVersion
item.setNote(contents, schemaVersion) - explicit version
```
2021-03-02 17:36:05 -05:00
Dan Stillman
1c366de546 Initial annotation support 2021-03-02 17:36:05 -05:00
Dan Stillman
92ba393488 Don't clear item type set in constructor when loading primary data
Generally, don't clear the change state for primaryData when calling
loadPrimaryData() on an item that doesn't exist. Specifically, this
fixes an issue where calling loadPrimaryData() on a nonexistent item
after setting a libraryID and key would also clear any item type set in
the constructor. The same would apply for props set in the
Collection/Search constructors.
2021-03-02 17:36:05 -05:00
Dan Stillman
b54d4e78b7 Save createdByUserID and lastModifiedByUserID for group items 2021-03-02 17:36:04 -05:00
Dan Stillman
bb0a1dab13 Don't normalize primary data values in Item.setField()
Just enforce proper data types
2021-03-02 17:36:04 -05:00
Dan Stillman
cca5dc0314 Don't send unnecessary format=json in sync download requests 2021-03-02 17:36:04 -05:00
Dan Stillman
d9cf53725a Make Find PDF test timing a bit more forgiving 2021-02-09 17:12:25 -05:00
Dan Stillman
843bcbb68a Show invalid-data warning in sync button tooltip for group libraries
Instead of showing warning button on toolbar

Otherwise if one person in a group upgrades to a beta with new fields,
everyone in the group will get a warning, even if they're already on the
latest release version. In a personal library, the user can upgrade to
the same version.
2021-02-09 16:36:06 -05:00
Dan Stillman
b919143630 Skip conflict resolution for remote objects that can't be saved
E.g., if a local item has been modified in a way that conflicts with a
remote item that also has a new, unknown field, don't show the CR
window -- just add the item to the sync queue and show the
some-data-could-not-be-downloaded error.
2021-02-07 16:49:01 -05:00
fletcherhaz
55acf9a0c0
Do not send back saveSingleFile: true if saving a PDF snapshot. (#1950) 2021-02-05 15:31:34 -05:00
Dan Stillman
b3220e83b1 Don't pass nested collections to translators as top-level collections
While we need to pass all items from descendant collections to
translators, only the immediate child collections of the selected
library or collection should be returned from Zotero.nextCollection().

I'm not sure how long we've been doing this wrong, but it resulted in
duplicated subcollections when round-tripping Zotero RDF. (TEI is the
only other translator that uses nextCollection(), so its exports were
probably similarly incorrect.)

https://forums.zotero.org/discussion/87135/export-import-creates-numerous-duplicate-subcollections
2021-01-25 03:58:04 -05:00
Dan Stillman
10ca05e22e Return array instead of iterator for Collection.getChildCollections(true) 2021-01-25 03:58:04 -05:00
Dan Stillman
9e6a2d2e50 Fix title mismatch for ISBN lookup tests
Not sure what this was using before (LOC isn't currently returning a
result), but GBV doesn't currently have a serial comma in the title.
2021-01-25 03:58:04 -05:00
Dan Stillman
cd63f96eee Increase DOI Content Negotiation timeout during lookup tests
Probably a temporary problem, but it's timing out on some runs at 10
seconds
2021-01-18 23:06:13 -05:00
Dan Stillman
4c048f6fd2 Relax HTML checking in SingleFile tests
Somehow the saved page starts with "<html style>" instead of "<html>" on
GitHub Actions
2021-01-18 23:06:13 -05:00
Dan Stillman
9152012368 Restore DB table reconciliation during integrity check
But skip it at startup, even if flagged on, if there are schema update
steps to perform, to avoid creating tables that aren't expected to exist
yet.

Originally added in 5b9e6497a but disabled in c4cc44528 and 7a434df53
2021-01-17 03:36:38 -05:00
Dan Stillman
e45ca4edad Support deleted property for collections and searches
This lays the groundwork for moving collections and searches to the
trash instead of deleting them outright. We're not doing that yet, so
the `deleted` property will never be set (except for items), but this
will allow clients from this point forward to sync collections and
searches with that property for when it's used in the future. For now,
such objects will just be hidden from the collections pane as if they
had been deleted.
2021-01-13 00:49:12 -05:00
fletcherhaz
98a75931b0
Fix double saving snapshots (#1937)
* Fix double saving snapshots

https://forums.zotero.org/discussion/86796/duplicated-snapshots

I was able to replicate it by adding a 5 second delay here:
a72ae14816/chrome/content/zotero/xpcom/translation/translate_item.js (L196)

This was caused from a race condition and (a72ae14) did not fully
solve the problem. Now this is tested and fixed.
2020-12-27 03:31:30 -05:00
Dan Stillman
7a434df539 Disable creation of missing tables/indexes in Schema.integrityCheck()
Originally added in 5b9e6497a. We stopped running the integrity check
before userdata upgrades in c4cc44528 because this new behavior was
breaking upgrades, but it could still be run when coming from the DB
Repair Tool, which could cause problems if, say, you recovered a
database from a computer that had an older version of Zotero and ran the
DB Repair Tool on it. Disabling this for now until we have a better
solution.
2020-12-21 01:50:13 -05:00
Dan Stillman
43e6493997 Don't convert "Type: article" in Extra to Document item
CSL 1.0.2 clarified that type 'article' is meant for preprints and
working papers, so until we have a Preprint item type, just leave it in
Extra.

Discussion: https://github.com/zotero/translators/pull/2248#discussion_r492009958
2020-12-21 01:11:28 -05:00
Dan Stillman
2b5a22c132 Add additional checks to test for HTTP.request() 5xx retry 2020-12-11 02:05:25 -05:00
Dan Stillman
87352822fa Delay properly on 429 response in syncAPIClient
ConcurrentCaller wasn't waiting properly if start() was called again
while it was pausing, so 429 caused an immediate retry, which is pretty
much exactly what you don't want a 429 to do.
2020-12-11 02:02:20 -05:00
Dan Stillman
84730e610c Don't delete valid item/CSL type from Type in Extra on sync
After a local item change, Zotero uploads the JSON and then applies the
saved JSON returned from the API to the local object using `fromJSON()`,
the same as it would apply any other remote change.

`fromJSON()` is meant to migrate Extra lines into real types and fields
after future item type/field changes. It calls
`Z.Utilities.Internal.extractExtraFields()`, which looks for valid item
type or CSL type values in Type lines in Extra, handles the rest of
parsing accordingly, and passes back the parsed item type. `fromJSON()`
wasn't handling `itemType` in the response object, so the item type
didn't get applied and the Type line was stripped. This fixes that.

Since valid type values are now parsed, if you have a Journal Article
item with a Pages field and enter "Type: song" into Extra and sync, the
item will be converted to Audio Recording and `Pages: 123` will be
placed in Extra.

https://forums.zotero.org/discussion/comment/369221/#Comment_369221
2020-11-27 17:50:20 -05:00
Dan Stillman
07874d84bd
Merge pull request #1909 from fletcherhaz/snapshot
Update SingleFile for bug fixes
2020-11-17 16:02:13 -05:00
Dan Stillman
5b9e6497af Schema integrity check improvements
- Create userdata tables and indexes that are missing
- Delete tables and triggers that should no longer exist
- Run schema integrity check before user data migration
- Run schema integrity check after restart error

This is meant to address two problems:

1) Database damage, and subsequent use of the DB Repair Tool, that
   results in missing tables

2) A small number of cases of schema update steps somehow not being
   reflected in users' databases despite their having updated userdata
   numbers, which are set within the same transaction. Until we figure
   out how that's happening, we should start adding conditional versions
   of schema update steps to the integrity check.

This is currently only running the update check after a restart error,
which might not occur for all missed schema update steps, so we might
want other triggers for calling setIntegrityCheckRequired().
2020-11-16 18:13:48 -05:00
Dan Stillman
6bcc8af86b Add Zotero.DB.columnExists(table, column) 2020-11-16 17:50:52 -05:00
Dan Stillman
b0e065a4ae Don't return relative path from .attachmentFilename for linked files
https://forums.zotero.org/discussion/86139/zotero-item-attachmentfilename-should-only-return-filename
2020-11-13 11:43:00 -05:00
Fletcher Hazlehurst
a2620b757d Update SingleFile and fix several bugs
- Using `sandboxPrototype` properly uses window as prototype
- This commit removes the need for our patch in babel-worker.js:
3d0bc4cf9f
- We properly inject into frames in the client if we ever include frames
2020-11-02 17:24:14 -07:00
Dan Stillman
367fea1847 Zotero.File.copyDirectory(): Fix copying of subdirectories on Windows
This fixes copying of the previous extracted SingleFileZ snapshots
between libraries.
2020-11-01 18:51:55 -05:00
Dan Stillman
29f48476a9 Use UnexpectedStatusException in Zotero.HTTP.loadDocuments()
Follow-up to 76ae5d9f59, which changed loadDocuments() to pass/throw an
Error on a non-2xx response code
2020-10-24 00:04:01 -04:00
fletcherhaz
76ae5d9f59
Switch back to SingleFile from SingleFileZ (#1904)
Our SingleFileZ integration would save images inside directories following the
SingleFileZ format. However, Zotero does not support syncing sub-directories of
attachments. This commit switch back to a single HTML file with base64 encoded
resources. We think that the 33% increase in resources will be offset by the
compression of HTML and removal of JavaScript and unused CSS.

This commit does not fix past snapshots that were saved using SingleFileZ.
2020-10-23 19:39:07 -04:00
Dan Stillman
828ec4010e Don't allow items to be set as their own parents, and correct existing 2020-10-14 19:33:46 -04:00
Fletcher Hazlehurst
bb8325ff9b Fix tests for new version of SingleFileZ 2020-10-13 11:08:23 -06:00
Fletcher Hazlehurst
bb0ddbd872 Fix server not handling empty body in multi-part request
An empty body is still valid. Was causing an issue for empty favicons.

https://forums.zotero.org/discussion/85600/bug-report-no-snapshot-in-zotero-beta
2020-10-12 15:06:11 -06:00
Fletcher Hazlehurst
1c5cefaffd Fix handling of network errors for SingleFile save 2020-09-28 10:43:32 -07:00
Dan Stillman
20c8cede4d
Merge pull request #1846 from fletcherhaz/snapshot
Use SingleFile to create snapshots of web pages
2020-09-24 18:08:34 -04:00
Dan Stillman
cae2aab70f Reduce startup time from retraction key caching 2020-09-23 22:58:53 -04:00
Fletcher Hazlehurst
0fba08b3c9 Use SingleFile to create snapshots of web pages 2020-09-23 09:37:09 -07:00
Dan Stillman
8fc316f727 Fix potential error dragging to library that hasn't been loaded
https://forums.zotero.org/discussion/85136/unable-to-copy-items-between-libraries

Zotero.Relations.getByPredicateAndObject() is now async.
2020-09-14 23:49:22 -04:00
Dan Stillman
0e74a91f6b Fix parsing of SQL dates without seconds
Previously, "2020-09-09 23:33" would be treated as a multipart date,
with "23:33" left in the visible field and "d" showing in the indicator.
2020-09-09 23:37:02 -04:00
Dan Stillman
4ac35ecda3 Fix invalid HTML in note from Quick Copy
A <body> was included with the note element outside of it.
2020-09-09 23:36:18 -04:00
Dan Stillman
8614e73aa8 Throw clearer error when 'storage' is a broken symlink
Instead of '(NS_ERROR_FILE_ALREADY_EXISTS) [nsIFile.create]', throw
"Broken symlink at <path>".

Closes #1834
2020-08-07 18:23:44 -04:00
Dan Stillman
abadbb2d77 Add fileInterface test for importing into selected collection
And update importFile() calls in tests to use current signature
2020-07-12 04:53:29 -04:00
Dan Stillman
6fd0b8245b Avoid logged error in retraction test from missing response data 2020-06-25 20:53:20 -04:00
Dan Stillman
7422c50076 Fix detection of retractions for items with DOI in Extra
Affected items should be detected on the next retraction updates check
2020-06-25 20:53:20 -04:00
Dan Stillman
bccf5ff0b2 Fix endless WebDAV loops if server has wrong mtimes but hash matches
Possibly caused by a third-party client uploading mtimes that then
aren't synced, or that differ from what get synced. When we detect this,
try to correct it by updating mtimes on WebDAV and the API to match the
local file.

https://forums.zotero.org/discussion/83554/zotero-loop-syncs-2000-items
2020-06-09 01:26:11 -04:00
Dan Stillman
a8c682bf4b Preserve linked-object and replaced-item relations when merging items
https://groups.google.com/d/msgid/zotero-dev/6f764822-ed7c-46eb-8068-ce9ed1a1538c%40googlegroups.com

Regression from 617564982c
2020-06-06 17:16:09 -04:00
Dan Stillman
b2e902746a Strip HTML tags from titles when generating filenames 2020-06-02 17:06:42 -04:00
Dan Stillman
d7ee3fdee2 strToDate(): A couple additional tests 2020-05-28 07:00:11 -04:00
Dan Stillman
0620b16d3e strToDate(): Don't parse 01/01/08 as the year 8
Regression from b7dc0d8b0a (5.0.81)
2020-05-28 06:19:42 -04:00
Dan Stillman
85bc5d168e strToDate(): Fix 'order' for just a number <=12
"m" instead of "m y"
2020-05-28 04:35:14 -04:00
Dan Stillman
70b361ed8b Fix stateful retractions tests
Closes #1824
2020-05-27 08:44:25 -04:00