Also:
* _finalizeErase in Zotero.DataObject is now inheritable
* Call _initErase before starting a DB transaction
* removes Zotero.Libraries.add and Zotero.Libraries.remove (doesn't seem like this is used any more)
This will appear much less frequently, since non-conflicting field changes on
both sides can be resolved automatically, but genuine field conflicts still
require manual conflict resolution.
The merge pane is no longer editable, since the itembox code to do that is
async and can't run in a modal window, but it's not really necessary,
particularly with conflicts happening less frequently.
TODO:
- Remote item deletions
- File conflicts
- Maybe handle some edge cases where the conflicted items fail to save
There's a lot more to do, and this isn't ready for actual usage, but the
basic functionality is mostly in place and has decent test coverage. It
can successfully upgrade a library last used with classic syncing and
pull down changes via the API. Uploading mostly works but is currently
disabled for safety until it has better test coverage.
Downloaded JSON is first saved to a cache table, which is then used to
populate other tables and later for generating PATCH requests and
automatically resolving conflicts (since it shows what was changed
locally and what was changed remotely). Objects with unmet dependencies
or unknown fields are skipped for now but don't block the rest of the
sync.
Some of the bigger remaining to-dos:
- Tests for uploading
- Re-do the preferences to get an API key
- File sync integration
- Full-text syncing integration
- Manual conflict resolution (though this already includes much smarter
conflict handling that automatically resolves many conflicts)
Since modal windows (e.g., the Create Bib window and the Quick Copy site
editor window) can't use yield, style retrieval
(Zotero.Styles.getVisible()/getAll()) is now synchronous, depending on a
previous async Zotero.Styles.init(). The translator list is generated in
the prefs window and passed into the Quick Copy site editor, but it's
possible the translators API should be changed to make getTranslators()
synchronous with a prior init() as well.
For new installations, a panel will show for the Z icon with a ">"
button to go to a second panel for the save icon. For existing
installations, a panel will show just for the save icon. In both cases,
the panels are non-auto-hiding but are dismissed if clicked on or when a
new content page is loaded.
There's also a new close button, but it's not enabled currently. Might
be useful for panels with embedded links where an accidental click on
the panel alone shouldn't close the panel (like the Firefox customize
mode wizard).
- When relinking a missing stored file, copy it into the attachment's
storage directory automatically
- Previously, selecting a file outside the attachment subdir would
just result in a missing attachment, since it only looks for stored
files within the subdir
- Display an error message if a Windows shortcut (.lnk) is added via
drag-and-drop or via a file dialog on non-Windows systems, until we
can figure out how to determine the original file
- Shortcuts can cause errors during syncing, for unclear reasons
- Neither nsIFile::copyToFollowingLinks() nor nsIFile::target work for
me to get the original file, even when nsIFile::isSymlink() returns
true
- Windows file dialogs seem to automatically resolve shortcuts, so
it's only an issue there for drag-and-drop
- Disallow hidden files from being selected in relink dialog
- I think some people on Windows with hidden files shown relink the
.zotero* files that show up when they click Locate, which causes
file sync errors. Which brings us to...
- Fix file sync errors for *.lnk and .zotero* files
- Ignore existing .zotero* attachment files, treating the files as
missing instead to encourage relinking
- Strip leading period in getValidFileName() to prevent added files from
being hidden
- This allows hidden files to be added explicitly; they just won't
stay that way in the storage directory
(These things should have tests, but that will have to happen on the 5.0
branch.)
- Simplified schema
- Tags are now added without reloading entire tag selector
- On my system, adding 400 tags to an item (separately, with the tag
selector updating each time) went from 59 seconds to 42. (Given that
it takes only 13 seconds with the tag selector closed, though,
there's clearly more work to be done.)
- Tag selector now uses HTML flexbox (in identical fashion, for now, but
with the possibility of fancier changes later, and with streamlined
logic thanks to the flexbox 'order' property)
- Various async fixes
- Tests
And use 'version' instead of 'itemVersion' for object version for items
Also add deferred foreign key checking to system.sql so that DROP TABLE
commands don't fail mid-transaction
Show a wizard after items are dragged to My Publications choosing
whether to include files and notes and choosing sharing settings for the
items. Sharing options are Creative Commons licenses, CC0, "All rights
reserved", or keeping the existing Rights field if available.
Also blocks collections, searches, linked file attachments, and
top-level attachments/notes from My Publications at the data layer, but
not yet from the UI in all places (so it can crash if you try).
Todo:
- Block certain UI actions with nice messages
- Show a nice scrollable list of items in the wizard to allow selecting
specific files/notes, instead of just having checkboxes for files and
notes that apply to all dragged items
- Show an explanation of My Publications in the right-hand pane when no
items are selected
- Maybe adjust handling when no attached files/notes, since it might be
a bit alarming at the moment to see sharing options for metadata
entries
Note that this loses conflicting changes to translate_item.js from 849803473a,
so those will need to be reapplied if applicable. /cc @aurimasv, @mtd91429
Adds a "My Publications" source after "My Library", implemented as a
separate library. Top-level items can be dragged in and removed.
(This doesn't currently work without disabling Quick Copy.)
Also:
- Make "Group Libraries" an unselectable header instead of a container,
and don't indent group libraries
- Fix relation purging, which maybe never worked
- Pass only libraryID/key on deletes (which should speed them up)
- Fix async item cloning/copying
- Fix miscellaneous other bugs
To-do:
- Confirmation dialog on drag
- API support
Fixes#426: this allows the localization of the Word Processors Add-in
button and messages displayed in the Prefs, under the "Word Processors"
tab. These strings were previously hardcoded in the integration plugins.
[and delete unused "#zotero-prefpane-keys checkbox" (there used to be a checkbox to "Try to override conflicting shortcuts", which has been definely removed by 5b34dce40f )]
Generated files for a more robust attached-link-dialog and localized strings
* AttachLink.js
* AttachLink.xul
zotero/xpcom/attachments.js
* created function cleanAttachmentURI
* in function linkFromURL, removed the regex constraints and the comment list of valid protocols
* removed outdated function declaration from beginning of script
* Improved automatic title generation mechanism
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
- Always allow "Report Errors...", even when no errors
- Show submitted diagnostic info in report
- Use white background and unitalicized text for report
- Make window larger by default
When opening the advanced search window, the current library is
selected, and a different library can be selected to change the search
scope. If a library is read-only, the saved search button is disabled.
For saved searches, the appropriate library is selected and the
drop-down is disabled.
Also:
- Close the advanced search window after a search is saved
- The default name for saved searches ("Untitled 2", etc.) was based on
collections rather than searches
- Once an initial search has been performed, the drop-downs and
checkboxes now update the results
- More consistent spacing in advanced search window
- (dev) Zotero.DB.getNextName() now takes a libraryID as its first
parameter instead of always using My Library; the old parameters are
deprecated but still work
- Each column in the middle pane can now have its own persistent
secondary sort column, configurable from a new submenu in the column
picker menu (top right of items list). The settings are stored in
extensions.zotero.secondarySort.[primaryField]. The submenu title
includes the current primary field (e.g., "Secondary Sort (Creator)"),
which is pretty weird, and I'm not sure I want to keep it, but it does
convey that the setting is specific to the selected column.
- The fallback sort fields (firstCreator, date, title, dateAdded) are
now configurable via the extensions.zotero.fallbackSort. Setting that
pref to an empty string avoids all fallback sorts, which
allows reverse-order clicking to set the order, as requested by
@aurimasv in #275.
- The previous behavior of sorting based on the exact Creator string
(rather than the actual creators) can now be restored with the
extensions.zotero.sortCreatorAsString pref. (It simply circumvents all
the newer code, so it's pretty safe.) This setting should result in
faster sorting in large libraries that have many items with the same
Creator string.
- Some of the lesser fields in the column picker menu are now in the
More Columns submenu (which is now alphabetical)
- The "Type" column is now the less-ambiguous "Item Type".
- This uses a different method to modify the column picker menu that is
simultaneously less and more hacky. (It no longer has to duplicate
Mozilla code in a custom XBL binding that wouldn't reflect future
upstream changes, and instead it bushwhacks its way through various
boxObject properties to get to the underlying menupopup.)
* Close window on blur after completion on Mac (revert previous change)
* Don't close window when canceling
* Add Esc handler to cancel/close window
* Allow columns to be resized
* Fixes#445
* Fixes#444
- Close Zotero pane before database is closed prior to reload, instead
of waiting until reload is complete
- Show an error message if Zotero Standalone is not accessible when it
should be
Avoid ambiguously implying that Linux users should be running Zotero as
root, and tell them to fix the permissions on the Zotero executable instead.
See https://forums.zotero.org/discussion/31020/
- Catch additional errors on Windows (too-long paths, maybe aliases)
- Fix "Show File" button (which didn't show in some cases, at least on
Windows)
- Clarify error message
- Remove all "Selected" from "Selected Item", because that's how a context menu works
- Change "Remove Selected Item" to "Remove Item from Collection" (fixes#122)
- Change "Delete Selected Item from Library" to "Move Item to Trash")
- Change "Remove" on Duplicate/Unfiled Items to "Hide" (fixes#122)
- Change "Create Parent Item from Selected Item" to "Create Parent Item"
- New tag colors support, with the ability to assign colors to up to 6
tags per library. Tags with colors assigned will show up at the top of
the tag selector and can be added to (and removed from) selected items
by pressing the 1-6 keys on the keyboard. The tags will show up as
color swatches before an item's title in the items list.
- Synced settings, with Notifier triggers when they change and
accessible via the API (currently restricted on the server to
'tagColors', but available for other things upon request)
- Silent DB upgrades for backwards-compatible changes. We'll do
something fancier with async DB queries in 4.0, but this will work for
changes that can be made without breaking compatibility with older
clients, like the creation of new tables. The 'userdata' value is
capped at 76, while further increments go to 'userdata2'.
TODO:
- Try to avoid jitter when redrawing swatches
- Optimize tag color images for retina displays
- Redo attachment dots in flat style?
- Clear all colors from an item with 0 (as in Thunderbird), but I don't
think we can do this without undo
- If attachment exists at same relative path in new base directory,
leave it alone so that it continues to work.
- If attachment doesn't exist in new base directory, revert it to an
absolute path.
- If new base directory is an ancestor or descendant of the previous
base directory, adjust relative paths below the new directory so that
they keep working.
- More dialog changes
- Select current base directory in file picker when changing directory
- Always use .persistentDescriptor instead of initWithPath(), though it
probably doesn't matter, and wrap in try/catch in case the old setting
is broken in some way.
- New function Zotero.File.directoryContains(dir, file), since
nsIFile.contains() isn't recursive
- Don't use a private Zotero.Item property from outside to force path changes.
- Changed placeholder to just "attachments:" for consistency with "storage:"
- Reworked dialog text
- Use a fancier (and undocumented) filefield XUL element for path
- A few small code tweaks
- New promise-based architecture
- Library-specific file sync queues, allowing other libraries to
continue if there's an error in one library
- Library-specific sync errors, with error icons next to each library
- Changed file uploading in on-demand download mode, which had been missing
- On-demand download progress indicator in middle pane
- More accurate progress indicator
- Various tweaks and bug fixes
- Various future tweaks and bug fixes
Currently generates multi-source citations for multiple items for note
styles, but a list of note citations might be better.
Also, context menu option is unchanged, since it becomes very long if
you add "Citation/Bibliography".
Adds three new preferences controlling whether a new collection is created upon import:
- extensions.zotero.import.createNewCollection.fromFile
Controls whether a new collection is created by import from cog menu
- extensions.zotero.import.createNewCollection.fromClipboard
Controls whether a new collection is created by import from clipboard
- extensions.zotero.import.createNewCollection.fromFileOpenHandler
Controls whether a new collection is created when import is initiated by
double-clicking a file or by dragging it over Zotero. This preference applies only to
Zotero Standalone, and is configurable in the dialog that appears asking the user
to confirm the import.
Closes#19, [papercuts] (26) Clipboard import into an open collection
- Open in current window if one is already open
- Use a button instead of a label
- Don't show button if citation item is not linked to a Zotero item
This doesn't currently work if the Zotero pane has never been opened in the given window, because ZoteroPane.selectItem() appears to fail in that circumstance. This needs to be fixed.
The button wasn't opening the WebDAV URL within Standalone, and
XULRunner doesn't provide the whitelist interface anyway.
Documentation page still needed
Can choose to download files "at sync time" or "as needed"
On-demand defaults to on, but remains off for existing users
To-do:
- Handling of local and remote file changes on on-demand download
(currently if a file exists it isn't downloaded, which means a
remotely modified file won't be redownloaded in on-demand mode)
- Additional control over file downloading and retention
Other changes:
- Overhauled entire file syncing architecture
- Replaced numAttachments column with Note and Attachment columns with
dynamic icons to indicate status
- Double-clicking a parent with a missing best attachment and on-demand
downloading off no longer loads the parent URL
- Bugs
TODO: limit to large syncs?
This needs a lot of testing.
Also:
- Tweak pumpGenerator() wait level behavior to match Zotero.wait()
- If Zotero is closed in the top-most window, show a popup instead of the pane-covering progress meter, and take an optional icon in Zotero.showZoteroPaneProgressMeter() for use in the popup
- Restore protection against opening Zotero pane when Zotero.locked is set
- Display a nicer error if Zotero.DB.commitTransaction() is called without an active transaction
- Allow text with icons to extend to multiple lines in progressWindow popup
- Automatically use current window if one isn't specified in Zotero.repaint()
- Adds a per-library "Duplicate Items" virtual search to the source list -- shows up by default for "My Library" but can be added to and removed from all libraries
- Current matching algorithm is very basic: finds exact title matches (after normalizing case/diacritics/punctuation/spacing) and DOI/ISBN matches (untested)
- In duplicates view, sets are selected automatically; in other views, duplicate items can be selected manually and the merge interface can be brought up with "Merge Items" in the context menu
- Can select a master item and individual fields to merge from other versions
- Word processor integration code will automatically find mapped replacements and update documents with new item keys
Possible future improvements:
- Improved detection algorithms
- UI tweaks
- Currently if any items differ, all available versions will be shown as master item options, even if only one item is different; probably the earliest equivalent item should be shown for each distinct version
- Caching of results for performance
- Confidence scale
- Creator version selection (currently the creators from the chosen master item are kept)
- Merging of matching child items
- Better sorting of duplicates if not clustered together by the selected sort column
- Relation path compression when merging items that are already mapped to previously removed duplicates
Other changes in this commit:
- Don't show Trash in word processor integration windows
- Consider items in trash to be missing in word processor documents
- Selection of special views (Trash, Unfiled, Duplicates) is now restored properly in new windows
- Disabled field transform context menu when item isn't editable
- Left/right arrow now expands/collapses all selected items instead of just the last-selected row
- Relation deletions are now synced
- The same items row is now reselected after item deletion
- (dev) Zotero.Item.getNotes(), Zotero.Item.getAttachments(), and Zotero.Item.getTags() now return empty arrays rather than FALSE if no matches -- tests on those return values in third-party code will need to be changed
- (dev) New function Zotero.Utilities.removeDiacritics(str, lowercaseOnly) -- could be used to generate ASCII BibTeX keys
- (dev) New 'tempTable' search condition can take a table to join against -- useful for implementing virtual source lists
- (dev) Significant UI code cleanup
- (dev) Moved all item pane content into itemPane.xul
- Probably various other things
Needless to say, this needs testing.
- Implement connector for Firefox (should switch in/out of connector mode automatically when Standalone is launched or closed, although this has only been tested extensively on OS X)
- Share core translation code between Zotero and connectors
Still to be done:
- Run translators in non-Fx connectors (this works in theory, but it's not currently enabled for any translators)
- Show translation results in non-Fx connectors
- Ability to translate to server when Zotero Standalone is not running
Adds "Show Unfiled Items" context menu to libraries, which adds a virtual saved search for unfiled items. Right-click, Remove to hide. Per-library visibility is saved in prefs and persists across restarts.
Implemented as 'unfiled' search condition, but not accessible via search UI
Should probably use a different color icon to differentiate from real saved searches
also adds a parameter to disable proxy redirection by domain (although if I had known how much work would be involved in implementing this before I started, I wouldn't have bothered)
closes#1650: suppress author does not work for multiple sources
closes#1505: Edit Biblography Button Strips Year Disambiguation
closes#1503: Editing a bibliography resets all reference numbers to 1 (new)
closes#1262: Broken pluralization with et al. + other issues
closes#1238: Localize quotation marks
closes#1191: Harmonize 'plural/pluralize' label attribute with CSL schema
closes#1154: Only one works page numbers are added to the citation are when citing multiple works by the same author
closes#1097: Disambiguation issues
closes#1083: Defect in IEEE CSL with Multiple Citations
closes#993: more sophisticated subsequent-author-substitute
closes#833: text-transform doesn't work with name
- Renamed 'Repository' to 'Library Catalog' and added 'Archive'
- Moved 'Repository' data to 'Archive' for item types other than books, periodicals, and theses
- Mapped 'Archive' to CSL 'archive'
- For "Book Section", added "Book Author" (needs to be mapped to CSL container-author) (closes#872)
- For A/V item types, "Recording Type" mapped to "Type" is now "Format" mapped to "Medium" (closes#1060)
- For "Radio Broadcast" and "TV Broadcast", replaced "Series Title" with "Program Title" mapped to "Publication Title" (CSL field container-title rather than collection-title) (closes#860)
- For "Patent", added "Country" field, removed (nonfunctional) base-map from "Place" to "Country", added "Issuing Authority", replaced "Date" with "Filing Date" (not currently mapped to CSL), and moved "Date" data to "Filing Date". Kept "Issue Date" base-mapped to "Date". (closes#1375)
- For "Film", mapped "Distributor" to "Publisher" (closes#861) and added "Genre" mapped to "Type"
- For "Podcast", mapped "Audio File Type" to "Medium" rather than "Type"
- For "Bill", added "Cosponsor" creator type (closes#655)
- For "Case", added "Author" as primary creator type instead of "Counsel" (#655)
- For "Computer Program", added "Date" (closes#1502)
- For "Thesis" and "Newspaper Article", added "Place" (closes#1274)
- For "Thesis", replaced 'pages' with 'numPages' (closes#1618)
This can't yet sync, so upping sync API version and delaying dev XPI build.
- Fix search for January dates
- Support 'yesterday'/'today'/'tomorrow' and localized equivalents (case-insensitive) in date searches (e.g., [Date Added] [is] ['yesterday'])
- Make "Saved Search" a separate search condition from 'Collection' in drop-down menu
- Better handle invalid search conditions
- (Dev) Saved search conditions are now 'collection' and 'savedSearch' rather than 'collectionID' and 'savedSearchID', though the latter can be used as shortcuts when adding
This could use some testing.
- Show appropriate message when moving items to trash
- Unbind delete action from Backspace on Windows, where it's non-standard
- Fixed some inconsistencies in Cmd/Ctrl-Delete behavior -- Cmd-Delete will move item to trash without prompt in Library and move items to trash with a prompt in collection or search
API changes:
- Zotero.Items.erase(), Zotero.Item.erase(), and Zotero.ItemTreeView.deleteSelection() no longer takes eraseChildren parameter
- Group file sync via Zotero File Storage
- Split file syncing into separate modules for ZFS and WebDAV
- Dragging items between libraries copies child notes, snapshots/files, and links based on checkboxes for each (enabled by default) in the Zotero preferences
- Sync errors now trigger an exclamation/error icon separate from the sync icon, with a popup window displaying the error and an option to report it
- Various errors that could cause perpetual sync icon spinning now stop the sync properly
- Zotero.Utilities.md5(str) is now md5(strOrFile, base64)
- doPost(), doHead(), and retrieveSource() now takes a headers parameter instead of requestContentType
- doHead() can now accept an nsIURI (with login credentials), is a background request, and isn't cached
- When library access or file writing access is denied during sync, display a warning and then reset local group to server version
- Perform additional steps (e.g., removing local groups) when switching sync users to prevent errors
- Compare hash as well as mod time when checking for modified local files
- Don't trigger notifications when removing groups from the client
- Clear relation links to items in removed groups
- Zotero.Item.attachmentHash property to get file MD5
- importFromFile() now takes libraryID as a third parameter
- Zotero.Attachments.getNumFiles() returns the number of files in the attachment directory
- Zotero.Attachments.copyAttachmentToLibrary() copies an attachment item, including files, to another library
- Removed Zotero.File.getFileHash() in favor of updated Zotero.Utilities.md5()
- Zotero.File.copyDirectory(dir, newDir) copies all files from dir into newDir
- Preferences shuffling: OpenURL to Advanced, import/export character set options to Export, "Include URLs of paper articles in references" to Styles
- Other stuff I don't remember
Suffice it to say, this could use testing.
Closes#884, final period missing when a citation is first added in note styles
Closes#1298, issues with footnotes and citations in OOo
Closes#1069, Use async HTTP calls for integration requests
Closes#1027, User-customizable integration port number
Closes#698, Migration away from VBA
Closes#1085, Migrate VBA plug-in to new XML-based API
Closes#792, Auto-updating of OO plugins
- Remove reference to Word from incompatible version message, which is also used with OpenOffice
- Add lines to Doc Prefs window to clarify the necessary file format (though enforcing this at save time would be best)
- closes#1513, RTF Scan: {Bib} Placemarker Flexibility
- switches citations from parentheses to brackets, to make it easier to distingush between what was reformatted and what wasn't
- adds a list of example citations to RTF scan dialog
- Make note font configurable through extensions.zotero.note.fontFamily hidden pref (for now)
- Additional CSS rules can be added through extensions.zotero.note.css hidden pref
- Give TinyMCE note body #zotero-tinymce-note id for more specific external customization
- Remove status bar position option from visible preferences...
(Smith, 2006)
The database is scanned for each citation, and positioning is adjusted automatically for footnotes. Currently, this won't work with names with accents, but I'll get to that.
- fixes NCBI PubMed translator type (should be 12, since it's a search translator as well as a web translator)
- fixes a bug passing multiple translators to translate.setTranslator()
- increases priority of CrossRef translator
- makes sync animated arrow progress indicator into a general CSS class (zotero-small-progress-indicator) so that the same progress indicator can be used for lookup. if this is confusing, we could use something else instead.
- Only one preference for recognizing proxies for transparent redirection
- Blacklists sites and http://%h/%p
- Fixes to EZProxy
This will clear all existing proxies from Zotero
Ben's duplicate detection code, with the integration reworked a bit
Very rough, so currently requires creation of a boolean extensions.zotero.debugShowDuplicates pref to view the Actions menu option
- Still experimental, but committing for testing
- Sync conflicts with deleted items aren't yet supported
Unrelated: deprecated ZoteroPane.deleteSelectedItem() in favor of more accurately named deleteSelectedItems()
- disable bookmarks mode for note-based styles
- display an error when attempting to insert a bookmark into a footnote
- closes#1201, Keyboard focus stays in Word when creating or editing a citation
