- Add high-res webpage icon
- Show webpage icon in grayscale when no translator (except on hover, for fun)
- Remove pre-Australis icons
- Switch to CustomizableUI API for toolbar icon
- Move icon generation code to separate file
- Add Zotero.hiRes flag for Retina/etc. displays (available only after a window
has loaded)
Known issues:
- While the gray is mostly to be less distracting, the gray/color distinction
will probably be lost on most people. A separate guidance panel for the gray
icon might help.
- On pages with frames, the webpage icon appears first and then is replaced
with a translator icon.
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
- Each column in the middle pane can now have its own persistent
secondary sort column, configurable from a new submenu in the column
picker menu (top right of items list). The settings are stored in
extensions.zotero.secondarySort.[primaryField]. The submenu title
includes the current primary field (e.g., "Secondary Sort (Creator)"),
which is pretty weird, and I'm not sure I want to keep it, but it does
convey that the setting is specific to the selected column.
- The fallback sort fields (firstCreator, date, title, dateAdded) are
now configurable via the extensions.zotero.fallbackSort. Setting that
pref to an empty string avoids all fallback sorts, which
allows reverse-order clicking to set the order, as requested by
@aurimasv in #275.
- The previous behavior of sorting based on the exact Creator string
(rather than the actual creators) can now be restored with the
extensions.zotero.sortCreatorAsString pref. (It simply circumvents all
the newer code, so it's pretty safe.) This setting should result in
faster sorting in large libraries that have many items with the same
Creator string.
- Some of the lesser fields in the column picker menu are now in the
More Columns submenu (which is now alphabetical)
- The "Type" column is now the less-ambiguous "Item Type".
- This uses a different method to modify the column picker menu that is
simultaneously less and more hacky. (It no longer has to duplicate
Mozilla code in a custom XBL binding that wouldn't reflect future
upstream changes, and instead it bushwhacks its way through various
boxObject properties to get to the underlying menupopup.)
FT sync is enabled by default for new users and configurable in the Sync
prefs.
Also disable downgrades once full-text sync is enabled, since otherwise
someone switching back and forth between old and new versions could miss
full-text content updates.
Trigger a proxy authentication prompt at startup if a PAC file is
installed and one of a few randomly chosen big sites requires a proxy.
This also improves general proxy detection by not making a request
to S3 unless it would actually be proxied.
It's not clear when this became an issue, but our usual background HTTP
requests (set that way to avoid triggering auth prompts when saving from
websites) weren't triggering proxy authentication dialogs, which was breaking
most network activity in Standalone. To fix this, we now make a foreground
request at startup to a file on S3 and resolve the Zotero.proxyAuthComplete
promise when we're done. Any network requests that want to wait for proxy
authentication can wait for that promise.
This behavior can be disabled via the triggerProxyAuthentication hidden pref.
Adds three new preferences controlling whether a new collection is created upon import:
- extensions.zotero.import.createNewCollection.fromFile
Controls whether a new collection is created by import from cog menu
- extensions.zotero.import.createNewCollection.fromClipboard
Controls whether a new collection is created by import from clipboard
- extensions.zotero.import.createNewCollection.fromFileOpenHandler
Controls whether a new collection is created when import is initiated by
double-clicking a file or by dragging it over Zotero. This preference applies only to
Zotero Standalone, and is configurable in the dialog that appears asking the user
to confirm the import.
Closes#19, [papercuts] (26) Clipboard import into an open collection
Falls back to /usr/bin/xdg-open by default, configurable via
extensions.zotero.fallbackLauncher.unix (or .windows, though that
should be much rarer). Also used for file revealing.
If xdg-open fails, files launch via the external helper app dialog and
URLs just don't work.
Can choose to download files "at sync time" or "as needed"
On-demand defaults to on, but remains off for existing users
To-do:
- Handling of local and remote file changes on on-demand download
(currently if a file exists it isn't downloaded, which means a
remotely modified file won't be redownloaded in on-demand mode)
- Additional control over file downloading and retention
Other changes:
- Overhauled entire file syncing architecture
- Replaced numAttachments column with Note and Attachment columns with
dynamic icons to indicate status
- Double-clicking a parent with a missing best attachment and on-demand
downloading off no longer loads the parent URL
- Bugs
Does not currently support sorting or citation editor, and has not been tested on any platform but OS X, so off by default for now. Can be enabled with the hidden pref extensions.zotero.integration.quickFormat.
- Closes#1831, Connectors should be able to save via API in the absence of Zotero Standalone
- Fixes Zotero.Utilities.deepCopy() for arrays
- Fixes some circumstances where an error would not be saved for future error reporting
- Fixes connector status checking
- Implement connector for Firefox (should switch in/out of connector mode automatically when Standalone is launched or closed, although this has only been tested extensively on OS X)
- Share core translation code between Zotero and connectors
Still to be done:
- Run translators in non-Fx connectors (this works in theory, but it's not currently enabled for any translators)
- Show translation results in non-Fx connectors
- Ability to translate to server when Zotero Standalone is not running
also adds a parameter to disable proxy redirection by domain (although if I had known how much work would be involved in implementing this before I started, I wouldn't have bothered)
- Store one item per IA bucket, with attachments stored as objects
- Use proper mediatype field based on Zotero item type
- Commons list is now pulled dynamically based on RDF stored at IA, without need for corresponding local item (which may have been deleted, etc.)
- Once available, OCRed PDFs can be pulled down by right-clicking on Commons and selecting Refresh
- Downloaded OCRed PDFs are now named the same as the existing attachment, with "(OCR)" appended
- The relations table is used to link downloaded OCRed PDFs to the IA file, so the downloaded file can be renamed without triggering another download
- The Commons view is marked for automatic refresh after an item is uploaded
- Added some progress notifications, though more are probably needed
- Other things
Also:
- Added Zotero.File.getBinaryContents(file)
- Erase an item's relations when the item is deleted, and purge orphaned ones
- Zotero.URI.eraseByPathPrefix(prefix) no longer prepends 'http://zotero.org' (which has been moved to Zotero.URI.defaultPrefix)
- New function Zotero.URI.eraseByURI(prefix)
Known Issues:
- Slow (some IA changes should be able to speed it up)
- Identifier format is likely temporary
- Sometimes it stops during setTimeout() calls for no apparent reason whatsoever
- Didn't test items with multiple attachments
- Not sure if Commons view will auto-refresh if you switch to it before the upload is done
- IA translator not yet updated
- Deleting items not supported by IA
- Date Added/Date Modified don't show up properly in Zotero for Commons items
- Group file sync via Zotero File Storage
- Split file syncing into separate modules for ZFS and WebDAV
- Dragging items between libraries copies child notes, snapshots/files, and links based on checkboxes for each (enabled by default) in the Zotero preferences
- Sync errors now trigger an exclamation/error icon separate from the sync icon, with a popup window displaying the error and an option to report it
- Various errors that could cause perpetual sync icon spinning now stop the sync properly
- Zotero.Utilities.md5(str) is now md5(strOrFile, base64)
- doPost(), doHead(), and retrieveSource() now takes a headers parameter instead of requestContentType
- doHead() can now accept an nsIURI (with login credentials), is a background request, and isn't cached
- When library access or file writing access is denied during sync, display a warning and then reset local group to server version
- Perform additional steps (e.g., removing local groups) when switching sync users to prevent errors
- Compare hash as well as mod time when checking for modified local files
- Don't trigger notifications when removing groups from the client
- Clear relation links to items in removed groups
- Zotero.Item.attachmentHash property to get file MD5
- importFromFile() now takes libraryID as a third parameter
- Zotero.Attachments.getNumFiles() returns the number of files in the attachment directory
- Zotero.Attachments.copyAttachmentToLibrary() copies an attachment item, including files, to another library
- Removed Zotero.File.getFileHash() in favor of updated Zotero.Utilities.md5()
- Zotero.File.copyDirectory(dir, newDir) copies all files from dir into newDir
- Preferences shuffling: OpenURL to Advanced, import/export character set options to Export, "Include URLs of paper articles in references" to Styles
- Other stuff I don't remember
Suffice it to say, this could use testing.
- dropping Zotero items into a bucket puts them in that IA bucket
- double clicking a bucket takes you to that IA bucket
In order to enable Zotero Commons:
1) Get an access key and secret key at http://www.archive.org/account/s3.php
2) Go to about:config
3) Search "commons" (no quotes)
4) Set "extensions.zotero.commons.enabled" to true
5) Enter your S3 access key into "extensions.zotero.commons.accessKey"
6) Enter your S3 secret key into "extensions.zotero.commons.secretKey"
7) Enter your buckets into "extensions.zotero.commons.buckets" as a comma separated list
Note: Steps 4-7 take effect in new windows
- Make note font configurable through extensions.zotero.note.fontFamily hidden pref (for now)
- Additional CSS rules can be added through extensions.zotero.note.css hidden pref
- Give TinyMCE note body #zotero-tinymce-note id for more specific external customization
- Remove status bar position option from visible preferences...
(Smith, 2006)
The database is scanned for each citation, and positioning is adjusted automatically for footnotes. Currently, this won't work with names with accents, but I'll get to that.
- Only one preference for recognizing proxies for transparent redirection
- Blacklists sites and http://%h/%p
- Fixes to EZProxy
This will clear all existing proxies from Zotero
- Disable unresponsive script warning during items list refresh
- Zotero.UnresponsiveScriptIndicator.disable() now returns true if it disables the indicator and false if it was already disabled
- Still experimental and incomplete, with no lock support and not much error handling
Also:
- New expiry date for sync functions
- Attachment character set was being dropped during syncing
- Possibly improves sizing issues with preferences window
- Fixes problems with attachment filenames with extended characters
- Fixes some problem with tags that I don't remember
- Makes XMLHTTPRequest calls are now background requests (no auth windows or other prompts)
- Z.U.HTTP.doOptions() now takes an nsIURI instead of a URL spec
- New methods:
- Zotero.Utilities.rand(min, max)
- Zotero.Utilities.probability(x)
- Zotero.Utilities.Base64.encode(str) and decode(str)
- Zotero.getTempDirectory()
- Zotero.Date.dateToISO(date) - convert JS Date object to ISO 8601 UTC date/time
- Zotero.Date.isoToDate(isoDate) - convert an ISO 8601 UTC date/time to a JS Date object
changes default behavior. transparent redirection is now enabled by default, and a caution dialog appears when proxies are first accessed. when transparent redirection is turned off, no dialog appears, and proxies get saved automatically. when the user switches transparent redirection on, there is a warning that s/he should check that there are only trusted proxies in the list. i'm not sure how well i've worded the messages, so feel free to suggest better alternatives.
also, a bit of jsdoc cleanup on proxy.js
closes#831, transparent EZProxy support
adds a proxy pane to the preferences
asks before saving proxies to the DB (to avoid the potential phishing risk #831 would otherwise pose)
- Inspired by Dan Chudnov's Python/MODS-based Zeroconf demo at THATcamp
- Enabled by extensions.zotero.zeroconf.enabled (off by default)
- Currently supports only OS X (tested on Leopard, not sure about earlier versions)
- Uses Apple's dns-sd and mDNS command-client clients, but should be able to be extended to other clients, though a native library would be far superior
- Discovery is on-demand for now via Actions menu ("Search for Shared Libraries")
- Includes rudimentary web server (code copied from integration.js) that serves items as sync XML -- no authentication yet!
- Only supports top-level items
- Remote libraries show up in left pane (under remote computer name, for now)
- Items can be dragged into collections (but not the library yet, for some reason)
- On first run, might cause a long pause and the "This file was downloaded from the Internet" message on Leopard -- can't manage to get around the quarantine for the script file that we need to access stdout from Firefox
- Needs a lot of work, and without a real JS (or otherwise Mozilla-native) Zeroconf library we can't do proper discovery without intermittent polling
- But it works, at least for me
Also includes some data/sync-layer changes that I needed along the way (and that we'll need for shared collections of any type)
Apologies for the massive (and, due to data_access.js splitting, difficult-to-follow) commit. Please note that external code that accesses the data layer may need to be tweaked for compatibility. Here's a comprehensive-as-possible changelog:
- Added server sync functionality (incomplete)
- Overhaul of data layer
- Split data_access.js into separate files (item.js, items.js, creator.js, etc.)
- Made creators and collections first-class objects, similar to items
- Constructors now take id as first parameter, e.g. new Zotero.Item(1234, 'book'), to allow explicit id setting and id changing
- Made various data layer operations (including attachment fields) require a save() rather than making direct DB changes
- Better handling of unsaved objects
- Item.setCreator() now takes creator objects instead of creator ids, and Item.save() will auto-save unsaved creators
- clone() now works on unsaved objects
- Newly created object instances are now disabled after save() to force refetch of globally accessible instance using Zotero.(Items|Creators|etc.).get()
- Added secondary lookup key to data objects
- Deprecated getID() and getItemType() methods in favor of .id and .itemTypeID properties
- toArray() deprecated in favor of serialize(), which has a somewhat modified format
- Added support for multiple creators with identical data -- currently unimplemented in interface and most of data layer
- Added Item.diff() for comparing item metadata
- Database changes
- Added SQLite triggers to enforce foreign key constraints
- Added Zotero.DB.transactionVacuum flag to run a VACUUM after a transaction
- Added Zotero.DB.transactionDate, .transactionDateTime, and transactionTimestamp to retrieve consistent timestamps for entire transaction
- Properly store 64-bit integers
- Set PRAGMA locking_mode=EXCLUSIVE on database
- Set SQLite page size to 4096 on new databases
- Set SQLite page cache to 8MB
- Do some database cleanup and integrity checking on migration from 1.0 branch
- Removed IF NOT EXISTS from userdata.sql CREATE statements -- userdata.sql is now processed only on DB initialization
- Removed itemNoteTitles table and moved titles into itemNotes
- Abstracted metadata edit box and note box into flexible XBL bindings with various modes, including read-only states
- Massive speed-up of item tree view
- Several fixes from 1.0 branch for Fx3 compatibility
- Added Notifier observer to log delete events for syncing
- Zotero.Utilities changes
- New methods getSQLDataType() and md5()
- Removed onError from Zotero.Utilities.HTTP.doGet()
- Don't display more than 1024 characters in doPost() debug output
- Don't display passwords in doPost() debug output
- Added Zotero.Notifier.untrigger() -- currently unused
- Added Zotero.reloadDataObjects() to reset all in-memory objects
- Added |chars| parameter to Zotero.randomString(len, chars)
- Added Zotero.Date.getUnixTimestamp() and Date.toUnixTimestamp(JSDate)
- Adjusted zotero-service.js to simplify file inclusion
Various things (such as tags) are temporarily broken.
At least some scrapers (NYT and WashPo, for sure) should be updated to follow this pref
Addresses #327, Scrapers should either take snapshots or use URL field
Changed "Scholar" to "Zotero", everywhere
Apologies to anyone with working copy changes, but there are probably the fewer at this moment than there will be again.
Hopefully this won't break anything, though existing prefs will be lost. I avoided scholar.google.com--if you know any other legitimate "scholar"s in the code, be sure to fix them once I'm done here.
This is a multi-commit change--there's at least one more coming. *Do not update to this version! It won't work!*
closes#160, cache regular expressions
closes#188, rewrite MARC handling functions
MARC-based translators should now produce item types besides "book." right now, artwork, film, and manuscript are available. MARC also has codes for various types of audio (speech, music, etc.) and maps.
the EBSCO translator does not yet produce attachments. i sent them an email because their RIS export is invalid (the URLs come after the "end of record" field) and i'm waiting to see if they'll fix it before i try to fix it myself.
the EBSCO translator is unfortunately a bit slow, because it has to make 5 requests in order to get RIS export. the alternative (scraping individual item pages) would be even slower.
regular expression caching can be turned off by disabling extensions.scholar.cacheTranslatorData in about:config. if you leave it on, you'll have to restart Firefox after updating translators.
closes#3, Overflow metadata dumps into "extra" field
add "extra" data where such data is useful and conveniently accessible (not available for XML-based export or MARC formats yet)
add links to permanent URLs
download associated files from full text sources (if extensions.scholar.downloadAssociatedFiles preference is enabled)
fix WorldCat translator
improve InnoPAC translator (it now works on Georgetown search results pages, albeit slowly, because it must first realize the catalog is misconfigured)
tag items from SIRSI and WorldCat
return to putting the full lengths of books into "pages," because some citation styles require it
fix COinS (broken a few revisions ago)
Scholar should now attempt to process citation information from EndNote download links (MIME types application/x-endnote-refer and application/x-research-info-systems). in situations where Scholar cannot process the information, a standard helper app dialog will appear. this behavior is controlled by the preference extensions.scholar.parseEndNoteMIMETypes.
Scholar.OpenURL.resolve(item) returns the URL that retrieves an item from the user's OpenURL resolver. this means we can implement a "find in my library" feature.
Scholar.OpenURL.discoverResolvers() returns a list of available resolvers for the user's current location (by IP address).
