- All string values are now trimmed going into the DB (with a migration step for existing values) -- this fixes erroneous conflicts due to leading/trailing whitespace in sync XML being ignored
- Case disambiguation fixed on server
- Added basic diff ability to collections and tags so that identical objects won't trigger conflicts
- Fixed various other bugs that could cause erroneous conflicts
- Moved string fields in serialize() objects into a 'fields' object for consistency with Zotero.Item
Upshot of most of the above is that identical pre-upgrade libraries should now merge cleanly
Also reorganized/simplified/modularized parts of the sync code
Also migrates remaining files for past upgrades on the trunk (assuming their ids haven't been changed by syncing--if the item that has a given id already has a directory or if there is no attachment item with that id, the directory will be moved to orphaned-files)
- Fix a scoping issue that could prevent some property files from being purged
- Properly handle systems with millisecond-level file mod time granularity (clearing of storage sync history is required)
- Fix issue that might have erroneously removed property files
- Fix potential security issues with symlinks in ZIP files due to Firefox brokenness
- Zotero.Utilities.HTTP.doGet() can now take a URI instead (and doesn't display the password that way for authenticated requests)
- For now, delete orphaned files immediately when using "Purge Orphaned Storage Files" instead of waiting a day
- Properly remove deleted files from delete log
- Better debugging of various things
- allow deletion of multiple styles simultaneously
- split Zotero.Styles/Zotero.Style and Zotero.CSL into style.js and csl.js respectively
- add Zotero.File.getBinaryContents for binary-safe file reading
- add Zotero.MIMETypeHandler to provide a unified interface for registering observers and capturing MIME types with Zotero
Files are copied from translators.zip and styles.zip (or, for SVN installs, 'translators' and (for now) 'csl' directories) in the installation directory to 'translators' and 'styles' directories in the data directory. A build_zip file is provided for testing translators.zip (which will take precedence over a 'translators' directory) but isn't required.
The timestamp stored in repotime.txt is stored in the database and is sent to the server for updates since that time.
Updating a file in [install-dir]/translators or [install-dir]/styles automatically copies all files in that directory to the data directory.
This will break translation for the dev XPI and on the trunk until Dan commits his part of this (or you copy the translators dir in SVN to the zotero dir in your profile dir)
- Still experimental and incomplete, with no lock support and not much error handling
Also:
- New expiry date for sync functions
- Attachment character set was being dropped during syncing
- Possibly improves sizing issues with preferences window
- Fixes problems with attachment filenames with extended characters
- Fixes some problem with tags that I don't remember
- Makes XMLHTTPRequest calls are now background requests (no auth windows or other prompts)
- Z.U.HTTP.doOptions() now takes an nsIURI instead of a URL spec
- New methods:
- Zotero.Utilities.rand(min, max)
- Zotero.Utilities.probability(x)
- Zotero.Utilities.Base64.encode(str) and decode(str)
- Zotero.getTempDirectory()
- Zotero.Date.dateToISO(date) - convert JS Date object to ISO 8601 UTC date/time
- Zotero.Date.isoToDate(isoDate) - convert an ISO 8601 UTC date/time to a JS Date object
The feature grabs relevant fragments from the document and then searches them on Google Scholar. This will only work with OCRed PDFs, so it doesn't work with JSTOR, but it should work fairly well with everything else.
changes default behavior. transparent redirection is now enabled by default, and a caution dialog appears when proxies are first accessed. when transparent redirection is turned off, no dialog appears, and proxies get saved automatically. when the user switches transparent redirection on, there is a warning that s/he should check that there are only trusted proxies in the list. i'm not sure how well i've worded the messages, so feel free to suggest better alternatives.
also, a bit of jsdoc cleanup on proxy.js
closes#831, transparent EZProxy support
adds a proxy pane to the preferences
asks before saving proxies to the DB (to avoid the potential phishing risk #831 would otherwise pose)
The remote object could legitimately not have been updated locally since the last sync if the server data was cleared and synched from a different client, potentially resulting in a different key condition on sync. The two sides should now be properly merged (at least in theory).
Adds a Python/py-appscript-based plug-in for Word 2008. To get this to work, you'll need to copy the Zotero directory (not its contents) to ~/Microsoft User Data/Word Script Menu Items and install py-appscript (sudo easy_install appscript)
Some caveats:
- Requires Word be installed at /Applications/Microsoft Office 2008/Microsoft Word 2008.app (this is fixable, but I'm still determining the best way to solve it)
- Still need to figure out what to do with items that have been deleted from the DB (right now, we just ignore them)
- Sometimes, Python.app launches with the script, which seems to slow execution time
Also moves orphaned directories into orphaned-files folder in data dir
Safety first: Keeps track of moved files, moving them back if there's an error before the end of the upgrade process (since the keys are generated randomly and would be different if recreated), and creates a zotero.moved-files.36.bak file with a list of id/key pairs
closes#704, EndNote to Zotero style converter (won't actually convert styles due to copyright concerns, but will load them into the DB)
also adds CSL style manager
- Inspired by Dan Chudnov's Python/MODS-based Zeroconf demo at THATcamp
- Enabled by extensions.zotero.zeroconf.enabled (off by default)
- Currently supports only OS X (tested on Leopard, not sure about earlier versions)
- Uses Apple's dns-sd and mDNS command-client clients, but should be able to be extended to other clients, though a native library would be far superior
- Discovery is on-demand for now via Actions menu ("Search for Shared Libraries")
- Includes rudimentary web server (code copied from integration.js) that serves items as sync XML -- no authentication yet!
- Only supports top-level items
- Remote libraries show up in left pane (under remote computer name, for now)
- Items can be dragged into collections (but not the library yet, for some reason)
- On first run, might cause a long pause and the "This file was downloaded from the Internet" message on Leopard -- can't manage to get around the quarantine for the script file that we need to access stdout from Firefox
- Needs a lot of work, and without a real JS (or otherwise Mozilla-native) Zeroconf library we can't do proper discovery without intermittent polling
- But it works, at least for me
Also includes some data/sync-layer changes that I needed along the way (and that we'll need for shared collections of any type)
Apologies for the massive (and, due to data_access.js splitting, difficult-to-follow) commit. Please note that external code that accesses the data layer may need to be tweaked for compatibility. Here's a comprehensive-as-possible changelog:
- Added server sync functionality (incomplete)
- Overhaul of data layer
- Split data_access.js into separate files (item.js, items.js, creator.js, etc.)
- Made creators and collections first-class objects, similar to items
- Constructors now take id as first parameter, e.g. new Zotero.Item(1234, 'book'), to allow explicit id setting and id changing
- Made various data layer operations (including attachment fields) require a save() rather than making direct DB changes
- Better handling of unsaved objects
- Item.setCreator() now takes creator objects instead of creator ids, and Item.save() will auto-save unsaved creators
- clone() now works on unsaved objects
- Newly created object instances are now disabled after save() to force refetch of globally accessible instance using Zotero.(Items|Creators|etc.).get()
- Added secondary lookup key to data objects
- Deprecated getID() and getItemType() methods in favor of .id and .itemTypeID properties
- toArray() deprecated in favor of serialize(), which has a somewhat modified format
- Added support for multiple creators with identical data -- currently unimplemented in interface and most of data layer
- Added Item.diff() for comparing item metadata
- Database changes
- Added SQLite triggers to enforce foreign key constraints
- Added Zotero.DB.transactionVacuum flag to run a VACUUM after a transaction
- Added Zotero.DB.transactionDate, .transactionDateTime, and transactionTimestamp to retrieve consistent timestamps for entire transaction
- Properly store 64-bit integers
- Set PRAGMA locking_mode=EXCLUSIVE on database
- Set SQLite page size to 4096 on new databases
- Set SQLite page cache to 8MB
- Do some database cleanup and integrity checking on migration from 1.0 branch
- Removed IF NOT EXISTS from userdata.sql CREATE statements -- userdata.sql is now processed only on DB initialization
- Removed itemNoteTitles table and moved titles into itemNotes
- Abstracted metadata edit box and note box into flexible XBL bindings with various modes, including read-only states
- Massive speed-up of item tree view
- Several fixes from 1.0 branch for Fx3 compatibility
- Added Notifier observer to log delete events for syncing
- Zotero.Utilities changes
- New methods getSQLDataType() and md5()
- Removed onError from Zotero.Utilities.HTTP.doGet()
- Don't display more than 1024 characters in doPost() debug output
- Don't display passwords in doPost() debug output
- Added Zotero.Notifier.untrigger() -- currently unused
- Added Zotero.reloadDataObjects() to reset all in-memory objects
- Added |chars| parameter to Zotero.randomString(len, chars)
- Added Zotero.Date.getUnixTimestamp() and Date.toUnixTimestamp(JSDate)
- Adjusted zotero-service.js to simplify file inclusion
Various things (such as tags) are temporarily broken.
Closes#285, Hitting Esc should close tags popup
Fixes various UI issues and problems adding tags on Minefield
Still getting autocomplete self-destruction
If anyone's out there, now would be the time to test this.
Missing icons for the new types -- I could try to get some of those now, or we could just launch. Maybe we'll just launch. But really, might want to test this. Just sayin'.
Use automaticSnapshots pref (which defaults to on and is changeable in the prefs window) rather than downloadAssociatedFiles (which defaults to off and is only settable through about:config at the moment) for now in translate.js
downloadAssociatedFiles should eventually be used for PDFs and other large files, whereas automaticSnapshots will be for HTML and the like -- in the meantime, I think it's OK for scrapers to just follow the visible pref for both, since otherwise they'd be totally confused when the NIFP button took a snapshot and the scrapers didn't
Simon, if there's any problem I'm not aware of with switching this for now (other than people getting some large PDFs on JStor), let me know.
closes#165, verify import/export can carry all data for all fields and item types
closes#168, make sure MODS import works with files from external sources
Attach Snapshot of Current Page
Attach Link to Current Page
I also removed "New Item" from the context menu, as it was the only option in the menu that wasn't specific to the selected item, it made the whole menu a lot more confusing, and it doesn't take all that much longer to click the (+) button and choose Book (assuming that's even the item you want).
Addresses #155, Localize strings
- Added "Search subfolders"
- Localized strings in search dialog
Known issue:
#342, "Search subfolders" checkbox should be greyed out until applicable
Including in the DB, which it turns out isn't really all that bad (thanks, among other things, to SQLite's ability to DROP tables within transactions without autocommitting (which MySQL can't do))
At least some scrapers (NYT and WashPo, for sure) should be updated to follow this pref
Addresses #327, Scrapers should either take snapshots or use URL field
Changed creator type <label> to <toolbarbutton> -- hopefully won't have any ill effects with other platforms' default themes (I haven't checked)
Moved some metadata pane CSS from overlay.css to zotero.css
Currently uses an annoying popup window, since the code used in the regular metadata pane has too much baggage to use here -- it really should be abstracted into an XBL binding, but that won't be an easy feat
Also fixes SQL error opening popup notes and JS strict warnings when saving them
- Currently requires user to enter dates in SQL format if they want to change the access date, but at least it doesn't mangle the dates anymore
- Uses new function ScholardammitZotero.Date.dateToSQL(Date date [, Boolean toUTC])
- Utilities.lpad() now forces _string_ to a string so that .length exists
- Unrelated: Item.save() now returns false if the item didn't change
closes#313, Blacklist known ad sites from scraper detection
closes#306, some New York Times ads prevent page from being recognized
closes#308, attachment import bug
currently, the ad site blacklist is located at the top of ingester/browser.js. at some point, we may want to switch this to a database table.
addresses #327, Scrapers should either take snapshots or use URL field
closes#309, Integration server prevents Zotero from loading in multiple instances of Firefox
In other words, show both "Shakespeare" and "Shakespeare, William" in the drop-down, and if the latter is chosen, save both fields
One issue is that since the autocomplete is by default limited to the width of the textbox, longer entries get truncated (though you can see them with a mouseover), and that may not be easy to fix.
