Our strategy: put arguments into a property of the function, and then
get them out unwrapped. This avoids security checks on arguments passed
to the function.
It's way too slow, though, since the whole list is regenerated after
merging.
Fixes#519
Also:
- The arguments to Zotero.Item.prototype.clone() have changed, and it no
longer takes an existing item or copies primary data. To create an
in-memory copy of an item, use the new Zotero.Item.prototype.copy().
- Zotero.Item.prototype.getUsedFields() now gets in-memory fields rather
than the fields in the database
Our modifications no longer include a custom yield handler to
automatically call all() on yielded arrays (which maintained Bluebird
1.x behavior). It's now necessary to call all() or similar explicitly.
Also fixed a few incorrect yields hidden by that behavior.
The state check errors were caused by 4812ab6f, which was a fix for
"Q.async(...)(...) is undefined" errors caused by ad8b81f4c, which was a
fix for "too much recursion" errors related to Task.spawn() on Windows
with JIT enabled.
This will hopefully fix some remaining issues with long filenames during
syncing, particularly on Linux with encrypted filenames (which have a
filename length of 143).
(This may have reintroduced some edge case bugs, so it needs some
testing.)
Previously, 'collection' and 'savedSearch' conditions used
"[libraryID]_[key]" format. Now, the condition should contain only a
key, and the libraryID will be drawn from either the search itself or a
libraryID search condition. Old-style conditions are still parsed if
provided.
- Protocol handler extensions can now handle promises and can also make
data available as it's ready instead of all at once (e.g., reports now
output one entry at a time)
- zotero:// URL syntaxes are now more consistent and closer to the web
API (old URLs should work, but some may currently be broken)
Also:
- Code to generate server API, currently available for testing via
zotero://data URLs but eventually moving to HTTP -- zotero://data URLs match
web API URLs, with a different prefix for the personal library (/library vs.
/users/12345)
- Miscellaneous fixes to data objects
Under the hood:
- Extensions now return an AsyncChannel, which is an nsIChannel implementation
that takes a promise-yielding generator that returns a string,
nsIAsyncInputStream, or file that will be used for the channel's data
- New function Zotero.Utilities.Internal.getAsyncInputStream() takes a
generator that yields either promises or strings and returns an async input
stream filled with the yielded strings
- Zotero.Router parsers URLs and extract parameters
- Zotero.Item.toResponseJSON()
And remove UDF setup code
Duplicates view also used REGEXP, so we'll need to figure out another approach
for the async rewrite of that (#519). There are no other current UDF consumers,
so this closes#528.
And catch other errors and throw StopIteration so that they stop the
search. (Non-StopIteration errors in onRow don't stop Sqlite.jsm
queries. We were logging them but then re-throwing them, which didn't do
anything.)
This required doing additional caching at startup (e.g., item types and fields)
so that various methods can remain synchronous.
This lets us switch back to using the current Sqlite.jsm. Previously we were
bundling the Fx24 version, which avoided freezes with locking_mode=EXCLUSIVE
with both sync and async queries.
Known broken things:
- Autocomplete
- Database backup
- UDFs (e.g., REGEXP function used in Zotero.DB.getNextName())
- Zotero.Item.prototype.getFilePath() is now synchronous, with a separate async getFilePathAsync()
- getFile() no longer takes a skipExistsCheck parameter, since that shouldn't happen synchronously
- Zotero.Items.getByLibraryAndKey() is now synchronous again, with a
separate Zotero.Items.getByLibraryAndKeyAsync() - I haven't fully
tested this, so I'm not sure if there will need to be any async
calls.
- Some of the full-text indexing functions now take file paths instead of nsIFile objects
- Zotero.File.getContentsAsync() can now take a string path as well
Promise-based rewrite of most of the codebase, with asynchronous database and file access -- see https://github.com/zotero/zotero/issues/518 for details.
WARNING: This includes backwards-incompatible schema changes.
An incomplete list of other changes:
- Schema overhaul
- Replace main tables with new versions with updated schema
- Enable real foreign key support and remove previous triggers
- Don't use NULLs for local libraryID, which broke the UNIQUE index
preventing object key duplication. All code (Zotero and third-party)
using NULL for the local library will need to be updated to use 0
instead (already done for Zotero code)
- Add 'compatibility' DB version that can be incremented manually to break DB
compatibility with previous versions. 'userdata' upgrades will no longer
automatically break compatibility.
- Demote creators and tags from first-class objects to item properties
- New API syncing properties
- 'synced'/'version' properties to data objects
- 'etag' to groups
- 'version' to libraries
- Create Zotero.DataObject that other objects inherit from
- Consolidate data object loading into Zotero.DataObjects
- Change object reloading so that only the loaded and changed parts of objects are reloaded, instead of reloading all data from the database (with some exceptions, including item primary data)
- Items and collections now have .parentItem and .parentKey properties, replacing item.getSource() and item.getSourceKey()
- New function Zotero.serial(fn), to wrap an async function such that all calls are run serially
- New function Zotero.Utilities.Internal.forEachChunkAsync(arr, chunkSize, func)
- Add tag selector loading message
- Various API and name changes, since everything was breaking anyway
Known broken things:
- Syncing (will be completely rewritten for API syncing)
- Translation architecture (needs promise-based rewrite)
- Duplicates view
- DB integrity check (from schema changes)
- Dragging (may be difficult to fix)
Lots of other big and little things are certainly broken, particularly with the UI, which can be affected by async code in all sorts of subtle ways.
* Accept detailed cookie information (including host, path, secureOnly, and hostOnly) from connectors so we can send correct cookies when fetching pages from different hosts. This way we also don't have to worry about exposing cookies to different hosts.
* Don't drop cookies that we receive from other hosts. Some pages (e.g. PDF URLs) result in redirects to other hosts (and even domains) that then set cookies, which are required to retrieve the PDF. (e.g. Cell Press stores PDFs on ScienceDirect, but their PDF links initially point to cell.com).
* Send detailed cookies where possible. Currently that's only for Chrome/Opera and Firefox in Connector mode. Does not seem to be possible in Safari.
Possibly to other places as well (but not Notepad, which doesn't work
from Firefox or Chrome either)
Unfortunately this requires going back to 'copy' cursor feedback when
dragging, even when Shift is used. We can only choose one on Windows (as
far as I can tell), and we were previously using the unadorned 'move'.
nsICollation broke for some locales. (Testing requires changing the
language setting in Language & Region and then restarting the computer.
The change seems to not fully go into effect until then, even though the
UI changes.) This is fixed in Nightly, but we can work around it by
using the new Intl.Collator.
With Task.spawn, regular expressions in Zotero.DB were causing "too much
recursion" errors on Windows with JIT enabled.
This requires a change to Q to allow async() to take a generator instead
of a generator-maker (which is the reason it was using Task.spawn to
begin with).
S3 upload timeouts were retried already with an exponential backoff, but
this adds retrying for other kinds of upload failures as well as failed
downloads. If 5 consecutive failures occur a file sync error is
thrown.
Failed file sync requests to the Zotero API are not currently retried,
but S3 accounts for the majority.
The download portion of this still needs further testing.
When starting in Connector mode (i.e. Standalone is open), Zotero first starts in Full mode, looks for Standalone, then "shuts down" and restarts in Connector mode. `Zotero.shutdown()` returns a promise which is then followed up by a `Zotero.init` call. Thus, when starting in Connector mode, Zotero initialization is asynchronous and makes it possible for `Zotero_Browser.init()` to be called before `Zotero.initialized` is true, which prevents `Zotero_Browser` from initializing. Additionally, even if `Zotero_Browser.init()` is called after Zotero is initialized in Connector mode, it is possible that `Zotero_Browser.init()` will be called _after_ the "load" event for browser.xul has already fired, so `chromeLoad` is never called. This patch ensures that both of these race conditions are taken into account.
Can be triggered by double-clicking or Return and also by F2 on
Windows/Linux
This does mean double-clicking no longer toggles the collection open and
closed. If we wanted to preserve that we could probably capture the
double-click.
Closes Trac ticket 231, only 8 years later
Previously, position/size was persisted for each item's note
individually, but that meant that there was no default position/size for
the note window and an entry was created in localstore.rdf for
every note opened in a new window. There's also a good chance people had
no idea what was going on.
- Always allow "Report Errors...", even when no errors
- Show submitted diagnostic info in report
- Use white background and unitalicized text for report
- Make window larger by default
That was easy. (@simonster, since you disabled this originally, any reason not
to do this, other than the extra tab stop? At least in current Firefox
versions, this is the default behavior.)
When opening the advanced search window, the current library is
selected, and a different library can be selected to change the search
scope. If a library is read-only, the saved search button is disabled.
For saved searches, the appropriate library is selected and the
drop-down is disabled.
Also:
- Close the advanced search window after a search is saved
- The default name for saved searches ("Untitled 2", etc.) was based on
collections rather than searches
- Once an initial search has been performed, the drop-downs and
checkboxes now update the results
- More consistent spacing in advanced search window
- (dev) Zotero.DB.getNextName() now takes a libraryID as its first
parameter instead of always using My Library; the old parameters are
deprecated but still work
- Each column in the middle pane can now have its own persistent
secondary sort column, configurable from a new submenu in the column
picker menu (top right of items list). The settings are stored in
extensions.zotero.secondarySort.[primaryField]. The submenu title
includes the current primary field (e.g., "Secondary Sort (Creator)"),
which is pretty weird, and I'm not sure I want to keep it, but it does
convey that the setting is specific to the selected column.
- The fallback sort fields (firstCreator, date, title, dateAdded) are
now configurable via the extensions.zotero.fallbackSort. Setting that
pref to an empty string avoids all fallback sorts, which
allows reverse-order clicking to set the order, as requested by
@aurimasv in #275.
- The previous behavior of sorting based on the exact Creator string
(rather than the actual creators) can now be restored with the
extensions.zotero.sortCreatorAsString pref. (It simply circumvents all
the newer code, so it's pretty safe.) This setting should result in
faster sorting in large libraries that have many items with the same
Creator string.
- Some of the lesser fields in the column picker menu are now in the
More Columns submenu (which is now alphabetical)
- The "Type" column is now the less-ambiguous "Item Type".
- This uses a different method to modify the column picker menu that is
simultaneously less and more hacky. (It no longer has to duplicate
Mozilla code in a custom XBL binding that wouldn't reflect future
upstream changes, and instead it bushwhacks its way through various
boxObject properties to get to the underlying menupopup.)
The previous version would keep only the last instance.
This version requires the array to contain only primitives of a single
data type, but I think that's OK for all of our uses. (This version
should also be faster.)
Cmd on OS X, Shift on Windows/Linux
How do I not get to close a ticket for this?
Unfortunately on Windows it doesn't seem possible to set the cursor
effect to arbitrary states (see note in libraryTreeView.js::
_setDropEffect() for the gory details), so this just uses the default
cursor there. On OS X and Linux the cursor reflects the requested
action.
Previously Cmd-Shift-Z worked for redo in text areas (Abstract, Extra)
but would toggle the Zotero pane in regular text fields. This change
allows Redo to work as expected.
The search bar gets focus when you first open the Zotero pane, though,
so we don't redo there, since it would be annoying if you couldn't close
the pane immediately with the same shortcut.
I think requiring everyone with Zotero installed to manually close the panel is
just too annoying. (Not allowing the auto-hide for a couple seconds after
opening to prevent accidental closes would be nice, but preventDefault() in the
panel's onpopuphiding() doesn't seem to work, nor does changing noautohide
after opening.)
Increase the delay to 2 seconds to try to show on top of the Firefox 28->29
upgrade panel on slower computers.
Using separate arrows from treetwisty.svg from Mozilla, since it doesn't seem
to be possible to use list-style-position in the tree
The reason we still have to do this at all is that coloring of the twisties in
the tree still gets messed up as you move around the tree. (This still happens
in Places and Thunderbird as well.)
Network errors (where the connection itself failed, rather than failed
HTTP requests) were being thrown directly in the stream listener, which
prevented the Zotero.Sync.Storage.Request -- and therefore the file sync
-- from ever completing.
"Your socket connection to the server was not read from or written to
within the timeout period."
I can't reproduce these reliably, so this is fairly untested, but it
seems to work. It backs off exponentially when the error occurs, and
halves the delay on successful requests.
Eventually all 50x file sync errors should also be retried
automatically, but that can't really happen in 4.0.
At least for me (in a VM), the text console has always been unusable on
Windows. Logging to the Browser Console is slower than dump() on OS X
and Linux, but on Windows it's much faster.
Sometime in the past SQL datetimes saved to the server could be
interpreted as multipart dates, causing the date portion to be stripped
from the visible field. I fixed this previously on the server, but a
full sync could result in conflicts for any such values that synced
down. Instead, overwrite the local version (keeping Date Modified the
same) if there are no other changes.
- Send modifier keys through to loadURI() when clicking Open Link in notes
- Open link in parent window from external note window
- Don't show both menus on right-click
Follow-up from #450
Follow up from #440
- Convert curly single quotes to straight quotes before inserting
- Add General and Supplemental Unicode punctuation ranges to getClass()
(so that fancy punctuation doesn't end up in words)
- Move single-quote test from getClass() to semanticSplitter(), and
consider it a letter only if in the middle of a word
- Add comments to semanticSplitter()
This might be ever-so-slightly slower, but it's neglible. (War and Peace
seems to now take ~1570ms instead of ~1500ms for me.)
* Close window on blur after completion on Mac (revert previous change)
* Don't close window when canceling
* Add Esc handler to cancel/close window
* Allow columns to be resized
* Fixes#445
* Fixes#444
Follow up from #440
- Convert curly single quotes to straight quotes before inserting
- Add General and Supplemental Unicode punctuation ranges to getClass()
(so that fancy punctuation doesn't end up in words)
- Move single-quote test from getClass() to semanticSplitter(), and
consider it a letter only if in the middle of a word
- Add comments to semanticSplitter()
This might be ever-so-slightly slower, but it's neglible. (War and Peace
seems to now take ~1570ms instead of ~1500ms for me.)
also deal with punctuation at the beginning of title
Also fix capitalizeTitle to work with quotation marks and Spanish beginning punctuation. Also adds ? and ! as punctuation after which it always capitalizes
switch sentence case conversion to a regex; I'm leaving capitalizeTitle as the substring routine, it's the same length and probably slightly more efficient.
document.location is null after the document is detached from its parent window (e.g. after we navigate to a different page in the same hidden browser)
If localstore.rdf has the tag selector persisted closed from before
zotero-persist, the 'state' attribute on the splitter that we now use
won't cause the tag selector to open.
This would previously have thrown an error. I'm not sure what these
documents would be, but it's a safe bet that they're not loaded in a
normal browser window.
Also fix weirdness trying to open collapsed tag selector after restart.
(The splitter's 'state' attribute has to be persisted, not the
'collapsed' state of the pane in question.)
If Zotero Standalone was opened before Firefox, closed, and opened
again, the user would see a message stating Zotero Standalone was open,
but the pane would not have closed. This was purely cosmetic.
- When opening the DB, always tell other Zotero instances to close it,
regardless of whether they are holding the lock.
- Don't let database re-open after it has been closed. This also fixes
some issues with connector switching.
This should make it much easier to debug startup errors, particularly in
Standalone.
This also adds a general mechanism to set Zotero initialization options via
command-line flags.
- Refresh Unfiled Items view when items are added
- Fix brief freeze ("too much recursion") adding an item to a search
where the new item doesn't appear. Now, select the library root
instead if a manually added item doesn't appear in the current view.
- Fix immediate closing of title field when adding an item to a
collection rather than the library root
nsISemanticUnitScanner doesn't seem to be able to deal with single
quotes, so protect those. (There might be other characters it doesn't
handle, but this is ancient code, so it stays as is for now.)
It might be possible to write non-UTF-8 data by passing another charset
to TextEncoder, but I haven't tried it.
Firefox 19+ only, and for now, at least, only if data is passed as
string rather than input stream
This isn't technically any slower than before, but if people were using
multiple computers, they might not have had their entire full-text index
on a single computer before full-text syncing.
"height: auto" (from another extension's fix for this) actually caused
the bottoms of tree rows to be cut off on Windows (and at least on one
Linux installation with Chinese characters).
- Close Zotero pane before database is closed prior to reload, instead
of waiting until reload is complete
- Show an error message if Zotero Standalone is not accessible when it
should be
FT sync is enabled by default for new users and configurable in the Sync
prefs.
Also disable downgrades once full-text sync is enabled, since otherwise
someone switching back and forth between old and new versions could miss
full-text content updates.
