There are currently two types of fulltext searching: an SQL-based word index and a file scanner. They each have their advantages and drawbacks.
The word index is very fast to search and is currently used for the find-as-you-type quicksearch. However, indexing files takes some time, so we should probably offer a preference to turn it off ("Index attachment content for quicksearch" or something). There's also an issue with Chinese characters (which are indexed by character rather than word, since there are no spaces to go by, so a search for a word with common characters could produce erroneous results). The quicksearch doesn't use a left-bound index (since that would probably upset German speakers searching for "musik" in "nachtmusik," though I don't know for sure how they think of words) but still seems pretty fast.
* Note: There will be a potentially long delay when you start Firefox with this revision as it builds a fulltext word index of your existing items. We obviously need a notification/option for this. *
The file scanner, used in the Attachment Content condition of the search dialog, offers phrase searching as well as regex support (both case-sensitive and not, and defaulting to multiline). It doesn't require an index, though it should probably be optimized to use the word index, if available, for narrowing the results when not in regex mode. (It does only scan files that pass all the other search conditions, which speeds it up considerably for multi-condition searches, and skips non-text files unless instructed otherwise, but it's still relatively slow.)
Both convert HTML to text before searching (with the exception of the binary file scanning mode).
There are some issues with which files get indexed and which don't that we can't do much about and that will probably confuse users immensely. Dan C. suggested some sort of indicator (say, a green dot) to show which files are indexed.
Also added (very ugly) charset detection (anybody want to figure out getCharsetFromString(str)?), a setTimeout() replacement in the XPCOM service, an arrayToHash() method, and a new header to timedtextarea.xml, since it's really not copyright CHNM (it's really just a few lines off from the toolkit timed-textbox binding--I tried to change it to extend timed-textbox and just ignore Return keypress events so that we didn't need to duplicate the Mozilla code, but timed-textbox's reliance on html:input instead of html:textarea made things rather difficult).
To do:
- Pref/buttons to disable/clear/rebuild fulltext index
- Hidden prefs to set maximum file size to index/scan
- Don't index words of fewer than 3 non-Asian characters
- MRU cache for saved searches
- Use word index if available to narrow search scope of fulltext scanner
- Cache attachment info methods
- Show content excerpt in search results (at least in advanced search window, when it exists)
- Notification window (a la scraping) to show when indexing
- Indicator of indexed status
- Context menu option to index
- Indicator that a file scanning search is in progress, if possible
- Find other ways to make it index the NYT front page in under 10 seconds
- Probably fix lots of bugs, which you will likely start telling me about...now.
Using Shift-Enter as the save keystroke within the Extra textbox so that people can use Enter to create multiple lines of text. Shift-Enter would normally be the newline command, but that's probably a convention that non-technical users of Zotero wouldn't know... Tab (and other triggers for blur()) also saves, and since Extra is the last field, tabbing away functions the same as hitting Enter does for other fields, so it's probably not that big of a deal.
Here's a shot at a single/double creator field toggle switch -- let me know what you think
A few issues:
- There's currently no comma between the last name and first name when in two-field mode -- I removed it to greatly simplify the code, hoping to be able to use the CSS :after pseudo-element, but that seems to not work with XUL -- I'll figure out a clean solution and add it back ( refs #288 )
- It's not very smart about switching between single-field mode and two-field mode, as it currently just keeps the last word (even if it's "Jr." or "III") as the last name and puts the rest in the first name field -- not a big deal, but it should at least be a bit smarter about this ( refs #289 )
- There are probably some other bugs
Addresses #260, Add auto-complete to search window
- New XPCOM autocomplete component for Zotero data -- can be used by setting the autocompletesearch attribute of a textbox to 'zotero' and passing a search scope with the autocompletesearchparam attribute. Additional parameters can be passed by appending them to the autocompletesearchparam value with a '/', e.g. 'tag/2732' (to exclude tags that show up in item 2732)
- Tag entry now uses more or less the same interface as metadata -- no more popup window -- note that tab isn't working properly yet, and there's no way to quickly enter multiple tags (though it's now considerably quicker than it was before)
- Autocomplete for tags, excluding any tags already set for the current item
- Standalone note windows now register with the Notifier (since tags needed item modification notifications to work properly), which will help with #282, "Notes opened in separate windows need item notification"
- Tags are now retrieved in alphabetical order
- Scholar.Item.replaceTag(oldTagID, newTag), with a single notify
- Scholar.getAncestorByTagName(elem, tagName) -- walk up the DOM tree from an element until an element with the specified tag name is found (also checks with 'xul:' prefix, for use in XBL), or false if not found -- probably shouldn't be used too widely, since it's doing string comparisons, but better than specifying, say, nine '.parentNode' properties, and makes for more resilient code
A few notes:
- Autocomplete in Minefield seems to self-destruct after using it in the same field a few times, taking down saving of the field with it -- this may or may not be my fault, but it makes Zotero more or less unusable in 3.0 at the moment. Sorry. (I use 3.0 myself for development, so I'll work on it.)
- This would have been much, much easier if having an autocomplete textbox (which uses an XBL-generated popup for the suggestions) within a popup (as it is in the independent note edit panes) didn't introduce all sorts of crazy bugs that had to be defeated with annoying hackery -- one side effect of this is that at the moment you can't close the tags popup with the Escape key
- Independent note windows now need to pull in itemPane.js to function properly, which is a bit messy and not ideal, but less messy and more ideal than duplicating all the dual-state editor and tabindex logic would be
- Hitting tab in a tag field not only doesn't work but also breaks things until the next window refresh.
- There are undoubtedly other bugs.
Closes#208, tags overflow pane
This is kind of cheating, but it works, and as long as we can live with a 170px min-height, I'm not going to take the time to figure out how (and whether) this can be fixed.
This needs to be replaced with a 24-bit PNG with a transparent (rather than white) background -- Dan, you can either send that to me or go ahead and replace it yourself (I'd do it but I don't know the kerning, etc.)
Using a -1px bottom margin to get infinite height (a la Fitts's Law) when maximized, though I haven't tested on Windows or Linux -- let me know if clicking the screen edge doesn't work for you
- Added support for tabbing between fields, which unfortunately we don't get for free from tabindex because of what we're doing with labels and textboxes. (The textbox being tabbed away from is deleted before the blur() completes, so it doesn't know where it's supposed to go next.) Basically replicated the built-in functionality for all the text fields -- can add on special tricks for creator adding/removing later.
- Increased spacing between rows slightly and adjusted margins to get rid of shifting when switching between label and textbox -- this also fixes the annoying clicking-off-a-textbox-to-the-lower-portion-of-a-label-below-it-doesn't-select-the-label problem.
Conditions can now offer drop-down menus rather than freeform text fields -- implemented for collections/saved searches and item types
Special handling to combine collections and saved searches into a single "Collection" menu (can we get away with calling them "Smart Collections"?) -- internally, values are stored as C1234 or S5678 in the interface and converted to/from regular collectionID and savedSearchID conditions for search.js
Use localized strings for conditions (tries searchConditions.* first, then itemFields.*)
Alphabetize condition list
Operator menu now fixed length for all conditions
Fixed a number of creator-editing-related issues (mostly interface-side, with a little help from the data layer):
- New row no longer disappears when clicking "+" and then clicking the existing or new creator (removed onselect="ScholarItemPane.loadPane(this.selectedIndex)" on the <deck> (from r371, with changelog "Individual tabs don't load their content unless selected"), which doesn't seem to be necessary as far as I can tell)
- New row no longer disappears when changing creator before editing names (setting the creator type now triggers an Item.save() with an otherwise blank creator (now allowed by the data layer), which isn't entirely ideal but is probably OK for now)
- Clicking the minus button on an unsaved row (i.e. one just created with the plus button) no longer throws an error (new method Item.hasCreatorAt(pos), and ScholarItemPane.removeCreator() just deletes the label directly, since it doesn't get a notify() event to reload the pane)
- The plus button is disabled on unsaved rows, since allowing the user to create multiple unsaved rows and then edit one in the middle is problematic (and the other alternatives have their own problems); the minus button is also disabled on the default row that shows when there are no creators
- Creator type is no longer reset when editing a name field
- Name field is no longer erased when clicking directly on creator type popup and changing creator type without blurring textbox
- Comma is appended to last name when switching from <textbox> to <label> without saving changes (before it was just appended to labels when the pane was loaded)
If you have attachments to the old terminology, feel free to file a complaint.
Changed interface code too, since David is gone (or at the very least has more important things to do with his remaining time)
Addresses #169, add OpenURL interface hooks
Addresses #170, Put "Link" option before "Import" in drop-down menu
Fixes some advanced search flaws (there are still bugs)
Closes#152, Saved Searches (interface layer)
For now, advanced search IS a saved search.
There are still bugs. The 'search' icon is ugly. I wanted to get it out there, however.
Closes#128, Display files as children of item in items list
Fixes#132, On Full-screen mode, browser content disappears but Scholar content does not stretch.
Fixes#133, When Scholar on top, pane automatically resizes depending on item info height.
Fixes#115, Windows should open with Scholar Pane closed.
Fixes#105, search box loses focus after search starts.
Retooled the interface a bit, and removed the top toolbar. The close and fullscreen buttons are located to the right of the items toolbar. The item pane cannot be collapsed.
This custom credits panel gives us more flexibility. Also, there is an About link which is certainly easier to find than right-clicking on the extension in the Add-ons window.
- changes scrapers table to translators table; all import/export/web translators now belong in this table
- adds Scholar.Translate to handle translation issues. eventually, Scholar.Ingester.Document will become part of this interface
- adds Scholar_File_Interface (in fileInterface.js) to handle UI for export and eventually import. (David, when you have time, please connect Scholar_File_Interface.exportFile to a button.)
- adds an export translator for MODS. all of our metadata, but not our hierarchy (projects, etc.) translates directly and unambiguously into valid MODS. eventually, we can use RDF or another format to handle hierarchy.
- adds utilities.getVersion() and utilities.inArray() for simplified scraper coding
- fixes minor interface issues with the nifty chrome scraping status window
- This required moving the icon to the title field so that the indent would work out right. The type column (which for new installs will be hidden) displays the type in text.
- I expect several small bugs in regard to this.
closes#26, notes list in notes pane
closes#79, add icons for new object types
fixes#71, Metadata pane should be refreshed on a notify() event for the selected item
Moved the capture icon into the URL bar (invisible until you visit a scrapable page. Currently just displays a Book, but will change to the correct item types in the future?)
