Apologies for the massive (and, due to data_access.js splitting, difficult-to-follow) commit. Please note that external code that accesses the data layer may need to be tweaked for compatibility. Here's a comprehensive-as-possible changelog:
- Added server sync functionality (incomplete)
- Overhaul of data layer
- Split data_access.js into separate files (item.js, items.js, creator.js, etc.)
- Made creators and collections first-class objects, similar to items
- Constructors now take id as first parameter, e.g. new Zotero.Item(1234, 'book'), to allow explicit id setting and id changing
- Made various data layer operations (including attachment fields) require a save() rather than making direct DB changes
- Better handling of unsaved objects
- Item.setCreator() now takes creator objects instead of creator ids, and Item.save() will auto-save unsaved creators
- clone() now works on unsaved objects
- Newly created object instances are now disabled after save() to force refetch of globally accessible instance using Zotero.(Items|Creators|etc.).get()
- Added secondary lookup key to data objects
- Deprecated getID() and getItemType() methods in favor of .id and .itemTypeID properties
- toArray() deprecated in favor of serialize(), which has a somewhat modified format
- Added support for multiple creators with identical data -- currently unimplemented in interface and most of data layer
- Added Item.diff() for comparing item metadata
- Database changes
- Added SQLite triggers to enforce foreign key constraints
- Added Zotero.DB.transactionVacuum flag to run a VACUUM after a transaction
- Added Zotero.DB.transactionDate, .transactionDateTime, and transactionTimestamp to retrieve consistent timestamps for entire transaction
- Properly store 64-bit integers
- Set PRAGMA locking_mode=EXCLUSIVE on database
- Set SQLite page size to 4096 on new databases
- Set SQLite page cache to 8MB
- Do some database cleanup and integrity checking on migration from 1.0 branch
- Removed IF NOT EXISTS from userdata.sql CREATE statements -- userdata.sql is now processed only on DB initialization
- Removed itemNoteTitles table and moved titles into itemNotes
- Abstracted metadata edit box and note box into flexible XBL bindings with various modes, including read-only states
- Massive speed-up of item tree view
- Several fixes from 1.0 branch for Fx3 compatibility
- Added Notifier observer to log delete events for syncing
- Zotero.Utilities changes
- New methods getSQLDataType() and md5()
- Removed onError from Zotero.Utilities.HTTP.doGet()
- Don't display more than 1024 characters in doPost() debug output
- Don't display passwords in doPost() debug output
- Added Zotero.Notifier.untrigger() -- currently unused
- Added Zotero.reloadDataObjects() to reset all in-memory objects
- Added |chars| parameter to Zotero.randomString(len, chars)
- Added Zotero.Date.getUnixTimestamp() and Date.toUnixTimestamp(JSDate)
- Adjusted zotero-service.js to simplify file inclusion
Various things (such as tags) are temporarily broken.
Closes#285, Hitting Esc should close tags popup
Fixes various UI issues and problems adding tags on Minefield
Still getting autocomplete self-destruction
This is a bit uglier, and the large markers are kinda annoying once you know the drop-down is there, but perhaps this will stop the forum posts. Please let them stop.
If anyone's out there, now would be the time to test this.
Missing icons for the new types -- I could try to get some of those now, or we could just launch. Maybe we'll just launch. But really, might want to test this. Just sayin'.
Use automaticSnapshots pref (which defaults to on and is changeable in the prefs window) rather than downloadAssociatedFiles (which defaults to off and is only settable through about:config at the moment) for now in translate.js
downloadAssociatedFiles should eventually be used for PDFs and other large files, whereas automaticSnapshots will be for HTML and the like -- in the meantime, I think it's OK for scrapers to just follow the visible pref for both, since otherwise they'd be totally confused when the NIFP button took a snapshot and the scrapers didn't
Simon, if there's any problem I'm not aware of with switching this for now (other than people getting some large PDFs on JStor), let me know.
closes#165, verify import/export can carry all data for all fields and item types
closes#168, make sure MODS import works with files from external sources
Attach Snapshot of Current Page
Attach Link to Current Page
I also removed "New Item" from the context menu, as it was the only option in the menu that wasn't specific to the selected item, it made the whole menu a lot more confusing, and it doesn't take all that much longer to click the (+) button and choose Book (assuming that's even the item you want).
Addresses #155, Localize strings
- Added "Search subfolders"
- Localized strings in search dialog
Known issue:
#342, "Search subfolders" checkbox should be greyed out until applicable
Including in the DB, which it turns out isn't really all that bad (thanks, among other things, to SQLite's ability to DROP tables within transactions without autocommitting (which MySQL can't do))
At least some scrapers (NYT and WashPo, for sure) should be updated to follow this pref
Addresses #327, Scrapers should either take snapshots or use URL field
Changed creator type <label> to <toolbarbutton> -- hopefully won't have any ill effects with other platforms' default themes (I haven't checked)
Moved some metadata pane CSS from overlay.css to zotero.css
Currently uses an annoying popup window, since the code used in the regular metadata pane has too much baggage to use here -- it really should be abstracted into an XBL binding, but that won't be an easy feat
Also fixes SQL error opening popup notes and JS strict warnings when saving them
- Currently requires user to enter dates in SQL format if they want to change the access date, but at least it doesn't mangle the dates anymore
- Uses new function ScholardammitZotero.Date.dateToSQL(Date date [, Boolean toUTC])
- Utilities.lpad() now forces _string_ to a string so that .length exists
- Unrelated: Item.save() now returns false if the item didn't change
closes#313, Blacklist known ad sites from scraper detection
closes#306, some New York Times ads prevent page from being recognized
closes#308, attachment import bug
currently, the ad site blacklist is located at the top of ingester/browser.js. at some point, we may want to switch this to a database table.
addresses #327, Scrapers should either take snapshots or use URL field
closes#309, Integration server prevents Zotero from loading in multiple instances of Firefox
In other words, show both "Shakespeare" and "Shakespeare, William" in the drop-down, and if the latter is chosen, save both fields
One issue is that since the autocomplete is by default limited to the width of the textbox, longer entries get truncated (though you can see them with a mouseover), and that may not be easy to fix.
Changed "Scholar" to "Zotero", everywhere
Apologies to anyone with working copy changes, but there are probably the fewer at this moment than there will be again.
Hopefully this won't break anything, though existing prefs will be lost. I avoided scholar.google.com--if you know any other legitimate "scholar"s in the code, be sure to fix them once I'm done here.
This is a multi-commit change--there's at least one more coming. *Do not update to this version! It won't work!*
Closes#304, change references to "website" to "web page"
More changes as per discussions with Dan:
- Linked URLs have been given a second chance at life, though they still shouldn't be used for (most, if any) scrapers (which should use snapshots or the URL field instead)
- Renamed the "website" item type to "webpage"
- Removed "web page" from the New Item menu
- Added Save Link To Current Page toolbar button
- Added toolbar separator between New Item buttons and link/attachment/note to differentiate
- Added limited metadata (URL and accessDate) for attachments
- URL for attachments now stored in itemData (itemAttachments.originalPath is no longer used, but I'm probably not gonna worry about it and just wait for SQLite to support dropping columns with ALTER TABLE) -- getURL() removed in favor of getField('url')
- Snapshots now say "View Snapshot"
- Added Show File button to file attachments to show in filesystem
- Added timed note field to attachments for single notes and adjusted Item.updateNote(), etc. to work with attachments
- Fixed bug with manually bound params in fulltext indexer and Item.save() (execute() vs. executeStep()) -- any recently added items probably aren't in the fulltext index because of this
Known bugs/issues:
- Attachment metadata and notes probably aren't properly imported/exported now (and accessDate definitely isn't)
- Scrapers don't save metadata properly
- Attachment title should be editable
- File attachments could probably use some more metadata (#275, more or less, though they won't be getting tabs)
And fixed a bug that caused the text of some search conditions to not display when editing a previously saved search
Also added autocomplete to more fields in metadata pane:
- creator
- publisher
- place
- publicationTitle
- journalAbbreviation
- seriesTitle
- seriesText
It should also do the type and medium fields, but they need to be separated out first--e.g. artworkType, interviewMedium--since they're describing different data
Overhaul of the item drag and drop functionality, allowing dragging notes and attachments into and out of other items and addressing all the related issues that arise when that's possible.
Should also be generally smarter about deciding what can be dragged and dropped where and in what modes.
Let me know if something doesn't work as you expect.
Icons from Ken
Of course, I realized just now that this is really quite similar to the stop sign, and anyone who puts this to the left of their URL bar and doesn't have a Home icon in their toolbar will get the two next to each other... Looks good otherwise, though.
Changes as per my discussions with Dan:
- Separated snapshot functionality into two individual buttons, Create New Item From Current Page and Take Snapshot of Current page
- Updated schema to support primary, secondary and hidden item types (and future user customizations)
- Reorganized New Item menu, moving secondary items into sub-menu
- Removed ability to create link attachments, since it never really made much sense -- will simply use the webpage item type instead. Underlying functionality still exists for the time being, as people have existing links in their libraries--I think we're gonna have to just warn beta testers and delete them in a transition step, as converting nested links really wouldn't be worth the effort.
- Moved file link/add functions into new item menu and removed attachment drop-down
- Large, prominent View and Locate buttons in edit pane for going to an associated URL and looking up in OpenURL, respectively -- buttons gray out as appropriate
- New Item from Page stores the URL and access date (Item.save() checks for the string "CURRENT_TIMESTAMP" for accessDate and doesn't bind it as a string)
- "Website" to "Web Page" (do we prefer "Webpage"? they both look a bit funky in uppercase)
More coming.
Bugs/Known Issues:
- Since snapshots from the toolbar are now top-level in the current collection, there needs to be a way to drag them into items
- The camera icon for adding snapshots, despite being a famfamfam icon, really doesn't read too well (or perhaps just clashes with the rest of our icons). Anybody have a better one? (It also may be able to just be lightened up a bit.)
- Trying the large View/Locate buttons after discussions with Dan, but this approach may not work -- 1) a large View button for the URL makes a lot less sense when you have a parent item with a child snapshot, since people will end up clicking it all the time when they really want to view the snapshot, and 2) the Locate button is awfully big for something that only applies to certain types of items, may not get used very often when it does, and probably won't work when it is
- The access date is stored in UTC and displayed with toLocaleString() like Date Added and Date Modified, but, unlike those two, it's also user-editable. This is clearly a problem. Probably need to parse to Date on blur() with strToDate() and insert as UTC, discarding anything left over.
- Item type itself is still "website" -- should probably change that while we still can
Closes#253, OpenURL arrow should provide visual feedback on mouseover and/or look more button-like
Addresses #304, change references to "website" to "web page"
Addresses #207, openurl arrow functionality
Differentiates between single and double fields for the search, but there's a problem in the current implementation in that only one field is editable at once, so displaying two-field names in a drop-down is a little problematic. While I could display the full names, comma-delimited, and get the discrete parts (which is what Scholar.Utilities.AutoComplete.getResultComment(), included in this commit, is for--the creatorID for the row would be hidden in the autocomplete drop-down comment field), it's a bit unclear what should happen when a user selects a comma-separated name from the drop-down of one of the fields. One option would be to have a row for the last name (in case that's all they want to complete) and other rows for "last, first" matches, and selecting one of the two-part names would replace whatever's in the opposite name field with the appropriate text (and save it to the DB, I'm afraid, unless I change how the creator fields work), keeping the focus in the current textbox for easy tabbing. Not great, but it might work.
Other ideas?
Adds tab handling to tags interface, which also addresses beta tester requests for a quicker way to enter multiple tags
This is getting pretty messy, but it's still probably better than repeating all the itemPane.js in tagsbox.xml.
One known issue is that, since it resorts the list of tags after a change, if you change an existing tag to a name that alters its current position and then tab away, you don't necessarily end up in the field you expect.
- Switched to manually repeated bound parameters in indexWords()
- Switched to the innerHTML regex used elsewhere instead of a more proper but nevertheless misguided DOM traverser to split elements in indexDocument
This may invalidate the fulltext progress indicator ticket
Still need #292, "Delete From Library" context menu option in collections
This may or may not help people understand that deleting from a collection doesn't by design delete from the library, but, regardless, this is the same behavior as iTunes.
Example usage:
var windowWatcher = Components.classes["@mozilla.org/embedcomp/window-watcher;1"].
getService(Components.interfaces.nsIWindowWatcher);
var progress = new Scholar.ProgressWindow(windowWatcher.activeWindow);
progress.changeHeadline('Indexing item...');
progress.addLines(['All About Foo'], ['chrome://scholar/skin/treeitem-book.png']);
progress.addDescription('Bar bar bar bar bar');
progress.show();
progress.fade();
There are currently two types of fulltext searching: an SQL-based word index and a file scanner. They each have their advantages and drawbacks.
The word index is very fast to search and is currently used for the find-as-you-type quicksearch. However, indexing files takes some time, so we should probably offer a preference to turn it off ("Index attachment content for quicksearch" or something). There's also an issue with Chinese characters (which are indexed by character rather than word, since there are no spaces to go by, so a search for a word with common characters could produce erroneous results). The quicksearch doesn't use a left-bound index (since that would probably upset German speakers searching for "musik" in "nachtmusik," though I don't know for sure how they think of words) but still seems pretty fast.
* Note: There will be a potentially long delay when you start Firefox with this revision as it builds a fulltext word index of your existing items. We obviously need a notification/option for this. *
The file scanner, used in the Attachment Content condition of the search dialog, offers phrase searching as well as regex support (both case-sensitive and not, and defaulting to multiline). It doesn't require an index, though it should probably be optimized to use the word index, if available, for narrowing the results when not in regex mode. (It does only scan files that pass all the other search conditions, which speeds it up considerably for multi-condition searches, and skips non-text files unless instructed otherwise, but it's still relatively slow.)
Both convert HTML to text before searching (with the exception of the binary file scanning mode).
There are some issues with which files get indexed and which don't that we can't do much about and that will probably confuse users immensely. Dan C. suggested some sort of indicator (say, a green dot) to show which files are indexed.
Also added (very ugly) charset detection (anybody want to figure out getCharsetFromString(str)?), a setTimeout() replacement in the XPCOM service, an arrayToHash() method, and a new header to timedtextarea.xml, since it's really not copyright CHNM (it's really just a few lines off from the toolkit timed-textbox binding--I tried to change it to extend timed-textbox and just ignore Return keypress events so that we didn't need to duplicate the Mozilla code, but timed-textbox's reliance on html:input instead of html:textarea made things rather difficult).
To do:
- Pref/buttons to disable/clear/rebuild fulltext index
- Hidden prefs to set maximum file size to index/scan
- Don't index words of fewer than 3 non-Asian characters
- MRU cache for saved searches
- Use word index if available to narrow search scope of fulltext scanner
- Cache attachment info methods
- Show content excerpt in search results (at least in advanced search window, when it exists)
- Notification window (a la scraping) to show when indexing
- Indicator of indexed status
- Context menu option to index
- Indicator that a file scanning search is in progress, if possible
- Find other ways to make it index the NYT front page in under 10 seconds
- Probably fix lots of bugs, which you will likely start telling me about...now.
Note that there's no code for user types and fields yet -- just the schema (actually there's a tiny bit of code in the item type manager, since we'll probably use some of the same methods for managing user types, but not much)
Templates for primary item types are currently only used by the item type manager to make creating new types easier and to prevent the removal of fields from an item type that are associated with its template item type -- the fields are all still recorded in itemTypeFields, since they might have different orders or default visibility settings from their templates
This lets us add tables to user.sql without writing migration steps for them yet still have the ability to change existing user tables and migrate data if necessary.
Also added _getDropCommands() to do a regex on the SQL file and create the DROP TABLE|INDEX steps necessary to use the DB_REBUILD flag without DROP commands in the SQL file itself, before I realized that it probably made the most sense to just delete the SQL file and storage directory. Changed _initializeSchema() to do that instead. Leaving _getDropCommands() in, in case there's ever a need for it.
Using Shift-Enter as the save keystroke within the Extra textbox so that people can use Enter to create multiple lines of text. Shift-Enter would normally be the newline command, but that's probably a convention that non-technical users of Zotero wouldn't know... Tab (and other triggers for blur()) also saves, and since Extra is the last field, tabbing away functions the same as hitting Enter does for other fields, so it's probably not that big of a deal.
This should make development much easier, as we can, for example, add 80 item types without having to write transition steps
Pretty sure this won't delete anyone's data. Might want to test that theory, though.
- closes#217, ability to exclude notes/attachments from select items window
- closes#244, ability to quick search from select items window
- fixes a bug with footnotes in Word integration
- fixes a bug in InnoPAC translator where items would sometimes appear twice
Here's a shot at a single/double creator field toggle switch -- let me know what you think
A few issues:
- There's currently no comma between the last name and first name when in two-field mode -- I removed it to greatly simplify the code, hoping to be able to use the CSS :after pseudo-element, but that seems to not work with XUL -- I'll figure out a clean solution and add it back ( refs #288 )
- It's not very smart about switching between single-field mode and two-field mode, as it currently just keeps the last word (even if it's "Jr." or "III") as the last name and puts the rest in the first name field -- not a big deal, but it should at least be a bit smarter about this ( refs #289 )
- There are probably some other bugs
New logic in itemTreeView notify() target:
- Items are only selected on add in the active window -- this fixes a fairly major flaw in the previous system that would cause new items to be selected in all open windows
- If a quicksearch is open in the active window and a new item is added, clear the search
- If quicksearch and active window and item modify, rerun search
- If quicksearch and not active window, rerun search
- If not quicksearch and not active window, update list but retain previous selection
Addresses #260, Add auto-complete to search window
- New XPCOM autocomplete component for Zotero data -- can be used by setting the autocompletesearch attribute of a textbox to 'zotero' and passing a search scope with the autocompletesearchparam attribute. Additional parameters can be passed by appending them to the autocompletesearchparam value with a '/', e.g. 'tag/2732' (to exclude tags that show up in item 2732)
- Tag entry now uses more or less the same interface as metadata -- no more popup window -- note that tab isn't working properly yet, and there's no way to quickly enter multiple tags (though it's now considerably quicker than it was before)
- Autocomplete for tags, excluding any tags already set for the current item
- Standalone note windows now register with the Notifier (since tags needed item modification notifications to work properly), which will help with #282, "Notes opened in separate windows need item notification"
- Tags are now retrieved in alphabetical order
- Scholar.Item.replaceTag(oldTagID, newTag), with a single notify
- Scholar.getAncestorByTagName(elem, tagName) -- walk up the DOM tree from an element until an element with the specified tag name is found (also checks with 'xul:' prefix, for use in XBL), or false if not found -- probably shouldn't be used too widely, since it's doing string comparisons, but better than specifying, say, nine '.parentNode' properties, and makes for more resilient code
A few notes:
- Autocomplete in Minefield seems to self-destruct after using it in the same field a few times, taking down saving of the field with it -- this may or may not be my fault, but it makes Zotero more or less unusable in 3.0 at the moment. Sorry. (I use 3.0 myself for development, so I'll work on it.)
- This would have been much, much easier if having an autocomplete textbox (which uses an XBL-generated popup for the suggestions) within a popup (as it is in the independent note edit panes) didn't introduce all sorts of crazy bugs that had to be defeated with annoying hackery -- one side effect of this is that at the moment you can't close the tags popup with the Escape key
- Independent note windows now need to pull in itemPane.js to function properly, which is a bit messy and not ideal, but less messy and more ideal than duplicating all the dual-state editor and tabindex logic would be
- Hitting tab in a tag field not only doesn't work but also breaks things until the next window refresh.
- There are undoubtedly other bugs.
- import translators no longer fail when trying to import an item with no name
- the T2/BT field becomes the publication title when no JO/JF field is available (fixes newspaper issues)
- Y2 is now treated as part of the date if and only if it is improperly formatted (seriously, why can't Thomson get their own specs straight?)
- work around EndNote's strange behavior of putting article titles into notes for no apparent reason
- RIS export gives dates as per specification
- fixed a bug that could have (potentially) caused problems formatting "January"
- allow translators to access strToDate function
MODS uses the encoding as specified in the <?xml tag, or else UTF-8
RIS uses IBM850, since the spec says "IBM Extended Character Set" and it's the only code page Mozilla supports. (should I do this? or just use unicode?)
MARC uses UTF-8, since I don't think there's any way to get full MARC-8 support, and UTF-8 is now the preferred encoding anyway
- improved bibliography (especially Chicago Manual of Style)
- improved error handling for import/export/bibliography
- bibliographic export now ignores notes and standalone attachment (before, they made export silently fail). an error appears if you try to generate a bibliography from only notes or standalone attachments.
positions "saving item" window in a slightly better place on Windows
the UMich bug was actually bigger than I though. as it turns out, the HiddenDOMWindow in Windows is not a chrome window, so i had to modify createHiddenBrowser() to attach the hidden browser object to an existing browser window. i don't believe this should have any adverse effects for snapshots, etc., but Dan, correct me if i'm wrong. it would be nice to be able to create a real chrome instance instead of a XUL element, but all of my attempts at doing so have failed.
i've fixed the Amazon.com bug (i think) and made the translator show a "Could Not Save Item" prompt rather than show an empty list, but if you see any other pages where this happens, let me know
- make EBSCO scraper work better through a proxy
- shorten Accession Number -> Accession No, Journal Abbreviation -> Journal Abbr, Publication Title -> Publication. it does look a bit stranger, but it also makes the interface more functional (especially for those of us without giant widescreen LCDs ;-)
(Apparently 'columnName NONE' in SQLite is functionally equivalent to 'columnName ILUVTIDDLYWINKS' in setting column affinity--as in, they both use the default of NUMERIC--which frankly I feel the docs could've been slightly more clear on, since they scatter the word NONE in capital letters all about the page with the other affinity keywords...)
modifies scrapers to use dates in the format that comes out of the page, rather than converting to SQL
adds Scholar.Date.formatDate() to provide a pretty representation of dates
- Scholar.strToDate() accepts a string date and returns an object containing year, month, day, and part
- capture access date whenever URL is captured
- updated Zotero.dot to use new namespaces
Closes#208, tags overflow pane
This is kind of cheating, but it works, and as long as we can live with a 170px min-height, I'm not going to take the time to figure out how (and whether) this can be fixed.
- About panel now gets version number automatically
- Change version from 1.0a1 to 1.0b1
* Important: If you're on an SVN install, you need to rename the scholar@chnm.gmu.edu text file in your profile extension directory to zotero@chnm.gmu.edu *
XPI installs will (I think) update automatically, since I kept an entry in updates.rdf with the old GUID
This needs to be replaced with a 24-bit PNG with a transparent (rather than white) background -- Dan, you can either send that to me or go ahead and replace it yourself (I'd do it but I don't know the kerning, etc.)
Using a -1px bottom margin to get infinite height (a la Fitts's Law) when maximized, though I haven't tested on Windows or Linux -- let me know if clicking the screen edge doesn't work for you
Proper labels for saved searches, hiding rather than disabling irrelevant menu options, added "New Saved Search..." to menu, made file and bibliography export work with saved searches, added "Export Library..." option to Library drop-down
addresses #214, add footnote support to word integration
- the third icon on the Zotero Word toolbar is now reserved for "Document Options," which, for now, means on the selection of styles
- the Document Options window will, for now, appear the first time you create a citation. the default style probably belongs in the Scholar preferences window.
- you can now generate citations in both footnote and in-text citation formats. you can't yet switch between them on the fly, but that should be coming soon...
- Ibid is not yet implemented. again, coming soon.
- select() new collections and saved searches on creation
- Show all collections in search dialog Collection condition drop-down, not just top-level ones -- eventually there should probably be some sort of level indication to show hierarchy
- Don't allow a search to define itself as a savedSearchID condition. Right.
- Fixed a couple bugs in Collection condition creation that would've made such searches often not work
- Added saved search 'add' support to collectionTreeView notify(), and send the right trigger from Scholar.Search
Implemented isBefore and isAfter operators and added to date conditions -- currently have to type dates in SQL format, but we'll have a chooser or something by Beta 2 (this should probably be a known issue)
Added isLessThan and isGreaterThan to 'pages', 'section', 'accessionNumber', 'seriesNumber', and 'issue' fields, though somebody should probably check me on that list
Removed JS strict warning on empty search results
- Added support for tabbing between fields, which unfortunately we don't get for free from tabindex because of what we're doing with labels and textboxes. (The textbox being tabbed away from is deleted before the blur() completes, so it doesn't know where it's supposed to go next.) Basically replicated the built-in functionality for all the text fields -- can add on special tricks for creator adding/removing later.
- Increased spacing between rows slightly and adjusted margins to get rid of shifting when switching between label and textbox -- this also fixes the annoying clicking-off-a-textbox-to-the-lower-portion-of-a-label-below-it-doesn't-select-the-label problem.
not yet implemented:
- formatted in-text citations, rather than placeholders
- footnotes
- selection of citation style (for now, only APA is available)
- support for non-ASCII characters
- exclusion of notes from select items window
- Windows support (although it shouldn't be difficult)
- probably much more...
"Notes" column changed to "#" and combined with the attachment count. "#" isn't exactly immediately self-explanatory, but "Notes/attachments" would be way too long, and once you realize what it's for (i.e. the first time you create a child item), you really don't want the column taking up any more room than that... If people have a better suggestion, do share.
The notes/attachment count wasn't going down on an attachment deletion because it's not actually a notes/attachment count at all--it's a notes count, and new attachments were incorrectly incrementing it.
This technically either fixes or invalidates #202 ("attachments categorized as notes"), depending on how you look at it, but I'll change that to reflect the desired goal of having a combined notes/attachments count.
Refs #202
The problem was twofold: 1) onselect="ScholarItemPane.loadPane(this.selectedIndex)" is (for some reason) triggered when clicking off of the "+" creator add button onto the rest of the deck pane (and not just when switching tabs) and 2) the selectedIndex passed into loadPane() from the deck during the non-tab triggers is (for some reason) an empty string rather than the index of the selected tab, which is why the creator problem only used to happen the first time you clicked away from a "+" (before the empty string was stored as a key in the _loaded array). (In short, onselect for decks is pretty broken.)
Now only calling loadPane if (this.selectedIndex!=='') -- can't just test for typeof=='number' because this.selectedIndex on the deck is always a string
refs #179 and #190
- Added 'fulltext' condition as shortcut to add an operator/value against all string-based conditions -- can be used for quicksearch within a view if collectionID/savedSearchID is added as a required condition along with it
Note that the 'fulltext' condition isn't stored internally and addCondition() doesn't return a searchConditionID for it, so it's not really meant to be used for saved searches.
Example:
var search = new Scholar.Search();
search.addCondition('collectionID', 'is', 6856, true);
search.addCondition('fulltext', 'contains', 'wellman');
Scholar.debug(search.search());
- Fixed isNot/doesNotContain for items table fields and collectionID
Conditions can now offer drop-down menus rather than freeform text fields -- implemented for collections/saved searches and item types
Special handling to combine collections and saved searches into a single "Collection" menu (can we get away with calling them "Smart Collections"?) -- internally, values are stored as C1234 or S5678 in the interface and converted to/from regular collectionID and savedSearchID conditions for search.js
Use localized strings for conditions (tries searchConditions.* first, then itemFields.*)
Alphabetize condition list
Operator menu now fixed length for all conditions
- Support for multiple id collection/search remove/modify notifications
- New method Scholar.Searches.get(id) to get 'id' and 'name' for a particular savedSearchID
Scholar.Attachments.importSnapshotFromFile(file, url, title, mimeType, charset, sourceItemID)
file is primary file within directory -- file.parent is copied into the storage directory
url is the original URL associated with the snapshot
Attachments.importFromURL() now first does a HEAD request to get the MIME type and passes that through Scholar.MIME.hasInternalHandler() (now abstracted from Scholar.File, along with the other MIME functions) -- if it can handle the MIME type, it uses a hidden browser; otherwise, it use a remote web page persist to save the file directly
closes#160, cache regular expressions
closes#188, rewrite MARC handling functions
MARC-based translators should now produce item types besides "book." right now, artwork, film, and manuscript are available. MARC also has codes for various types of audio (speech, music, etc.) and maps.
the EBSCO translator does not yet produce attachments. i sent them an email because their RIS export is invalid (the URLs come after the "end of record" field) and i'm waiting to see if they'll fix it before i try to fix it myself.
the EBSCO translator is unfortunately a bit slow, because it has to make 5 requests in order to get RIS export. the alternative (scraping individual item pages) would be even slower.
regular expression caching can be turned off by disabling extensions.scholar.cacheTranslatorData in about:config. if you leave it on, you'll have to restart Firefox after updating translators.
On Simon's suggestion, appending .html to all text/html files--I can't reproduce the problem, and I'm not sure what's causing the ExternalHelperAppService to get confused, but this should at least avoid the issue...
import/export of file data should work for all file types _except_ snapshots (in this situation, export is working, but import is not yet complete; see #193)
also, fixes a potential security issue that could have allowed malicious web translators to post local data to remote sites (although, given we maintain the central repository and there's no easy way to install a translator, the risk would have been minimal to begin with).
closes#3, Overflow metadata dumps into "extra" field
add "extra" data where such data is useful and conveniently accessible (not available for XML-based export or MARC formats yet)
add links to permanent URLs
download associated files from full text sources (if extensions.scholar.downloadAssociatedFiles preference is enabled)
fix WorldCat translator
improve InnoPAC translator (it now works on Georgetown search results pages, albeit slowly, because it must first realize the catalog is misconfigured)
tag items from SIRSI and WorldCat
return to putting the full lengths of books into "pages," because some citation styles require it
fix COinS (broken a few revisions ago)
closes#186, stop translators from hanging
when a document loads inside a frameset, we now check whether we can scrape each individual frame.
all functions involving tabs have been vastly simplified, because in the process of figuring this out, i discovered Firefox 2's new tab events.
if a translator throws an exception inside loadDocument(), doGet(), doPost(), or processDocuments(), a translate error message will appear, and the translator will not hang
(Also fixes bug from r379 where changing creator type on a blank unsaved creator would actually insert "(first)" and "(last)")
Localized '(first)' and '(last)'
Refs #179, adding a new creator then clicking an existing creator makes the creator field disappear
closes#180, make all contextual menu export/create bibliography options work right
also:
- add Chicago Note style output
- unregister RDF data sources from cache after import
Fixed a number of creator-editing-related issues (mostly interface-side, with a little help from the data layer):
- New row no longer disappears when clicking "+" and then clicking the existing or new creator (removed onselect="ScholarItemPane.loadPane(this.selectedIndex)" on the <deck> (from r371, with changelog "Individual tabs don't load their content unless selected"), which doesn't seem to be necessary as far as I can tell)
- New row no longer disappears when changing creator before editing names (setting the creator type now triggers an Item.save() with an otherwise blank creator (now allowed by the data layer), which isn't entirely ideal but is probably OK for now)
- Clicking the minus button on an unsaved row (i.e. one just created with the plus button) no longer throws an error (new method Item.hasCreatorAt(pos), and ScholarItemPane.removeCreator() just deletes the label directly, since it doesn't get a notify() event to reload the pane)
- The plus button is disabled on unsaved rows, since allowing the user to create multiple unsaved rows and then edit one in the middle is problematic (and the other alternatives have their own problems); the minus button is also disabled on the default row that shows when there are no creators
- Creator type is no longer reset when editing a name field
- Name field is no longer erased when clicking directly on creator type popup and changing creator type without blurring textbox
- Comma is appended to last name when switching from <textbox> to <label> without saving changes (before it was just appended to labels when the pane was loaded)
references #178, changes to various date fields
- updates CSL to work with the latest schema. we can now (almost) generate completely valid APA style. the only issue is that there's no syntax for specifying short forms for page and creator type labels.
- updates scrapers to use date field rather than year field.
- removes now-unnecessary translation engine code pertaining to year field.
Added isUTC parameter to Scholar.Date.sqlToDate() and now run the two columns through that
Side effect: now using toLocaleString() to format the date strings
Closes#111, minor modifications to field list schema
Changes per above tickets and comments at http://chnm.grouphub.com/projects/310105/msg/cat/2333974/2995041/comments
- date, year and lastModified merged into date
- added date field to journalArticle
- added series, seriesTitle and seriesText to journalArticle
- added seriesNumber (along with series) to book and bookSection
(xmlhttp.status seems to turn into a very large integer when there's a network error--I can't imagine this a fairly reliable test, but, then, neither was the previous one, and at least at the moment we're not actually using the test for anything other than an appropriate debug message)
- Fixed bug in File.hasInternalHandler() (no access to navigator from XPCOM)
- Changed "View Attachment" action to check File.hasInternalHandler() and use window.loadURI() for internally handled files and nsIFile.launch() for external -- this prevents the user from getting a helper app dialog when they try to view external files. I basically had to duplicate most of Mozilla's content detection logic and "guess" whether or not it will be able to handle the file internally, which seems a little silly, but, while I feel there are probably better ways to do various parts of this, what's here seems to do the trick. Let me know if you notice it guessing incorrectly (i.e. you get a helper app dialog rather than having a file just open or it launches a file that should've just been loaded into the window). Also look for text files that should be launched rather than opened, especially XML-based data files, as this is a chance for Scholar to be smarter than Firefox itself--for example, OmniGraffle files, which are actually just XML files, normally open up in Firefox as an XML tree, but Scholar will launch them instead. (I imagine the same will need to be done for OmniOutliner, among other things...)
If you have attachments to the old terminology, feel free to file a complaint.
Changed interface code too, since David is gone (or at the very least has more important things to do with his remaining time)
- 'isInstitution' parameter added to Item.setCreator(), Creators.getID(), Creators.add()
- 'isInstitution' property added to return from Creators.get() and Item.getCreator()
var obj = Scholar.Items.getNewItemByType(1);
obj.setField('title', 'Digital History for Dummies');
obj.setCreator(0, '', 'Center for History and New Media', 1, true); // true == institutional creator
var id = obj.save();
Note: 'firstName' field is ignored when 'isInstitution' is true
Conditions in ANY queries can be made required by passing 'true' as an extra parameter to addCondition() and updateCondition() -- this can be used for limiting ANY queries to particular collections (in place of the removed 'context' condition), but if there was an elegant way to expose it to the user for all ANY queries, it's something users might find very useful.
- Implemented 'collectionID' and 'savedSearchID' conditions (a.k.a. search within a search) and removed special 'context' condition. Per my conversation with Dan, the 'recursive' flag is now a global flag that applies to all specified collectionIDs, which is less than ideal but probably better than the alternatives (e.g. having condition-specific recursive checkboxes). It does mean, however, that a "Search subfolders" checkbox is irrelevant if there are no collectionID conditions and should probably be greyed out until applicable.
Another side effect is that it's no longer possible to do an ANY search and return results only within a specific folder (though it can now be done by putting the ANY conditions in a subsearch). Since ANY searches are always annoying in this regard, what I might do is add a way to mark particular conditions as required even in ANY mode, which would allow for quite a lot of flexibility...
Note also that while 'collectionID' and 'savedSearchID' are standard conditions, they should probably be combined into a single condition on the interface side (like playlists and smart playlists under just 'Playlist' in iTunes).
- Now skips invalid/obsolete saved conditions in load()
- Remaining searchConditionIDs are no longer affected by removeCondition() (i.e. they now act like autoincrements), which should make interface code simpler
- Changed default join mode to ALL
- Fixed loading of saved searches with no search conditions
Closes#174, Don't load images and attached files when detecting content type in linkFromURL()
If mime type not provided, Scholar.Files.linkFromURL() now uses XMLHTTPRequest HEAD request to get the content type without loading file (thanks Simon for the idea)
If title not provided, try to figure it out from URL, though not particularly intelligently (last slash)
Note that order of title and mimeType parameters is now swapped
This code should be a bit smarter about unexpected conditions
Addresses #169, add OpenURL interface hooks
Addresses #170, Put "Link" option before "Import" in drop-down menu
Fixes some advanced search flaws (there are still bugs)
Closes#152, Saved Searches (interface layer)
For now, advanced search IS a saved search.
There are still bugs. The 'search' icon is ugly. I wanted to get it out there, however.
Scholar should now attempt to process citation information from EndNote download links (MIME types application/x-endnote-refer and application/x-research-info-systems). in situations where Scholar cannot process the information, a standard helper app dialog will appear. this behavior is controlled by the preference extensions.scholar.parseEndNoteMIMETypes.
Implemented advanced/saved search architecture -- to use, you create a new search with var search = new Scholar.Search(), add conditions to it with addCondition(condition, operator, value), and run it with search(). The standard conditions with their respective operators can be retrieved with Scholar.SearchConditions.getStandardConditions(). Others are for special search flags and can be specified as follows (condition, operator, value):
'context', null, collectionIDToSearchWithin
'recursive', 'true'|'false' (as strings!--defaults to false if not specified, though, so should probably just be removed if not wanted), null
'joinMode', 'any'|'all', null
For standard conditions, currently only 'title' and the itemData fields are supported -- more coming soon.
Localized strings created for the standard search operators
API:
search.setName(name) -- must be called before save() on new searches
search.load(savedSearchID)
search.save() -- saves search to DB and returns a savedSearchID
search.addCondition(condition, operator, value)
search.updateCondition(searchConditionID, condition, operator, value)
search.removeCondition(searchConditionID)
search.getSearchCondition(searchConditionID) -- returns a specific search condition used in the search
search.getSearchConditions() -- returns search conditions used in the search
search.search() -- runs search and returns an array of item ids for results
search.getSQL() -- will be used by Dan for search-within-search
Scholar.Searches.getAll() -- returns an array of saved searches with 'id' and 'name', in alphabetical order
Scholar.Searches.erase(savedSearchID) -- deletes a given saved search from the DB
Scholar.SearchConditions.get(condition) -- get condition data (operators, etc.)
Scholar.SearchConditions.getStandardConditions() -- retrieve conditions for use in drop-down menu (as opposed to special search flags)
Scholar.SearchConditions.hasOperator() -- used by Dan for error-checking
closes#76, implement extensible search/retrieval architecture for obtaining metadata
OpenURL COinS lookup is now implemented using a real search architecture system. at the moment, it works with Open WorldCat for books, CrossRef for journal articles (provided the COinS object contains a DOI or an ISSN), and PubMed when a PMID is available.
If you are simple removing an item from a project, it won't ask you if you want to delete files and notes.
ItemTreeView:
- notify() now works with multiple ids for action=modify.
- saveSelection() returns an array of selected IDs instead of saving it to a class variable
- rememberSelection(selection) takes an array of IDs.
OpenURL lookup now works for books. this means that all that's necessary to add scrapable book metadata to a page is an ISBN, as shown below:
<span class="Z3988" title="ctx_ver=Z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.isbn=1579550088"></span>
also, we can now scrape Open WorldCat and Wikipedia Book Sources pages with no specialized code involved.
i'm still looking for a better way of looking up journal article metadata. it's currently implemented with CrossRef, but CrossRef simply will not work without a DOI, and is also incomplete (only holds the last name of the first author).
Scholar.OpenURL.resolve(item) returns the URL that retrieves an item from the user's OpenURL resolver. this means we can implement a "find in my library" feature.
Scholar.OpenURL.discoverResolvers() returns a list of available resolvers for the user's current location (by IP address).
closes#163, make translator API allow creator types besides author
import and export in the multi-ontology RDF format should now work properly. collections, notes, and see also are all preserved. more extensive testing will be necessary later.
Fixed ignoreCase logic (and also set all but CharacterSets to false, since there's no reason for them to be true)
Also made CachedTypes.getID() and getName() return false and '', respectively, on unknown types rather than letting them hit the error (there's still the 'invalid * type' debug message)
- We might want to do this for regular files as well? I think we need a discussion on how Citation will be presented to the user, and if that will save the web page, and all that jazz.
- eliminates "unresponsive script" message on import/export
i tried to make a progress bar that actually provides useful information, but for some reason, XUL interface updates are done asynchronously, and thus don't actually happen as long as the import/export operation continues. the code is there, but disabled, if there's some solution to this issue, but i searched and couldn't find one.
Temporarily added in a check of the backup file on startup, since I'm not entirely convinced that the backup mechanism on shutdown couldn't create a corrupt file under certain conditions
If you run with debug output on and notice the "Backup file was corrupt" message, let me know.
The Scholar database is backed up on browser close. On startup, if the main database is damaged, the extension saves a copy of the damaged file and tries to restore from the last automatic backup. If it fails, it creates a new database file.
New methods:
Scholar.getScholarDatabase(string [ext])
Scholar.backupDatabase()
Scholar.moveToUnique(file, newFile) -- find a unique filename using the leafName of newFile as the suggested name (using the built-in Mozilla functionality) and move the file there
Scholar.Date.getFileDateString(file)
Scholar.Date.getFileTimeString(file)
Closes#146, ScholarPane.selectItem(id)
Closes#147, "Edit in a Separate Window" on a note should select the note's parent item.
Addresses #143, Scholar toolbar button to save current page as an independent file in the selected project.
- Standalone Files are now added to the current Project.
Files added to an item are now actually attached to the item.
Closes#118, add "translator" creator type
Closes#122, add DOI and abbreviated journal title fields
Addresses #45, reorder item fields -- source/rights moved down to bottom; date fields not yet moved
Returns the first 80 characters of the note content as the title
Also changed setField() to use the loadIn parameter for primary fields so it can be used instead of this._data without affected _changedItems
for the moment, this feature is disabled on the mac, since firefox can't handle HTML on the clipboard (and thus we can't copy the fully formatted bibliography). i can re-enable it if it would be useful regardless.
closes#4, Make printable version
- moves functions for creating and deleting hidden browser objects to scholar.js (from ingester.js), since these are necessary for printing as well
- allows saving bibliography in HTML or printing bibliography. style support is not yet complete (pending finalization of 0.9 version of CSL specification).
Closes#128, Display files as children of item in items list
Fixes#132, On Full-screen mode, browser content disappears but Scholar content does not stretch.
Fixes#133, When Scholar on top, pane automatically resizes depending on item info height.
Not finished, but enough to give David something to work with
No BLOBs -- just linking/importing of files and loaded documents
New Scholar.Item methods:
incrementFileCount() (used internally)
decrementFileCount() (used internally)
isFile()
numFiles()
getFile() -- returns nsILocalFile or false if associated file doesn't exist (note: always returns false for items with LINK_MODE_LINKED_URL, since they have no files -- use getFileURL() instead)
getFileURL() -- returns URL string
getFileLinkMode() -- compare to Scholar.Files.LINK_MODE_* constants: LINKED_FILE, IMPORTED_FILE, LINKED_URL, IMPORTED_URL
getFileMimeType() -- mime type of file (e.g. text/plain)
getFileCharset() -- charsetID of file
getFiles() -- array of file itemIDs this file is a source for
New Scholar.Files methods:
importFromFile(nsIFile file [, int sourceItemID])
linkFromFile(nsIFile file [, int sourceItemID])
importFromDocument(nsIDOMDocument document [, int sourceItemID])
linkFromDocument(nsIDOMDocument document [, int sourceItemID])
New class Scholar.FileTypes -- partially implemented, not yet used
New class Scholar.CharacterSets -- same as other *Types classes:
getID(idOrName)
getName(idOrName)
getTypes() (aliased to getAll(), which I'll probably change the others to as well)
Charsets table with all official character sets (copied from Mozilla source)
Renamed Item.setNoteSource() to setSource() and Item.getNoteSource() to getSource() and adjusted to handle both notes and files
- ScholarPane.getSelectedCollection() returns collection object, or null if Library is selected.
- ScholarPane.getSelectedItem() returns selected item/note/item-like-thing, or null if no item is selected.
add Scholar.Cite and Scholar.CSL for parsing items into a bibliography using CSL. unfortunately, the output is not very good at the moment, and the format likely needs some changes, but I'm working with a few other people on getting it to that point.
Fixes#115, Windows should open with Scholar Pane closed.
Fixes#105, search box loses focus after search starts.
Retooled the interface a bit, and removed the top toolbar. The close and fullscreen buttons are located to the right of the items toolbar. The item pane cannot be collapsed.
closes#100, migrate ingester to Scholar.Translate
closes#88, migrate scrapers away from RDF
closes#9, pull out LC subject heading tags
references #87, add fromArray() and toArray() methods to item objects
API changes:
all translation (import/export/web) now goes through Scholar.Translate
all Scholar-specific functions in scrapers start with "Scholar." rather than the jumbled up piggy bank un-namespaced confusion
scrapers now longer specify items through RDF (the beginning of an item.fromArray()-like function exists in Scholar.Translate.prototype._itemDone())
scrapers can be any combination of import, export, and web (type is the sum of 1/2/4 respectively)
scrapers now contain functions (doImport, doExport, doWeb) rather than loose code
scrapers can call functions in other scrapers or just call the function to translate itself
export accesses items item-by-item, rather than accepting a huge array of items
MARC functions are now in the MARC import translator, and accessed by the web translators
new features:
import now works
rudimentary RDF (unqualified dublin core only), RIS, and MARC import translators are implemented (although they are a little picky with respect to file extensions at the moment)
items appear as they are scraped
MARC import translator pulls out tags, although this seems to slow things down
no icon appears next to a the URL when Scholar hasn't detected metadata, since this seemed somewhat confusing
apologizes for the size of this diff. i figured if i was going to re-write the API, i might as well do it all at once and get everything working right.
caveats:
- it's not human readable. mozilla doesn't nest blank nodes, so everything's scattered throughout the file. it would be relatively easy to do post-processing with E4X or even regexps to correct this.
- there's no generic callNumber field, so all callNumbers are encoded as LCC.
adds container creation routines to dataMode rdf
changes Dublin Core export to Unqualified Dublin Core, and removes DC Terms qualifiers
This custom credits panel gives us more flexibility. Also, there is an About link which is certainly easier to find than right-clicking on the extension in the Add-ons window.
adds export of seeAlso info and project hierarchy to RDF. for now, this is embedded in the modsCollection root element.
uses nodeIDs for Dublin Core RDF.
adjusts the Google Books translator to work with the latest revision of the site
renames the MODS translator to just MODS, because "Metadata Object Description Schema (MODS)" was too long for the export dialog
- fixes a bug that could result in the Scrape Progress chrome thingy sticking around forever
- makes chrome thingy disappear when URL changes or when tabs are switched
Item.addSeeAlso(itemID)
Item.removeSeeAlso(itemID)
Item.getSeeAlso() -- array of linked items
Relationships are mutual
Closes#82, Notes: see also table
- changes scrapers table to translators table; all import/export/web translators now belong in this table
- adds Scholar.Translate to handle translation issues. eventually, Scholar.Ingester.Document will become part of this interface
- adds Scholar_File_Interface (in fileInterface.js) to handle UI for export and eventually import. (David, when you have time, please connect Scholar_File_Interface.exportFile to a button.)
- adds an export translator for MODS. all of our metadata, but not our hierarchy (projects, etc.) translates directly and unambiguously into valid MODS. eventually, we can use RDF or another format to handle hierarchy.
- adds utilities.getVersion() and utilities.inArray() for simplified scraper coding
- fixes minor interface issues with the nifty chrome scraping status window
New methods:
Item.addTag(tag)
Item.getTags() -- array of tagIDs
Item.removeTag(tagID)
Tags.getName(tagID)
Tags.getID(tag)
Tags.add(text) -- returns tagID of new tag
Tags.purge() -- purge obsolete tags
The last two are for use by Item.addTag() and Item.removeTag(), respectively, and probably don't need to be used elsewhere.
Item.save() changed to not call notify() if a transaction is in progress, which is totally going to trip someone up at some point, and probably me, but it's better than a new parameter
Item.toArray() implemented -- builds up a multidimensional array of item data, converting all type ids to their textual equivalents -- currently has empty placeholder arrays for tags and seeAlso
Sample source output:
'itemID' => "2"
'itemType' => "book"
'title' => "Computer-Mediated Communication: Human-to-Human Communication Across the Internet"
'dateAdded' => "2006-03-12 05:25:50"
'dateModified' => "2006-03-12 05:25:50"
'publisher' => "Allyn & Bacon Publishers"
'year' => "2002"
'pages' => "347"
'ISBN' => "0-205-32145-3"
'creators' ...
'0' ...
'firstName' => "Susan B."
'lastName' => "Barnes"
'creatorType' => "author"
'notes' ...
'0' ...
'note' => "text"
'tags' ...
'seeAlso' ...
'1' ...
'note' => "text"
'tags' ...
'seeAlso' ...
'tags' ...
'seeAlso' ...
Sample note output:
'itemID' => "17"
'itemType' => "note"
'dateAdded' => "2006-06-27 04:21:16"
'dateModified' => "2006-06-27 04:21:16"
'note' => "text"
'sourceItemID' => "2"
'tags' ...
'seeAlso' ...
sourceItemID won't exist if it's an independent note.
We'll use the same format in reverse for fromArray, so Simon, let me know if you need more data (preserving type ids, etc) or want anything in a different form.
- Added 'note' item type
- Updated API to support independent note creation
Notes are, more or less, just regular items, with an item type of 1. They're created through Scholar.Notes.add(text, sourceItemID), which returns the itemID of the new note. sourceItemID is optional--if left out, an independent note will be created. (There's currently nothing stopping you from doing getNewItemByType(1) yourself, but the note would be contentless and broken, so you shouldn't do that.) Note data could've been stuffed into itemData, but I kept it separate in itemNotes to keep metadata searching faster and to keep things cleaner.
Methods calls that can be called on all items:
isNote() (same as testing for itemTypeID 1)
Method calls that can be called on source items only:
numNotes()
getNotes() (array of note itemIDs for a source)
Method calls that can be called on note items only:
updateNote(text)
setNoteSource(sourceItemID) (for changing source--use empty or false to make independent, which is currently what happens when you delete a note--will get option with #91)
getNote() (note content)
getNoteSource() (sourceItemID of a note)
Calling the above methods on the wrong item types will throw an error.
*** This will break note creation/display until David updates interface code. ***
- Move processDocuments, a function for loading a DOM representation of a document or set of documents, to Scholar.Utilities.HTTP
- Add Scholar.Ingester.ingestURL, a simplified function to scrape a URL (closes#33)
- This required moving the icon to the title field so that the indent would work out right. The type column (which for new installs will be hidden) displays the type in text.
- I expect several small bugs in regard to this.
closes#26, notes list in notes pane
closes#79, add icons for new object types
fixes#71, Metadata pane should be refreshed on a notify() event for the selected item
- Added detection for network failure -- debug message is output and noNetwork property is added to the xmlhttp object
- Removed onStatus callback from HTTP.doGet and HTTP.doPost -- that was copied over from the Piggy Bank API, but the onDone callback has to handle errors anyway, so it can just check the status code if it actually cares to differentiate non-200 status codes from any other error
- Added error handling for empty responseXML to Schema._updateScrapersRemoteCallback
- Renamed SCHOLAR_CONFIG['REPOSITORY_CHECK_RETRY'] to SCHOLAR_CONFIG['REPOSITORY_RETRY_INTERVAL']
Scholar.Prefs also registers itself as a preferences observer and can be used to trigger actions when certain prefs are changed by editing the switch statement in the observe() method
Updated preferences.js to use Scholar.Prefs
- Added methods getID(idOrName) and getName(idOrName) to Scholar.CreatorTypes and Scholar.ItemTypes to take either typeID or typeName
- Removed getTypeName() in each and changed references accordingly
- Streamlined both classes to be as similar as possible
- Make Amazon scraper work with multiple documents
- Fix bugs in processDocuments
- Make Scholar.Ingester.Utilities.getItemArray() willing to take an array of DOM nodes to search for links, and finally take advantage of the fact that objects have no length
- Multiple item detection code is now a part of the scraperJavaScript, rather than the scrapeDetectCode, and code to choose which items to add is part of Scholar.Ingester.Utilities, accessible from inside scrapers. The alternative approach would result in one request (or, in the case of JSTOR, three requests) per new item, while in some cases (e.g. Voyager) only one request is necessary to get all of the items.
Moved the capture icon into the URL bar (invisible until you visit a scrapable page. Currently just displays a Book, but will change to the correct item types in the future?)
- When possible, corporate creators/contributors are categorized with their own RDF types (prefixDummy + "corporateCreator/corporateContributor)
- Remove extraneous debug code in extensions
- Don't try to display an SQLite error when it's "not an error" (i.e. when the error is in something else)
- Switch to nsIFile instead of nsILocalFile to retrieve the profile directory
Fix in Collection.erase() -- when the DB methods started returning values in their native type, the collection id became an int rather than a string and "new Array(this._id)" became a length declaration rather than an elements declaration
- Ingester lets callback function save items, rather than saving them itself.
- Better handling of multiple items in API, although no scrapers currently implement this.
- Fix item modify notify() on 1st row.
- Ensure that new items are visible when added.
- New functionality for creating new items (prevents a lot of problems).
- Number-based fields display properly.
- Fixed bug when creating and saving the first notes on an item.
- New notes won't save empty.
[fix] You shouldn't lose your changes if you select another item in the middle of editing a field.
[fix] The dropdown menu to select notes doesn't steal the focus
[interface] Multi-notes functionality (waiting on data layer)
[docs] Major internal documentation written for itemTreeView.js and collectionTreeView.js (this actually does work ;-))
Added a separate retry interval so that the extension retries sooner after failures (browser offline, request failure, etc.)
Revision 200 -- w00t i am victorious
- Removed localLastUpdated field from scrapers table and renamed centralLastUpdated to lastUpdated; updated scraper queries accordingly
- Added query in scrapers.sql to update version table 'repository' row to prevent immediate downloads of newly installed scrapers
- Get version property from extension manager in Scholar.init() and assign to Scholar.version
[Drag and Drop] in Items Tree: You can drag items from one window into another, directly into the Items list.
[Editing] Close the edit box and save when you click on its label
The collections list does not resize randomly now.
The pane on the right stays open all the time - even when 0/multiple items are selected. This is to avoid frequent resizing of the items pane.
Temporarily, if the first "word" of a field's value is more than 29 characters long, it will set it to crop. This is for the long URLs, etc.
Scholar.HTTP.doGet(url, onStatus, onDone) and Scholar.HTTP.doPost(url, body, onStatus, onDone) -- onStatus and onDone are callbacks to call on non-200 responses and the response body, respectively
Assigned guids to scrapers, replaced INSERT queries with REPLACE queries, and removed table DELETE query at top -- this will allow scrapers to be updated without deleting any others that may exist (e.g. that someone is developing, third-party, etc.)
[style] Better add/remove Creator buttons.
[fix] The sorting should not randomly switch the order of two items with the same sort value (eg, Barnes vs. Barnes).
[fix] The browser should not open with two sorted columns.
scholar.properties addition is just to keep metadata panel from breaking and could probably be removed once interface is hard-coded to not display the notes field there
- Send a 'delete' rather than a 'remove' to itemViews when items are actually deleted (removals from collections still get 'remove' unless it's part of a collection erase)
1) Items into collections.
2) Collections into collections.
Dan S, please check this out, I am getting some exceptions from the data access portion.
[interface] Temporarily, "metadata" pane on the right is not resizable.
Added creatorTypeID to the PK in itemCreators, because a creator could conceivably have two roles on an item
Throw an error if attempt to save() an item with two identical creator/creatorTypeID combinations, since, assuming this shouldn't be allowed (i.e. we don't have a four-column PK), there's really no way to handle this elegantly on my end -- interface code can use new method Item.creatorExists(firstName, lastName, creatorTypeID, skipIndex), with skipIndex set to the current index, to make sure this isn't done
Integrated the scrapers with the schema update mechanism. Changed a bunch of schema methods to handle both schema.sql and scrapers.sql (or others, if need be) and altered the version table to track mu
ltiple versions for different files. This theoretically should detect that the version table has changed and force a reinitialization of the DB--let me know if there are problems.
- Broke schema functions into separate object and got rid of DB_VERSION config constant in favor of a toVersion variable in the _migrateSchema command (which isn't technically necessary either, since the version number at the top of schema.sql is now always compared to the DB version at startup) but will help reduce the chance that someone will update the schema file without adding migration steps)
- Removed Amazon scraper from schema.sql, as it will be loaded with the rest of the scrapers
- Lots of work to be done. For example, the present way of showing a textbox is sort of a hack - taking a label, assigning certain properties to a textbox, then removing the label and placing the textbox in its place. I will be looking into our options.
- Also, I need to figure out adding/editing/deleting creators.
- When the textbox loses focus, it updates and saves the item (like iCal).
Date: removed Scholare.Date functions, as they are overkill. We can use the built-in Date.toLocaleString()
- Implemented loadDocument API, for loading and parsing the DOMs of HTML documents in the background
- Added scraper code to SVN repository (now includes 12 scrapers, see Writeboard for details)
To update to the latest versions of all scrapers, ensure you have an up-to-date version of sqlite3, then run:
sqlite3 ~/Library/Application\ Support/Firefox/Profiles/profileName/scholar.sqlite < scrapers.sql
The real search function will be considerably more advanced/flexible, but this should work as a placeholder for the moment. Probably quick enough for FAYT, at least with a ~0.5 second delay to avoid unnecessary calls while people are typing (which is probably a good idea anyway). This search doesn't use indexes at all, so if more speed is needed, one option would be to maintain a manual FULLTEXT-type index (using triggers, ideally) that could be quickly searched, but we'd lose intra-word filtering, which people would probably expect...
parentCollectionID is optional and defaults to moving item to root
Returns true on success, false on attempt to move collection into its existing parent, itself or a descendent collection; throws exception on invalid parentCollectionID
Sends a columnTree notify() with previous parent ID, id of collection itself, and new parent ID (unless the previous or new are the root, in which case it's omitted) -- that may or may not make sense for the interface code and can be changed if needed
New items are automatically selected.
When adding a new item, or editing an item without a creator, display an empty creator box.
Edit button changed from toggling to Save/Cancel buttons.
Simple "Do you want to save changes?" dialog if you select a different item while in edit mode.
- Collection.add( name [, parentCollectionID] ) -- saves new collection in DB and returns new Collection object; defaults to root if parent not provided
Currently some exceptions in folderTreeView.js when notified on collection add
- Added back Item.isCollection() and Collection.isCollection(), as they seem to still be used
- Collection._load() now takes optional ids for loading new collections
Close buttons on the Metadata and Notes.
The Collections list no longer throws an exception when the header is clicked.
More minor interface improvements.
This behavior of JS has some implications for how interface code handles things like deletes. The following won't do what one might expect/desire (and my apologies if this is obvious or redundant, but I figure it's worth pointing out):
var item = Scholar.Items.get(1);
Scholar.debug(item);
Scholar.DB.query("DELETE FROM items WHERE itemID=1");
Scholar.Items.reloadAll();
Scholar.debug(item);
The last line will still display the old object, even though _items[1] has been deleted inside Scholar.Items. The following does work, however:
var item = Scholar.Items.get(1);
Scholar.debug(item);
Scholar.DB.query("DELETE FROM items WHERE itemID=1");
Scholar.Items.reloadAll();
var item = Scholar.Items.get(1);
Scholar.debug(item);
Now item is properly undefined. Moral of the story: object references need to be deleted manually after receiving delete notifications, for if external code still has references to deleted data objects, the data layer can't do much about it.
Revised the way the dateAdded and dateModified columns are displayed.
Views are now unregistered on window close.
A few functions in overlay.js were renamed.
Began better commenting of the interface code.
Although I do not yet have interface code for it, collections do not have any sort of setName(), save() functions.
Collections list now implements notify()
Made some fixes on itemTreeView to allow for notify() on multiple deletes
(shifts the row identifier up 1 for each previous delete, etc.)
One more thing -- this might not happen because the tree calls each Item.erase() or Collection.removeItem() individually.
Bug fix on notifier.js so that register*Tree() actually returns the hash. :-)
Unfortunately, unregister*Tree() does not seem to actually work, there is still an object at _observers['itemTree'][hash]
(temporary): When you select an item Scholar no longer loads a page into the browser.
The javascript bugs on those HTML pages were frustrating the debugger.
- Added Notifier triggers to Item.erase() (which only runs if not in a nested transaction) and Collections.erase()
- notify() in ItemTreeView updated with example of taking multiple ids, though it doesn't actually work (and notify() implementations may decide just to refresh the whole tree when ids.length>1 rather than dealing with changes individually)
- When deleting collections, use DB tables that actually exist
Scholar.Notifier framework to handle update notifications between data layer and interface
- Notifier.registerColumnTree(ref) and Notifier.registerItemTree(ref) pass back a unique hash that can be used to unregister the tree later with unregister*(hash) (and must be, lest we leak treeViews and probably entire browser windows if the browser window is closed without unregistering the treeView)
- Data layer calls Scholar.Notify.trigger(event, type, id) after various events (only two calls in the data layer at the moment--more coming later)
- Notify.trigger() calls notify(event, type, id) on all registered trees of the appropriate type -- the data layer usually knows what collection the action pertains to, but we decided that it's cleaner to just let the tree decide what it wants to do rather than add all that logic into the data layer)
(Note: Item collection adds appear to be buggy on the interface side, but removes seem to be working)
Overhauled data access layer to support new model:
- Changed Item.getParent to Item.getCollections() to get ids of parent collections
- Removed Item.setPosition() -- item positions are set internally when items are added to and deleted from collections, but the orderIndex is not currently used or manipulatable externally
- Item constructor/Items.getNewItemByType()/Items.add() no longer take folderID and orderIndex as parameters
- Split getTreeRows() into Scholar.getCollections(parent) and Scholar.getItems(parent), which return root collections or all library items, respectively, if no parent given
- All references to folders in object/method/property names changed to collections
- New methods Collection.addItem(itemID), Collection.hasItem(itemID), Collection.removeItem(itemID)
- Collection.erase() takes optional deleteItems parameter to delete items from DB -- otherwise just removes association and leaves item in library (does, however, delete all descendent collections from DB regardless)
* Note: This will break displaying of items until interface code is updated. *
Scholar.TreeView is now Scholar.FolderTreeView (SourcesTreeView later? depends)
Some changes to the way FolderTreeView works.
New Scholar.ItemGroup -- used by FolderTreeView and ItemTreeView.
the root folder is not currently displayed -- doesn't really matter because we are changing the way it all works.
(note: the Scholar.Items.getAll() function does not seem to be working correctly -- try clicking on the library)
Switching to scholar@chnm.gmu.edu -- SVN folks should rename the text file in their FF profile directories. (Used to be scholar@chnm, so you need to make the change even if you hadn't changed to the GUID yet.) Anyone who's tried the XPI yet will need to reinstall just once from the URL (http://chnm.gmu.edu/firefoxscholar/download/scholar-dev.xpi), as the old version won't find updates with its id.
* Renamed treeView.js to folderTreeView.js (more changes on this file forthcoming) -- this will soon be specifically for folders.
* Added itemTreeView.js, with new Scholar.ItemTreeView class -- specifically for items.
* The items list now supports 3 additional columns: rights, date added, and date modified.
* For now, selecting an item loads the "I'm Feeling Lucky" result of a Google search on the item title.
Added update.rdf for auto-update mechanism (which will get moved out of the extension root by the build system but should stay with a particular tree)
Assigned a new GUID to extension
*** Important: The extension GUID has changed, so you should rename the scholar@chnm text file in your Firefox profile exensions directory to our new GUID "{e007c81c-eb2a-4e1d-b381-d25a45cd11ed}" and restart Firefox ***
More details coming on Basecamp: http://chnm.grouphub.com/W161222
*** Important note: after checking out this code, be sure to delete the compreg.dat and xpti.dat files in your FF profile directory or else the extension will not load. They'll be regenerated when you start FF again, and you won't have to do it again unless we add interfaces or other components. Once I set up the XPI packaging system, this step won't be necessary. ***
- Localization properties are now loaded directly via nsIStringBundleService and available via Scholar.getString(name) -- removed stringbundle from XUL
- Updated status line in bottom right to reflect whether Scholar is loaded correctly or not (temporary, obviously)
Scholar.Folder.erase() -- deletes a folder and all descendent folders and items
Scholar.Folder._getDescendents() - returns an array of descendent folders and items (rows of 'id' and 'isFolder') ("private" for now--could be useful as public later)
Scholar.Folders.unload() - clear folder from internal cache (used by Scholar.Folder.erase())
DB transactions are now automatically nested if beginTransaction() is called when one is already in progress -- the queries aren't committed until the outermost commitTransaction() is called, which is handy for calling methods with their own transactions inside others when the entire process should be a transaction (e.g. Folder.erase() calling Item.erase() for the folder's descendents). Rollbacks are a little trickier -- currently, a rollback in a nested transaction flags the entire transaction for rollback when the outermost commit is reached, which for our purposes will probably be fine.
ScholarLocalizedStrings moved out of sidebar.js and into Scholar.LocalizedStrings
Rudimentary creator adding/editing. lots of things to work on, because it doesn't work.
Added Scholar.CreatorTypes with methods getTypes() (multi-dim array with 'id' and 'name') and getTypeName(creatorTypeID)
Fixed bug in Scholar.Creators.purge() causing SQL error when deleting a non-existent creator
Fixed incorrect field order for itemCreators INSERT queries in save()
Changed setCreator() to take empty creator names (causing DELETE in itemCreators but without shifting up creators below it (or down and above, depending on your perspective) like removeCreators())
The selection no longer lost or moved on folder opening/closing.
The erase functionality works, on Mac (Windows) use the delete (backspace) or forward delete (delete) keys.
unfortunately there are sometimes exceptions so I am not calling erase() on each item until we figure out what the problem is.
view._deleteItem() and view._insertItem() are now view._hideItem() and view._showItem() to better reflect functionality.
Scholar.ItemTypes.getTypes() -- returns associative array of all itemTypes, with 'id' and 'name'
Scholar.ItemTypes.getTypeName(itemTypeID) -- returns pre-l10n type name (e.g. 'book', 'journalArticle')
Removed getLevel()
Updated Object.setPosition() to handle setting the position of new objects without committing, so it can be used within save(). May or may not actually work--my money's on no.
Added Folder.isEmpty()
No Folder.setPosition() yet, but, then, no editing or saving of folders at all--coming soon
Added getLevel() functions to Scholar.Object and Scholar.Folder -- save() and setPosition() not yet updated
Updated sample data to include levels
Added orderIndex (from treeOrder table, now) back into Scholar.Object data
New Scholar.Objects method for treeView: getTreeRows() -- returns mixed array of all Folder and Object objects in the proper order, type testable with instanceof (e.g. if (obj instanceof Scholar.Folder))
Scholar.Objects.get() and getAll() return arrays hashed by objectID again, since they're no longer used directly by treeView
Scholar.Objects.get() now returns the object directly if only one argument and scalar (i.e. one id)
Updated schema and sample data to handle folder ordering
Added include.js for use in non-browser.xul windows to bring in the SCHOLAR_CONFIG and Scholar objects
Adjusted sidebar to use shared object
Added test line (displayable by clicking "Hello, World" before and after sidebar load to demonstrate cross-window access
I also started implementing some of the localization. There is a lot to do still, and I am still learning a lot of the Firefox extension technologies (XUL, adv. Javascript, etc.) but things are going great.
Fixed object retrieval queries to use orderIndex is 0 rather than 1, since that's the default now
Scholar_Objects.getAll() now uses loaded.push() rather than loaded[objectID], since _for (objectID in objects)_ isn't an option with treeview
Scholar_Object.getField() now returns an empty string rather than false for empty values
Added some new core functions:
- Scholar.varDump(), after PHP's var_dump()
- Scholar.flattenArguments(), to flatten mixed array/literal argument lists into a single array
- Scholar.join() -- a version of join() that operates externally, for use on, for example, the arguments object (safer than extending Object)
- Scholar.Hash, a slightly smarter associative array -- not perfect, but brings a proper length property and a few convenience methods (and allows for other additions) -- should probably be limited to places where the length property or other additional additions are needed, since its use is a little non-standard (e.g. you have to remember to do _for (i in arr.items)_ rather than just _for (i in arr)_, to use set(), etc.)
Moved schema file out of chrome and removed XML tags, since it will no longer be accessed by XMLHTTP
Changed schema updater to use XPCOM components rather than XMLHTTP
Converted DB class to a singleton named Scholar_DB
Scholar_DB.query() now returns associative array similar to mysql_fetch_assoc() for SELECT statements rather than a mozIStorageStatementWrapper -- the mozStorage executeDataSet() function doesn't work yet, so we do it this way
Added DB functions:
- beginTransaction(), commitTransaction(), rollbackTransaction()
- columnQuery() -- one column, multiple rows, returned as array indexed by row
- getColumns() and getColumnHash
DB query functions can now handle bind parameters passed as an array of object literals: e.g. [{'int':2},{'string':'foobar'}]
valueQuery() now returns an int for COUNT(*) queries, so the result can be tested without a "0" being true
Changed _initializeSchema to drop existing tables before creating new ones, hacked with try/catch until DROP TABLE IF EXISTS works with the mozStorage extension (it's already in the latest SQLite release)
Added DB_REBUILD config flag to manually trigger schema regeneration
Added debug logging at level 5 for all SQL queries
Updated sample data
