closes#26, notes list in notes pane
closes#79, add icons for new object types
fixes#71, Metadata pane should be refreshed on a notify() event for the selected item
- Added detection for network failure -- debug message is output and noNetwork property is added to the xmlhttp object
- Removed onStatus callback from HTTP.doGet and HTTP.doPost -- that was copied over from the Piggy Bank API, but the onDone callback has to handle errors anyway, so it can just check the status code if it actually cares to differentiate non-200 status codes from any other error
- Added error handling for empty responseXML to Schema._updateScrapersRemoteCallback
- Renamed SCHOLAR_CONFIG['REPOSITORY_CHECK_RETRY'] to SCHOLAR_CONFIG['REPOSITORY_RETRY_INTERVAL']
Scholar.Prefs also registers itself as a preferences observer and can be used to trigger actions when certain prefs are changed by editing the switch statement in the observe() method
Updated preferences.js to use Scholar.Prefs
- Added methods getID(idOrName) and getName(idOrName) to Scholar.CreatorTypes and Scholar.ItemTypes to take either typeID or typeName
- Removed getTypeName() in each and changed references accordingly
- Streamlined both classes to be as similar as possible
- Make Amazon scraper work with multiple documents
- Fix bugs in processDocuments
- Make Scholar.Ingester.Utilities.getItemArray() willing to take an array of DOM nodes to search for links, and finally take advantage of the fact that objects have no length
- Multiple item detection code is now a part of the scraperJavaScript, rather than the scrapeDetectCode, and code to choose which items to add is part of Scholar.Ingester.Utilities, accessible from inside scrapers. The alternative approach would result in one request (or, in the case of JSTOR, three requests) per new item, while in some cases (e.g. Voyager) only one request is necessary to get all of the items.
Moved the capture icon into the URL bar (invisible until you visit a scrapable page. Currently just displays a Book, but will change to the correct item types in the future?)
- When possible, corporate creators/contributors are categorized with their own RDF types (prefixDummy + "corporateCreator/corporateContributor)
- Remove extraneous debug code in extensions
- Don't try to display an SQLite error when it's "not an error" (i.e. when the error is in something else)
- Switch to nsIFile instead of nsILocalFile to retrieve the profile directory
Fix in Collection.erase() -- when the DB methods started returning values in their native type, the collection id became an int rather than a string and "new Array(this._id)" became a length declaration rather than an elements declaration
- Ingester lets callback function save items, rather than saving them itself.
- Better handling of multiple items in API, although no scrapers currently implement this.
- Fix item modify notify() on 1st row.
- Ensure that new items are visible when added.
- New functionality for creating new items (prevents a lot of problems).
- Number-based fields display properly.
- Fixed bug when creating and saving the first notes on an item.
- New notes won't save empty.
[fix] You shouldn't lose your changes if you select another item in the middle of editing a field.
[fix] The dropdown menu to select notes doesn't steal the focus
[interface] Multi-notes functionality (waiting on data layer)
[docs] Major internal documentation written for itemTreeView.js and collectionTreeView.js (this actually does work ;-))
Added a separate retry interval so that the extension retries sooner after failures (browser offline, request failure, etc.)
Revision 200 -- w00t i am victorious
- Removed localLastUpdated field from scrapers table and renamed centralLastUpdated to lastUpdated; updated scraper queries accordingly
- Added query in scrapers.sql to update version table 'repository' row to prevent immediate downloads of newly installed scrapers
- Get version property from extension manager in Scholar.init() and assign to Scholar.version
[Drag and Drop] in Items Tree: You can drag items from one window into another, directly into the Items list.
[Editing] Close the edit box and save when you click on its label
The collections list does not resize randomly now.
The pane on the right stays open all the time - even when 0/multiple items are selected. This is to avoid frequent resizing of the items pane.
Temporarily, if the first "word" of a field's value is more than 29 characters long, it will set it to crop. This is for the long URLs, etc.
Scholar.HTTP.doGet(url, onStatus, onDone) and Scholar.HTTP.doPost(url, body, onStatus, onDone) -- onStatus and onDone are callbacks to call on non-200 responses and the response body, respectively
Assigned guids to scrapers, replaced INSERT queries with REPLACE queries, and removed table DELETE query at top -- this will allow scrapers to be updated without deleting any others that may exist (e.g. that someone is developing, third-party, etc.)
[style] Better add/remove Creator buttons.
[fix] The sorting should not randomly switch the order of two items with the same sort value (eg, Barnes vs. Barnes).
[fix] The browser should not open with two sorted columns.
scholar.properties addition is just to keep metadata panel from breaking and could probably be removed once interface is hard-coded to not display the notes field there
- Send a 'delete' rather than a 'remove' to itemViews when items are actually deleted (removals from collections still get 'remove' unless it's part of a collection erase)
1) Items into collections.
2) Collections into collections.
Dan S, please check this out, I am getting some exceptions from the data access portion.
[interface] Temporarily, "metadata" pane on the right is not resizable.
Added creatorTypeID to the PK in itemCreators, because a creator could conceivably have two roles on an item
Throw an error if attempt to save() an item with two identical creator/creatorTypeID combinations, since, assuming this shouldn't be allowed (i.e. we don't have a four-column PK), there's really no way to handle this elegantly on my end -- interface code can use new method Item.creatorExists(firstName, lastName, creatorTypeID, skipIndex), with skipIndex set to the current index, to make sure this isn't done
Integrated the scrapers with the schema update mechanism. Changed a bunch of schema methods to handle both schema.sql and scrapers.sql (or others, if need be) and altered the version table to track mu
ltiple versions for different files. This theoretically should detect that the version table has changed and force a reinitialization of the DB--let me know if there are problems.
- Broke schema functions into separate object and got rid of DB_VERSION config constant in favor of a toVersion variable in the _migrateSchema command (which isn't technically necessary either, since the version number at the top of schema.sql is now always compared to the DB version at startup) but will help reduce the chance that someone will update the schema file without adding migration steps)
- Removed Amazon scraper from schema.sql, as it will be loaded with the rest of the scrapers
- Lots of work to be done. For example, the present way of showing a textbox is sort of a hack - taking a label, assigning certain properties to a textbox, then removing the label and placing the textbox in its place. I will be looking into our options.
- Also, I need to figure out adding/editing/deleting creators.
- When the textbox loses focus, it updates and saves the item (like iCal).
Date: removed Scholare.Date functions, as they are overkill. We can use the built-in Date.toLocaleString()
- Implemented loadDocument API, for loading and parsing the DOMs of HTML documents in the background
- Added scraper code to SVN repository (now includes 12 scrapers, see Writeboard for details)
To update to the latest versions of all scrapers, ensure you have an up-to-date version of sqlite3, then run:
sqlite3 ~/Library/Application\ Support/Firefox/Profiles/profileName/scholar.sqlite < scrapers.sql
The real search function will be considerably more advanced/flexible, but this should work as a placeholder for the moment. Probably quick enough for FAYT, at least with a ~0.5 second delay to avoid unnecessary calls while people are typing (which is probably a good idea anyway). This search doesn't use indexes at all, so if more speed is needed, one option would be to maintain a manual FULLTEXT-type index (using triggers, ideally) that could be quickly searched, but we'd lose intra-word filtering, which people would probably expect...
parentCollectionID is optional and defaults to moving item to root
Returns true on success, false on attempt to move collection into its existing parent, itself or a descendent collection; throws exception on invalid parentCollectionID
Sends a columnTree notify() with previous parent ID, id of collection itself, and new parent ID (unless the previous or new are the root, in which case it's omitted) -- that may or may not make sense for the interface code and can be changed if needed
New items are automatically selected.
When adding a new item, or editing an item without a creator, display an empty creator box.
Edit button changed from toggling to Save/Cancel buttons.
Simple "Do you want to save changes?" dialog if you select a different item while in edit mode.
- Collection.add( name [, parentCollectionID] ) -- saves new collection in DB and returns new Collection object; defaults to root if parent not provided
Currently some exceptions in folderTreeView.js when notified on collection add
- Added back Item.isCollection() and Collection.isCollection(), as they seem to still be used
- Collection._load() now takes optional ids for loading new collections
Close buttons on the Metadata and Notes.
The Collections list no longer throws an exception when the header is clicked.
More minor interface improvements.
This behavior of JS has some implications for how interface code handles things like deletes. The following won't do what one might expect/desire (and my apologies if this is obvious or redundant, but I figure it's worth pointing out):
var item = Scholar.Items.get(1);
Scholar.debug(item);
Scholar.DB.query("DELETE FROM items WHERE itemID=1");
Scholar.Items.reloadAll();
Scholar.debug(item);
The last line will still display the old object, even though _items[1] has been deleted inside Scholar.Items. The following does work, however:
var item = Scholar.Items.get(1);
Scholar.debug(item);
Scholar.DB.query("DELETE FROM items WHERE itemID=1");
Scholar.Items.reloadAll();
var item = Scholar.Items.get(1);
Scholar.debug(item);
Now item is properly undefined. Moral of the story: object references need to be deleted manually after receiving delete notifications, for if external code still has references to deleted data objects, the data layer can't do much about it.
Revised the way the dateAdded and dateModified columns are displayed.
Views are now unregistered on window close.
A few functions in overlay.js were renamed.
Began better commenting of the interface code.
Although I do not yet have interface code for it, collections do not have any sort of setName(), save() functions.
Collections list now implements notify()
Made some fixes on itemTreeView to allow for notify() on multiple deletes
(shifts the row identifier up 1 for each previous delete, etc.)
One more thing -- this might not happen because the tree calls each Item.erase() or Collection.removeItem() individually.
Bug fix on notifier.js so that register*Tree() actually returns the hash. :-)
Unfortunately, unregister*Tree() does not seem to actually work, there is still an object at _observers['itemTree'][hash]
(temporary): When you select an item Scholar no longer loads a page into the browser.
The javascript bugs on those HTML pages were frustrating the debugger.
- Added Notifier triggers to Item.erase() (which only runs if not in a nested transaction) and Collections.erase()
- notify() in ItemTreeView updated with example of taking multiple ids, though it doesn't actually work (and notify() implementations may decide just to refresh the whole tree when ids.length>1 rather than dealing with changes individually)
- When deleting collections, use DB tables that actually exist
Scholar.Notifier framework to handle update notifications between data layer and interface
- Notifier.registerColumnTree(ref) and Notifier.registerItemTree(ref) pass back a unique hash that can be used to unregister the tree later with unregister*(hash) (and must be, lest we leak treeViews and probably entire browser windows if the browser window is closed without unregistering the treeView)
- Data layer calls Scholar.Notify.trigger(event, type, id) after various events (only two calls in the data layer at the moment--more coming later)
- Notify.trigger() calls notify(event, type, id) on all registered trees of the appropriate type -- the data layer usually knows what collection the action pertains to, but we decided that it's cleaner to just let the tree decide what it wants to do rather than add all that logic into the data layer)
(Note: Item collection adds appear to be buggy on the interface side, but removes seem to be working)
Overhauled data access layer to support new model:
- Changed Item.getParent to Item.getCollections() to get ids of parent collections
- Removed Item.setPosition() -- item positions are set internally when items are added to and deleted from collections, but the orderIndex is not currently used or manipulatable externally
- Item constructor/Items.getNewItemByType()/Items.add() no longer take folderID and orderIndex as parameters
- Split getTreeRows() into Scholar.getCollections(parent) and Scholar.getItems(parent), which return root collections or all library items, respectively, if no parent given
- All references to folders in object/method/property names changed to collections
- New methods Collection.addItem(itemID), Collection.hasItem(itemID), Collection.removeItem(itemID)
- Collection.erase() takes optional deleteItems parameter to delete items from DB -- otherwise just removes association and leaves item in library (does, however, delete all descendent collections from DB regardless)
* Note: This will break displaying of items until interface code is updated. *
Scholar.TreeView is now Scholar.FolderTreeView (SourcesTreeView later? depends)
Some changes to the way FolderTreeView works.
New Scholar.ItemGroup -- used by FolderTreeView and ItemTreeView.
the root folder is not currently displayed -- doesn't really matter because we are changing the way it all works.
(note: the Scholar.Items.getAll() function does not seem to be working correctly -- try clicking on the library)
Switching to scholar@chnm.gmu.edu -- SVN folks should rename the text file in their FF profile directories. (Used to be scholar@chnm, so you need to make the change even if you hadn't changed to the GUID yet.) Anyone who's tried the XPI yet will need to reinstall just once from the URL (http://chnm.gmu.edu/firefoxscholar/download/scholar-dev.xpi), as the old version won't find updates with its id.
* Renamed treeView.js to folderTreeView.js (more changes on this file forthcoming) -- this will soon be specifically for folders.
* Added itemTreeView.js, with new Scholar.ItemTreeView class -- specifically for items.
* The items list now supports 3 additional columns: rights, date added, and date modified.
* For now, selecting an item loads the "I'm Feeling Lucky" result of a Google search on the item title.
Added update.rdf for auto-update mechanism (which will get moved out of the extension root by the build system but should stay with a particular tree)
Assigned a new GUID to extension
*** Important: The extension GUID has changed, so you should rename the scholar@chnm text file in your Firefox profile exensions directory to our new GUID "{e007c81c-eb2a-4e1d-b381-d25a45cd11ed}" and restart Firefox ***
More details coming on Basecamp: http://chnm.grouphub.com/W161222
*** Important note: after checking out this code, be sure to delete the compreg.dat and xpti.dat files in your FF profile directory or else the extension will not load. They'll be regenerated when you start FF again, and you won't have to do it again unless we add interfaces or other components. Once I set up the XPI packaging system, this step won't be necessary. ***
- Localization properties are now loaded directly via nsIStringBundleService and available via Scholar.getString(name) -- removed stringbundle from XUL
- Updated status line in bottom right to reflect whether Scholar is loaded correctly or not (temporary, obviously)
Scholar.Folder.erase() -- deletes a folder and all descendent folders and items
Scholar.Folder._getDescendents() - returns an array of descendent folders and items (rows of 'id' and 'isFolder') ("private" for now--could be useful as public later)
Scholar.Folders.unload() - clear folder from internal cache (used by Scholar.Folder.erase())
DB transactions are now automatically nested if beginTransaction() is called when one is already in progress -- the queries aren't committed until the outermost commitTransaction() is called, which is handy for calling methods with their own transactions inside others when the entire process should be a transaction (e.g. Folder.erase() calling Item.erase() for the folder's descendents). Rollbacks are a little trickier -- currently, a rollback in a nested transaction flags the entire transaction for rollback when the outermost commit is reached, which for our purposes will probably be fine.
ScholarLocalizedStrings moved out of sidebar.js and into Scholar.LocalizedStrings
Rudimentary creator adding/editing. lots of things to work on, because it doesn't work.
Added Scholar.CreatorTypes with methods getTypes() (multi-dim array with 'id' and 'name') and getTypeName(creatorTypeID)
Fixed bug in Scholar.Creators.purge() causing SQL error when deleting a non-existent creator
Fixed incorrect field order for itemCreators INSERT queries in save()
Changed setCreator() to take empty creator names (causing DELETE in itemCreators but without shifting up creators below it (or down and above, depending on your perspective) like removeCreators())
The selection no longer lost or moved on folder opening/closing.
The erase functionality works, on Mac (Windows) use the delete (backspace) or forward delete (delete) keys.
unfortunately there are sometimes exceptions so I am not calling erase() on each item until we figure out what the problem is.
view._deleteItem() and view._insertItem() are now view._hideItem() and view._showItem() to better reflect functionality.
Scholar.ItemTypes.getTypes() -- returns associative array of all itemTypes, with 'id' and 'name'
Scholar.ItemTypes.getTypeName(itemTypeID) -- returns pre-l10n type name (e.g. 'book', 'journalArticle')
Removed getLevel()
Updated Object.setPosition() to handle setting the position of new objects without committing, so it can be used within save(). May or may not actually work--my money's on no.
Added Folder.isEmpty()
No Folder.setPosition() yet, but, then, no editing or saving of folders at all--coming soon
Added getLevel() functions to Scholar.Object and Scholar.Folder -- save() and setPosition() not yet updated
Updated sample data to include levels
Added orderIndex (from treeOrder table, now) back into Scholar.Object data
New Scholar.Objects method for treeView: getTreeRows() -- returns mixed array of all Folder and Object objects in the proper order, type testable with instanceof (e.g. if (obj instanceof Scholar.Folder))
Scholar.Objects.get() and getAll() return arrays hashed by objectID again, since they're no longer used directly by treeView
Scholar.Objects.get() now returns the object directly if only one argument and scalar (i.e. one id)
Updated schema and sample data to handle folder ordering
Added include.js for use in non-browser.xul windows to bring in the SCHOLAR_CONFIG and Scholar objects
Adjusted sidebar to use shared object
Added test line (displayable by clicking "Hello, World" before and after sidebar load to demonstrate cross-window access
I also started implementing some of the localization. There is a lot to do still, and I am still learning a lot of the Firefox extension technologies (XUL, adv. Javascript, etc.) but things are going great.
Fixed object retrieval queries to use orderIndex is 0 rather than 1, since that's the default now
Scholar_Objects.getAll() now uses loaded.push() rather than loaded[objectID], since _for (objectID in objects)_ isn't an option with treeview
Scholar_Object.getField() now returns an empty string rather than false for empty values
Added some new core functions:
- Scholar.varDump(), after PHP's var_dump()
- Scholar.flattenArguments(), to flatten mixed array/literal argument lists into a single array
- Scholar.join() -- a version of join() that operates externally, for use on, for example, the arguments object (safer than extending Object)
- Scholar.Hash, a slightly smarter associative array -- not perfect, but brings a proper length property and a few convenience methods (and allows for other additions) -- should probably be limited to places where the length property or other additional additions are needed, since its use is a little non-standard (e.g. you have to remember to do _for (i in arr.items)_ rather than just _for (i in arr)_, to use set(), etc.)
Moved schema file out of chrome and removed XML tags, since it will no longer be accessed by XMLHTTP
Changed schema updater to use XPCOM components rather than XMLHTTP
Converted DB class to a singleton named Scholar_DB
Scholar_DB.query() now returns associative array similar to mysql_fetch_assoc() for SELECT statements rather than a mozIStorageStatementWrapper -- the mozStorage executeDataSet() function doesn't work yet, so we do it this way
Added DB functions:
- beginTransaction(), commitTransaction(), rollbackTransaction()
- columnQuery() -- one column, multiple rows, returned as array indexed by row
- getColumns() and getColumnHash
DB query functions can now handle bind parameters passed as an array of object literals: e.g. [{'int':2},{'string':'foobar'}]
valueQuery() now returns an int for COUNT(*) queries, so the result can be tested without a "0" being true
Changed _initializeSchema to drop existing tables before creating new ones, hacked with try/catch until DROP TABLE IF EXISTS works with the mozStorage extension (it's already in the latest SQLite release)
Added DB_REBUILD config flag to manually trigger schema regeneration
Added debug logging at level 5 for all SQL queries
Updated sample data
