Commit graph

130 commits

Author SHA1 Message Date
Simon Kornblith
67f6ae3ed2 - closes #69, notification system for broken scrapers
- don't put "Page" before page in WaPo scraper
2006-09-09 19:47:47 +00:00
Simon Kornblith
d4576d3d55 addresses #69, notification system for broken scrapers
thanks to Dan for his help on the repository side of things
2006-09-09 00:12:09 +00:00
Simon Kornblith
60422e032e - closes #261, work around content-disposition: attachment on endnote links. this workaround is far from the most elegant, but it seemed nicer than writing a stream converter component that didn't really convert streams
- fixes bugs in RIS import
2006-09-08 22:26:59 +00:00
Simon Kornblith
7b7d3d85e3 - added Washington Post translator
- translation works properly even when a user has switched to a different page
2006-09-08 05:47:47 +00:00
Simon Kornblith
b8ddba3a67 CiteSeer translator 2006-09-08 01:59:22 +00:00
Simon Kornblith
5028880d38 closes #280, BibTeX translator
- fixes date bugs
- fixes (again) an issue that would cause the "unresponsive script" dialog to appear when importing or exporting
2006-09-07 22:10:26 +00:00
Simon Kornblith
cf8dc232b1 - new translators: New York Review of Books, Chronicle of Higher Education
- more useful errors in utilities
- fixes minor bugs in citation styling
2006-09-07 01:23:13 +00:00
Simon Kornblith
89cf0c7235 closes #276, fix RIS bugs
- import translators no longer fail when trying to import an item with no name
- the T2/BT field becomes the publication title when no JO/JF field is available (fixes newspaper issues)
- Y2 is now treated as part of the date if and only if it is improperly formatted (seriously, why can't Thomson get their own specs straight?)
- work around EndNote's strange behavior of putting article titles into notes for no apparent reason
- RIS export gives dates as per specification
- fixed a bug that could have (potentially) caused problems formatting "January"
- allow translators to access strToDate function
2006-09-06 04:45:19 +00:00
Simon Kornblith
b3bb6b9013 remove unnecessary debug code 2006-09-05 07:59:25 +00:00
Simon Kornblith
045780d9ac closes #250, figure out proper text encodings for import/export
MODS uses the encoding as specified in the <?xml tag, or else UTF-8
RIS uses IBM850, since the spec says "IBM Extended Character Set" and it's the only code page Mozilla supports. (should I do this? or just use unicode?)
MARC uses UTF-8, since I don't think there's any way to get full MARC-8 support, and UTF-8 is now the preferred encoding anyway
2006-09-05 07:51:55 +00:00
Simon Kornblith
cec35d7566 closes #272, problems with Library of Congress ingest 2006-09-05 03:06:22 +00:00
Simon Kornblith
e0f6f023d8 various fixes to citation formatting (mostly Chicago Manual of Style) 2006-09-05 01:09:04 +00:00
Simon Kornblith
7d93903e2d closes #239, fix embedded RDF translator 2006-09-04 21:43:23 +00:00
Simon Kornblith
370fe48388 - remove extraneous debug code
- update scrapers.sql version (do not put into the repository)
2006-09-04 20:21:38 +00:00
Simon Kornblith
aa6e2cfab1 closes #264, UMich lib catalog doesn't work on Windows; other issues related to Mirlyn
positions "saving item" window in a slightly better place on Windows

the UMich bug was actually bigger than I though. as it turns out, the HiddenDOMWindow in Windows is not a chrome window, so i had to modify createHiddenBrowser() to attach the hidden browser object to an existing browser window. i don't believe this should have any adverse effects for snapshots, etc., but Dan, correct me if i'm wrong. it would be nice to be able to create a real chrome instance instead of a XUL element, but all of my attempts at doing so have failed.
2006-09-04 20:19:38 +00:00
Simon Kornblith
2b0bebe7a4 closes #258, MARC translator should capitalize titles 2006-09-04 18:16:50 +00:00
Simon Kornblith
e5404f4938 closes #269, For some COinS pages "could not save item" error 2006-09-04 17:37:07 +00:00
Simon Kornblith
0ab9e8b36c references #268, occasional problems with ingest of pages with multiple references
i've fixed the Amazon.com bug (i think) and made the translator show a "Could Not Save Item" prompt rather than show an empty list, but if you see any other pages where this happens, let me know
2006-09-04 17:09:44 +00:00
Simon Kornblith
ed6650c4e7 closes #218, Windows support for Word integration. this solution seems to work with both Word 2003 and Word 2007. i have not tested with earlier versions. Zotero.dot is the Windows verison; Zotero.dot.dmg is the Mac version. the only difference is the function call used to perform SOAP requests.
to get this to work right, you'll need the SOAP toolkit from http://www.microsoft.com/downloads/details.aspx?FamilyID=ba611554-5943-444c-b53c-c0a450b7013c&DisplayLang=en
I may replace the SOAP object with a simple XMLHTTP object, since that page says that the SOAP toolkit is deprecated.
2006-09-04 08:06:04 +00:00
Simon Kornblith
10f4b28c63 closes #214, add footnote support to word integration
closes #215, allow user to select desired citation style and change citation styles on the fly
2006-09-04 04:13:12 +00:00
Simon Kornblith
59a1628e5b fixes #254, NY Times scraper fails (thanks Sean) 2006-09-01 02:45:31 +00:00
Simon Kornblith
6f885c9cb0 make Amazon.com translator work on book pages linked from other book pages 2006-08-31 22:36:05 +00:00
Simon Kornblith
438ff82955 - replace storage streams with plain old strings for translate IO. there's not much of a reason to use storage streams now, and it was screwing up non-ASCII characters.
- make EBSCO scraper work better through a proxy
- shorten Accession Number -> Accession No, Journal Abbreviation -> Journal Abbr, Publication Title -> Publication. it does look a bit stranger, but it also makes the interface more functional (especially for those of us without giant widescreen LCDs ;-)
2006-08-31 07:45:03 +00:00
Simon Kornblith
4b756d700b fixed an issue that could prevent MARC fields below 100 (ISBN and call number) from appearing in records 2006-08-31 05:21:41 +00:00
Simon Kornblith
1c8e3fcb02 closes #239, fix embedded RDF translator
modifies scrapers to use dates in the format that comes out of the page, rather than converting to SQL
adds Scholar.Date.formatDate() to provide a pretty representation of dates
2006-08-31 00:04:11 +00:00
Simon Kornblith
0cd3021cf3 closes #241, improved date handling
- Scholar.strToDate() accepts a string date and returns an object containing year, month, day, and part
- capture access date whenever URL is captured
- updated Zotero.dot to use new namespaces
2006-08-30 21:56:52 +00:00
Simon Kornblith
27617ee152 closes #236, Export windows should offer a default filename with extension
closes #238, present dialog if no import translator is available for a file
closes #240, change XML namespaces
2006-08-30 19:57:23 +00:00
Simon Kornblith
a75c5df70c - add MLA style
- put text-only version of bibliography on the clipboard, in addition to HTML version (Windows-only).
2006-08-30 06:12:26 +00:00
Simon Kornblith
1c21bddbfc - modifications to citation engine's handling of localized strings
- added the missing integrationDocPrefs.xul file
2006-08-30 04:00:19 +00:00
Simon Kornblith
68c480b7b5 - closes #232, University of Michigan library site does not work
- improved handling of scraper errors (hopefully, the hanging should be gone)
2006-08-30 01:41:51 +00:00
Simon Kornblith
d8171f775c closes #223, citing the same item multiple times should produce only one bibliography entry 2006-08-29 17:29:35 +00:00
Simon Kornblith
0c24beee3f closes #213, add in-text citation support to citation engine
fixes date and et al. handling bugs in citation engine
permits citation of multiple items in Word integration
2006-08-29 04:24:11 +00:00
Simon Kornblith
7385ba4df2 ABC-CLIO fixes 2006-08-26 21:57:02 +00:00
Simon Kornblith
d3fc9866b9 - add ABC-CLIO (America: History and Life) translator
- fix a potential issue with COinS support
2006-08-26 21:36:49 +00:00
Simon Kornblith
a457cdb493 added New York Times translator 2006-08-26 07:27:02 +00:00
Simon Kornblith
ddb754839c make google scholar translator turn on export automatically 2006-08-26 06:04:29 +00:00
Simon Kornblith
72da6e412e add Google Scholar translator 2006-08-26 05:51:41 +00:00
Simon Kornblith
53aae7751c support FirstSearch databases besides WorldCat 2006-08-26 04:59:30 +00:00
Simon Kornblith
f07cb5a5bc adds an InfoTrac OneFile translator
fixes a bug in ingester progress window handling
2006-08-26 03:50:15 +00:00
Simon Kornblith
0e63958f96 - make proquest work better behind proxies
- improved frame support
2006-08-24 18:00:48 +00:00
Simon Kornblith
04d05548b2 closes #103, figure out how to store captured pages in native export format
fixes ampersands in citation COinS
fixes tags and seeAlso in import/export (should now work for all items)
2006-08-20 04:35:04 +00:00
Simon Kornblith
a55b035761 update scrapers.sql version (oops) 2006-08-19 23:15:38 +00:00
Simon Kornblith
94bd2415da adds short roles to CSL (Ed. instead of Editor)
adds COinS to exported HTML
uses real lists in HTML output
fixes other small citation style issues
2006-08-19 23:14:27 +00:00
Simon Kornblith
26668a6e73 closes #194, EBSCO translator
closes #160, cache regular expressions
closes #188, rewrite MARC handling functions

MARC-based translators should now produce item types besides "book." right now, artwork, film, and manuscript are available. MARC also has codes for various types of audio (speech, music, etc.) and maps.
the EBSCO translator does not yet produce attachments. i sent them an email because their RIS export is invalid (the URLs come after the "end of record" field) and i'm waiting to see if they'll fix it before i try to fix it myself.
the EBSCO translator is unfortunately a bit slow, because it has to make 5 requests in order to get RIS export. the alternative (scraping individual item pages) would be even slower.
regular expression caching can be turned off by disabling extensions.scholar.cacheTranslatorData in about:config. if you leave it on, you'll have to restart Firefox after updating translators.
2006-08-19 18:58:09 +00:00
Simon Kornblith
20486d5053 addresses #103, figure out how to store captured pages in native export format
import/export of file data should work for all file types _except_ snapshots (in this situation, export is working, but import is not yet complete; see #193)
also, fixes a potential security issue that could have allowed malicious web translators to post local data to remote sites (although, given we maintain the central repository and there's no easy way to install a translator, the risk would have been minimal to begin with).
2006-08-18 05:58:14 +00:00
Simon Kornblith
10ba568ee8 closes #39, auto-ingest of associated files (as recognizable)
closes #3, Overflow metadata dumps into "extra" field

add "extra" data where such data is useful and conveniently accessible (not available for XML-based export or MARC formats yet)
add links to permanent URLs
download associated files from full text sources (if extensions.scholar.downloadAssociatedFiles preference is enabled)
fix WorldCat translator
improve InnoPAC translator (it now works on Georgetown search results pages, albeit slowly, because it must first realize the catalog is misconfigured)
tag items from SIRSI and WorldCat
return to putting the full lengths of books into "pages," because some citation styles require it
fix COinS (broken a few revisions ago)
2006-08-17 07:56:01 +00:00
Simon Kornblith
51108446e3 closes #187, make berkeley's library work
closes #186, stop translators from hanging

when a document loads inside a frameset, we now check whether we can scrape each individual frame.
all functions involving tabs have been vastly simplified, because in the process of figuring this out, i discovered Firefox 2's new tab events.
if a translator throws an exception inside loadDocument(), doGet(), doPost(), or processDocuments(), a translate error message will appear, and the translator will not hang
2006-08-15 19:46:42 +00:00
Simon Kornblith
dac5bbb3f3 closes #183, export bibliography to RTF 2006-08-14 21:54:45 +00:00
Simon Kornblith
feff0aa531 closes #53, export to footnote or bibliography
closes #180, make all contextual menu export/create bibliography options work right

also:
- add Chicago Note style output
- unregister RDF data sources from cache after import
2006-08-14 20:34:13 +00:00
Simon Kornblith
bb07710b34 okay, i actually tested it this time, so i'm pretty sure i got all the bugs out. no, i am not just cheating to get closer to the illusive r500. 2006-08-14 05:23:39 +00:00