Reimagining Discovery
- 1 Vision
- 2 Project Goals
- 3 Problem and Value Statements
- 4 Alignment with Harvard Library Multi-Year Goals and Objectives
- 5 Scope
- 5.1 In Scope
- 5.2 Out of Scope
- 6 Deliverables and Work Products
- 7 Definition of Done
- 8 Stakeholders
- 9 Technical Project Team
- 10 Estimated Schedule
- 11 Assumptions, Constraints, Dependencies, and Risks
- 11.1 Project Assumptions
- 11.2 Project Constraints
- 11.3 Project Dependencies
- 11.4 Project Risks
- 12 Acceptance
Vision
Revolutionize how researchers, students, and the global community access and explore Harvard's extensive collections, making all kinds of information easily discoverable and accessible.
Project Goals
Enhance user experience
Improve discovery and accessibility of special and archival collections and all types of digital collections including but not limited to image, text, audio, video, born digital, immersive (3d, XR, VR, MR), GIS, etc.
Integrate distinct digital collections discovery platforms, including developing a new one
Investigate and use AI-powered tools to enhance user experience and metadata
Problem and Value Statements
Problem Statement
Since its founding, Harvard Library has been a guardian of the University’s memory and a gateway to the world's knowledge. We currently host an array of discovery systems that use different design approaches, organizational priorities, and technology standards. Scholars and the public expect to be able to find trustworthy information and discover resources easily regardless of the system that is managing and providing access to it.
Solution Business Value
By enabling rich cross-collection search, this project will offer end users intuitive, contextual discovery of special collections, archives and digital collections, through a mix of conversational interfaces, browsing that emphasizes the visual nature of materials when appropriate, and recommendations for similar or related resources, all informed by ongoing user research.
Alignment with Harvard Library Multi-Year Goals and Objectives
This projects aligns with FY 24 HL Goals:
Diversify and expand access to knowledge
Maximize the breadth of tangible and digital collections across Harvard and peer institutions, for the benefit of all partners
Increase our focus on acquiring, accessing, and creating digital content that is accessible to all, as open as possible, and permits creative uses of collections as data
Invest in open access infrastructure and services that support equitable, sustainable models for scholarly communication and open knowledge
Scope
In Scope
Cross collection search for special and archival collections, focusing on the end user experience and making clear the relationships between archival objects/items and larger collections. Deliverables: ArcLight/new HOLLIS for Archival Discovery UI & Collections Explorer.
Incorporate AI/ML technologies to offer natural language search, and generative AI features like summarization, while retaining baseline search and browse functions. Deliverable: Collections Explorer.
Access to digital content, and act as a replacement HOLLIS for Images and Harvard Digital Collections, extending their use cases to meet project goals: full text searching, born digital, GIS, A/V. Deliverable: to be agreed upon at end of Year 1.
Reimagine metadata pipeline using new technologies from AI/ML. Deliverable: Requirements gathering and scoping for a metadata hub.
Out of Scope
Discovery and access to licensed resources (articles, databases) and general collections
Deliverables and Work Products
Key Tasks and Outcomes
View our Sprint Outcomes and Demos
Definition of Done
Discovery platform, including access to digital assets, is released on production environment and in use by Harvard constituents and the public.
Stakeholders
Executive Stakeholders | Title |
|---|---|
Martha Whitehead | VP for Harvard Library and University Librarian |
Stu Snydman | AUL & Managing Director for Library Technology Services |
Salwa Ismail | AUL for Discovery and Access (Jan. 2025) |
Tom Hyry | AUL for Archives and Special Collections |
The Library Stakeholders are acting as an extended project team, meeting weekly to help inform and prioritize the work.
Library Stakeholders | Title |
|---|---|
Amy Deschenes | Head of UX and Digital Accessibility |
Kai Fay | Discovery & Access Strategic Projects Manager |
Adrien Hilton | Director of Technical Services for Archives and Special Collections |
Chelcie Rowell | Associate Head of Digital Collections Discovery |
Shalimar Fojas White | Herman & Joan Suit Librarian, Fine Arts Library |
Student interns, as needed | Harvard grad and undergraduate students |
Technical Project Team
Team Member | Title | Project Role(s) |
|---|---|---|
Enrique Diaz | Manager of Library Software Engineering | Product Owner (LTS) |
JJ Chen | Digital Library Data Engineer | Developer (LTS) |
Chris Axon | Senior Digital Library Software Engineer | Developer (LTS) |
Maura Meagher | Associate UX Developer | Developer (LTS) |
Angela Kilsdonk | Senior IT Project Manager | Project Manager/ Scrum Lead (LTS) |
Jenny Rae Bailey | UX Researcher | UX Researcher/Designer (HL) |
Past Contributors
Team Member | Title | Project Role(s) |
|---|---|---|
Katie Amaral | Technical Project Lead | Developer, Architecture (LTS) |
Meg McMahon | UX Researcher | UX Researcher/Designer (HL) |
Carolyn Caizzi | Senior IT Project Manager | Project Manager/ Scrum Lead (LTS) |
Doug Simon | Senior Digital Library Software Engineer | Developer (LTS) |
Estimated Schedule
Project is managed by using the Scrum framework and these phases/milestones will be adjusted. Below is a a high level schedule.
Phase | Phase Start | Phase End | Completion Milestone |
|---|---|---|---|
1 | July 2024 | September 2024 | Natural language discovery platform with generative AI features for discovering digitized, special and archival collections is built and released to QA for testing. |
2 | October 2024 | December 2024 | Platform is tested by end users and improvements are recommended. Research into scaling platform for production is completed. Data pipeline is scoped and work begins. Design process for digitized collections (images) component is completed. |
3 | January 2025 | March 2025 | Data pipeline and digitized collections components begin to be built. Consulting with vendor for vector database begins. First iteration of pipeline with Finding Aids and index is created. Decision to soft launch discovery platform is made depending on data pipeline. DECISION: product will not be ready for a soft launch by end of March. |
4 | April 2025 | June 2025 | Finalize data pipeline for Finding Aids and digitized collections components and release to QA (include Alma records for collections that need mediated access if time allows). Platform is monitored for costs and analytics are gathered and reviewed to plan for full launch September 2025. |
5 | July 2025 | September 2025 | Search relevancy evaluation and tuning; LLM evaluation and implementation of final selection. LLM guardrail implementation. Enhancing system security and observability. Production launch September 15. |
6 | October 2025 | December 2025 | Build out pipelines for Alma special collections records. Build out functionality for image records, including front end enhancements, authentication. Continuous improvement of the platform. Metadata Hub discovery completed. |
7 | January 2026 | March 2026 | Finalize pipeline for Alma special collections records and deliver record ingest. Implementation of Item Detail Page in Collections Explorer. Enhancements of search result relevancy. Metadata Hub development begins. |
8 | April 2026 | June 2026 | Enhancements of search result relevancy. Build out pipeline for image records. Implementation of Image Detail Page, including Viewer and thumbnails. Enhancements to search results and filters. Metadata Hub planning. |
9 | July 2026 | September 2026 | Build out pipeline and release of JSTOR Digital Stewardship image records in Collections Explorer. Release of new Collections Explorer front end design based on the new Harvard Library design system. |
10-12 | Year 3 will focus on building out Metadata Hub, as well as continuously improve the Collections Explorer platform. Investigation into and possible rollout of workflows for using AI to improve quality of metadata. |
Assumptions, Constraints, Dependencies, and Risks
Project Assumptions
Stakeholders either have or have identified the appropriate subject matter experts to advise on prioritization of work and other project matters
Stakeholders will have made available the time required to participate in project activities and to complete tasks as requested
Project sponsor and other stakeholders are empowered to make the decision required for the project to be a success
Project sponsor will provide written approval to move forward with system development when requested as part of incremental/iterative system demonstrations
Project Constraints
Scope - Flexible (all types of digital collections depends on unknowns)
Time - Fixed 3 year project
Cost - Fixed 3 year budget
Project Dependencies
ArcLight implementation project
Media Presentation Service upgrade
LibraryCloud reimagine or defining a new data pipeline
DRS Futures project
Rapidly changing LLM industry
Project Risks
Description | Plan | Impact | Owner |
|---|---|---|---|
Rapidly changing Generative AI space | Build system to be flexible, swap out models easily | Cost, trust | Technical Project Team |
Library metadata quality is varied and semantic retrieval works with unstructured data | See if metadata fields can help the quality of embeddings; experiment with different embedding models, focusing on full text content and multi-modal models for digital images | Quality of retrieval | Metadata creators and Technical Project Team |
Unexpected changes to other library systems like Aeon, JSTOR Forum | Account for and expect changes from external systems in design of data pipeline | Timeline delays | Technical Project Team |
Staff capacity to support work of the project | Meeting weekly with stakeholders to ensure there is enough time to plan for bouts of work that include time from broader staff | Overall project success | Library Stakeholders |
Acceptance
Accepted by: Library Stakeholders August 8 2024. In scope deliverables confirmed and defined April 3 2025 by Library Stakeholders and AUL Executive Sponsors.
Prepared by: Carolyn Caizzi
Effective Date: August 9 2024