Page Comparison

...

I. Problem/Value Statement

Problem Statement

A part Part of the DRS Futures project consists of optimizing the data pipelines that feed the LTS delivery services and creating deliverable images that are separate from their archival and production masters, using a format that is optimized for delivery rather than for long-time preservation. This, combined with an extensive remediation of images that are currently undeliverable or very slow inefficient because of the way they were originally generated originally, should result in a drastic reduction of error rates and improvement resource consumption of media delivery timingsservices.

We have identified a format into which we want to convert all the still images in DRS, starting from the most problematic ones. This format is High -Troughput JPEG200 Throughput JPEG2000 (HTJ2K), a quite recent standard that is supported by a few encoding and decoding tools. (Note: this conversion would leave the archival images in DRS unchanged, it would only replace the delivery images.)

The leading solution for HTJ2K in terms of encoding speed and efficiency is Kakadu, a proprietary software that LTS is currently licensed to use and is using currently engaging to generate traditional JPEG2000 images. Kakadu, however, has a few disadvantages when it comes to bulk conversion: it offers command-line tools that were never meant to be used in large-scale batch jobs. Additionally, these tools can only convert JPEG2000 to or from another format, not from JPEG2000 to JPEG2000. Since we want to convert old-style JP2 to HTJ2K, we would have to convert each image twice, effectively doubling the conversion time. Multiplied by many millions of images, this can have a dramatic effect on the time scale of the project.

...

The most flexible and efficient image library is Vips, which comes with Python bindings and command-line utilities. We have reached out to the Vips maintainer and proposed to add a plug-in to support JPEG2000 read/write using Kakadu. Vips has currently a similar functionality that uses OpenJPEG, which uses much slower agorithmsalgorithms. Happily, the Vips developer agreed to contribute to the development of such plugin and keep it compatible with future Vips releases.

...

Our vision aligns with the following Harvard Library multi-year goals and objectives (MYGOs):

• MYGO #8: Focus technical services on effective workflows and metadata that matter the most
  • By offering a centralized service for efficient, high-quality, and scalable processing of highly visible (public) images that can be used across campus we would encourage discontinuing one-off solutions that individual departments have been developing for lack of a better alternative, incurring in additional maintenance costs and inconsistent, often substandard output quality.
• MYGO #10: Focus on space as a service, considering the most cost-effective approaches to user interests, collections security and preservation, and staff needs in HL and HCL facilities
  • HTJ2K is the most space- and computationally efficient format for Web quality images today. By converting existing lossless DRS images into HTJ2K we would significantly reduce storage, computing and I/O usage on the most traffic-heavy (and expensive) tier of our infrastructure.
• MYGO #14: Minimize the environmental impact of collections, services, and spaces
  • As described in MYGO #10, the goal of this project is to save computing resources, thus reducing the environmental footprint of our services.

...

• The Vips Kakadu plugin code is completed and committed to a Harvard-owned , public Git repository
• We are consistently able to compile and run the plugin
• Comprehensive tests are written for the key functions
• All tests pass
• Exhaustive relevant documentation is provided
• We are able to integrate the delivered code into our imgconv project and verify that the features and options satisfy our needs.
  

V. Stakeholders

VI. Project Team

* Pierre-Anthony Lemieux deferred to John Cupitt for carrying out the development, remaining available for advising on Kakadu-specific topics. We have not yet established whether such consulting will be pro bono or for a fee. In the latter case, we should request a cost estimate, but ideally we would like to have one contractor billing for the whole project.

VII. Estimated Schedule (tentative)

VIII. Assumptions, Constraints, Dependencies, and Risks

...

• • The code delivered by the contractor will be agnostic to external integrations.
    
  • The plugin developer will be provided a free and renewable Kakadu SDK license to perform development, testing, and long-term maintenance (this process is currently underway).
  • Integration with imgconv and DRS pipelines, including deployment infrastructure and long-term maintenence, will be the DRS Futures engineering team.
  • Stakeholders will be available to participate in project activities and to complete tasks as requested.
  • The Executive Sponsor and other stakeholders are empowered to make the decisions required for the project to be a success.

Project Constraints

• • Contractor availability
  • Team DRS Futures team availability
  • Scope
  • Time
  • CostBudget

Project Dependencies

• • The new v3 IIIF manifest design, required for development of Viewer architecture and functionality, is not yet available for informing development of the upgraded Viewer application and plug-insplugin developer will be need a Kakadu SDK license to perform development, testing, and long-term maintenance of the requested software. Kakadu Software has agreed to provide John Cupitt a free and renewable Kakadu SDK license. The handing-off of that license is underway.

Project Risks

IX. Acceptance

Accepted by  [ TODO  ]

...