Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

This is an umbrella page for project charters specific to media transformation tools work as part of the DRS Integration work. Full charters are developed for more specific projects in sub-pages.

I. Problem/Value Statement

...

Part of the DRS Futures project consists of optimizing the data pipelines that feed the LTS delivery services and creating deliverable images copies of files that are separate from their archival and production masters, using a format that is optimized for delivery rather than for long-time preservation. This , combined with an extensive remediation of images that are currently undeliverable or very inefficient because of the way they were originally generated, should result in a drastic reduction of error rates and resource consumption of media delivery services.

Strictly related to improving existing substandard images is resolving their root cause, which is the workflows with which these images were produced and introduced into DRS. Many of these images were Many deliverable files are currently produced by individual departments with ad-hoc developed or outsourced tools, without a central oversight on the quality of the resulting derivatives. A central service for creating derivatives specifically for delivery using community best practices and standards would drastically improve both the overall published media quality and Harvard staff's content production workflows.

Business Value

Creating images delivery copies specifically optimized for delivery would have several advantages:

...

b. Archival and production master images files would have a clearly defined purpose and their quality control would be in the content managers' hands, while the quality optimization of delivery images files would be guaranteed controlled by LTS.

c. If a better encoding for non-archival images delivery files were found in the future or requirements for delivery formats change, delivery images files could be recreated from their masters, without the need to change the latter.

Moreover, campus-wide adoption of an LTS-managed, centralized image file conversion tool to consistently produce high quality images deliverables at high speed would remove the need for individual department departments to come up with their own bespoke solution solutions for conversion, and would make it possible to completely automate the derivative generation, further relieving HL staff DRS stakeholders from repetitive and error-prone work.

II. Vision and Approach

We have identified a format into which we want to convert all the still images in DRS, starting from the most problematic ones. This format is High Throughput JPEG2000 (HTJ2K), a quite recent standard that is supported by a few encoding and decoding tools. (Note: this conversion would leave the archival images in DRS unchanged, it would only replace the delivery images.)[ To Do ]



Our vision aligns with the following Harvard Library multi-year goals and objectives (MYGOs):

  • MYGO #8: Focus technical services on effective workflows and metadata that matter the most
    • By offering a centralized service for efficient, high-quality, and scalable processing of highly visible (public) images files that can be used across campus we would encourage discontinuing one-off solutions that individual departments have been developing for lack of a better alternative, incurring in additional maintenance costs and inconsistent, often substandard output quality.
  • MYGO #10: Focus on space as a service, considering the most cost-effective approaches to user interests, collections security and preservation, and staff needs in HL and HCL facilities
    • HTJ2K is the most space- and computationally efficient format for Web quality images today. By converting existing lossless DRS images into HTJ2K we would significantly reduce storage, computing and I/O usage on the most traffic-heavy (and expensive) tier of our infrastructure.[ To Do ]
  • MYGO #14: Minimize the environmental impact of collections, services, and spaces
    • As described in MYGO #10, the goal of this project is to save computing resources, thus reducing the environmental footprint of our services.

...

III. In Scope/Out of Scope

In Scope

  • Developing a plug-in for the Vips image library that uses Kakadu for reading and writing JPEG2000 images.
  • Integrating the plug-in into the currently developed imgconv project.
  • Maintaining the plug-in up to speed with future Vips upgrades
  • Documentation and tests for the developed code

Note that this project is limited to still images. Other media may need a different workflow and approach.

Out of Scope

The following items are out of scope because they are achievable without this project; however, an optimized HTJ2K converter would significantly improve their quality:

  • Development of a microservice for converting images based on configurable profiles (imgconv)
  • Integration of imgconv into an automation framework for large scale processing (drs-pipelines)
  • Conversion of defective and/or substandard delivery images into delivery-optmized HTJ2K

IV. Deliverables and Work Products

  • Code, documentation and tests for a Kakadu plugin for Vips in a Git repository.

Definition of Done

This project will be considered done once:

  • The Vips Kakadu plugin code is completed and committed to a Harvard-owned Git repository
  • We are consistently able to compile and run the plugin
  • Comprehensive tests are written for the key functions
  • All tests pass
  • Exhaustive relevant documentation is provided
  • We are able to integrate the delivered code into our imgconv project and verify that the features and options satisfy our needs.

V. Stakeholders

...

VI. Project Team

...

Software engineers

...

QA

...

Stefano Cossu, Brian Hoffman

...

Functional documentation

...

John Cupitt

...

Scrum Master

...

Stefano Cossu

...

Project Manager

...

Vitaly Zakuta (LTS)

* Pierre-Anthony Lemieux deferred to John Cupitt for carrying out the development, remaining available for advising on Kakadu-specific topics. We have not yet established whether such consulting will be pro bono or for a fee. In the latter case, we should request a cost estimate, but ideally we would like to have one contractor billing for the whole project.

VII. Estimated Schedule (tentative)

...

Development & Release

...

Develop code

...

Development & Release

...

Unit and integration tests

...

Development & Release

...

Complete & validate documentation

...

Development & Release

...

imgconv integration and release

VIII. Assumptions, Constraints, Dependencies, and Risks

Project Assumptions

    • The code delivered by the contractor will be agnostic to external integrations.
    • Integration with imgconv and DRS pipelines, including deployment infrastructure and long-term maintenence, will be the DRS Futures engineering team.
    • Stakeholders will be available to participate in project activities and to complete tasks as requested.
    • The Executive Sponsor and other stakeholders are empowered to make the decisions required for the project to be a success.

Project Constraints

    • Contractor availability
    • DRS Futures team availability
    • Scope
    • Time
    • Budget

Project Dependencies

    • The plugin developer will be need a Kakadu SDK license to perform development, testing, and long-term maintenance of the requested software. Kakadu Software has agreed to provide John Cupitt a free and renewable Kakadu SDK license. The handing-off of that license is underway.

Project Risks

...

IX. Acceptance

Accepted by  [ TODO  ]

Prepared by Stefano Cossu

...

  • [ To Do ]


Out of Scope

  • [ To Do ]