Exchanging Data with External Partners

Image File Data Organization and Data Transfer Guide

(This is how Imaging Services formats and organizes data to share with project partners.)

Providing digital images and associated metadata to external partners (e.g., funders, commercial publishers, other cultural heritage organizations)

Imaging Services will work with Harvard libraries, museums, and archives to share digitized content with external partners.

However, Imaging Services will not:

Produce images or associated metadata files in formats, or to special format-encodings or technical specifications required by external partners when these specification differ standard workflows developed to make digital content that is to be stored by Library's Digital Repository Service.
Responsibilities for transferring data, converting data, and the provision of any supplemental data need to satisfy external partner requirements are to be borne by the owning repository that has contracted with Imaging Services.

Description of files

Image files are encoded in compliance with the JPEG2000 standard
- Data compression: irreversible 9-7 wavelet transform for lossy compression, or reversible 5-3 wavelet transform for lossless compression.
- Photometric interpretation:
  - RGB (color), 8-bits per channel, embedded sRGB ICC profile.
  - Grayscale, 8 or 16-bits per channel.
Structural metadata files (optional) conform to the METS standard and Harvard's METS profile for page-turned objects.
MARC record files (optional) are in MarcXML or MODS format.
Packaging tag files generated by the packaging application (Bagit) describe the package.

File naming

Image file names consist of three or four components: [UNIQUE_ID]_[Sequence_number].jp2 or [UNIQUE_ID]_[Volume_number]_[Sequence_number].jp2

[UNIQUE_ID]: Usually this will be a catalog record ID¹. For example, a 9-digit HOLLIS ID (e.g. 011835322) will be recommended for an item that has been cataloged in Harvard’s HOLLIS catalog system. In the case of manuscript collections, the library or archives repository will provided a unique identifier (e.g., morgan_601_705_volIV).
[Volume_number]: (multi-volume sets only) 3-digit or 4-digit volume sequence number with a prefix 'v' (e.g. v005 or v0005).
[Sequence_number]: 4-digit or 5-digit page sequence number (e.g. 0045, 00045).
.jp2: File format (JPEG 2000, part 1) extension.

Examples

Single volume file name example: 010010723_0001.jp2
Multi-volume file name example: _008105127_v0007-METS.xml
Manuscript collection file name example: _morgan_601_705_volIV-METS.xml

MarcXML file names consist of two components: [HOLLIS_ID].xml²

[HOLLIS_ID]: Hollis system identifier (e.g. 011835322)
.xml: File format (XML) extension

Packaging tag file names (see next section – Use of the “bagit” file-packaging and –
interchange protocol).

Use of the “Bagit” file-packaging and -interchange protocol

The data files provided will be arranged and inventoried in accordance with the “Bagit” specification promoted by the Preservation Directorate of the Library of Congress.

To learn more about Bagit and to investigate the freely available tools available for checking the integrity of the Bagit-packaged data, we suggest you consult the following online resources:

A Library of Congress produced video designed to introduce the “Bagit” specification: http://www.youtube.com/watch?v=l3p3ao_JSfo
Opensource Bagit software tools: https://github.com/LibraryOfCongress
Wikipedia entry: https://en.wikipedia.org/wiki/BagIt

Organization of files and file system on portable media (i.e, portable hard drive)³

<root directory>
| bag-info.txt  
| bagit.txt
| manifest-md5.txt
| tagmanifest-md5.txt
|
\--/data/
    |
    \--/[BATCH ID] (see note 4)
        | _[UNIQUE_ID]-METS.xml(single volume METS file example:_007984492-METS.xml)
        | 
        | _[UNIQUE_ID]_[VOLUME_ID]-METS.xml(multivolume METS file example: _000652831_v0002-METS.xml)
        |
        | _[UNIQUE_ID]-METS.xml (manuscript collection METS file example: _morgan_601_705_volIV-METS.xml)
        |
        | _batch.xml(see note 5) (technical metadata file)
        |
        | [HOLLIS_ID].xml(see note 6) (MarcXML file, e.g., 000652831.xml)
        |
        \--/[UNIQUE_ID(see note 7)]/(manuscript collection example, e.g. morgan_601_705_volIV)
            |
            | [UNIQUE_ID]_[####].jp2
            | morgan_601_705_volIV_0001.jp2
            | morgan_601_705_volIV_0002.jp2
            | morgan_601_705_volIV_0003.jp2
            | morgan_601_705_volIV_0004.jp2
            ...
            | morgan_601_705_volIV_0099.jp2
        \--/[HOLLIS_ID]/(single volume monograph example, e.g. 007984492)
            |
            | [HOLLIS_ID]_[####].jp2
            | 007984492_0001.jp2
            | 0079984492_0002.jp2
            ...
            | 007984492_0099.jp2
        \--/[UNIQUE_ID]_[VOLUME_ID]/(see note 8) (multi-volume example, e.g. 000652831_v0002)
            |
            | [VOLUME_ID]_[####].jp2
            | 000652831_v0002_0001.jp2
            | 000652831_v0002_0002.jp2
            | 000652831_v0002_0003.jp2
            ...
            | 000652831_v0002_0099.jp2

NOTES

1. In cases where the record identifier includes space, the spaces will be replaced by underscores.			
2. MARCXML records are only available for items that have been cataloged in Harvard’s bibliographic database, HOLLIS.			
3. If more than one disk is needed, a batch may span more than one disk; the corresponding metadata files for the batch will appear on each disk.
4. Batch level identifiers are assigned to groups of titles prepared and submitted together for scanning. These named “batches” will be maintained from scanning all the way through deposit to Harvard's Digital Repository Service and transfer of data to project partners beyond the Harvard libraries.
Inclusion of technical metadata is optional.
5. Inclusion of MARCXML files are optional.
6. The title's unique identifier is used as the directory name.
7. Individual volumes or fascicles will be labeled using a two- or three-digit sequence number (e.g., v001, v002, v099, v123).