Creating Video Object Batches

Accepted Formats

See: https://harvardwiki.atlassian.net/wiki/x/zoHeAg##FormatsSupportedbytheDRS-VideoFormats

Note: Video files intended for delivery in Harvard SDS streaming delivery service should be in Quicktime format using the H.264 video codec and the AAC audio codec and have a file extension ".mp4".

Methods of creating video batches using Batch Builder

There are four methods you can use to create video objects in Batch Builder:

  1. Manually build objects from template. This is best when you have only one or two video objects to work with. Point and click to create a new batch and then create new objects.

  2. Automatically build objects from template. This is best when you have three or more video objects to work with. In this scenario BB generates object structure for each of your objects and moves files into object directories. Before this can happen you need to rename your files to add special "object name" file prefixes or provide mapping files that associate files with their objects.

  3. Manually build objects without a template. In this method, rather than using the template, you fill out object and file metadata and create object directories for each individual object one-object-at-a-time. As this method is rarely used in workflows, it is not currently documented.

  4. Use Batch Builder advanced video functionality to build an advanced video batch. Use this method if you plan to deposit video files along with additional process files and video captions. See Video Advanced CLI workflow for the procedure.

Manually build objects from template

This is a description of the procedure to manually create a batch of Video objects from template in Batch Builder 2 (BB). Use this method when only Video objects need to be created, processed in Batch Builder (BB) and deposited into the DRS. If you have many (over 10) objects to create, use the procedure to build objects automatically.

Procedure summary:
  1. Create a new project.
  2. Enter DRS deposit settings.
  3. Create object template.
  4. Add directories to template.
  5. Create new object batch.
  6. Move/copy files into corresponding directories on disk.
  7. Process the batch.
  8. Upload the batch to your dropbox
  9. Check the email report
What you need before you start:
  • Prepare several video files and put them in a directory of your choice somewhere on local hard disk or network drive. If you have more than one video file per recording (e.g.: an archival master and a deliverable or an archival master, a production master and a deliverable) make sure file names match (e.g.: video-1.mov, video-1.mov and video-1.mp4). Consult Naming Rules for more information on file and directory naming.

  • Decide what you will use for Owner Supplied Names for your objects and the files they consist of. For instance, you could use local classification or local accession numbers. See Naming Rules for more information about owner supplied names.
Procedure:
  1. Create a new project.

    1. Open BB by double clicking on the executable.
    2. From the Project menu select New.
    3. In the dialog enter Project Name (used for internal tracking); select the directory on disk in which the project will be saved (click the ellipsis button to browse to it on disk); select Content Model "video" from the dropdown; click OK to continue.

  2. Enter DRS deposit settings.

    1. Click on "Deposit Settings" in Project Panel tree.
    2. Enter deposit data in the form:

      Batch Name Pattern: default - {owner}{batchDir}{yyyy}{mo}{dd}_{hh24}{mm}{ss}

      (This is the name for a batch that appears on DRS deposit reports. Batch name must be at least 3 characters long.)

      Success Email: type email and press enter.
      Failure Email:
      type email and press enter.
      Success Method:
      choose how you will receive load report.
      Deposit Agent:
      type your HUID.
      Deposit Agent Email: type your official Harvard email address.

  3. Create object template (called "batch template" in BB 1).

    1. Select "Object Template" in Project Panel on the left and then enter information in the Content Panel on the right.
    2. Enter Owner Code for objects you will be depositing (use all caps).
    3. Enter Billing Code for objects you will be depositing (use all caps).
    4. Select DRS Access Flag value for objects you will be depositing.
    5. Enter the URN Authority Path for your objects (Look up your Path here if needed: http://nrs.harvard.edu/urn-3:hul.ois:nrsstatusprod) – use all caps.
    6. Enter URN Resource Name Pattern.
    7. (Optional) switch to "optional" tab to enter any optional metadata (for example descriptive metadata, roles or/and relationships that will apply to your objects). For more about optional metadata see Sections 8. Adding Relationships, 9. Adding Descriptive Metadata, 10. Adding Supporting Content.

  4. Add directories to template.

    If there is one video file per object
    (e.g.: one file that serves as archival master and as deliverable)

    1. Right click on "Object Template" in the Project Panel on the left and add a directory and select the 'add a directory…' menu option
    2. Type in a name for your directory in the text box. The directory name starts with "video", but the name you type will be appended to it. Tip: to make it easier to read, start your directory name with an underscore (_). So, typing '_mov_files' will result in a directory called 'video_mov_files'.
    3. In the Project Panel on the left select the directory you just created.
    4. In the Content Panel on the right, choose "yes" from the dropdown list for the field "First Generation in DRS".
    5. Choose "HIGHUSE" for Usage Class.
    6. In the Content Panel switch to Optional tab, scroll down to video File Metadata and use Ctrl Click on your keyboard to select role ARCHIVAL_MASTER and role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.

    If there are two or more video files per object
    (e.g.: a separate file for archival master and a separate file for deliverable)

    Create a directory for an archival master and nest one or more deliverable directories inside it (this is needed for BB to determine the derivative relationship).

    1. Right click on "OBJECT TEMPLATE" in the Project Panel on the left and select the 'add a directory …' menu option.
    2. In the text box, type '-archival_master' or a similar suffix that describes the role of your directory. The directory name will start with "video" but the name you type will be appended to it. So, typing '-archival_master' will result in a directory called 'video-archival_master'
    3. In the Project Panel on the left select the directory you just created.
    4. In the Content Panel on the right choose "yes" from the dropdown list for the field "First Generation in DRS".
    5. Choose "LOWUSE" for Usage Class.
    6. In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role ARCHIVAL_MASTER.
    7. In the Project Panel on the left, right click on the directory you just created and select the 'add a directory …' menu option.
    8. In the text box, type '-deliverable' or a similar suffix that describes the role for your directory.
    9. In the Project Panel on the left, select the directory you just created.
    10. In the Content Panel choose "no" from the dropdown list for the field "First Generation in DRS".
    11. Choose "HIGHUSE" for Usage Class.
    12. In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
    13. If you have more deliverables that are derived from the archival master right click on the topmost video directory you created, add a custom suffix if needed and follow steps g-i once again.

  5. Save your work (Project > Save on main menu)

  6. Create new object batch.

    1. Select "Batch>New" from the main menu. Enter a name for the batch (will be used as the batch directory name on disk) Click OK.
      Tip: including the word "batch" in your batch name will help you remember the directory's purpose on the file system.
    2. Select the batch (with the red letter B) in the project Panel.
    3. From the main menu select Object > New; leave Content Model at default (FROM TEMPLATE) and enter the name of the object (Object Owner Supplied Name). Click OK.
    4. Click on the "+" next to the Batch icon to expand it – it will show the icon for the new object.
    5. (Optional) Select each object icon in the Project Panel and edit any needed metadata (for instance, if each object has a separate corresponding HOLLIS record, you can switch to the Optional tab in the Content Panel on the right while the object icon is selected in the Project Panel on the left and enter the Aleph or Alma ID for each object in the "Aleph/Alma ID for MODS import" field).
    6. Repeat if you have more than object.

  7. Move/copy files into corresponding file directories on disk.

    (Can be done outside Batch Builder)

    The directory structure on disk is the same as what you see in the Project Panel in Batch Builder if you expand the batch icon. If you need to check where on disk Batch Builder built the directory structure, click on Deposit Settings in the Project Panel on the left and look at the Project Path field at the bottom of the Content Panel on the right.

    1. Copy the files for into corresponding file directories in the object directory.
    2. Repeat if you have more than one object.

  8. Process the batch.

    1. In the Project Panel on the left right-click on the batch you want to process (batch icon with a red letter B) and select "Create descriptors and batch.xml."
    2. Builder will start processing the batch.
    3. When the batch is successfully processed you will see the following message: FINISHED - Creation of batch.xml and descriptors complete for batch: …

    If you see any error messages, remedy the errors and re-process the batch.

  9. Upload batch to dropbox.

    1. Open your SFTP client and log into your DRS2 training dropbox account.
    2. Change to the "incoming" directory.
    3. Upload the batch: copy the entire batch directory to the "incoming" directory.
    4. Close the SFTP client. Batch processing will start.

  10. Check load report in email.

    1. Open your email inbox.
    2. Look for message from "drs2-support" with subject "DRS LOAD REPORT …"

    For a more readable report, open the attached text file in Excel, or just copy the file listing and paste into Excel. If you see any error messages, remedy the errors and re-process the batch.

Automatically build objects from template

This is a description of the procedure to automatically build a batch of video objects from template using Batch Builder 2 (BB). Use this method if you have a large number of objects (over 10) that need to be created and your source files are not broken into directories by object.

Procedure summary:
  1. Create a new project.
  2. Enter DRS deposit settings.
  3. Create object template.
  4. Add directories to template.
  5. Move/copy files into template directories.
  6. Add object name prefix to file names or supply a mapping.txt file.
  7. Create new object batch.
  8. Process the batch.
  9. Upload the batch to your dropbox
  10. Check the load report in email
What you need before you start:
  • Prepare video files and put them in a directory of your choice somewhere on local hard disk or network drive. If you have more than one file per object (e.g.: an archival master and a deliverable or an archival master, a production master and a deliverable) make sure the file names match (e.g.: video-1.mov, video-1.mov and video-1.mp4). Consult Naming Rules section of this Guide for more information on file and directory naming.

  • Decide what you will use for Owner Supplied Names for your objects and the files they consist of. For instance, you could use local classification or local accession numbers. See Naming Rules for more information about owner supplied names.

  • Make sure you either provide object name prefix for each file to specify which objects it should be assigned to (The syntax is: [obj-osn]--[file_name].[extension]) or provide object assignment for your file in an external mapping file. For more details about providing object prefixes for your files see the File name pattern options section of Installation and Setup. For more details about external mapping files see Using mapping.txt File.
Procedure:
  1. Create a new project.

    1. Open BB by double clicking on the executable.
    2. From the Project menu select New.
    3. In the dialog enter Project Name (used for internal tracking); select the directory on disk in which the project will be saved (click the ellipsis button to browse to it on disk); select Content Model "video" from the dropdown; click OK to continue.

  2. Enter DRS deposit settings.

    1. Click on "Deposit Settings" in Project Panel tree.
    2. Enter deposit data in the form:

      Batch Name Pattern: default -
      {owner}{batchDir}{yyyy}{mo}{dd}_{hh24}{mm}{ss}
      (This is the name for a batch that appears on DRS deposit reports. Batch name must be at least 3 characters long.)

      Success Email: type email and press enter. 
      Failure Email: type email and press enter.
      Success Method: choose how you will receive load report. 
      Deposit Agent: type your HUID.
      Deposit Agent Email: type your official Harvard email address.

  3. Create object template(called "batch template" in BB 1).

    1. Select "Object Template" in Project Panel on the left and then enter information in the Content Panel on the right.
    2. Enter Owner Code for objects you will be depositing (use all caps).
    3. Enter Billing Code for objects you will be depositing (use all caps).
    4. Select DRS Access Flag value for objects you will be depositing.
    5. Enter the URN Authority Path for your objects (Look up your Path here if needed: http://nrs.harvard.edu/urn-3:hul.ois:nrsstatusprod) – use all caps.
    6. Enter File URN Resource Name Pattern.
    7. (Optional) switch to "optional" tab to enter any optional metadata (for example descriptive metadata, roles or/and relationships that will apply to your objects). For more about optional metadata see Sections 8. Adding Relationships, 9. Adding Descriptive Metadata, 10. Adding Supporting Content.

  4. Add directories to template.

    If there is one video file per object
    (e.g.: one file that serves as archival master and as deliverable)

    1. Right click on "Object Template" in the Project Panel on the left and add a directory (The directory name starts with "video" but you can append a custom suffix).
    2. In the Project Panel on the left select the directory you just created.
    3. In the Content Panel on the right, choose "yes" from the dropdown list for the field "First Generation in DRS".
    4. Choose "HIGHUSE" for Usage Class.
    5. In the Content Panel switch to Optional tab, scroll down to video File Metadata and use Ctrl Click on your keyboard to select role ARCHIVAL_MASTER and role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.

    If there are two or more video files per object
    (e.g.: a separate file for archival master and a separate file for deliverable)

    Create a directory for an archival master and nest one or more deliverable directories inside it (this is needed for BB to determine the derivative relationship).

    1. Right click on "Object Template" in the Project Panel on the left to add a directory (this will be the directory for archival master). The directory name starts with "video" but you can append a custom suffix.
    2. In the Project Panel on the left select the directory you just created.
    3. In the Content Panel on the right choose "yes" from the dropdown list for the field "First Generation in DRS".
    4. Choose "LOWUSE" for Usage Class.
    5. In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role ARCHIVAL_MASTER.
    6. In the Project Panel on the left right click on the directory you just created and create another directory nested inside it (this will be the directory for deliverable video). Once again, append a custom suffix to directory name if needed (e.g. "video-deliverable").
    7. In the content panel choose "no" from the dropdown list for the field "First Generation in DRS".
    8. Choose "HIGHUSE" for Usage Class.
    9. In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
    10. If you have more deliverables that are derived from the archival master (an mp3 deliverable and an mp4 deliverable for instance) right click on the topmost video directory you created, add a custom suffix if needed and follow steps g-i once again.

    Go to Project menu in the top menu bar and choose Save to save your project.

  5. Move/Copy files into template directory.

    (Done outside Batch Builder)
    Once you added directories to Object Template in BB, they were also created on disk. The directory structure on disk is the same as what you see in the Project Panel in Batch Builder when you expand the Object Template icon, but on disk the Object Template directory is called "template." If you need to check where on disk Batch Builder built the directory structure, click on Deposit Settings in the Project Panel and take a look at the Project Path field at the bottom of the main metadata panel.

    • If you have one video file per object, copy each file to the corresponding video directory inside the template directory.
    • If you have more than one file per object, copy the files for each object into corresponding nested video directories in the template directory.

      Important: the file names for all files belonging to the same object need to match (e.g.: video-1.mov; video-1.mov; video-1.mp4).

6.  Add object name prefix to file names or supply a mapping.txt file.

In order to tell BB which object each file should end up with you need to either add object name prefixes to each file name or supply a mapping.txt file that associates each file with its future object.

    • Add an object owner supplied name as a prefix to the name of each file. The prefix needs to be separated from the file name by the special separator string "--". E.g.: obj1--file1.mov is a file name that tells Batch Builder that this particular file needs to be part of object obj1. Make sure that the file name pattern setting in the BB Options dialog is set to the correct setting. See more information about file Owner Supplied Name pattern settings in Setting Options in Installation and Setup.
    • Provide a mapping file mapping.txt placed in the project /_aux/template directory which maps each file to an object owner supplied name.

      Syntax: relative_file_path,file_OwnerSuppliedName,PDS_sequence number(optional),object_OwnerSuppliedName

      Notes on syntax: There should be no spaces between a comma "," and the next character. If an optional element value is skipped it still needs to be designated by a comma "," so Batch Builder can associate the right value with the right element. For example, for objects that do not have PDS sequence numbers, an extra comma needs to be inserted between the FILE OSN value and Object OS.

7.  Create a new object batch.

You can create your new object batch in the Graphical User Interface or using the Command Line Interface (CLI).

In the Graphical User Interface

    1. From the Object menu in BB select "Create a new batch with objects from template"; enter the name of the batch directory to be created on disk and click OK. A new batch icon will be added to the Project Panel on the right (icon with a red letter B) and new objects will be built that are going to be part of this batch.
    2. In the Project Panel click on the "+" next to the Batch icon of the batch you just created to expand the tree – it will show the icons for the new objects (a blue letter O).
    3. (Optional) Select each object icon in the Project Panel and edit any needed metadata (for instance, if each object has a separate corresponding HOLLIS record, you can switch to the Optional tab in the Content Panel on the right while the object icon is selected in the Project Panel on the left and enter the Aleph or Alma ID for each object in the "Aleph/Alma ID for MODS import" field).

In the Command Line Interface

    1. In the command line window (Terminal on Mac or Linux or CMD on Windows) change to the BB installation directory, e.g.: cd C:\Program Files\BatchBuilder\BatchBuilder-2.0.45
      Type bathcbuildercli –a buildtemplate –p [project_directory_path] –b [batch_directory_name], e.g.: batchbuildercli -a buildtemplate -p "E:\My Project" -b batch1

      Note that the batch directory name is the name of your new batch directory, to be created by BB.

    2. If the batch creation is successful you will see the following messages:

      Initializing BB...
      Done!
      Creating batch1 and objects from files in the template...
      Success!

8.  Process the batch.

You can process your batch in the Graphical User Interface (GUI) or using the Command Line Interface (CLI)

In the Graphical User Interface

    1. In the Project Panel on the left right-click on the batch you want to process (batch icon with a red letter B) and select "Create descriptors and batch.xml."
    2. Builder will start processing the batch.
    3. When the batch is successfully processed you will see the following message: FINISHED - Creation of batch.xml and descriptors complete for batch: 
    4. If you see any error messages, remedy the errors and re-process the batch.

In the Command Line Interface

    1. In the command line window (Terminal on Mac or Linux or CMD on Windows) change to the BB installation directory, e.g.: cd C:\Program Files\BatchBuilder\BatchBuilder-2.0.45
    2. Type batchbuildercli –a build –p [project_directory_path] –b [batch_directory_name], e.g.: batchbuildercli -a build -p "E:\My Project" -b batch1
    3. (Optional) add additional parameters to the bathcbuildercli command to substitute batch, object and directory level properties. For more information see Creating Batches Using Command Line Interface.

    4. When the batch is successfully processed you will see the following messages:

      Building PREMIS file metadata...
      Building HulDrsAdmin file metadata...
      Success!

If you see any error messages at the end of batch processing, remedy the errors and re-process the batch (you can disregard any errors showing during batch processing on command line as long as you get the final "Success!" message)

Video Advanced CLI workflow

Supplying metadata using properties and sidecar files

Default video properties (set by Batch Builder)

Default video properties for Batch Builder advanced CLI video mode are stored in the “/conf/advanced/video” directory of Batch Builder install directory. These files are never changed under normal circumstances. They govern default metadata assignments for BB advanced CLI video mode. Most default assignments can be overwritten by object-specific, and file-specific properties files.

The following default video properties files are present in the “/conf/advanced/video” directory:

  • archival_directory.properties
  • default.properties
  • deliverable_directory.properties
  • misc_directory.properties
  • production_directory.properties


The following settings are set in each file:

archival_directory.properties

#properties for the archival directory
usageClass=LOWUSE
isFirstGenerationInDrs=yes
role=ARCHIVAL_MASTER

default.properties

#directories to scan for content. Each directory could have a
#  corresponding _directory.properties file
dirs=archival,production,deliverable,misc
cmid= CMID-12.0

deliverable_directory.properties

#properties for the deliverable directory
usageClass=HIGHUSE
isFirstGenerationInDrs=no
role=DELIVERABLE

misc_directory.properties

#properties for the misc directory
usageClass=LOWUSE
isFirstGenerationInDrs=unspecified

playlists_directory.properties

#properties for the playlists directory
usageClass=HIGHUSE
isFirstGenerationInDrs=unspecified

production_directory.properties

#properties for the production directory
usageClass=LOWUSE
isFirstGenerationInDrs=no
role=PRODUCTION_MASTER

Batch-specific video properties (set by depositor)

Batch-level video.properties file is a required file that needs to be inserted at the root of the batch directory for the advanced CLI video batch. The file contains all required batch level and object level settings.  The format of it must be:

  • US-ASCII text
  • Each line in “property=value” format
  • Lines can be commented out by starting the line with the ‘#’ symbol.
  • Can’t use reserved XML characters (see the table below for full list).


Property

Valid Values

Example

Meaning

accessFlag

one of: P,R,N

R

The default access flag to use for this object’s files. Can be overridden by the access flag specified in file level properties files.

adminCategory

A comma-delimited list of one or more Wordshack URIs

adminCategory=http\://id.lib.harvard.edu/wordshack/adminCategory/12345Administrative categories to associate with the object

alephID

a valid Aleph or Alma Identifier

009513791

or

990086637850203941

Aleph or Alma ID for the related bibliographic record in HOLLIS

authorityPath

a valid NRS authority path

FHCL.Loeb

Authority path for any URNs that will be created for this object’s files

batchName

100 character limit; Valid values: ASCII alpha-numeric characters,  / "-" / "." / "_" / "~"/


The  following reserved XML characters should be excluded:

gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "=" /

Valid value examples: Project_137_4a~5.1a

Batch-123_23yux-a.21


Invalid values examples:

batch&&<<>>><!

project清代盐政与盐稅

Name for the batch

billingCode

a valid Billing Code

FHCL.MUSI.MUS_004

Billing code for the object

depositAgent

a valid 8 digit HUID

xxxxxxxx

Person depositing the object.

depositAgentEmaila valid email addressjohn_harvard@harvard.eduEmail of person depositing the object. This email has to exactly match the depositor's email as entered in Wordshack.

objectOSN

A unique value within the content’s Owner Code

A299ff

Owner supplied name of the object (this overrides the object directory name, which is treated as object OSN by default)

ownerCode

a valid Owner Code

FHCL.MUSI

DRS Owner Code for the batch

resourcePattern

an NRS resource pattern, see Section 16.1 Assigning File URN Resource Name Patterns

{n}

Resource name pattern for any URNs created for this object’s deliverable files

role

currently no roles for video object are used


One or more roles for the object (not likely to be used - included to support future uses)

failureEmail

one or more valid email addresses separated by commas

example1@fas.harvard.edu, example2@fas.harvard.edu 

Emails to which batch ingest failure reports will be sent

successEmail

one or more valid email addresses separated by commas

example1@fas.harvard.edu, example2@fas.harvard.edu

Emails to which batch ingest success reports will be sent

successMethod

one of: EMAIL, DROPBOX, ALL

EMAIL

Distribution method for the loader reports

embargoDuration

integer

80

Embargo duration value in days, months or years. Not needed if embargoGrantEnd property is used

embargoBasis

One of the controlled values: 

  • Harvard policy
  • license
  • risk assessment
  • statute
  • copyright

Harvard policy

Basis for an embargo

embargoLicense

DRS URN or object Owner Supplied Name or DRS ID of license object in DRS

urn-3:DRS.OBJECT:12345

or objLicense

or 400001

This is where a depositor provides a link to an existing rights object in DRS that will be linked to embargo rights block for the deposited object

embargoLicenseTerms

free text

My license terms

Corresponds to BB GUI field Documentation Note

embargoGrantStart

date in format yyyy-mm-dd

2012-01-01

Corresponds to BB GUI field Start Date

embargoGrantEnd

date in format yyyy-mm-dd

2012-01-01

Corresponds to BB GUI field End Date

harvardPolicy

DRS URN or object Owner Supplied Name or DRS ID of  Harvard Policy object in DRS

urn-3:DRS.OBJECT:12345

or objHarvardPolicy

or 400001

Link to  an existing Harvard policy object in DRS (the next three properties are not related to embargo and are created as separate rights blocks for the deposited object)

license

DRS URN or object Owner Supplied Name or DRS ID of License object in DRS

urn-3:DRS.OBJECT:12345

or objLic

or 400001

Link to  an existing License policy object in DRS

statute

DRS URN or object Owner Supplied Name or DRS ID of Statute object in DRS

urn-3:DRS.OBJECT:12345

or objStatute

or 400001

Link to  an existing Statute object in DRS

metsLabel

free text

my mets label

METS Label display value. This will display in SDS for deliverable video

methodology

free text

this is how this object was created

Object methodology

nonPublicNote

Value for the non-public note field


Value for the non-public note field

producer

A comma-delimited list of one or more Wordshack URIs:

producer=[wordshack URI],[another wordshack URI].


producer=http://id.lib.harvard.edu/wordshack/organization/742

Object producer – controlled value from Word Shack

relatedLinks

relatedLinks="Relationship=value --- URI=value"

Note the space after the relationship value, three dashes and a space before the URI value. The "space three dashes space" is a required delimiter.

relatedLinks="Relationship=LTS --- URI=http://library.harvard.edu/lts"

Related link to any URI - displays in SDS and PDS.

harvardMetadataLinks

harvardMetadataLinks="Type=LinkType --- Identifier=value --- Label=DisplayLabel"

Note the space after the Harvard Metadata Links type value, and then three dashes and a space before the Identifier value. The "space three dashes space" is a required delimiter.

The LinkType value for harvardMetadataLinks comes from a controlled list. The following are valid link types (case sensitive):

Aleph

Alma

Finding Aid

Gale

HLSL

HULPR

Local

OCLC

RLIN

URI

DASH

Shared Shelf

harvardMetadataLinks="Type=Aleph --- Identifier=000012345 --- Label=HOLLIS


For multiple links the syntax is:

harvardMetadataLinks=Type\=Finding Aid --- Identifier\=13527824 --- Label\=Finding Aid,Type\=Aleph --- Identifier\=13527824 --- Label\=HOLLIS,Type\=Alma --- Identifier\=90086637850203941 --- Label\=HOLLIS_ALMA

or

harvardMetadataLinks="Type=Aleph --- Identifier=000012345 --- Label=HOLLIS”, "Type=Alma --- Identifier=90086637850203941 --- Label=HOLLIS”

Harvard Metadata Links - links to Harvard systems. Provide an identifier type, identifier and a label. This displays in SDS for video and PDS

has_documentation

Object URN of a related Documentation object in DRS

urn-3:DRS.OBJECT:12345

Link to  an existing documentation object in the DRS

has_methodology

Object URN of a related Methodology object in DRS

urn-3:DRS.OBJECT:12345

Link to  an existing methodology object in the DRS

Sample video.properties file:

#properties for the object

alephID=009513791

authorityPath=FHCL.Loeb

batchName=Project_137_4a

billingCode= FHCL.MUSI.MUS_004

depositAgent=12345678

depositAgentEmail=john_harvard@harvard.edu

failureEmail=example1@fas.harvard.edu,example2@fas.harvard.edu

objectOSN= A299ff

ownerCode=FHCL.MUSI

resourcePattern={n}

successEmail=example1@fas.harvard.edu,example2@fas.harvard.edu

successMethod=EMAIL


Place the video.properties file at the root of the batch directory, e.g.:
…/batch/video.properties

File-specific video properties files (set by depositor)

Provide file-level properties files for files in the batch (optional: if needed to override Batch Builder-configured default metadata values). Each properties file contains properties for a corresponding video file. The format of it must be:

  • US-ASCII text
  • File level properties are shown in the table below
  • Each line in “property=value” format
  • Properties with repeatable values should be delimited by a comma
  • Lines can be commented out by starting the line with the ‘#’ symbol
  • Can’t use reserved XML characters



Property

Valid values

Example

Meaning

fileAccessFlag

one of: P,R,N

P

Access level of file

fileAdminCategory

A comma-delimited list of one or more Wordshack URIs –
You can alternatively use the same string formatting used in the BB GUI

http://id.lib.harvard.edu/wordshack/adminCategory/12345

Administrative categories to associate with the file

fileosn

Alpha-numeric ASCII

34561-a7-bach

Name for the file (not necessarily unique)

isFirstGenerationInDrs

one of: yes, no, unspecified

yes

whether or not the file is the closest version in the DRS to the original capture or creation.

isPreferredDeliverableSource

one of: yes, no, unspecified

yes

whether or not the file is the recommended file to use for generating future deliverables

fileNonPublicNote

Value for the non-public note field

my note


Processing

comma-delimited list of one or more of:
CROPPED
BIT_REDUCED
BRIGHTNESS_ADJUSTED
COLOR_CORRECTED
COMPOSITED
CONTRAST_ADJUSTED
DENOISED
DESKEWED
DOWN_MIXED
DYNAMIC_RANGE_COMPRESSED
EDITED
EQUALIZED
GAMMA_CORRECTED
KEYED
LEVEL_ADJUSTED
PHASE_INVERTED
PROCESSED
PULL_DOWN_APPLIED
PULL_UP_APPLIED
RETIMED
ROTOSCOPED
SAMPLERATE_CONVERTED
SCALED
SHARPENED

SCALED, COMPOSITED, ROTOSCOPED, KEYED, EDITED, RETIMED, PULL_DOWN_APPLIED, PULL_UP_APPLIED

Key actions taken on the file before it was deposited to the DRS

fileproducer

A comma-delimited list of one or more Wordshack URIs – You can alternatively use the same string formatting used in the BB GUI

Currently not supported in BB

File producer – controlled value from Word Shack

quality

one of: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10

5

Locally-defined measure of quality

usageClass

One of: HIGHUSE, LOWUSE

LOWUSE

Estimate of how frequently the file will be accessed by library users

filemethodology

(any text)


Notes about the methods performed and/or tools used in the creation, processing or preparation of the content prior to DRS deposit

filerole

comma-delimited list of one or more of:

ARCHIVAL_MASTER

video_DECISION_LIST

DELIVERABLE

DOCUMENTATION

EDIT_DECISION_LIST

LICENSE

PROCESS_FILES

PROCESS_HISTORY

PRODUCTION_MASTER

THUMBNAIL

DELIVERABLE


downloadProhibited

(for video files)

yes

yes

If value set to yes:

Request only streaming delivery URN. If value is No or NA, request URN for streaming and download delivery.

Video file must be streamed - direct download will not be permitted. Note that you this property is required only if value is set to ‘yes’. If the value is set to ‘no’ simply do not include the property. It is set at ‘no’ by default. If the property is provided and set to ‘yes’ you also need to set the downloadProhibitedBasis property listed below.

downloadProhibitedBasis* (for video files)

One of the controlled values: 

  • Harvard policy
  • license
  • risk assessment
  • statute
  • copyright

license

Basis for the restrictions or terms of use.


*Required if downloadProhibited property is supplied.

downloadProhibitedLicense

(for video files)

DRS object URN or DRS Object OSN or DRS Object ID of the license object

urn-3:HUL.DRS.OBJECT:197548

or ObjMyLicense

or 12345678

ID of the documentation within the DRS, e.g. a deposited donor agreement

downloadProhibitedLicenseTerms

(for video files)

(any text)

50 year policy

A note about the documentation, e.g. the applicable section

downloadProhibitedGrantStart

(for video files)

A date/timestamp

yyyy-mm-dd

yyyy-mm

yyyy

2012-02-12

2012-02

2012

Embargo start date

downloadProhibitedGrantEnd

(for video files)

A date/timestamp

yyyy-mm-dd

yyyy-mm

yyyy

2062-02-12

2062-02

2062

Embargo end date

file_has_documentation

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyDoc

or 12345678

Has documentation in DRS

fileHarvardPolicy

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyPolicy

or 12345678

related Harvard policy object in DRS

filelicense

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyLicense

or 12345678

related license object in DRS

filestatute

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyStatute

or 12345678

related statute object in DRS


Place the properties file in the same directory as the video file. It needs to have the same file base name as the video  For example, the file

batch/object/archival/sub_dir1_Media/file1.properties

would contain metadata for the file

batch/object/archival/sub_dir1_Media/file1.mov


 

File-specific captions properties files (set by depositor)

 

Provide file-level properties files for captions files in the batch 

Each properties file contains properties for a corresponding captions file. The format of it must be:

 

  • US-ASCII text
  • File level properties are shown in the table below
  • Each line in “property=value” format
  • Properties with repeatable values should be delimited by a comma
  • Lines can be commented out by starting the line with the ‘#’ symbol
  • Can’t use reserved XML characters

 

Property

Valid values

Example

Meaning

fileAccessFlag (required)

one of: P,R,N

P

Access level of file

fileAdminCategory

A comma-delimited list of one or more Wordshack URIs –
You can alternatively use the same string formatting used in the BB GUI

http://id.lib.harvard.edu/wordshack/adminCategory/12345

Administrative categories to associate with the file

fileosn

Alpha-numeric ASCII

34561-a7-bach

Name for the file (not necessarily unique). If not supplied in the properties file the fileosn is set as equal to file basename by Batch Builder

isFirstGenerationInDrs (required)

one of: yes, no, unspecified

yes

whether or not the file is the closest version in the DRS to the original capture or creation.

isPreferredDeliverableSource

one of: yes, no, unspecified

yes

whether or not the file is the recommended file to use for generating future deliverables

fileNonPublicNote

Value for the non-public note field

my note


fileproducer

A comma-delimited list of one or more Wordshack URIs – You can alternatively use the same string formatting used in the BB GUI

Currently not supported in BB

File producer – controlled value from Word Shack

usageClass (required)

One of: HIGHUSE, LOWUSE

LOWUSE

Estimate of how frequently the file will be accessed by library users

filemethodology

(any text)


Notes about the methods performed and/or tools used in the creation, processing or preparation of the content prior to DRS deposit

file_has_documentation

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyDoc

or 12345678

Has documentation in DRS

fileHarvardPolicy

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyPolicy

or 12345678

related Harvard policy object in DRS

filelicense

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyLicense

or 12345678

related license object in DRS

filestatute

DRS object URN or DRS Object OSN or DRS Object ID

urn-3:HUL.DRS.OBJECT:197548

or ObjMyStatute

or 12345678

related statute object in DRS

 


 

Place the properties file in the same directory as the captions file. It needs to have the same file basename as the captions. For example, the file

 

batch/object/captions/file1.properties

 

would contain metadata for the file

 

batch/object/captions/file1.vtt

Relationships.csv file (set by depositor)

The required relationships.csv file specifies relationships that are not automatically created by Batch Builder or on ingest. This file should be placed at the root of the batch directory. The file format must be comma-separated CSV with a new line for each relationship. The file paths inside the file need to be in the form supported by the OS on which you are running Batch Builder (POSIX for MAC/Linux and DOS-style for Windows).

  • US-ASCII text
  • Each line is a comma-separated list of attributes for each relationship as described below
  • File paths need to be in the form supported by the OS on which you are running Batch Builder (POSIX for MAC/Linux and DOS-style for Windows)
  • Lines can be commented out by starting the line with the ‘#’ symbol.
  • Can’t use reserved XML characters (see the table below for full list).


The syntax for each line must be:

[Source_filePath],[relationship_name],[target_type],[target]

where

  • Source_filePath:
    • for files in this object: relative path to the source file from the object directory for this object
  • relationship_name, one of:
    • HAS_SOURCE (indicates derivative relationships between files in this object)
    • HAS_DOCUMENTATION (for pointing from files or the object to DRS-stored documentation files or objects)
    • HAS_METHODOLOGY (for pointing from the object to a DRS-stored methodology object)
    • HAS_RIGHTS (for pointing from files or the object to DRS-stored rights files or objects)
    • HAS_SUPPLEMENT (points to a closed captions text object in the same batch or in DRS) - if in the same batch use target_type OBJECT_OSN
  • target_type, one of:
    • RELATIVE_PATH (Use for a relative file path)
    • HUL_DRS_OBJECT_URN (Use for the URN of a DRS object)
    • OBJECT_OSN (Use for owner supplied name of object in the same batch or object in DRS)
    • HUL_DRS_OBJECT_ORACLE (Use for DRS ID of the target object in DRS)
    • HUL_DRS_FILE_ORACLE – (Use for DRS ID of the target file in DRS)
  • Target, one of:
    • relative path to a file,
    • DRS object URN,
    • DRS object owner supplied name (Object OSN)
    • DRS ID of object in DRS
    • DRS ID of file in DRS

Sample relationships.csv file:

# Derivative video content relationships

deliverable\Media\elephants-dream-delivery.mp4,HAS_SOURCE,RELATIVE_PATH,production\sub_dir_1_Media\elephants-dream-medium.mov

production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SOURCE,RELATIVE_PATH,archival\sub_dir_1_Media\elephants-dream-hd.mov

archival\sub_dir_1_Media\elephants-dream-hd.mov,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz

production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz

deliverable\Media\elephants-dream-delivery.mp4,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz


Place this file at the root of the batch directory, e.g.:

…/batch/
             video.properties
             relationships.csv

Batch-specific MODS file (set by depositor

If supplying descriptive metadata, create a MODS file called mods.xml complying with MODS schema versions 3.5 or earlier (DRS does not currently support MODS 3.6).  Seehttp://www.loc.gov/standards/mods/ for more information about MODS metadata and schemas.

Place this file at the root of the batch directory, e.g.:

…/batch/
               video.properties
               mods.xml
               relationships.csv

Note that as an alternative to supplying a MODS file, descriptive metadata can be imported from HOLLIS through Batch Builder by providing an Aleph or Alma ID in the video.properties file.

EBUCore, Digital Provenance metadata and Source metadata

EBUCore is technical metadata for video. It should be supplied in sidecar XML files that contain EBUCore output for each of the video files that are being deposited. EBUCore can be generated by running the FITS tool on each of the video files. Each EBUCore file should have the same file base name as the video file and should be placed in the same directory as the video file. Instructions. Example EBUCore file: elephants-dream-hd.xml

Digital Provenance metadata should be supplied in a sidecar XML file called digiprov.xml using the ReVTMD metadata schema. The digital provenance files should be placed in 'archival' directory or 'production' directory of the batch. Example file: digiprov.xml

Source file metadata should be supplied in a sidecar file with any name using the UTVideoSrc schema. Source metadata file should be placed in the directory 'original' inside the batch directory. Example file: VideoSrc10-01-2015-DA.xml

VTT Captions for video files

File format: WebVTT (a plain text format that is part of the HTML5 standard.)

File extension: .vtt

Object Role: CLOSED_CAPTION_DATA (set automatically by Batch Builder)

Content Model: Text (one VTT file per object)

Delivery URN: request FDS URN for each closed captions VTT file

Usageclass: “HIGHUSE” (set for file – set in .properties file in the “captions” directory)

FirstGeneration: “Yes” (set for file – set in .properties file in the “captions” directory)

Sample .properties file for captions:

#properties for mycaps1
fileosn= mycaps1-fileosn
isFirstGenerationInDrs=unspecified
usageClass=HIGHUSE
accessFlag=P

Relationships: The text object generated by Batch Builder will contain one VTT file which will be a target of a HAS_SUPPLEMENT file-to-object relationship from the video file to the text object. In order for relationship to be created it needs to be specified by the depositor in the relationships.csv file, where it will point from the video file to the text object containing the VTT file. The syntax is the following:
path_to_video_file,HAS_SUPPLEMENT,OBJECT_OSN,text_obj_OSN (note that in this case ‘obj_OSN’ is derived from the file basename, e.g. for a file mycaps1.vtt, the derived ‘obj_OSN’ is ‘mycaps1’)

Relationships.csv example:

archival\sub_dir_1_Media\elephants-dream-hd.mov,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1
production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1
deliverable\Media\elephants-dream-delivery.mp4,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1

This relationship points to a text object that will be created with the file mycaps1.vtt

Note that multiple video files may be the sources of HAS_SUPPLEMENT relationships pointing to the same text object containing a VTT file. Additionally, a single video file may be the source of HAS_SUPPLEMENT relationships to multiple Text objects, each contacting a VTT file.

Placement: Closed captions WebVTT files with extension “.vtt” should be placed into “captions” directory in the Batch Builder video object directory structure. Corresponding .properties files should be placed in the same directory. All captions files and corresponding .properties files for a batch should be placed in the same directory. See sample directory structure below.

Object OSN: object owner supplied name should be the generated based on the file basename (basename.vtt) of the caption VTT file.

File OSN: file owner supplied name should be the generated based on the file basename (basename.vtt) of the caption VTT file unless it is overridden in the file-level properties file.

Workflow:

VTT files with extension “.vtt” should be placed into the “captions” directory in advanced CLI directory structure.  The “captions” directory should be located directly inside the object directory in the batch. The HAS_SUPPLEMENT relationships between the text objects containing the VTT files and the video files should be specified in the relationships.csv file, located in the batch directory.

Each VTT file will be assembled by Batch Builder into a single Text object with object-level role CLOSED_CAPTION_DATA. For each text object, a file-to-object HAS_SUPPLEMENT relationship will be created from the video file to the text object (as modeled in relationships.csv file).

Preparing content files

Prepare content files:

  • consult Harvard Library Digital Preservation wiki for DRS-accepted video formats: https://harvardwiki.atlassian.net/wiki/x/zoHeAg
  • generate video files in one of the accepted formats
  • generate video files for delivery according to DRS Streaming Delivery Service video specifications: H264 encoded video and AAC encoded video in Quicktime container with .mp4 extension
  • generate EBUCore files for each video file by running the FITS tool on each file and creating EBUCore standard output
  • generated captions files in VTT format
  • generate digital provenance metadata in a sidecar XML file called digiprov.xml using the ReVTMD metadata schema
  • generate original source file metadata in a sidecar file with any name using the UTVideoSrc schema

Arrange the content in the directory structure required by Batch Builder

Below is a sample directory structure created in preparation for use with Batch Builder. Directory names and file names (or parts file names) in bold must be named as shown.

Figure 1: Sample Directory Structure


/batch/
                video.properties
                mods.xml
                relationships.csv

object/

     archival/ [1..n]

         digiprov.xml (ReVTMD)

                      sub_dir1_Media/ [1..n]

                                               File1.mov
                                               File1.properties
                                               File1.xml (EBUCore metadata for video file)
                                               File2.mov
                                               File2.properties
                                               File2.xml (EBUCore metadata for video file)

    production/ [1..n] REQUIRED EVEN IF EMPTY
         digiprov.xml (ReVTMD)

                 sub_dir1_Media/ [1..n]

                                               File1.mov
                                               File1.properties
                                               File1.xml (EBUCore metadata for video file)
                                               File2.mov
                                               File2.properties
                                               File2.xml (EBUCore metadata for video file)

     deliverable/ [0..n]

                            Media/ [1..n]

                                               File1.mov
                                               File1.properties
                                               File1.xml (EBUCore metadata for video file)
                                               File2.mov
                                               File2.properties
                                               File2.xml (EBUCore metadata for video file)

  

   captions/

                        mycaps.vtt
                        mycaps.properties

    original/

                source_metadata.xml (UTVIDEOSrc)

  misc/ (miscellaneous process files placed in this directory are zipped by Batch Builder and moved to '/workspace/moved_content'
            directory)

Command Line Syntax

The existing Batch Builder Windows bat and Unix shell scripts are used to launch Batch Builder in advanced mode. The syntax is:

  • On PC: batchbuildercli –advanced video –d [object directory path]
    • Use Windows style directory separator '\'
  • On Mac or Linux: ./batchbuildersli.sh -advanced video –d [object directory path]
    • use UNIX style directory separator '/'

Note that by default the object directory name serves as the object Owner Supplied Name (you can override that in video.properties file).

Example 1 (Windows): batchbuildercli –advanced video –d "C:\mybatch\004390909"

Example 2 (Mac / Linux): ./batchbuildercli.sh -advanced video -d /mybatch/004390909

Additional command line optional properties

All command line optional properties for deposit settings, object settings and file settings that are available for "simple CLI" in Batch Builder are also now available in advanced CLI. See more here: https://harvardwiki.atlassian.net/wiki/x/xw2VAg

Process the Batch

Use command line syntax above to call Batch Builder to process your batch.

The program will:

  • Create a directory at workspace/moved_content
  • Put a copy of the files that were in the misc directory in the workspace/moved_content directory
  • Zip up the contents of the misc directory and name it misc_files.zip
  • Create the object descriptor for the video object
    • The contents of EBUCore files supplied with each video file will be inserted into the descriptor
    • The ReVTMD digiprov.xml digital provenance files will be referenced in the filesec and structmap of the descriptor
    • The contents of the UTVideoSrc original object metadata file will be inserted into the descriptor
    • Build a separate Text object with role CLOSED_CAPTION_DATA for each captions file
      • The file name will be used as the object owner-supplied-name
      • Create the object descriptor for each object
      • The descriptor will be named {object-OSN}_descriptor.xml and will be written to the batch directory
      • Relationships between the text object and video files will be parsed from the relationships.csv file

After executing Batch Builder the following changes should be visible in the batch directory:

/my_batch-19a326/

     batch.xml  (DRS batch loading file)  
     my_object-A299ff/      
          misc/         
               misc_files.zip  (files that were in the misc directory)      
          workspace/      
          moved_content/  (contains content in the misc directory)   
       descriptor.xml  (object descriptor for the video object)
       mycaps_descriptor.xml

Depositing to the DRS

After executing the script, copy the following parts of the batch directory to your DRS dropbox.

/my_batch-19a326/  (example batch directory)      
     batch.xml           
     my_object-A299ff/  (example object directory)   

          descriptor.xml
          mycaps_descriptor.xml

          archival/            
          deliverable/             
          misc/            
          original/            
          production/            
          captions/


Notes:

  • Object owner-supplied-names (Object OSNs)

    Object OSNs are required in DRS 2 and must be unique within a Owner Code.
  • URNs
    • A DRS object URN is automatically assigned to all objects. These URNs will have a HUL.DRS.OBJECT authority path, e.g.urn-3:HUL.DRS.OBJECT:175844
    • File-level delivery URNs are automatically given to video files with the role DELIVERABLE and two VTT captions files (no file role is needed for VTT captions files). These URNs will have a local authority path, e.g.urn-3:FHCL.Loeb.Faids:mus00023. Each video file that gets a delivery URN will get an SDS (streaming) URN and an FDS (download) URN unless DOWNLOAD_PROHIBITED is specified in the rights for the video file. Each VTT captions file will get an FDS URN.



On this page:

In this section: