Creating Video Object Batches
- Vitaly Zakuta
- Paul Aloisio
- Julie Wetherill
Accepted Formats
See: https://harvardwiki.atlassian.net/wiki/x/zoHeAg##FormatsSupportedbytheDRS-VideoFormats
Note: Video files intended for delivery in Harvard SDS streaming delivery service should be in Quicktime format using the H.264 video codec and the AAC audio codec and have a file extension ".mp4".
Methods of creating video batches using Batch Builder
There are four methods you can use to create video objects in Batch Builder:
- Manually build objects from template. This is best when you have only one or two video objects to work with. Point and click to create a new batch and then create new objects.
- Automatically build objects from template. This is best when you have three or more video objects to work with. In this scenario BB generates object structure for each of your objects and moves files into object directories. Before this can happen you need to rename your files to add special "object name" file prefixes or provide mapping files that associate files with their objects.
- Manually build objects without a template. In this method, rather than using the template, you fill out object and file metadata and create object directories for each individual object one-object-at-a-time. As this method is rarely used in workflows, it is not currently documented.
- Use Batch Builder advanced video functionality to build an advanced video batch. Use this method if you plan to deposit video files along with additional process files and video captions. See Video Advanced CLI workflow for the procedure.
Manually build objects from template
This is a description of the procedure to manually create a batch of Video objects from template in Batch Builder 2 (BB). Use this method when only Video objects need to be created, processed in Batch Builder (BB) and deposited into the DRS. If you have many (over 10) objects to create, use the procedure to build objects automatically.
Procedure summary:
- Create a new project.
- Enter DRS deposit settings.
- Create object template.
- Add directories to template.
- Create new object batch.
- Move/copy files into corresponding directories on disk.
- Process the batch.
- Upload the batch to your dropbox
- Check the email report
What you need before you start:
- Prepare several video files and put them in a directory of your choice somewhere on local hard disk or network drive. If you have more than one video file per recording (e.g.: an archival master and a deliverable or an archival master, a production master and a deliverable) make sure file names match (e.g.: video-1.mov, video-1.mov and video-1.mp4). Consult Naming Rules for more information on file and directory naming.
- Decide what you will use for Owner Supplied Names for your objects and the files they consist of. For instance, you could use local classification or local accession numbers. See Naming Rules for more information about owner supplied names.
Procedure:
- Create a new project.
- Open BB by double clicking on the executable.
- From the Project menu select New.
- In the dialog enter Project Name (used for internal tracking); select the directory on disk in which the project will be saved (click the ellipsis button to browse to it on disk); select Content Model "video" from the dropdown; click OK to continue.
- Enter DRS deposit settings.
- Click on "Deposit Settings" in Project Panel tree.
- Enter deposit data in the form:
Batch Name Pattern: default - {owner}{batchDir}{yyyy}{mo}{dd}_{hh24}{mm}{ss}
(This is the name for a batch that appears on DRS deposit reports. Batch name must be at least 3 characters long.)
Success Email: type email and press enter.
Failure Email: type email and press enter.
Success Method: choose how you will receive load report.
Deposit Agent: type your HUID.
Deposit Agent Email: type your official Harvard email address.
- Create object template (called "batch template" in BB 1).
- Select "Object Template" in Project Panel on the left and then enter information in the Content Panel on the right.
- Enter Owner Code for objects you will be depositing (use all caps).
- Enter Billing Code for objects you will be depositing (use all caps).
- Select DRS Access Flag value for objects you will be depositing.
- Enter the URN Authority Path for your objects (Look up your Path here if needed: http://nrs.harvard.edu/urn-3:hul.ois:nrsstatusprod) – use all caps.
- Enter URN Resource Name Pattern.
- (Optional) switch to "optional" tab to enter any optional metadata (for example descriptive metadata, roles or/and relationships that will apply to your objects). For more about optional metadata see Sections 8. Adding Relationships, 9. Adding Descriptive Metadata, 10. Adding Supporting Content.
- Add directories to template.
If there is one video file per object
(e.g.: one file that serves as archival master and as deliverable)- Right click on "Object Template" in the Project Panel on the left and add a directory and select the 'add a directory…' menu option
- Type in a name for your directory in the text box. The directory name starts with "video", but the name you type will be appended to it. Tip: to make it easier to read, start your directory name with an underscore (_). So, typing '_mov_files' will result in a directory called 'video_mov_files'.
- In the Project Panel on the left select the directory you just created.
- In the Content Panel on the right, choose "yes" from the dropdown list for the field "First Generation in DRS".
- Choose "HIGHUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and use Ctrl Click on your keyboard to select role ARCHIVAL_MASTER and role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
(e.g.: a separate file for archival master and a separate file for deliverable)
Create a directory for an archival master and nest one or more deliverable directories inside it (this is needed for BB to determine the derivative relationship).- Right click on "OBJECT TEMPLATE" in the Project Panel on the left and select the 'add a directory …' menu option.
- In the text box, type '-archival_master' or a similar suffix that describes the role of your directory. The directory name will start with "video" but the name you type will be appended to it. So, typing '-archival_master' will result in a directory called 'video-archival_master'
- In the Project Panel on the left select the directory you just created.
- In the Content Panel on the right choose "yes" from the dropdown list for the field "First Generation in DRS".
- Choose "LOWUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role ARCHIVAL_MASTER.
- In the Project Panel on the left, right click on the directory you just created and select the 'add a directory …' menu option.
- In the text box, type '-deliverable' or a similar suffix that describes the role for your directory.
- In the Project Panel on the left, select the directory you just created.
- In the Content Panel choose "no" from the dropdown list for the field "First Generation in DRS".
- Choose "HIGHUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
- If you have more deliverables that are derived from the archival master right click on the topmost video directory you created, add a custom suffix if needed and follow steps g-i once again.
- Save your work (Project > Save on main menu)
- Create new object batch.
- Select "Batch>New" from the main menu. Enter a name for the batch (will be used as the batch directory name on disk) Click OK.
Tip: including the word "batch" in your batch name will help you remember the directory's purpose on the file system. - Select the batch (with the red letter B) in the project Panel.
- From the main menu select Object > New; leave Content Model at default (FROM TEMPLATE) and enter the name of the object (Object Owner Supplied Name). Click OK.
- Click on the "+" next to the Batch icon to expand it – it will show the icon for the new object.
- (Optional) Select each object icon in the Project Panel and edit any needed metadata (for instance, if each object has a separate corresponding HOLLIS record, you can switch to the Optional tab in the Content Panel on the right while the object icon is selected in the Project Panel on the left and enter the Aleph or Alma ID for each object in the "Aleph/Alma ID for MODS import" field).
- Repeat if you have more than object.
- Select "Batch>New" from the main menu. Enter a name for the batch (will be used as the batch directory name on disk) Click OK.
- Move/copy files into corresponding file directories on disk.
(Can be done outside Batch Builder)
The directory structure on disk is the same as what you see in the Project Panel in Batch Builder if you expand the batch icon. If you need to check where on disk Batch Builder built the directory structure, click on Deposit Settings in the Project Panel on the left and look at the Project Path field at the bottom of the Content Panel on the right.- Copy the files for into corresponding file directories in the object directory.
- Repeat if you have more than one object.
- Process the batch.
- In the Project Panel on the left right-click on the batch you want to process (batch icon with a red letter B) and select "Create descriptors and batch.xml."
- Builder will start processing the batch.
- When the batch is successfully processed you will see the following message: FINISHED - Creation of batch.xml and descriptors complete for batch: …
- Upload batch to dropbox.
- Open your SFTP client and log into your DRS2 training dropbox account.
- Change to the "incoming" directory.
- Upload the batch: copy the entire batch directory to the "incoming" directory.
- Close the SFTP client. Batch processing will start.
- Check load report in email.
- Open your email inbox.
- Look for message from "drs2-support" with subject "DRS LOAD REPORT …"
Automatically build objects from template
This is a description of the procedure to automatically build a batch of video objects from template using Batch Builder 2 (BB). Use this method if you have a large number of objects (over 10) that need to be created and your source files are not broken into directories by object.
Procedure summary:
- Create a new project.
- Enter DRS deposit settings.
- Create object template.
- Add directories to template.
- Move/copy files into template directories.
- Add object name prefix to file names or supply a mapping.txt file.
- Create new object batch.
- Process the batch.
- Upload the batch to your dropbox
- Check the load report in email
What you need before you start:
- Prepare video files and put them in a directory of your choice somewhere on local hard disk or network drive. If you have more than one file per object (e.g.: an archival master and a deliverable or an archival master, a production master and a deliverable) make sure the file names match (e.g.: video-1.mov, video-1.mov and video-1.mp4). Consult Naming Rules section of this Guide for more information on file and directory naming.
- Decide what you will use for Owner Supplied Names for your objects and the files they consist of. For instance, you could use local classification or local accession numbers. See Naming Rules for more information about owner supplied names.
- Make sure you either provide object name prefix for each file to specify which objects it should be assigned to (The syntax is: [obj-osn]--[file_name].[extension]) or provide object assignment for your file in an external mapping file. For more details about providing object prefixes for your files see the File name pattern options section of Installation and Setup. For more details about external mapping files see Using mapping.txt File.
Procedure:
- Create a new project.
- Open BB by double clicking on the executable.
- From the Project menu select New.
- In the dialog enter Project Name (used for internal tracking); select the directory on disk in which the project will be saved (click the ellipsis button to browse to it on disk); select Content Model "video" from the dropdown; click OK to continue.
- Enter DRS deposit settings.
- Click on "Deposit Settings" in Project Panel tree.
- Enter deposit data in the form:
Batch Name Pattern: default - {owner}{batchDir}{yyyy}{mo}{dd}_{hh24}{mm}{ss}
(This is the name for a batch that appears on DRS deposit reports. Batch name must be at least 3 characters long.)
Success Email: type email and press enter.
Failure Email: type email and press enter.
Success Method: choose how you will receive load report.
Deposit Agent: type your HUID.
Deposit Agent Email: type your official Harvard email address.
- Create object template(called "batch template" in BB 1).
- Select "Object Template" in Project Panel on the left and then enter information in the Content Panel on the right.
- Enter Owner Code for objects you will be depositing (use all caps).
- Enter Billing Code for objects you will be depositing (use all caps).
- Select DRS Access Flag value for objects you will be depositing.
- Enter the URN Authority Path for your objects (Look up your Path here if needed: http://nrs.harvard.edu/urn-3:hul.ois:nrsstatusprod) – use all caps.
- Enter File URN Resource Name Pattern.
- (Optional) switch to "optional" tab to enter any optional metadata (for example descriptive metadata, roles or/and relationships that will apply to your objects). For more about optional metadata see Sections 8. Adding Relationships, 9. Adding Descriptive Metadata, 10. Adding Supporting Content.
- Add directories to template.
If there is one video file per object
(e.g.: one file that serves as archival master and as deliverable)- Right click on "Object Template" in the Project Panel on the left and add a directory (The directory name starts with "video" but you can append a custom suffix).
- In the Project Panel on the left select the directory you just created.
- In the Content Panel on the right, choose "yes" from the dropdown list for the field "First Generation in DRS".
- Choose "HIGHUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and use Ctrl Click on your keyboard to select role ARCHIVAL_MASTER and role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
(e.g.: a separate file for archival master and a separate file for deliverable)
Create a directory for an archival master and nest one or more deliverable directories inside it (this is needed for BB to determine the derivative relationship).- Right click on "Object Template" in the Project Panel on the left to add a directory (this will be the directory for archival master). The directory name starts with "video" but you can append a custom suffix.
- In the Project Panel on the left select the directory you just created.
- In the Content Panel on the right choose "yes" from the dropdown list for the field "First Generation in DRS".
- Choose "LOWUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role ARCHIVAL_MASTER.
- In the Project Panel on the left right click on the directory you just created and create another directory nested inside it (this will be the directory for deliverable video). Once again, append a custom suffix to directory name if needed (e.g. "video-deliverable").
- In the content panel choose "no" from the dropdown list for the field "First Generation in DRS".
- Choose "HIGHUSE" for Usage Class.
- In the Content Panel switch to Optional tab, scroll down to video File Metadata and select role DELIVERABLE. Note that in order to get a delivery URN on deposit an video file needs to have a role DELIVERABLE set in BB.
- If you have more deliverables that are derived from the archival master (an mp3 deliverable and an mp4 deliverable for instance) right click on the topmost video directory you created, add a custom suffix if needed and follow steps g-i once again.
- Move/Copy files into template directory.
(Done outside Batch Builder)
Once you added directories to Object Template in BB, they were also created on disk. The directory structure on disk is the same as what you see in the Project Panel in Batch Builder when you expand the Object Template icon, but on disk the Object Template directory is called "template." If you need to check where on disk Batch Builder built the directory structure, click on Deposit Settings in the Project Panel and take a look at the Project Path field at the bottom of the main metadata panel.
- If you have one video file per object, copy each file to the corresponding video directory inside the template directory.
- If you have more than one file per object, copy the files for each object into corresponding nested video directories in the template directory.
Important: the file names for all files belonging to the same object need to match (e.g.: video-1.mov; video-1.mov; video-1.mp4).
6. Add object name prefix to file names or supply a mapping.txt file.
In order to tell BB which object each file should end up with you need to either add object name prefixes to each file name or supply a mapping.txt file that associates each file with its future object.
- Add an object owner supplied name as a prefix to the name of each file. The prefix needs to be separated from the file name by the special separator string "--". E.g.: obj1--file1.mov is a file name that tells Batch Builder that this particular file needs to be part of object obj1. Make sure that the file name pattern setting in the BB Options dialog is set to the correct setting. See more information about file Owner Supplied Name pattern settings in Setting Options in Installation and Setup.
- Provide a mapping file mapping.txt placed in the project /_aux/template directory which maps each file to an object owner supplied name.
Syntax: relative_file_path,file_OwnerSuppliedName,PDS_sequence number(optional),object_OwnerSuppliedName
Notes on syntax: There should be no spaces between a comma "," and the next character. If an optional element value is skipped it still needs to be designated by a comma "," so Batch Builder can associate the right value with the right element. For example, for objects that do not have PDS sequence numbers, an extra comma needs to be inserted between the FILE OSN value and Object OS.
7. Create a new object batch.
You can create your new object batch in the Graphical User Interface or using the Command Line Interface (CLI).
In the Graphical User Interface
- From the Object menu in BB select "Create a new batch with objects from template"; enter the name of the batch directory to be created on disk and click OK. A new batch icon will be added to the Project Panel on the right (icon with a red letter B) and new objects will be built that are going to be part of this batch.
- In the Project Panel click on the "+" next to the Batch icon of the batch you just created to expand the tree – it will show the icons for the new objects (a blue letter O).
- (Optional) Select each object icon in the Project Panel and edit any needed metadata (for instance, if each object has a separate corresponding HOLLIS record, you can switch to the Optional tab in the Content Panel on the right while the object icon is selected in the Project Panel on the left and enter the Aleph or Alma ID for each object in the "Aleph/Alma ID for MODS import" field).
In the Command Line Interface
- In the command line window (Terminal on Mac or Linux or CMD on Windows) change to the BB installation directory, e.g.: cd C:\Program Files\BatchBuilder\BatchBuilder-2.0.45
Typebathcbuildercli –a buildtemplate –p [project_directory_path] –b [batch_directory_name], e.g.: batchbuildercli -a buildtemplate -p "E:\My Project" -b batch1
Note that the batch directory name is the name of your new batch directory, to be created by BB. - If the batch creation is successful you will see the following messages:
Initializing BB...
Done!
Creating batch1 and objects from files in the template...
Success!
- In the command line window (Terminal on Mac or Linux or CMD on Windows) change to the BB installation directory, e.g.: cd C:\Program Files\BatchBuilder\BatchBuilder-2.0.45
8. Process the batch.
You can process your batch in the Graphical User Interface (GUI) or using the Command Line Interface (CLI)
In the Graphical User Interface
- In the Project Panel on the left right-click on the batch you want to process (batch icon with a red letter B) and select "Create descriptors and batch.xml."
- Builder will start processing the batch.
- When the batch is successfully processed you will see the following message: FINISHED - Creation of batch.xml and descriptors complete for batch:
- If you see any error messages, remedy the errors and re-process the batch.
In the Command Line Interface
- In the command line window (Terminal on Mac or Linux or CMD on Windows) change to the BB installation directory, e.g.: cd C:\Program Files\BatchBuilder\BatchBuilder-2.0.45
- Type
batchbuildercli –a build –p [project_directory_path] –b [batch_directory_name], e.g.: batchbuildercli -a build -p "E:\My Project" -b batch1
- (Optional) add additional parameters to the bathcbuildercli command to substitute batch, object and directory level properties. For more information see Creating Batches Using Command Line Interface.
- When the batch is successfully processed you will see the following messages:
Building PREMIS file metadata...
Building HulDrsAdmin file metadata...
Success!
If you see any error messages at the end of batch processing, remedy the errors and re-process the batch (you can disregard any errors showing during batch processing on command line as long as you get the final "Success!" message)
Video Advanced CLI workflow
Supplying metadata using properties and sidecar files
Default video properties (set by Batch Builder)
Default video properties for Batch Builder advanced CLI video mode are stored in the “/conf/advanced/video” directory of Batch Builder install directory. These files are never changed under normal circumstances. They govern default metadata assignments for BB advanced CLI video mode. Most default assignments can be overwritten by object-specific, and file-specific properties files.
The following default video properties files are present in the “/conf/advanced/video” directory:
- archival_directory.properties
- default.properties
- deliverable_directory.properties
- misc_directory.properties
- production_directory.properties
The following settings are set in each file:
archival_directory.properties
#properties for the archival directory
usageClass=LOWUSE
isFirstGenerationInDrs=yes
role=ARCHIVAL_MASTER
default.properties
#directories to scan for content. Each directory could have a
# corresponding _directory.properties file
dirs=archival,production,deliverable,misc
cmid= CMID-12.0
deliverable_directory.properties
#properties for the deliverable directory
usageClass=HIGHUSE
isFirstGenerationInDrs=no
role=DELIVERABLE
misc_directory.properties
#properties for the misc directory
usageClass=LOWUSE
isFirstGenerationInDrs=unspecified
playlists_directory.properties
#properties for the playlists directory
usageClass=HIGHUSE
isFirstGenerationInDrs=unspecified
production_directory.properties
#properties for the production directory
usageClass=LOWUSE
isFirstGenerationInDrs=no
role=PRODUCTION_MASTER
Batch-specific video properties (set by depositor)
Batch-level video.properties file is a required file that needs to be inserted at the root of the batch directory for the advanced CLI video batch. The file contains all required batch level and object level settings. The format of it must be:
- US-ASCII text
- Each line in “property=value” format
- Lines can be commented out by starting the line with the ‘#’ symbol.
- Can’t use reserved XML characters (see the table below for full list).
Property | Valid Values | Example | Meaning |
accessFlag | one of: P,R,N | R | The default access flag to use for this object’s files. Can be overridden by the access flag specified in file level properties files. |
adminCategory | A comma-delimited list of one or more Wordshack URIs | adminCategory=http\://id.lib.harvard.edu/wordshack/adminCategory/12345 | Administrative categories to associate with the object |
alephID | a valid Aleph or Alma Identifier | 009513791 or 990086637850203941 | Aleph or Alma ID for the related bibliographic record in HOLLIS |
authorityPath | a valid NRS authority path | FHCL.Loeb | Authority path for any URNs that will be created for this object’s files |
batchName | 100 character limit; Valid values: ASCII alpha-numeric characters, / "-" / "." / "_" / "~"/ The following reserved XML characters should be excluded: gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@" | Valid value examples: Project_137_4a~5.1a Batch-123_23yux-a.21 Invalid values examples: batch&&<<>>><! project清代盐政与盐稅 | Name for the batch |
billingCode | a valid Billing Code | FHCL.MUSI.MUS_004 | Billing code for the object |
depositAgent | a valid 8 digit HUID | xxxxxxxx | Person depositing the object. |
depositAgentEmail | a valid email address | john_harvard@harvard.edu | Email of person depositing the object. This email has to exactly match the depositor's email as entered in Wordshack. |
objectOSN | A unique value within the content’s Owner Code | A299ff | Owner supplied name of the object (this overrides the object directory name, which is treated as object OSN by default) |
ownerCode | a valid Owner Code | FHCL.MUSI | DRS Owner Code for the batch |
resourcePattern | an NRS resource pattern, see Section 16.1 Assigning File URN Resource Name Patterns | {n} | Resource name pattern for any URNs created for this object’s deliverable files |
role | currently no roles for video object are used | One or more roles for the object (not likely to be used - included to support future uses) | |
failureEmail | one or more valid email addresses separated by commas | Emails to which batch ingest failure reports will be sent | |
successEmail | one or more valid email addresses separated by commas | Emails to which batch ingest success reports will be sent | |
successMethod | one of: EMAIL, DROPBOX, ALL | Distribution method for the loader reports | |
embargoDuration | integer | 80 | Embargo duration value in days, months or years. Not needed if embargoGrantEnd property is used |
embargoBasis | One of the controlled values:
| Harvard policy | Basis for an embargo |
embargoLicense | DRS URN or object Owner Supplied Name or DRS ID of license object in DRS | urn-3:DRS.OBJECT:12345 or objLicense or 400001 | This is where a depositor provides a link to an existing rights object in DRS that will be linked to embargo rights block for the deposited object |
embargoLicenseTerms | free text | My license terms | Corresponds to BB GUI field Documentation Note |
embargoGrantStart | date in format yyyy-mm-dd | 2012-01-01 | Corresponds to BB GUI field Start Date |
embargoGrantEnd | date in format yyyy-mm-dd | 2012-01-01 | Corresponds to BB GUI field End Date |
harvardPolicy | DRS URN or object Owner Supplied Name or DRS ID of Harvard Policy object in DRS | urn-3:DRS.OBJECT:12345 or objHarvardPolicy or 400001 | Link to an existing Harvard policy object in DRS (the next three properties are not related to embargo and are created as separate rights blocks for the deposited object) |
license | DRS URN or object Owner Supplied Name or DRS ID of License object in DRS | urn-3:DRS.OBJECT:12345 or objLic or 400001 | Link to an existing License policy object in DRS |
statute | DRS URN or object Owner Supplied Name or DRS ID of Statute object in DRS | urn-3:DRS.OBJECT:12345 or objStatute or 400001 | Link to an existing Statute object in DRS |
metsLabel | free text | my mets label | METS Label display value. This will display in SDS for deliverable video |
methodology | free text | this is how this object was created | Object methodology |
nonPublicNote | Value for the non-public note field | Value for the non-public note field | |
producer | A comma-delimited list of one or more Wordshack URIs: producer=[wordshack URI],[another wordshack URI]. | producer=http://id.lib.harvard.edu/wordshack/organization/742 | Object producer – controlled value from Word Shack |
relatedLinks | relatedLinks="Relationship=value --- URI=value" Note the space after the relationship value, three dashes and a space before the URI value. The "space three dashes space" is a required delimiter. | relatedLinks="Relationship=LTS --- URI=http://library.harvard.edu/lts" | Related link to any URI - displays in SDS and PDS. |
harvardMetadataLinks | harvardMetadataLinks="Type=LinkType --- Identifier=value --- Label=DisplayLabel" Note the space after the Harvard Metadata Links type value, and then three dashes and a space before the Identifier value. The "space three dashes space" is a required delimiter. The LinkType value for harvardMetadataLinks comes from a controlled list. The following are valid link types (case sensitive): Aleph Alma Finding Aid Gale HLSL HULPR Local OCLC RLIN URI DASH Shared Shelf | harvardMetadataLinks="Type=Aleph --- Identifier=000012345 --- Label=HOLLIS For multiple links the syntax is: harvardMetadataLinks=Type\=Finding Aid --- Identifier\=13527824 --- Label\=Finding Aid,Type\=Aleph --- Identifier\=13527824 --- Label\=HOLLIS,Type\=Alma --- Identifier\=90086637850203941 --- Label\=HOLLIS_ALMA or harvardMetadataLinks="Type=Aleph --- Identifier=000012345 --- Label=HOLLIS”, "Type=Alma --- Identifier=90086637850203941 --- Label=HOLLIS” | Harvard Metadata Links - links to Harvard systems. Provide an identifier type, identifier and a label. This displays in SDS for video and PDS |
has_documentation | Object URN of a related Documentation object in DRS | urn-3:DRS.OBJECT:12345 | Link to an existing documentation object in the DRS |
has_methodology | Object URN of a related Methodology object in DRS | urn-3:DRS.OBJECT:12345 | Link to an existing methodology object in the DRS |
Sample video.properties file:
#properties for the object
alephID=009513791
authorityPath=FHCL.Loeb
batchName=Project_137_4a
billingCode= FHCL.MUSI.MUS_004
depositAgent=12345678
depositAgentEmail=john_harvard@harvard.edu
failureEmail=example1@fas.harvard.edu,example2@fas.harvard.edu
objectOSN= A299ff
ownerCode=FHCL.MUSI
resourcePattern={n}
successEmail=example1@fas.harvard.edu,example2@fas.harvard.edu
successMethod=EMAIL
Place the video.properties file at the root of the batch directory, e.g.:
…/batch/video.properties
File-specific video properties files (set by depositor)
Provide file-level properties files for files in the batch (optional: if needed to override Batch Builder-configured default metadata values). Each properties file contains properties for a corresponding video file. The format of it must be:
- US-ASCII text
- File level properties are shown in the table below
- Each line in “property=value” format
- Properties with repeatable values should be delimited by a comma
- Lines can be commented out by starting the line with the ‘#’ symbol
- Can’t use reserved XML characters
Property | Valid values | Example | Meaning |
fileAccessFlag | one of: P,R,N | P | Access level of file |
fileAdminCategory | A comma-delimited list of one or more Wordshack URIs – | Administrative categories to associate with the file | |
fileosn | Alpha-numeric ASCII | 34561-a7-bach | Name for the file (not necessarily unique) |
isFirstGenerationInDrs | one of: yes, no, unspecified | yes | whether or not the file is the closest version in the DRS to the original capture or creation. |
isPreferredDeliverableSource | one of: yes, no, unspecified | yes | whether or not the file is the recommended file to use for generating future deliverables |
fileNonPublicNote | Value for the non-public note field | my note | |
Processing | comma-delimited list of one or more of: | SCALED, COMPOSITED, ROTOSCOPED, KEYED, EDITED, RETIMED, PULL_DOWN_APPLIED, PULL_UP_APPLIED | Key actions taken on the file before it was deposited to the DRS |
fileproducer | A comma-delimited list of one or more Wordshack URIs – You can alternatively use the same string formatting used in the BB GUI | Currently not supported in BB | File producer – controlled value from Word Shack |
quality | one of: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 | 5 | Locally-defined measure of quality |
usageClass | One of: HIGHUSE, LOWUSE | LOWUSE | Estimate of how frequently the file will be accessed by library users |
filemethodology | (any text) | Notes about the methods performed and/or tools used in the creation, processing or preparation of the content prior to DRS deposit | |
filerole | comma-delimited list of one or more of: ARCHIVAL_MASTER video_DECISION_LIST DELIVERABLE DOCUMENTATION EDIT_DECISION_LIST LICENSE PROCESS_FILES PROCESS_HISTORY PRODUCTION_MASTER THUMBNAIL | DELIVERABLE | |
downloadProhibited (for video files) | yes | yes | If value set to yes: Request only streaming delivery URN. If value is No or NA, request URN for streaming and download delivery. Video file must be streamed - direct download will not be permitted. Note that you this property is required only if value is set to ‘yes’. If the value is set to ‘no’ simply do not include the property. It is set at ‘no’ by default. If the property is provided and set to ‘yes’ you also need to set the downloadProhibitedBasis property listed below. |
downloadProhibitedBasis* (for video files) | One of the controlled values:
| license | Basis for the restrictions or terms of use. *Required if downloadProhibited property is supplied. |
downloadProhibitedLicense (for video files) | DRS object URN or DRS Object OSN or DRS Object ID of the license object | urn-3:HUL.DRS.OBJECT:197548 or ObjMyLicense or 12345678 | ID of the documentation within the DRS, e.g. a deposited donor agreement |
downloadProhibitedLicenseTerms (for video files) | (any text) | 50 year policy | A note about the documentation, e.g. the applicable section |
downloadProhibitedGrantStart (for video files) | A date/timestamp yyyy-mm-dd yyyy-mm yyyy | 2012-02-12 2012-02 2012 | Embargo start date |
downloadProhibitedGrantEnd (for video files) | A date/timestamp yyyy-mm-dd yyyy-mm yyyy | 2062-02-12 2062-02 2062 | Embargo end date |
file_has_documentation | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyDoc or 12345678 | Has documentation in DRS |
fileHarvardPolicy | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyPolicy or 12345678 | related Harvard policy object in DRS |
filelicense | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyLicense or 12345678 | related license object in DRS |
filestatute | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyStatute or 12345678 | related statute object in DRS |
Place the properties file in the same directory as the video file. It needs to have the same file base name as the video For example, the file
batch/object/archival/sub_dir1_Media/file1.properties
would contain metadata for the file
batch/object/archival/sub_dir1_Media/file1.mov
File-specific captions properties files (set by depositor)
Provide file-level properties files for captions files in the batch
Each properties file contains properties for a corresponding captions file. The format of it must be:
- US-ASCII text
- File level properties are shown in the table below
- Each line in “property=value” format
- Properties with repeatable values should be delimited by a comma
- Lines can be commented out by starting the line with the ‘#’ symbol
- Can’t use reserved XML characters
Property | Valid values | Example | Meaning |
fileAccessFlag (required) | one of: P,R,N | P | Access level of file |
fileAdminCategory | A comma-delimited list of one or more Wordshack URIs – | Administrative categories to associate with the file | |
fileosn | Alpha-numeric ASCII | 34561-a7-bach | Name for the file (not necessarily unique). If not supplied in the properties file the fileosn is set as equal to file basename by Batch Builder |
isFirstGenerationInDrs (required) | one of: yes, no, unspecified | yes | whether or not the file is the closest version in the DRS to the original capture or creation. |
isPreferredDeliverableSource | one of: yes, no, unspecified | yes | whether or not the file is the recommended file to use for generating future deliverables |
fileNonPublicNote | Value for the non-public note field | my note | |
fileproducer | A comma-delimited list of one or more Wordshack URIs – You can alternatively use the same string formatting used in the BB GUI | Currently not supported in BB | File producer – controlled value from Word Shack |
usageClass (required) | One of: HIGHUSE, LOWUSE | LOWUSE | Estimate of how frequently the file will be accessed by library users |
filemethodology | (any text) | Notes about the methods performed and/or tools used in the creation, processing or preparation of the content prior to DRS deposit | |
file_has_documentation | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyDoc or 12345678 | Has documentation in DRS |
fileHarvardPolicy | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyPolicy or 12345678 | related Harvard policy object in DRS |
filelicense | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyLicense or 12345678 | related license object in DRS |
filestatute | DRS object URN or DRS Object OSN or DRS Object ID | urn-3:HUL.DRS.OBJECT:197548 or ObjMyStatute or 12345678 | related statute object in DRS |
Place the properties file in the same directory as the captions file. It needs to have the same file basename as the captions. For example, the file
batch/object/captions/file1.properties
would contain metadata for the file
batch/object/captions/file1.vtt
Relationships.csv file (set by depositor)
The required relationships.csv file specifies relationships that are not automatically created by Batch Builder or on ingest. This file should be placed at the root of the batch directory. The file format must be comma-separated CSV with a new line for each relationship. The file paths inside the file need to be in the form supported by the OS on which you are running Batch Builder (POSIX for MAC/Linux and DOS-style for Windows).
- US-ASCII text
- Each line is a comma-separated list of attributes for each relationship as described below
- File paths need to be in the form supported by the OS on which you are running Batch Builder (POSIX for MAC/Linux and DOS-style for Windows)
- Lines can be commented out by starting the line with the ‘#’ symbol.
- Can’t use reserved XML characters (see the table below for full list).
The syntax for each line must be:
[Source_filePath],[relationship_name],[target_type],[target]
where
- Source_filePath:
- for files in this object: relative path to the source file from the object directory for this object
- relationship_name, one of:
- HAS_SOURCE (indicates derivative relationships between files in this object)
- HAS_DOCUMENTATION (for pointing from files or the object to DRS-stored documentation files or objects)
- HAS_METHODOLOGY (for pointing from the object to a DRS-stored methodology object)
- HAS_RIGHTS (for pointing from files or the object to DRS-stored rights files or objects)
- HAS_SUPPLEMENT (points to a closed captions text object in the same batch or in DRS) - if in the same batch use target_type OBJECT_OSN
- target_type, one of:
- RELATIVE_PATH (Use for a relative file path)
- HUL_DRS_OBJECT_URN (Use for the URN of a DRS object)
- OBJECT_OSN (Use for owner supplied name of object in the same batch or object in DRS)
- HUL_DRS_OBJECT_ORACLE (Use for DRS ID of the target object in DRS)
- HUL_DRS_FILE_ORACLE – (Use for DRS ID of the target file in DRS)
- Target, one of:
- relative path to a file,
- DRS object URN,
- DRS object owner supplied name (Object OSN)
- DRS ID of object in DRS
- DRS ID of file in DRS
Sample relationships.csv file:
# Derivative video content relationships
deliverable\Media\elephants-dream-delivery.mp4,HAS_SOURCE,RELATIVE_PATH,production\sub_dir_1_Media\elephants-dream-medium.mov
production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SOURCE,RELATIVE_PATH,archival\sub_dir_1_Media\elephants-dream-hd.mov
archival\sub_dir_1_Media\elephants-dream-hd.mov,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz
production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz
deliverable\Media\elephants-dream-delivery.mp4,HAS_SUPPLEMENT,OBJECT_OSN,elephants-dream-subtitles-en-vz
Place this file at the root of the batch directory, e.g.:
…/batch/
video.properties
relationships.csv
Batch-specific MODS file (set by depositor
If supplying descriptive metadata, create a MODS file called mods.xml complying with MODS schema versions 3.5 or earlier (DRS does not currently support MODS 3.6). Seehttp://www.loc.gov/standards/mods/ for more information about MODS metadata and schemas.
Place this file at the root of the batch directory, e.g.:
…/batch/
video.properties
mods.xml
relationships.csv
Note that as an alternative to supplying a MODS file, descriptive metadata can be imported from HOLLIS through Batch Builder by providing an Aleph or Alma ID in the video.properties file.
EBUCore, Digital Provenance metadata and Source metadata
EBUCore is technical metadata for video. It should be supplied in sidecar XML files that contain EBUCore output for each of the video files that are being deposited. EBUCore can be generated by running the FITS tool on each of the video files. Each EBUCore file should have the same file base name as the video file and should be placed in the same directory as the video file. Instructions. Example EBUCore file: elephants-dream-hd.xml
Digital Provenance metadata should be supplied in a sidecar XML file called digiprov.xml using the ReVTMD metadata schema. The digital provenance files should be placed in 'archival' directory or 'production' directory of the batch. Example file: digiprov.xml
Source file metadata should be supplied in a sidecar file with any name using the UTVideoSrc schema. Source metadata file should be placed in the directory 'original' inside the batch directory. Example file: VideoSrc10-01-2015-DA.xml
VTT Captions for video files
File format: WebVTT (a plain text format that is part of the HTML5 standard.)
File extension: .vtt
Object Role: CLOSED_CAPTION_DATA (set automatically by Batch Builder)
Content Model: Text (one VTT file per object)
Delivery URN: request FDS URN for each closed captions VTT file
Usageclass: “HIGHUSE” (set for file – set in .properties file in the “captions” directory)
FirstGeneration: “Yes” (set for file – set in .properties file in the “captions” directory)
Sample .properties file for captions:
#properties for mycaps1
fileosn= mycaps1-fileosn
isFirstGenerationInDrs=unspecified
usageClass=HIGHUSE
accessFlag=P
Relationships: The text object generated by Batch Builder will contain one VTT file which will be a target of a HAS_SUPPLEMENT file-to-object relationship from the video file to the text object. In order for relationship to be created it needs to be specified by the depositor in the relationships.csv file, where it will point from the video file to the text object containing the VTT file. The syntax is the following:
path_to_video_file,HAS_SUPPLEMENT,OBJECT_OSN,text_obj_OSN (note that in this case ‘obj_OSN’ is derived from the file basename, e.g. for a file mycaps1.vtt, the derived ‘obj_OSN’ is ‘mycaps1’)
Relationships.csv example:
archival\sub_dir_1_Media\elephants-dream-hd.mov,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1
production\sub_dir_1_Media\elephants-dream-medium.mov,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1
deliverable\Media\elephants-dream-delivery.mp4,HAS_SUPPLEMENT,OBJECT_OSN,mycaps1
This relationship points to a text object that will be created with the file mycaps1.vtt
Note that multiple video files may be the sources of HAS_SUPPLEMENT relationships pointing to the same text object containing a VTT file. Additionally, a single video file may be the source of HAS_SUPPLEMENT relationships to multiple Text objects, each contacting a VTT file.
Placement: Closed captions WebVTT files with extension “.vtt” should be placed into “captions” directory in the Batch Builder video object directory structure. Corresponding .properties files should be placed in the same directory. All captions files and corresponding .properties files for a batch should be placed in the same directory. See sample directory structure below.
Object OSN: object owner supplied name should be the generated based on the file basename (basename.vtt) of the caption VTT file.
File OSN: file owner supplied name should be the generated based on the file basename (basename.vtt) of the caption VTT file unless it is overridden in the file-level properties file.
Workflow:
VTT files with extension “.vtt” should be placed into the “captions” directory in advanced CLI directory structure. The “captions” directory should be located directly inside the object directory in the batch. The HAS_SUPPLEMENT relationships between the text objects containing the VTT files and the video files should be specified in the relationships.csv file, located in the batch directory.
Each VTT file will be assembled by Batch Builder into a single Text object with object-level role CLOSED_CAPTION_DATA. For each text object, a file-to-object HAS_SUPPLEMENT relationship will be created from the video file to the text object (as modeled in relationships.csv file).
Preparing content files
Prepare content files:
- consult Harvard Library Digital Preservation wiki for DRS-accepted video formats: https://harvardwiki.atlassian.net/wiki/x/zoHeAg
- generate video files in one of the accepted formats
- generate video files for delivery according to DRS Streaming Delivery Service video specifications: H264 encoded video and AAC encoded video in Quicktime container with .mp4 extension
- generate EBUCore files for each video file by running the FITS tool on each file and creating EBUCore standard output
- generated captions files in VTT format
- generate digital provenance metadata in a sidecar XML file called digiprov.xml using the ReVTMD metadata schema
- generate original source file metadata in a sidecar file with any name using the UTVideoSrc schema
Arrange the content in the directory structure required by Batch Builder
Below is a sample directory structure created in preparation for use with Batch Builder. Directory names and file names (or parts file names) in bold must be named as shown.
Figure 1: Sample Directory Structure
/batch/ object/ archival/ [1..n] digiprov.xml (ReVTMD) sub_dir1_Media/ [1..n] File1.mov production/ [1..n] REQUIRED EVEN IF EMPTY sub_dir1_Media/ [1..n] File1.mov deliverable/ [0..n] Media/ [1..n] File1.mov captions/ mycaps.vtt original/ source_metadata.xml (UTVIDEOSrc) misc/ (miscellaneous process files placed in this directory are zipped by Batch Builder and moved to '/workspace/moved_content' |
Command Line Syntax
The existing Batch Builder Windows bat and Unix shell scripts are used to launch Batch Builder in advanced mode. The syntax is:
- On PC: batchbuildercli –advanced video –d [object directory path]
- Use Windows style directory separator '\'
- On Mac or Linux: ./batchbuildersli.sh -advanced video –d [object directory path]
- use UNIX style directory separator '/'
Note that by default the object directory name serves as the object Owner Supplied Name (you can override that in video.properties file).
Example 1 (Windows): batchbuildercli –advanced video –d "C:\mybatch\004390909"
Example 2 (Mac / Linux): ./batchbuildercli.sh -advanced video -d /mybatch/004390909
Additional command line optional properties
All command line optional properties for deposit settings, object settings and file settings that are available for "simple CLI" in Batch Builder are also now available in advanced CLI. See more here: https://harvardwiki.atlassian.net/wiki/x/xw2VAg
Process the Batch
Use command line syntax above to call Batch Builder to process your batch.
The program will:
- Create a directory at workspace/moved_content
- Put a copy of the files that were in the misc directory in the workspace/moved_content directory
- Zip up the contents of the misc directory and name it misc_files.zip
- Create the object descriptor for the video object
- The contents of EBUCore files supplied with each video file will be inserted into the descriptor
- The ReVTMD digiprov.xml digital provenance files will be referenced in the filesec and structmap of the descriptor
- The contents of the UTVideoSrc original object metadata file will be inserted into the descriptor
- Build a separate Text object with role CLOSED_CAPTION_DATA for each captions file
- The file name will be used as the object owner-supplied-name
- Create the object descriptor for each object
- The descriptor will be named {object-OSN}_descriptor.xml and will be written to the batch directory
- Relationships between the text object and video files will be parsed from the relationships.csv file
After executing Batch Builder the following changes should be visible in the batch directory:
/my_batch-19a326/
batch.xml (DRS batch loading file)
my_object-A299ff/
misc/
misc_files.zip (files that were in the misc directory)
workspace/
moved_content/ (contains content in the misc directory)
descriptor.xml (object descriptor for the video object)
mycaps_descriptor.xml
Depositing to the DRS
After executing the script, copy the following parts of the batch directory to your DRS dropbox.
/my_batch-19a326/ (example batch directory)
batch.xml
my_object-A299ff/ (example object directory)
descriptor.xml
mycaps_descriptor.xml
archival/
deliverable/
misc/
original/
production/
captions/
Notes:
- Object owner-supplied-names (Object OSNs)
Object OSNs are required in DRS 2 and must be unique within a Owner Code.
- URNs
- A DRS object URN is automatically assigned to all objects. These URNs will have a HUL.DRS.OBJECT authority path, e.g.urn-3:HUL.DRS.OBJECT:175844
- File-level delivery URNs are automatically given to video files with the role DELIVERABLE and two VTT captions files (no file role is needed for VTT captions files). These URNs will have a local authority path, e.g.urn-3:FHCL.Loeb.Faids:mus00023. Each video file that gets a delivery URN will get an SDS (streaming) URN and an FDS (download) URN unless DOWNLOAD_PROHIBITED is specified in the rights for the video file. Each VTT captions file will get an FDS URN.
On this page:
In this section: