6. Working with Email Metadata
On this page:
Working with Email Metadata
EASi allows you to assign administrative and descriptive metadata to email messages and attachments before this content is uploaded to the DRS. Metadata can be added to a single item via the item's Full Display, or added to a group of items via the Brief Display.
Tip: Attachments have their own metadata fields and do not inherit metadata from their associated message. But you can assign the same metadata to message and attachment by using bulk metadata editing.
For definitions and values of individual metadata fields, consult the Metadata Reference section in Appendix A.
Bulk editing a group of items
Bulk editing involves making the same changes to a group of items using options on the Brief Display. Bulk editing can be a time saver, especially when you need to add the same metadata to a message and related attachments. Edits are saved in real time and are viewable/searchable immediately.
Tips about bulk editing:
- Visibility of existing metadata is limited. When editing metadata on an item's Full Display, you can see existing values easily before making changes. When bulk editing a group of items, your view of metadata assigned to those items is more limited. To improve your view, try exposing additional metadata columns in the results list (hover over column headings to get a list). Also, click the lightbulb icon next to a field in the metadata editing panel to see the values assigned to items in the selected set.
- Limits on removing metadata. Complex fields (review date and rights) can be removed in their entirety but individual components cannot be removed separately. Public and non-public notes cannot be removed during bulk editing.
- Saving edits to large groups takes time. For large groups of items (more than a few hundred), there may be a brief delay while the system saves metadata changes. EASi will display an Update Item pop-up window while saving is in process.
Here is the procedure for bulk editing from the Brief Display.
1. Perform a search and/or apply filters until you have an appropriate result set.
2. Select items in the results list to be edited.
* To select individually, click on the row or checkbox of eligible items.
* To auto-select all items on the current results page, click the check box above the Select column.
* To auto-select all items in the results set, click the Select check box above the list and then click the "Select all xxx search results" link.
* Tip: When all items in the results set are auto-selected, avoid deselecting individual items or any selections beyond the first page will be lost.
3. Open the metadata panel by clicking the edit metadata link above the results list.
4. [Optional] Click the lightbulb icon next to a metadata field to expose any existing values assigned to items in the selected set.
5. Make metadata selections as needed. Within a single bulk edit activity, you can perform multiple edits.
Some metadata fields can be edited directly, but for most you need to click an icon to open an editing window. Here is a key to the edit icons:
Add metadata values to items in the selected set.
Remove a metadata value from items in the selected set.
Replace metadata values. Use this option to replace metadata values in a selected set of items. You can use Replace to add, change, or remove values for all items in a set, or for items in the set with a particular metadata value. In this release, Replace has been implemented for single-value fields: Accession ID, Billing code, Collection, DRS access flag, Processing level, and Series.
Clear all values in the selected metadata field for items in the selected set.
6. Click the Save button to commit the changes. An Update Item pop-up window will display.
If the Save button is not highlighted, click anywhere on the metadata panel to activate it.
7. When the pop-up changes to Update Complete, click OK to clear the window.
8. If you need to close the metadata panel, click the close form link next to the Save button.
Editing a single item
The metadata for messages and attachments can be edited from the Full Display of the item. Edits are saved in real time and are viewable/searchable immediately. Here is the procedure:
- On the results list, click the EASi ID of the item to be edited. A Full Display of the item will open in a new browser window or tab.
- Make metadata selections as needed. Within a single bulk edit activity, you can perform multiple edits. See the Metadata Reference appendix for definitions of each metadata element.
- Click the Save button to commit the changes. If the Save button is not highlighted, click anywhere on the Full Display to activate it.
Handling rights metadata
This section describes the initial EASi implementation of rights and access metadata, which is part of a broader project to add support for rights metadata to the new release of DRS ("DRS 2").
The "Rights" segment of email metadata is where the curator specifies the terms and restrictions on email content stored in the DRS. Rights metadata is optional and should be applied to email messages and/or attachments as needed.
About rights metadata
An EASi rights segment includes:
- Rights basis: the statute, license, policy, etc. that is the basis for the rights granted and any restrictions associated with the content. These values for rights basis have been defined so far:
- Harvard policy: rights and restrictions that derive from Harvard policies (e.g., HU Archive's "Corporation Vote of March 13, 1995").
- License: legal agreement between two or more parties (e.g., contracts, donor agreements).
- Risk assessment: rights and restrictions that arise from a curator's judgement (e.g., risk assessment in the absence of specific license, statute, or policy).
- Statute: governmental regulations and laws (federal, state, city, etc.).
- Document: a link to a document stored in the DRS that documents the selected rights basis. The document dropdown list contains the DRS owner supplied names of rights documents available in your owner code or the HL shared owner code (H.LIB.SHARED).
- Document note: a free text document note if additional information about the basis documentation is needed.
- Restrictions that apply to the content as derived from the basis. Two types of restrictions can be specified:
- Secure storage restriction: If content is deemed sensitive, the curator can specify that it be stored on a secure network. Selecting secure storage will initiate an automated system action (storage on secure network, or not, as indicated).
By default, all content imported into EASi will have a default rights segment that designates it as requiring "secure storage (unconfirmed)" using the "risk assessment" rights basis. If review indicates the content is indeed sensitive, you should change the value to "secure storage (confirmed)". If deemed not sensitive, you should delete the rights segment.
Note: Both the "confirmed" and "unconfirmed" options will trigger secure storage. Leaving this option set to "unconfirmed" is an indication that the content was not reviewed before being archived.
- Delivery embargo restriction: Includes start date, end date, and duration of an embargo on access to the content. Note that currently, embargo metadata is informational and there is no automated system action associated with it. Access is still determined by the DRS access code ("P" for Public, "R" for Restricted to Harvard Community, or "N" for no access except to an authorized DRS account-holder). Also, in a later prototype a curator will be able to supply two embargo values and EASi will derive the third value.
If there is more than one basis that applies to a message or attachment, you can create multiple rights segments. For example, if a statue and a license both apply to a message, create a separate rights segment for each.
Viewing rights metadata
In full display, look for the "Rights " area in the metadata panel. Each rights segment is represented by a display label comprised of the rights basis plus the related rights documentation. Mouse over this label to view the details. Note that when you mouse over, only the fields actually completed by the curator will display.
This example illustrates the default "risk assessment" rights segment that is assigned to all content imported to EASi:
This example illustrates a rights segment that uses the "Harvard policy" basis.
In the brief display, there are six rights-related metadata columns: rights doc, rights doc note, secure storage, embargo start date, embargo end date, and embargo duration. Expose some or all of these columns if you want to see rights metadata associated with items in the results list.
Adding or deleting a rights segment
You can add or delete rights segments but it is not possible to edit an existing rights segment. Instead of editing an existing segment, you will need to create a new segment with the needed values and delete the unwanted segment.
Adding a new rights segment:
You can add a rights segment from the brief or full display.
- Click +add to open the rights information data entry form.
- Select a rights basis. This field is required to save the rights segment.
- Select a rights document that corresponds to the basis. This field is required if the selected basis is policy, license, or statue.
Click the down arrow to display a list of DRS2 object owner supplied names (OSN) of rights documents available in your owner code or in the HL shared owner code (H.LIB.SHARED). Click to select the appropriate document. - Add a document note if needed. This might be a note that clarifies the applicable section of the rights document.
- Add embargo data if needed.
Note: Enter data in all fields (embargo duration, start date, end date). - Select a secure storage option if needed. Select “required: confirmed” if you verified the need for secure storage. Select “required: unconfirmed” if you did not review content before assigning secure storage. Selecting either value will result in the use of DRS2 secure storage.
- Click the Add button to complete this rights segment.
- Back on the metadata panel, click the Save button to save this change.
Deleting a rights segment:
- Open the full display of a message or attachment.
- Click the red x next to the rights segment you want to delete. The segment label will display as crossed out.
- .Click the Save button to save this change.
Sample rights use cases
The following sample use cases illustrate how rights metadata might be assigned in specific circumstances. Coding of the DRS "access code" is included in each sample -- although not technically part of rights metadata, the access code ("P" for public, "R" for restricted to Harvard, "N" for not accessible) controls access to content stored in the DRS.
A. Curator finds sensitive information (e.g., credit card number) in a two year old email message.
The message is too valuable to delete, but the recent credit card data is a concern. The curator decides to assign the secure storage restriction and a 15 year delivery embargo. In this case, no other rights bases apply, so curator selects the "risk assessment" basis.
Rights and access metadata values:
Access code: | N |
Rights basis: | risk assessment |
Document: | [blank] |
Document note: | creditcard# |
Embargo duration: | 15 |
Embargo start date: | 05/01/2009 |
Embargo end date: | 05/01/2024 |
Secure storage: | required confirmed |
Note: If the message had been 15 years old and credit card owner was deceased, the curator might decide there is no risk. In the absence of other rights bases, rights metadata would not be needed.
B. Collection of business email from a retired Harvard employee.
This is a case of university administrative records, which are restricted from delivery for 50 years from the date of their creation. Curator selects the rights basis "Harvard policy" and assigns a delivery embargo with start date that matches the message sent date. In lieu of reviewing individual messages in the collection, the curator decides to assign the secure storage restriction.
Note: If individual messages were reviewed and deemed free of sensitive information, the secure storage restriction would not be necessary.
Rights and access metadata values:
Access code: | N |
Rights basis: | Harvard policy |
Document: | Vote of the President and Fellows of Harvard College, October 3, 1988 |
Document note: | 50 year rule |
Embargo duration: | 50 |
Embargo start date: | 05/01/2011 |
Embargo end date: | 05/01/2061 |
Secure storage: | required confirmed |
C. Email collection has donor agreement with restrictions.
The donor agreement specifies that messages from family members must be restricted until January 1, 2050. The curator selects the "License" rights basis and sets an embargo end date as specified by the agreement. Given sensitivity of the content, curator also decides to apply the secure storage restriction.
Rights and access metadata values:
Access code: | N |
Rights basis: | License |
Document: | Holly Near donor agreement |
Document note: | family messages restricted until January 1 2050 |
Embargo duration: | [blank] |
Embargo start date: | [blank] |
Embargo end date: | 12/31/2049 |
Secure storage: | required confirmed |
D. Email collection has donor agreement without restrictions.
The content does not involve university records, is not sensitive, and the donor agreement places no restrictions on delivery. Contents of this collection do not require rights metadata. The curator deletes the default "risk assessment" rights segment that was automatically added upon import to EASi.
Handling import flags
During import to EASi, email messages are filtered for social security numbers, credit card numbers (and in the future, viruses). When EASi detects the potential presence of this data, a message is assigned an import flag. This flag is a notification to the curator to review the content during EASi processing and take appropriate action (apply access and rights metadata, or delete the message).
About EASi import filters and flags
A "filter" is the utility used by EASi during import to scan messages for sensitive data. When a filter detects sensitive information, EASi sets an import flag in the message. In this prototype, EASi filters check data in the subject line and body of the message, but not attachments.
- Credit card filter: Messages identified as containing credit card numbers are assigned the "CREDITCARD#" flag. Since credit card number formats can vary, the EASi import filter is set to a loose match, which means there is a good chance of getting false positives (messages flagged as containing sensitive data when they do not).
- Social security filter: Messages identified as containing social security numbers are assigned the "SS#" flag. Format of social security numbers is well known so the EASi import filter performs tight matching, which means false positives are much less likely.
Import flags are always assigned by the system and cannot be assigned manually by curators. A curator can and should remove a flag if review of the message indicates the data is not present.
Reviewing flagged messages
- On the Brief Display, expose the "Flag" column. Mouse over any column and click the down arrow to open the column list. Scroll down the list and check the "Flag" choice.
- Search or filter on the import flag.
To search, enter a fielded search following any of these examples:
* flag:creditcard# (will retrieve messages with this flag value)
* flag:ss# (will retrieve messages with this flag value)
* flag:[* TO *] (will retrieve messages with any flag value)
To filter, enter a specific flag value (creditcard# or ss#) or [* TO *] (any flag value) and select "Flag" from the dropdown list. - Review the subject line and body of each flagged message to verify the presence of credit card or social security numbers. You will need to open the full display of a message to view the full message body.
- If nothing is found, you can delete the import flag. In the metadata panel on the left, click the red X to mark the flag for deletion.
- If you find a credit card or social security number:
- Apply the appropriate access and rights metadata to the message. Or, you can delete the message (not supported in this prototype). See Sample Rights Use Cases for guidance on the appropriate access and rights metadata choices.
- Delete the import flag.
- Click the "Save" button to save all metadata changes.
Using the "Needs Review" flag
The "Needs Review" flag is a metadata field associated with email messages and attachments. Curators can use this field to flag messages or attachments that need additional review. This might be review performed on items in EASi, or review on items after the content is archived in the DRS.
Viewing the review flag
In full display. The "Needs Review" flag has two components -- a review date and free text note. EASi uses the review date as the flag's display value (if no date is supplied, the flag will display as "no date"). To see the note, mouse over the flag label and the note opens in a callout.
In brief display. In the brief display, expose the "review date" and "review note" columns to see any review flags assigned to items in the results list.
Searching and filtering on review flag
You can search or filter on keywords in the note text component of a review flag. In this prototype, retrieving on review date is not well supported yet because of the complex ISO8601 date format currently used.
To filter by keywords in a review note, select the "Review note" filter. To search a note, use the search label "review_note:".
Adding and removing review flags
One or more review flags can be added to an item from the full display or when bulk editing on the brief display. Note that neither the date nor the note component is required, but one of these must be present to set the flag.
In this prototype, you can delete a review flag only from the full item display. Editing an existing review flag is not supported, so you will need to delete the flag and add a new one instead.
Checking process history
EASi tracks significant processing events for individual messages and attachments. This information is available on the full display, by clicking the View process history link below the metadata panel.
Processing events tracked in this release include conversion to the RFC-2822 internet message format (when imported to EASi) and removal of an import flag.
When messages or attachments are deleted, deletion history is recorded in the packet’s process history. To view deletion history aggregated by packet, go to Packets > Process History from the main menu.