DevSecOps Transformation Project Charter

PDF Version:

I. Problem/Value Statement

Problem Statement

Applications supported by Library Technology Services (LTS) are a combination of custom-developed, open source, and commercial systems hosted in both cloud and on-premises server environments. Our team of 37 staff provides an ambitious level of maintenance, support, and development activity to sustain operational excellence and continuous service improvements.   

The lack of modern, robust, and automated DevOps tooling and practices is severely hampering our ability to efficiently deliver high-quality services.  We estimate that LTS software engineers dedicate about 30% of their hours to administrative activities such as software patching, deploying, and upgrading. We estimate that our LTS Operations engineers spend about 25% of their time managing patching, deployments, and redeployments. With modern tooling, the aforementioned activities should account for less than 10% of staff time, which would facilitate the delivery of enhancements and new services at an accelerated pace.

​Of greater concern than efficiency, is security. Currently, our development practices are characterized by largely-manual patching, upgrading, and redeployment procedures. This way of working not only hampers our ability to quickly address critical information security vulnerabilities, it also takes engineers’ attention away from strategic project initiatives. Implementing automated processes to replace manual activities will significantly increase the rapidity with which we can resolve security concerns while also reducing project schedule disruptions. ​​​The climate of increasing cybersecurity threats underscores the urgent need to automate our DevOps practices. 

Business Value

We have an opportunity to make a critical investment in new infrastructure and innovative practices that will support operational excellence, system integrity, security, and our ability to deliver new services with agility and speed. ​​ 

With the implementation of a “DevSecOps transformation,” development and operations engineers will gain productivity efficiencies that will allow for simpler application deployments, regular patching to keep projects (middleware and system components) up-to-date, and faster responses to critical security vulnerabilities when they are discovered. Some specific examples include:  

• ​Less time spent manually performing tests or running test suites. A new continuous integration/continuous delivery (CI/CD) pipeline will handle test runs automatically.  

• ​Increased development velocity. 
• There will be a reduced need to take engineers away from strategically-important projects to address urgent upgrades.  
• There will be reduced time engineers need to take on manual image creation and deployment, freeing up time to focus on application development.
• ​Significant decrease in time dedicated to manual patching. The goal is 100% automated patching. For scenarios where automated patching fails, we will have the earliest indicator possible to allow us to plan and prioritize any development work needed to support the latest versions.  
• ​Reduced service disruption due to unanticipated security patching and deployment. Automation and scheduling will afford proactive rather than reactive responses to CISA Known Exploited Vulnerabilities (CKEV) notices.  
• Enhanced cross-team communication for more efficient resolution of issues across functions.  
• ​Increased time for engineers to focus on more complex problems rather than handling middleware and OS patching manually.  
• Cost savings. We will be less likely to experience security incidents that could result in financially (or reputationally) costly issues. ​​ 

In addition, we hope that the LTS DevSecOps transformation will serve as a model that will benefit our colleagues in other departments across HUIT as they endeavor to solve similar issues.

Vision and approach

LTS will have a robust, comprehensive, end-to-end CI/CD pipeline using Kubernetes and associated tools that will allow us to continually deploy new software and patches easily and quickly.

Approach

We anticipate this will be rolled out in two phases of work.

• Phase 1: assessment and planning. LTS will collaborate with a vendor partner and domain expert, Oteemo,  to assess our current software and CI/CD landscape. Oteemo will prepare an implementation plan for a full CI/CD pipeline utilizing Kubernetes.
• Phase 2: Implementation. LTS will implement the CI/CD plan across the LTS portfolio.

LTS has created a detailed roadmap for what the CI/CD pipeline plan should cover. Specific targets include:

• Best practices for CI/CD pipelines
• Moving off of Docker Swarm into a Kubernetes-based environment
• Migrating some of our more specialized/legacy apps into Kubernetes (taking into account some of our outliers)
• Recommendations and implementation guidelines for open-source tools when possible
• Realistically achievable by the LTS team

Alignment with Harvard Library Multi-Year Goals and Objectives (MYGOs)

• Adopt a sustainable organizational approach to technology lifecycle management and preparing for the future
• Simplify and advance systems to preserve, store, and access library digital assets

Alignment with HUIT Objectives and Key Results (OKRs) and HUIT IT Priorities

This work aligns with the following HUIT priorities:

1. Information Security: Continue to invest in information security efforts related to awareness, risk assessment, and operations.
2. Change Management: Improve ability to deliver projects and programs to the community through best practices and training related to change management and business process assessment.
3. Operational Excellence: Develop a plan for automation for critical, frequently used or heavily manual workflows.

In Scope/Out of Scope

In Scope

• All containerized LTS applications run using Docker Swarm

Out of Scope

• Non-containerized applications (such as DASH)
• Implementation/application to services outside the LTS portfolio

Deliverables and Work Products

Key Tasks and Outcomes

Definition of Done

This project will be considered done when LTS has completed implementing the CI/CD pipeline plan accepted from Oteemo.

Stakeholders

Project Team

Estimated Schedule

Assumptions, Constraints, Dependencies, and Risks

Project Assumptions

• Library Technology Services Leadership Team (LTSLT) have identified the appropriate subject matter experts to participate in the Working Group and who can accurately and completely define the business requirements for the project
• LTSLT will make available the time required to participate in project activities and to complete tasks as requested
• Project sponsor and other stakeholders are empowered to make the decisions required for the project to be a success

Project Constraints

• Scope
• The scope is only for LTS projects, and as we evaluate some bespoke items may be excluded from the CI/CD process (example: DASH, which is not containerized)
• Time
• Phase 1 has a time limit of approximately one month (give or take the time needed for plan review and acceptance)

Project Dependencies

• Staff availability: we must have staff available for Oteemo to work with as needed and then for the implementation phase (phase 2)
• This includes potential development work on systems that are unable to run in Kubernetes, which may represent a significant time commitment
• Oteemo staffing
• HUIT requirements for security and software
• Service availability (we can’t take things offline)

Project Risks

Acceptance

Accepted by Stu Snydman, 3/17/2023

Prepared by Maura Carbone

Edited by Sara Rubinow

Effective Date: 3/17/2023