Generative AI STEM Pedagogy (GAISTEMP) Initiative

Our 2 year goal, by the end of AY 24-25, is to produce GAI-enabled course structure across the introductory STEM curriculum.
The tools should be agnostic to the textbook(s) used in the course, and should be implementable with a minimal time investment in courses nationwide.
Initial backend is likely to be ChatGPT, currently at version 4, but we should strive to build an architecture that can evolve and adapt to new and better and different tools. We need to have an overall architecture philosophy evolve with our development spirals.

This effort will take place in stages, as follows:

Summer 2023 - establish structure for the process, secure resources, build team, delineate clear goals for the Fall 2023 term, define a subset of experiments for Fall 2023. Initial tool development for API-interface exploitation.
Develop an experimental active learning lecture series that incorporates GPT-4 capabilities and measure its effectiveness in student comprehension and retention.
Devise 8 experiments to do in Fall 2023
First development sprint for 8 experiments

Fall 2023 - Make all STEM courses and instructors at least GAI-aware.
Conduct initial set of 8 GAIPedagogy experiments across the STEM curriculum. Tracking of pedagogical experimental outcomes.
Oct 2023 workshop for Harvard STEM faculty.
Planning and invitations for GAISTEMP-1 conference.
Development sprint for API-interface tools

January 2024- Three day national Workshop on STEM GAI pedagogy.
Development sprint for GAI active learning and HW tools.
First draft of GAISTEMP experimental-results paper from first 8 examples

Spring 2023 - General Education course offered.
Rollout of prototype active learning and HW modules in 4 intro STEM courses, with assessments.

Summer 2024 - Extension of prototype active learning and HW modules to all of intro STEM curriculum, with assessments.
Development sprint for course modules.

Fall 2024 - First offering of GAI-empowered courses across entire introductory Harvard STEM curriculum. Prepare for second national workshop.

January 2025 - Second national workshop: GAISTEMP-2.

Spring 2025 -

January 2025 - Third national workshop: GAISTEMP-3.

Summer 2025

Management structure and resources.

We will coordinate and perhaps embark on joint projects with MIT. We will designate an MIT liaison subgroup. Cadence of those meetings TBD.
We will coordinate across the various elements of the Harvard community by participating in quarterly stakeholder meetings.
Our divisional coordinator will be Assistant Dean for Science Education Logan McCarty.
We will engage undergraduates, graduate students, postdocs, staff, and faculty in this effort.
Our primary perspective will be to support faculty in the incorporation of GAI tools into our learning program.

Fall term 2023

Instructional methodology viewpoint: (pick 8 experiments for Fall 2023)

Item	experiments	courses	GPT aspect needed
A: Incorporation of GAI into lecture-format STEM learning: Develop experimental active learning lecture modules that incorporate GAI capabilities, and devise methods to measure their effectiveness in student comprehension and retention. Use multi-shot learning to have GAI produce 'clicker' questions and active learning modules. Have GAI do assessments of online active learning modules. Increase accessibility of course materials	Assess utility of synthesis of student-provided question content, in-class just-in-time learning Assess utility of synthesize summaries of open-response clicker questions rather than multiple choice, for adaptive lectures Assess utility of peer-instruction that includes GAI virtual partners Assess utility of GAI-produced active learning modules Compare restricted-content (textbook and lecture materials) vs. all-content GAI generated active learning modules.		shared-edit document fed into GAI, results available to instructor. Multi-shot learning using restricted content and open PPT and math content upload as tokens
B: Develop and Exploit short-cycle adaptive problem sets with real-time feedback produce a 'virtual tutor' that uses a script of desired learning goals to promote student learning distinguish between conceptual mastery and ability to solve quantitative problems. To what extent can GAI do the math part?	Assess learning effectiveness of rapid-feedback interactive learning, outside of lecture. assess zero-shot vs multishot training for virtual tutor effectiveness. Assess instructor burden compared to current methods.		Ability to identify main topics, and embed in GAI interaction. Arithmetic needs to work.
C: Assist with analysis and gain insights from lab data.	Assess effectiveness of using the combination of GAI and Jupiter notebooks to do guided data exploration.		notebook and GAI integration ability to access data files from instructional lab data collection. Does this mean a browser interface with access to local files?

D: interactive student self-assessments.	Assess utility of GAI in helping student prioritize where to focus to enhance their mastery of material.
F: GAI-assisted student assessments and grading			ability to capture GAI sessions, with unique student identifiers
G: Temperature-dependence of learning effectiveness	Assess impact of 'temperature' setting for student learning
H: Ascertain subject-level mastery needed to exploit natural-langauge-driven code development. Try out a non-analytic problem and assess the results.	Use things like modulated-friction example, Modulated Friction example , to help students learn how to assess and refine GAI produced material.	15 a,b,c Math 22	numerical solution code
I: incorporate into HW and assessments the ability to perform calculations, as pioneered by Khan Academy	same as B		arithmetic capability
J: incorporate course-specific training inputs and give that high weighting			custom training inputs
K: Automation grading and assessments of student competence.			sequential prompts run open loop, no adjustment
L: dynamic tutoring	same as D		sequential prompts with iterative adjustment
M: GAI assisted generation and refinement of course instructional and assessment materials- HW, exams, quizzes, etc.	request a critique of exam questions request answer key to HW and exam questions
N: Course management- facilitating course selection	upload Canvas course materials and any ppt from previous year, including student CUE evaluation narratives and restrict responses to that material.		Need to restrict dissemination of uploaded CUE score materials.
O: Course management-	restricted-scope utility using uploaded content integration with grading systems integration with Canvas automated sectioning tool integration with Slack? multi-shot		need to mesh student user account on GAI system with their HUID and course record. Instructor-level overview of
P. Validation and verification tools and methods Develop methods for assessing and refining GAI-produced code	Assess level of expertise needed to get credible answers Conduct iterative-engagement tests		real-time access to URL's provenance tracking code validation and verification

Deliverable	impact (1-3 with 5 being high) 1: single course 2: 3-5 courses 3: >5 courses	effort scope (1-3 with 3 being low) 1: > 1 calendar month 2: ~ 1 calendar month 3: < 1 month	merit
classroom synthesis: Google form gathers student input, sent to canned prompts and displays on screen, 1-button workflow for instructor	3	1	3
implement nano-GPT in jupyter notebook	2	3	6
implement ability to execute queries on course-specific uploaded materials	3	1	3
facilitate upload of student sessions as parsable text into grading system	3	2	6
implement course management tools- grade sheet analysis and interrogation	3	2	6
incorporate chat functionality into course Slack channel	3	3	9
new-course development toolkit	3	2	6
assessment of student preparation, based on instructor expectations	3	2	6
assessment of student preparation, based only on uploaded course materials	3	1	3
instructional lab data browsing, tailored to individual courses	1	1	1
generation of lecture notes from recorded transcripts and video capture of blackbaord?	3	1	3
Dynamic HW system, multishot training to guide student interaction	3	1	3
Assess optimal temperature setting to maximize learning effectiveness (this is an experiment)	3	1	3
Assess performance of image capture HW assessment. "work this problem then upload image of your work"	3	2	6

respondent	email	notes
Carlos Argueles	carguelles@fas.harvard.edu	Hola Chris: Thanks for sending this email. I am very excited about this new technology. In fact, with Louis Deslauriers, we used some of it in my Physics 15B last semester. I am currently writing a proposal for the Cottrell Scholars, where this will be featured, and I intend to use them in my incoming classes. I will look at the links in your email. Have a wonderful summer, cheers, Carlos

David Malin		using it now in CS50 with custom interface

Course-based viewpoint

Assess our assessments: run midterm and final exams of science courses through GPT-4 and grade the results. Compare to overall student performance.
Enhance our assessments. Solicit constructive feedback on the exam questions we submit.
Assess our homework: run homework assignments through GPT-4 and grade the results. Compare to overall student performance.
Enhance our assignments. Solicit constructive feedback on the homework we submit.
Ask (require?) students to use GPT-4 on selected assignments to get feedback and examples of how it can be used.
Course-specific chat-bots- what training data?
For large lecture classes- merge active learning with GPT
For sections- aggregation of questions,
For labs- try out data analysis methods and inference
Customized training assembly of material - what do we need to start to capture?
Khan academy like adaptive tutorials
How does this shift the workload in our non-ladder teaching capacity? Especially sections and TFs and grading?

Curriculum-based viewpoint

What are the high-level learning goals for our students, in a GAI world?
How can we critical thinking skills and a truth-validation mindset?
What progression of instruction and course expectations are appropriate?

Student-centric viewpoint

Learning how to craft a prompt that gets what you want
GAI as a consultant
GAI for self-assessment
iterative refinements in GAI interactions
Learning how to validate and verify results
Honing critical thinking skills in the GAI context.
Ethical, responsible, thoughtful use of powerful tools.
Accommodating disabilities and ensuring equitable access.

IT-centric viewpoint

How do we integrate these learning tools with existing platforms? Examples include Canvas, grade sheets, Sharepoint, Jupiter notebooks, Open OnDemand, data repositories, assignments, work-uploading tools, etc?
What is the best approach to licensing and token-purchasing?
How do we throttle and regulate non-course abuse?
How do we develop, curate, and support the use of this new toolkit?
What are institutional roles, responsibilities, accountabilities, and authorities?
What staffing is needed, at what levels of the organization?
Who pays for what?

Specific examples :

Modulated Friction example

Summer 2023

Aug 23 professional development session

Spring term 2024

Gen Ed 1188 https://gened.fas.harvard.edu/classes/catching-tsunami-riding-gpt-wave

Gen ed 1188 course management

GAISTEM core team

Who?	From?	email
Christopher Stubbs	FAS Sci Div	stubbs@g.harvard.edu
Logan McCarty	FAS Sci Div	mccarty@fas.harvard.edu
Greg Kestin	FAS Sci Div	kestin@fas.harvard.edu
Erin Collins	FAS Sci Div	erin_collins@fas.harvard.edu
Jefferson Burson	HUIT	jefferson_burson@harvard.edu
Ventz Petkov	HUIT	ventz_petkov@harvard.edu
David LaPorte	HUIT	david_laporte@harvard.edu
Colin Murtaugh	HUIT	colin_murtaugh@harvard.edu.
Rebecca Nesson	SEAS	nesson@g.harvard.edu
Eske Pedersen	FAS Science	eskepedersen@fas.harvard.edu
Lawrence Eribarne	HUIT	lawrence_eribarne@harvard.edu

Departmental

GAISTEM stakeholder group, quarterly meetings

stakeholder	representative
College OUE	Amanda Claybaugh Anne Harrington
Harvard College	Rakesh Khurana
Bok Center	Tamara Brenner, Adam Beaver
HGSE
humanities div.	Robin Kelsey Jeffrey Schnapp
social sci div.
VPAL	Bharat Anand
HUIT	Klara Jelinkova

Links

https://bokcenter.harvard.edu/artificial-intelligence Bok center AI page

https://science.fas.harvard.edu/chatgpt divisional resource page

Meetings:
June 16 kickoff with HUIT and Bok:

David J. Malan to Everyone (Jun 16, 2023, 9:01 AM)
Here's a recording of GitHub Copilot "solving" via tab-completion, essentially, one of CS50's first problem sets, https://youtu.be/W2BmQuBMOD4. And one of our last, https://www.youtube.com/watch?v=veON9rxhJl0.
Thanks so much, all. Happy to connect anytime, malan@harvard.edu.

Ventz Petkov to Everyone (Jun 16, 2023, 9:52 AM)
Sharing a few things that others may find potentially interesting:

1.) Video of a voice interaction with OpenAI over custom data: http://tmp.vpetkov.net/ask-hr-audio.mp4

2.) Video of what a more natural “neural net” voice sounds like — this is not a real person: http://tmp.vpetkov.net/voice-readout.mp4

3.) An OpenAI presentation that talks about the tokenization/other challenges with using OpenAI directly (compared to LangChains for example): https://files.vpetkov.net/talks/OpenAI.pptx
#2 is Eleven Labs that Dave mentioned - without training from a real person/not cloning someone’s voice (using a default model)

Logan McCarty to Everyone (Jun 16, 2023, 10:03 AM)
Would it be possible to ingest the entire contents of a Canvas course, i.e. all of the PDFs, every page, all the videos, discussion questions, announcements etc.

Adam Beaver (Bok Center) to Everyone (Jun 16, 2023, 10:06 AM)
LL is building a HackMD here: https://hackmd.io/nOTyDO12SpOZExTjm6rj8Q?view

June 23

High-level questions-

are we going to focus on GAi-assisted learning or GAI-facilitated assessments?
Expectations of privacy and information flow.
What are the ways the GAI tools can increase (rather than decrease) efficiency of our teachers?
Need to clarify what assessments of student learning will incorporate GAI and which ones will not.
Assessment needs for this undertaking. How do we define success here?
structure and milestones - need to break into sub-teams?
How does aero-shot vs. one-shot vs. multi-shot learning impact learning effectiveness? Can we teach GAI how to grade student work?
What about a policy that at least 50% of student grades will be based on no-GAI student work, graded by humans? Hybrid homework with, say, half done using GAI and half submitted for assessment?
guidance and training for faculty- need another broadcast email to instructors.

Review categories of use cases and experiments above. Pick a few to focus on

GAI plus notebook for lab data exploration and numerical work. Consider use for both in-lab instruction and also offline learning. Shift from HW to virtual tutor. Do we even need/want to assess those interactions?
Virtual Tutor, with guided content. Khamingo at Khan Academy is doing exactly this. Need a way to have instructor be able to efficiently describe leaning goals and then implement that as a learning session. Should that be assessed?

Do we need a hierarchical access structure where instructors have access to selected student GAI sessions?
Do we need course-specific student accounts?
What are privacy expectations?
How would student GAI sessions be 'submitted' and 'graded'?

propose 3 subgroups:

1) virtual tutor, mainly conceptual content
2) GAI + notebook for data exploration and quantitative problems
3) Course support and faculty guidance.

Meeting discussion on above topics led to augmentation by 2: one student-focused and one looking for appropriate applications of restricted-domain tools.

Also decided we'd identify a handful of courses for pedagogy experimentation in the Fall term.

Decided to maintain a Table of Policy Questions GAI Policy questions that we can all edit as we go.

July 14 meeting

action items:

1. how do we provide licensing, and are there enterprise-level options (HUIT)

2. IF we decide to use a notebook interface, how does the use of API keys scale? One key per course? One key per person? (HUIT)

3. We should identify some undergraduates who can assist with developing course management tools (Brenner and Nesson)

4. Finalize FAQ document (Stubbs)

5. set up working session on notebook interface (Stubbs, in progress)

6. finalize date of STEM instruction session- proposed for 4-5 pm on Tuesday Aug 8. (Stubbs)

7. prep for STEM faculty instructional session (McCarty, Kestin, Neeson)

8. address issues that pertain to confidentiality and privacy of uploads. Enumerate the options (HUIT).