Our 2 year goal, by the end of AY 24-25, is to produce GAI-enabled course structure across the introductory STEM curriculum.
The tools should be agnostic to the textbook(s) used in the course, and should be implementable with a minimal time investment in courses nationwide.
Initial backend is likely to be ChatGPT, currently at version 4, but we should strive to build an architecture that can evolve and adapt to new and better and different tools. We need to have an overall architecture philosophy evolve with our development spirals.
...
| Item | experiments | courses | lead | GPT aspect needed | Validation criteria |
|---|---|---|---|---|---|
A: Incorporation of GAI into lecture-format STEM learning:
|
| shared-edit document fed into GAI, results available to instructor. Multi-shot learning using restricted content and open PPT and math content upload as tokens | |||
B: Develop and Exploit short-cycle adaptive problem sets with real-time feedback
|
| Ability to identify main topics, and embed in GAI interaction. Arithmetic needs to work. | |||
| C: Assist with analysis and gain insights from lab data. |
| notebook and GAI integration ability to access data files from instructional lab data collection. Does this mean a browser interface with access to local files? | |||
| D: interactive student self-assessments. |
| ||||
| F: GAI-assisted student assessments and grading | ability to capture GAI sessions, with unique student identifiers | ||||
| G: Temperature-dependence of learning effectiveness |
| ||||
| H: Ascertain subject-level mastery needed to exploit natural-langauge-driven code development. Try out a non-analytic problem and assess the results. |
| 15 a,b,c | numerical solution code | ||
| I: incorporate into HW and assessments the ability to perform calculations, as pioneered by Khan Academy | same as B | arithmetic capability | |||
| J: incorporate course-specific training inputs and give that high weighting | custom training inputs | ||||
| K: Automation grading and assessments of student competence. | sequential prompts run open loop, no adjustment | ||||
| L: dynamic tutoring | same as D | sequential prompts with iterative adjustment | |||
| M: GAI assisted generation and refinement of course instructional and assessment materials- HW, exams, quizzes, etc. |
| ||||
| N: Course management- facilitating course selection |
| Need to restrict dissemination of uploaded CUE score materials. | |||
| O: Course management- |
| need to mesh student user account on GAI system with their HUID and course record. Instructor-level overview of | |||
P. Validation and verification tools and methods
|
|
| respondent
| Deliverable |
|---|
| impact (1-3 with 5 being high) 1: single course 2: 3-5 courses 3: >5 courses | effort scope (1-3 with 3 being low) 1: > 1 calendar month 2: ~ 1 calendar month 3: < 1 month | merit | |
|---|---|---|---|
| classroom synthesis: Google form gathers student input, sent to canned prompts and displays on screen, 1-button workflow for instructor | 3 | 1 | 3 |
| implement nano-GPT in jupyter notebook | 2 | 3 | 6 |
| implement ability to execute queries on course-specific uploaded materials | 3 | 1 | 3 |
| facilitate upload of student sessions as parsable text into grading system | 3 | 2 | 6 |
| implement course management tools- grade sheet analysis and interrogation | 3 | 2 | 6 |
| incorporate chat functionality into course Slack channel | 3 | 3 | 9 |
| new-course development toolkit | 3 | 2 | 6 |
| assessment of student preparation, based on instructor expectations | 3 | 2 | 6 |
| assessment of student preparation, based only on uploaded course materials | 3 | 1 | 3 |
| instructional lab data browsing, tailored to individual courses | 1 | 1 | 1 |
| generation of lecture notes from recorded transcripts and video capture of blackbaord? | 3 | 1 | 3 |
| Dynamic HW system, multishot training to guide student interaction | 3 | 1 | 3 |
| Assess optimal temperature setting to maximize learning effectiveness (this is an experiment) | 3 | 1 | 3 |
| Assess performance of image capture HW assessment. "work this problem then upload image of your work" | 3 | 2 | 6 |
| respondent | notes | |
|---|---|---|
| Carlos Argueles | carguelles@fas.harvard.edu | Hola Chris: Thanks for sending this email. I am very excited about this new technology. In fact, with Louis Deslauriers, we used some of it in my Physics 15B last semester. Carlos |
| David Malin | using it now in CS50 with custom interface | |
...
| Who? | From? | |
|---|---|---|
| Christopher Stubbs | FAS Sci Div | stubbs@g.harvard.edu |
| Logan McCarty | FAS Sci Div | mccarty@fas.harvard.edu |
| Greg Kestin | FAS Sci Div | kestin@fas.harvard.edu |
| Erin Collins | FAS Sci Div | erin_collins@fas.harvard.edu |
| Jefferson Burson | HUIT | jefferson_burson@harvard.edu |
| Ventz Petkov | HUIT | ventz_petkov@harvard.edu |
| David LaPorte | HUIT | david_laporte@harvard.edu |
| Colin Murtaugh | HUIT | colin_murtaugh@harvard.edu. |
| Rebecca Nesson | SEAS | nesson@g.harvard.edu |
| Eske Pedersen | FAS Science | eskepedersen@fas.harvard.edu |
| Lawrence Eribarne | HUIT | lawrence_eribarne@harvard.edu |
Departmental
GAISTEM stakeholder group, quarterly meetings
| stakeholder | representative |
|---|---|
| College OUE | Amanda Claybaugh Anne Harrington |
| Harvard College | Rakesh Khurana |
| Bok Center | Tamara Brenner, Adam Beaver |
| HGSE | |
| humanities div. | Robin Kelsey Jeffrey Schnapp |
| social sci div. | |
| VPAL | Bharat Anand |
| HUIT | Klara Jelinkova |
Links
| Anand | |
| HUIT | Klara Jelinkova |
Links
https://bokcenter.harvard.edu/artificial-intelligence Bok center AI page
https://science.fas.harvard.edu/chatgpt divisional resource page
Meetings:
June 16 kickoff with HUIT and Bok:
David J. Malan to Everyone (Jun 16, 2023, 9:01 AM)
Here's a recording of GitHub Copilot "solving" via tab-completion, essentially, one of CS50's first problem sets, https://bokcenteryoutu.harvard.edu/artificial-intelligence Bok center AI pagebe/W2BmQuBMOD4. And one of our last, https://science.fas.harvard.edu/chatgpt divisional resource page
Meetings:
June 16 kickoff with HUIT and Bok:
David J. Malan www.youtube.com/watch?v=veON9rxhJl0.
Thanks so much, all. Happy to connect anytime, malan@harvard.edu.
Ventz Petkov to Everyone (Jun 16, 2023, 9:01 52 AM)
Here's a recording of GitHub Copilot "solving" via tab-completion, essentially, one of CS50's first problem sets, https://youtu.be/W2BmQuBMOD4. And one of our last, https://www.youtube.com/watch?v=veON9rxhJl0.
Thanks so much, all. Happy to connect anytime, malan@harvard.edu.
Ventz Petkov to Everyone (Jun 16, 2023, 9:52 AM)
Sharing a few things that others may find potentially interesting:
1.) Video of a voice interaction with OpenAI over custom data: http://tmp.vpetkov.net/ask-hr-audio.mp4
2.) Video of what a more natural “neural net” voice sounds like — this is not a real person: http://tmp.vpetkov.net/voice-readout.mp4
3.) An OpenAI presentation that talks about the tokenization/other challenges with using OpenAI directly (compared to LangChains for example): https://files.vpetkov.net/talks/OpenAI.pptx
#2 is Eleven Labs that Dave mentioned - without training from a real person/not cloning someone’s voice (using a default model)
Logan McCarty to Everyone (Jun 16, 2023, 10:03 AM)
Would it be possible to ingest the entire contents of a Canvas course, i.e. all of the PDFs, every page, all the videos, discussion questions, announcements etc.
Adam Beaver (Bok Center) to Everyone (Jun 16, 2023, 10:06 AM)
LL is building a HackMD here: https://hackmd.io/nOTyDO12SpOZExTjm6rj8Q?view
June 23
High-level questions-
- are we going to focus on GAi-assisted learning or GAI-facilitated assessments?
- Expectations of privacy and information flow.
- What are the ways the GAI tools can increase (rather than decrease) efficiency of our teachers?
- Need to clarify what assessments of student learning will incorporate GAI and which ones will not.
- Assessment needs for this undertaking. How do we define success here?
- structure and milestones - need to break into sub-teams?
- How does aero-shot vs. one-shot vs. multi-shot learning impact learning effectiveness? Can we teach GAI how to grade student work?
- What about a policy that at least 50% of student grades will be based on no-GAI student work, graded by humans? Hybrid homework with, say, half done using GAI and half submitted for assessment?
- guidance and training for faculty- need another broadcast email to instructors.
Review categories of use cases and experiments above. Pick a few to focus on
- GAI plus notebook for lab data exploration and numerical work. Consider use for both in-lab instruction and also offline learning. Shift from HW to virtual tutor. Do we even need/want to assess those interactions?
- Virtual Tutor, with guided content. Khamingo at Khan Academy is doing exactly this. Need a way to have instructor be able to efficiently describe leaning goals and then implement that as a learning session. Should that be assessed?
Do we need a hierarchical access structure where instructors have access to selected student GAI sessions?
Do we need course-specific student accounts?
What are privacy expectations?
How would student GAI sessions be 'submitted' and 'graded'?
propose 3 subgroups:
1) virtual tutor, mainly conceptual content
2) GAI + notebook for data exploration and quantitative problems
3) Course support and faculty guidanceSharing a few things that others may find potentially interesting:
1.) Video of a voice interaction with OpenAI over custom data: http://tmp.vpetkov.net/ask-hr-audio.mp4
2.) Video of what a more natural “neural net” voice sounds like — this is not a real person: http://tmp.vpetkov.net/voice-readout.mp4
3.) An OpenAI presentation that talks about the tokenization/other challenges with using OpenAI directly (compared to LangChains for example): https://files.vpetkov.net/talks/OpenAI.pptx
#2 is Eleven Labs that Dave mentioned - without training from a real person/not cloning someone’s voice (using a default model)
Logan McCarty to Everyone (Jun 16, 2023, 10:03 AM)
Would it be possible to ingest the entire contents of a Canvas course, i.e. all of the PDFs, every page, all the videos, discussion questions, announcements etc.
Adam Beaver (Bok Center) to Everyone (Jun 16, 2023, 10:06 AM)
LL is building a HackMD here: https://hackmd.io/nOTyDO12SpOZExTjm6rj8Q?view
June 23
High-level questions-
- are we going to focus on GAi-assisted learning or GAI-facilitated assessments?
- Expectations of privacy and information flow.
- What are the ways the GAI tools can increase (rather than decrease) efficiency of our teachers?
- Need to clarify what assessments of student learning will incorporate GAI and which ones will not.
- Assessment needs for this undertaking. How do we define success here?
- structure and milestones - need to break into sub-teams?
- How does aero-shot vs. one-shot vs. multi-shot learning impact learning effectiveness? Can we teach GAI how to grade student work?
- What about a policy that at least 50% of student grades will be based on no-GAI student work, graded by humans? Hybrid homework with, say, half done using GAI and half submitted for assessment?
- guidance and training for faculty- need another broadcast email to instructors.
Review categories of use cases and experiments above. Pick a few to focus on
- GAI plus notebook for lab data exploration and numerical work. Consider use for both in-lab instruction and also offline learning. Shift from HW to virtual tutor. Do we even need/want to assess those interactions?
- Virtual Tutor, with guided content. Khamingo at Khan Academy is doing exactly this. Need a way to have instructor be able to efficiently describe leaning goals and then implement that as a learning session. Should that be assessed?
Do we need a hierarchical access structure where instructors have access to selected student GAI sessions?
Do we need course-specific student accounts?
What are privacy expectations?
How would student GAI sessions be 'submitted' and 'graded'?
propose 3 subgroups:
1) virtual tutor, mainly conceptual content
2) GAI + notebook for data exploration and quantitative problems
3) Course support and faculty guidance.
Meeting discussion on above topics led to augmentation by 2: one student-focused and one looking for appropriate applications of restricted-domain tools.
Also decided we'd identify a handful of courses for pedagogy experimentation in the Fall term.
Decided to maintain a Table of Policy Questions GAI Policy questions that we can all edit as we go.
July 14 meeting
action items:
1. how do we provide licensing, and are there enterprise-level options (HUIT)
2. IF we decide to use a notebook interface, how does the use of API keys scale? One key per course? One key per person? (HUIT)
3. We should identify some undergraduates who can assist with developing course management tools (Brenner and Nesson)
4. Finalize FAQ document (Stubbs)
5. set up working session on notebook interface (Stubbs, in progress)
6. finalize date of STEM instruction session- proposed for 4-5 pm on Tuesday Aug 8. (Stubbs)
7. prep for STEM faculty instructional session (McCarty, Kestin, Neeson)
8. address issues that pertain to confidentiality and privacy of uploads. Enumerate the options (HUIT).