Versions Compared

Version Old Version 25 New Version 26
Changes made by

Christopher Stubbs

Christopher Stubbs

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Our 2 year goal, by the end of AY 24-25, is to produce GAI-enabled course structure across the introductory STEM curriculum.
The tools should be agnostic to the textbook(s) used in the course, and should be implementable with a minimal time investment in courses nationwide.
Initial backend is likely to be ChatGPT, currently at version 4, but we should  strive to build an architecture that can evolve and adapt to new and better and different tools. We need to have an overall architecture philosophy evolve with our development spirals.  

...

ItemexperimentscoursesleadGPT aspect neededValidation criteria

A: Incorporation of GAI into lecture-format STEM learning: 

  1. Develop experimental active learning lecture modules that incorporate GAI capabilities, and devise methods to measure their effectiveness in student comprehension and retention. 
  2. Develop first-generation tools 
  3. Use multi-shot learning to have GAI produce 'clicker' questions and active learning modules.
  4. Have GAI do assessments of online active learning modules. 
  5. Increase accessibility of course materials
  1. Assess utility of synthesis of student-provided questions, in real timequestion content, in-class just-in-time learning
  2. Assess utility of synthesize summaries of open-response quizzes clicker questions rather than multiple choice, for adaptive lectures
  3. Assess utility of peer-instruction including GAISequential interaction on course promptsthat includes GAI virtual partners
  4. Assess utility of GAI-produced active learning modules
  5. Compare restricted-content (textbook and lecture materials) vs. all-content GAI generated active learning modules. 




shared-edit document fed into GAI, results available to instructor. 

Multi-shot learning using restricted content and open 

PPT and math content upload as tokens


B: Develop and Exploit short-cycle adaptive problem sets with real-time feedback

  1. produce a 'virtual tutor' that uses a script of desired learning goals to promote student learning
  1. Assess learning effectiveness of rapid-feedback interactive learning, outside of lecture. 
  2. assess zero-shot vs multishot training for virtual tutor effectiveness. 
  3. Assess instructor burden compared to current methods. 


Ability to identify main topics, and embed in GAI interaction. 

Arithmetic needs to work. 


C: Assist with analysis and gain insights from lab data. 
  1. import data collected Assess effectiveness of using the combination of GAI and Jupiter notebooks to do guided data exploration. 


notebook and GAI integration

ability to access data files from instructional lab data collection. Does this mean a browser interface with access to local files? 








D: interactive student self-assessments. E: In-class group consultation (peer instruction) with ChatGPT participationF: capturing and submitting student work for GAI evaluation by course staffG: automated evaluation of understanding of material, by evaluating answers to questions we provide. 
  1. Assess utility of GAI in helping student prioritize where to focus to enhance their mastery of material. 




F: GAI-assisted student assessments and grading


ability to capture GAI sessions, with unique student identifiers






H: Ascertain subject-level mastery needed to exploit natural-langauge-driven code development.
         Try out a non-analytic problem and assess the results. 
  1. Use things like modulated-friction example. , Modulated Friction example , to help students learn how to assess and refine GAI produced material. 

15 a,b,c
Math 22


numerical solution code
I: incorporate into HW and assessments the ability to perform calculations, as pioneered by Khan Academysame as B

arithmetic capability
J: incorporate course-specific training inputs and give that high weighting


custom training inputs
K: Automation grading and assessments of student competence. 


sequential prompts run open loop, no adjustment
L: dynamic tutoringsame as D

sequential prompts with iterative adjustment 
M: GAI assisted generation and refinement of course instructional and assessment materials- HW, exams, quizzes, etc. 
  1. request a critique of exam questions
  2. request answer key to HW and exam questions




N: Course management- facilitating course selection
  1. upload Canvas course materials and any ppt from previous year, including student CUE evaluation narratives and restrict responses to that material. 


Need to restrict dissemination of uploaded CUE score materials. 
O: Course management- 
  1. restricted-scope utility using uploaded content
  2. integration with grading systems
  3. integration with Canvas
  4. automated sectioning tool
  5. integration with Slack? 
  6. multi-shot 


need to mesh student user account on GAI system with their HUID and course record. 

Instructor-level overview of 


P. Validation and verification tools and methods

  1. Develop methods for assessing and refining GAI-produced code 
  1. Assess level of expertise needed to get credible answers
  2. Conduct iterative-engagement tests 


  1. real-time access to URL's 
  2. provenance tracking
  3. code validation and verification 

respondentemail notes
Carlos Arguelescarguelles@fas.harvard.eduHola Chris:

Thanks for sending this email. I am very excited about this new technology. In fact, with Louis Deslauriers, we used some of it in my Physics 15B last semester.
I am currently writing a proposal for the Cottrell Scholars, where this will be featured, and I intend to use them in my incoming classes.
I will look at the links in your email. Have a wonderful summer, cheers,

Carlos




David Malin
using it now in CS50 with custom interface


















Course-based viewpoint

  1. Assess our assessments: run midterm and final exams of science courses through GPT-4 and grade the results. Compare to overall student performance. 
  2. Enhance our assessments. Solicit constructive feedback on the exam questions we submit. 
  3. Assess our homework: run homework assignments through GPT-4 and grade the results. Compare to overall student performance. 
  4. Enhance our assignments. Solicit constructive feedback on the homework we submit. 
  5. Ask (require?) students to use GPT-4 on selected assignments to get feedback and examples of how it can be used. 
  6. Course-specific chat-bots- what training data? 
  7. For large lecture classes- merge active learning with GPT
  8. For sections- aggregation of questions, 
  9. For labs- try out data analysis methods and inference
  10. Customized training assembly of material - what do we need to start to capture?
  11. Khan academy like adaptive tutorials
  12. How does this shift the workload in our non-ladder teaching capacity? Especially sections and TFs and grading? 

...

https://science.fas.harvard.edu/chatgpt divisional resource page


Meetings: 
June 16 kickoff with HUIT and Bok: 

...

3.) An OpenAI presentation that talks about the tokenization/other challenges with using OpenAI directly (compared to LangChains for example): https://files.vpetkov.net/talks/OpenAI.pptx
#2 is Eleven Labs that Dave mentioned - without training from a real person/not cloning someone’s voice (using a default model)
 
Logan McCarty to Everyone (Jun 16, 2023, 10:03 AM)
Would it be possible to ingest the entire contents of a Canvas course, i.e. all of the PDFs, every page, all the videos, discussion questions, announcements etc.
 
Adam Beaver (Bok Center) to Everyone (Jun 16, 2023, 10:06 AM)
LL is building a HackMD here: https://hackmd.io/nOTyDO12SpOZExTjm6rj8Q?view

June 23

High-level questions-

  1. are we going to focus on GAi-assisted learning or GAI-facilitated assessments? 
  2. Expectations of privacy and information flow.
  3. What are the ways the GAI tools can increase (rather than decrease) efficiency of our teachers? 
  4. Need to clarify what assessments of student learning will incorporate GAI and which ones will not. 
  5. Assessment needs for this undertaking. How do we define success here? 
  6. structure and milestones - need to break into sub-teams? 

Review categories of use cases and experiments above. Pick a few to focus on

  • GAI plus notebook for lab data exploration and numerical work. Consider use for both in-lab instruction and also offline learning. Shift from HW to virtual tutor. Do we even need/want to assess those interactions? 
  • Virtual Tutor, with guided content. Khamingo at Khan Academy is doing exactly this. Need a way to have instructor be able to efficiently describe leaning goals and then implement that as a learning session. Should that be assessed? 

Do we need a hierarchical access structure where instructors have access to selected student GAI sessions?
Do we need course-specific accounts?
What are privacy expectations?
How would student GAI sessions be 'submitted' and 'graded'?