/

2023-07-20 Meeting notes

2023-07-20 Meeting notes

Owned by Lawrence Eribarne

Last updated: Aug 11, 2023

Date

20 Jul 2023

Attendees

Lawrence Eribarne
Christopher Stubbs, Jefferson Burson, Logan McCarty, Gregory Kestin, Eske Pedersen, Rebecca Neeson, Ventz Petkov, David LaPorte, Colin Murtaugh

Goals

Focus today is figure out how to implement primary LLM interaction - web browser / api. Api provides more tokens and security

Discussion items

Time	Item	Who	Notes
	Requirements	Chris	Harvard ID / access when scales Statement on privacy implications of free versus paid access Need 4 examples with specific tools. 1. assignments. 2 assessments, 3 code generation, 4. active lectures. On top - re-evaluate the learning goals of your courses. The capability will advance quickly - may not want to drag students through that. What we need to do is take this different category of uses cases in courses and take advantage of cohort of students. Rebecca Nesson - narrowed it down to a group of 5 ready to start. Action: need paperwork through the office of academic programs. They can assess the current approach and generate ideas? Mostly Juniors - in computer science or applied math. Amanda may be able to find others from creativity class. Chris - immediate next step - restructure wiki page (action for Lawrence). Sketch out 3 or 4 things to try in each category (action for faculty). Pick something for life science, active learning session, mid term use case. By Monday 31st, put examples in front of students for feedback. Work on something jointly and have students reconvene to show what they did individually and setup sessions for collective efforts. Chris - Need principles - use, confidentiality, etc. 3 or 4 bedrock grounding principles. Lawrence Eribarne draft Greg, Logan McCarty and Rebecca Nesson - grab representative curriculum material, life science, physical science, math. Leverage with students in coming 2 weeks to see what works for assignments, asessments, active lectures. Doesnt' need to be large. Upload to wiki to store in one place. Turn around into gpt-ish, 2 to 3 different ways, what works, what doesnt, share w/ students for feedback. HUIT side - Jefferson Burson Aug. 8 driven, what can we provide for all term beyond browser base interface? Unsolved problems, authentication, scale, sys admin, using to one API keys, one to one, etc. HUIT figures out what works and what doesn’t and how do we test at scale. Could use 8th to find out partly. Action: What experiment to see what peole could do to see if it falls over and dies. David - we have tools to load test web service. We have tools inhouse to test methodically.
	Pedagogy	Chris, Greg, Logan, Rebecca	Chris - minority of courses will adopt something - how do we equip students and faculty? How to use and truth appraisal. Logan - a practical example of how you can use it would be good. Rebecca - ask gen ai answer and then critique it is a different activity than was students are doing with ai. Doesn’t' address learning objectives that everyone has. Need more. One additional category - feedback of two sorts - gai helping to receive and process feedback from students and producing useful feedback and the form of turoring and grading and how to address with faculty. Chris - office of undergrad education has FAQ questions hosted on their site. One quest on topic of grading -- current statement is discouraged use for that purpose. 1. untested and 2. confidentiality of student work produced for courses. Can students waive privacy? Don't know answer. (path to answer this question?Lawrence Eribarne ) Rebecca - with license agreement - should be in good shape, like gradescope Chris in principal - not in public domain and not in data set. Will need relevant for uses 8 august. For those doing summer session - capture feedback on what works and what doesn’t? (Lawrence Eribarne way to capture feedback? My question) Greg - in initial tests, got some answers wrong- frustrated students. Lesson learned capture area / testing feedback? Jefferson - in his testing, sometimes accuracy is not fully false or accurate. Depends on definitions used. ChatGPT not aware of nuances - true and false only not the only measure in experiment. Chris - for a lot of classes using chatgpt for language but pointed at their data. Need to make sure people are aware of this capability. Need to answer question for when will be ready for primetime or to scale. Chris - we need to help faculty understand - such as making a new course or privacy considerations (not uploading student CV). Chris - same as take home exam. Hesitates to implement polity that in unenforceable or unmonitorable. Rebeccas - already an issue with honor council. Logan - solution to students using more restrictive internal versus external for all answers. Sanctioned internal tools for violating academic integrity with outside tools. Rebecca - active learning is key. Chris - could create burden of special exception requests. Collisions of expectations. Rebecca - currently does take home and in class - it’s a technique , no rebellion. Maybe due to undergrad vs. graduate. Gregory - not a short-term solution, but paradign might shift to working conditions for assessments with a more withholding version of AI. Could be fine even with oral exam with ai that logs time, etc. to help teach critical thinking skills, memorization, etc. Chris - do exams as a learning experience. Do it, turn it in, provide it back to groups and group review is part of grade. Have faculty think broadly about how assessments will be done. ChatGPT crises could be a tool to elevate thought for what is put into courses. Logan question to Rebecca - any examples of providing guidance how to use the tool? Rebeccas no - but they are asked to submit a transcript to help assess how they used the tool. Rebecca - there are faculty completely unfamiliar and faculty ready to experiment. Should include the first group in experiments. \ Chris. How do I get an account, hold my hand, etc. - direct to Boks center function. Bigger issue in writing course than STEM courses. What do we do with faculty uncomfortable with this, what is our approach (action, Lawrence, change management awareness / adoption plan WIIFM).
	Tech	All	Gregory created a middle layer integration to leverage a single API. Does not maintain history. David - History can be achieved with IP address, user sessions, etc. to pass with requests via the API Ventz - Demoed chat bot interface in azure. Can interact with multipole people in same thread. Azure limits more higher. Difficult doing similar load in chatgpt directly. (@lawrence question for later - Microsoft demo on capabiliies for scaled chat bot support, prompt quality, output quality, etc.?) Chris -- Khan academy seems to be most significant with math and dynamic tutoring. Jefferson - what we are doing with containers, jupyter, etc. in line with other scalable development efforts. Need to discuss long-term support for custom chatbot management. Ventz / Jefferson - can scale backend with a common console that allows people to add data and controls. Canvas could select token to pick database. Chris - we need to confront licensing / financial planning requirements Need criteria for evaluating tools Lawrence Eribarne David - we have tools to load test web service. We have tools inhouse to test methodically. ACTION: put plan together for this. Chris - load test notebook based ools? (Action Jefferson)
	Collaboration	All	FAS has built a framework of committees that intersect with this effort. Lawrence question - any prep required to present this effort / progress to committees? Rebecca - do we want to include third-party tools or student projects? Chris - Engage others for feedback. Tools need to consider support, scale, integration, etc. Consider student pitch presentations. Chris - Need to connect with MIT efforts. Lawrence Eribarne o help with outreach / scheduling. David - There is also overlap from other requests for AI support coming into HUIT - no plans on partnering for larger use.
		in session chage	In session Chat: Hugging Face has some great alternatives to ChatGPT. But they are slower and not as good. However, the offline aspect is also appealing. There are things like GPT4ALL, Vicuna, etc. Azure’s implementation of OpenAI looks like it could be very promising down the line. It’s not quite as good currently, but very close and getting better. Llama2 (FB/Meta) is also interesting. Christopher William Stubbs to Everyone 8:20 AM Also Claude II Logan McCarty to Everyone 8:21 AM can anyone in the world use that url to send queries using your api key? Gregory Michael Kestin 8:22 AM Yes, indeed You to Everyone 8:22 AM If anyone is curious Azure pricing and details: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/ Logan McCarty to Everyone 8:28 AM In the API you essentially send back and forth a cumulative dialogue (with previous queries and responses) Colin Murtaugh to Everyone 8:29 AM I’ve got to hop off for another meeting for a bit Logan McCarty 8:25 AM Does Harvard have an Azure OpenAI enterprise contract (or presumably one is in the works)? David LaPorte 8:36 AM OpenAI is still early access, but it is easy to enable on an Azure subscription, falls under our existing MSA, and ties into existing billing and processing. Ventz Petkov 8:36 AM It’s slightly “complicated”. It’s currently in Beta, but you can apply and link to the current enterprise environments via Microsoft (same for GPT4). MS is limited it to only 5 tenants currently (I believe 3-4 are already activated). Supposedly the limit is going away soon. David LaPorte 8:37 AM @Ventz Petkov our account team informed me that the subscription/tenant limit has been removed In the API you essentially send back and forth a cumulative dialogue (with previous queries and responses) Ventz Petkov 8:39 AM I am using the LangChain’s ConversationBufferWindowMemory module, which allows you not to pass anything back and forth. The engine simply uses the previous user send+its reply to build its own history and then execute a summary/link for the next user send. It’s very elegant and minimal. It effectively does this in memory: user: Question AI: reply <implicit -> compile user’s Question and reply into a history chain> user: Follow up Question <implicit> Feed history chain AI: reply to Follow up given History chain Eske Pedersen (he/him) 9:00 AM Maybe this is a non-issue, but how do actually get students to use the HUIT generated AI tools rather than just go to external AI tool? to Everyone Maybe this is a non-issue, but how do actually get students to use the HUIT generated AI tools rather than just go to external AI tool? jburson to Everyone 9:03 AM At the community level, the external stuff is inevitable. Inside of a specific course, you’d like likely be able to follow David M’s example and focus on a specific tool to be employed Rebecca Nancy Nesson to Everyone 9:12 AM Devil’s advocate to Eske: A closed-book exam isn’t as problematic when in combination with a broad range of activities and assessments. In addition, if the type of problems on the exams aren’t focused on rote recall, the studying is a pretty impactful part of student learning. In response to Chris, it would be very helpful to have our registrar’s schedule for exams support that pedagogy. Currently, the exam has to be before the end of term to do that. Logan McCarty to Everyone 9:18 AM Rebecca when you say “exams that support that pedagogy” what type of exams do you have in mind? Rebecca Nancy Nesson 9:19 AM Where the individual proctored exam is then followed by a class session where group work occurs. It would be hard for me to do it in a single 3-hour exam session — but maybe that’s how you do it?

Action items

- Rebecca Nesson Students need paperwork through office of academic programs.
- Lawrence Eribarne Christopher Stubbs Restructure wiki page
- Christopher Stubbs Lawrence Eribarne Draft of grounding principles (Lawrence to find examples for consideration)
- Greg, Logan McCarty and Rebecca Nesson - grab representative curriculum material, life science, physical science, math. Leverage with students in coming 2 weeks to see what works for assignments, asessments, active lectures. Doesnt' need to be large. Upload to wiki to store in one place. Turn around into gpt-ish, 2 to 3 different ways, what works, what doesnt, share w/ students for feedback.
- HUIT team - tool comparison framework, options for Fall, ability to use internal data
- HUIT team - fall term options for operational / scalable use cases
- HUIT team - Awareness for capability of chatbot to leverage internal data.
- Christopher Stubbs Lawrence Eribarne - connect with MIT on common pursuits in this space.
- Lawrence Eribarne draft high level approach on communications, awareness, longer term adoption / change support, internal options vs. external, etc.

Copyright © 2024 The President and Fellows of Harvard College * Accessibility * Support * Request Access * Terms of Use