ashandoak.github.io

__CSC595 Term Portfolio Project__ Ashley Blacquiere V00134139

Overview

This document is a record of my learning journey in CSC 595 Research Methods with Dr. Hausi Muller.

If you’d like to view this document live, check out my GitHub Page.

Table of contents

  1. Lecture notes, activities & post-class reflections
  2. Research elevator pitch & discussion
  3. Notes & reflections on assigned homework
  4. Research talk best practices & discussion
  5. Proposed talk: How to ask good questions
  6. Paper review: Constructive Mathematics and Computer Programming
  7. Thesis introduction
  8. Literature review & Bruce’s suggestions
  9. A consolidated discussion of CSC595 guest lectures
  10. Learning outcomes
  11. References
  12. Useful resources

Lecture notes, activities & post-class reflections

Lecture 1 (Jan. 10, 2022)

Class notes: I liked that Dr. Muller has asked everyone to speak in the class - I think this is an important hurdle to get over and helps to encourage more consistent contributions in future classes. I elected not to speak in the first class, primarily because I’m not really sure what my story is at this point. The parameters around my research project has only really started to become well-formulated in the last month or so, but the specifics of what my role is lack a clear motivation - other that that I need to complete some particular tasks in order to support those over-arching parameters.

There were plenty of people who stated that their projects were not fully identified yet, and I don’t really mind saying this, but I would rather give it a bit more serious thought - knowing that I can make an attempt in lecture 2. I did notice in listening to the other pitches that the most interesting were those that had some clear connection to broader issues or research disciplines, or bigger humanist concepts. This is something that I think I lack.

I was also struck (my ‘aha!’ moment for the day) by the advice to offer a call to action at the end of an elevator pitch. This is a key idea for sure - not so much perhaps for a research talk, but definitely for an elevator pitch. I think this is something I’d like to try to incorporate into my pitch on Thursday.

General reflection: I can’t say that I was entirely sure what to expect from CSC 595: I have heard varying reports about the course in previous years, but I was also aware that this was Dr. Muller’s first time [recently?] teaching the course. I was, however, pleasantly surprised by the first lecture, particularly on the use of Jupyter Notebooks as a tool for literate programming and the focus on each student’s personal learning journey.

My own interest in literate programming began about four years ago as I was revamping some dated curriculum that was part of the program I teach at North Island College. I had come across the term while studying scholarship of teaching and learning papers from ACM SIGCSE and it struck me as an excellent pedagogical tool that could be brought to bear in my classrooms: my students typically have broad ranging backgrounds and levels of proficiency and many are coming to programming for the first time, or after a lengthy career in another field and so literate programming seemed to me to be an excellent tool for the communication of programming concepts embedded in the very object they’re using in practice.

Unfortunately, our programs at that time focused primarily on web technologies in the first year of instruction, and I struggled to find an appropriate fit with JavaScript that didn’t require that the students be forced to adopt a new interface. I was able to implement something in this vein using Swift Playgrounds for my second year students studying Swift, but the first year students were left without. I haven’t lost interest, however, and would like to incorporate something like this into my newer Kotlin and Android Studio based courses.

Fast forward to fall 2020 as I was beginning my MSc studies at UVic: The funding I have received for tuition costs has been provided as part of my professional development fund at North Island College, and part of my commitment to justifying this funding has been to record my learning journey in the form of a Jupyter Notebook. So I’m happy to be continuing this activity in a more official capacity in CSC595, where I’m finding myself at the intersection between literate programming and my own learning journey.

Lecture 2 (Jan. 13, 2022)

Class notes: I offered my pitch today. I had given it some thought over the intervening days and had arrived at a fairly short pitch that sounded something like the following:

My name is Ashley Blacquiere, and I am an MSc student in the computational theory group under the supervision of Dr. Bruce Kapron. My research interests are in the are of mathematical proof assistants and particularly the proof assistant Lean. For my research project I’ll be working on the development of new proof tactics in Lean for the purpose of demonstrating the correctness of certain cryptographic primitives.

Then, because my ‘aha!’ moment after lecture 1 was the call to action idea, I dropped a link to the LeanProver site in the chat.

Alas, maybe someone clicked on the link, but no one mentioned if they did. I felt like my pitch fell somewhat flat and it occurred to me afterwards that I didn’t do those things that I realized after lecture 1 were fairly critical: including having a clear connection to a meaningful research domain or humanist concern. I threw out a mention of cryptographic, but I was intentionally cryptic (haha, pun intended) because I don’t really know a lot about cryptography and I didn’t want to imply that my work would be cryptographic in nature (Bruce and our research partners are handling the crypto stuff - I’m the Lean expert). Now, however, I feel I’m at a loss as to how to communicate that critical touchpoint to others. This requires more thought…

My ‘aha!’ moment in class today was the idea of pointing to two seminal papers in my field as part of my elevator pitch. Clearly this applies only to elevator pitches given to researchers, and not to laypeople.

General reflection: There is clearly a balance to be struck in terms of the length of an elevator pitch and its content. There were a couple pitches in class today that were really quite good, but that were surprisingly long. That said, one can fit quite a lot of content into a minute if determined to do so (I think my pitch was probably 20 seconds or less). I wonder if having a number of exit points in a longer pitch could be a good strategy for developing multiple pitches that could be deployed at any one time?

Lecture 3 (Jan. 17, 2022)

Class notes: Dr. Muller’s point regarding the importance of slides when there is no textbook is well taken. In many of my classes I don’t rely on textbooks, but I do collect large amounts of material that students can engage with. These days I usually even make clear that all that extra material is optional, since not all students engage with it anyway. That said, I’ve always had a dislike for slides, and I often find myself trying to identify alternative ways of communication the same information but in a more interactive means. Boardwork is one method, but I’m rarely disciplined enough to make good notes, and so I usually end up going a little tangential. Most times, I rely on live coding and code walkthroughs, since that is primarily what my students want to see anyway.

I’ve always been enamoured with the idea of a good example providing critical motivation to understanding a concept. And I very much envious of those instructors who have some combination of the aptitude, the time, and / or the right class and content to make meaningful pithy examples for many lectures. It puts me in mind of that quote of Mark Twain’s (which I’m paraphrasing): “I didn’t have time to write you a short letter, so I’ve written you a long one…”, which serves to point out the amount of work it can be to produce something meaningful and concise.

Winston’s big four are:

I think I do the first, second and third pretty well already (though I’m sure I could use some work on any one of them). The ‘near miss’ is kind of brilliant (this is my ‘aha!’ moment for the day). I think it could be the right kind of way of thinking to address the idea of the ‘motivating example’ (discussed below).

General reflection: Winston’s idea of being cyclic in a presentation reminds me of the advice I once had from a theatre instructor some years ago: He spoke of the concepts of reuse and reincorporation as they relate to performance and presentation. Perhaps it was more the former though, because as I think of it I believe this idea came initially from improv and theatre games, where a performer is expected to accept what is offered and attempt to reuse and reincorporate it and other previously used material into their improvisation. The logic being that something familiar to the audience is funnier because they can predict the outcome when it’s well-telegraphed and so the audience feels like they’re part of the joke - an in-joke between the performers and the audience. Although Winston made a good point regarding jokes and one’s ability to pull off a joke naturally, I think the idea itself has merit; I’ve often found myself calling back on examples from early in a class to help elucidate a point - cyclic, like Winston, but sometimes improvisational, like reuse and reincorporate.

Lecture 4 (Jan. 20, 2022)

Class notes: We continued with Patrick Winston’s ‘How to Speak’ videos today. There are certainly lots of great ideas present in these videos and I think a good strategy for anyone new to classroom instruction or speaking would be best served by trying to pick one or two ideas to try out, rather than attempt to adopt the entire suite of practices at once. For me, I think that idea of the near miss is one that I’d like to concentrate on in the short term.

I really liked the formula: \(\phi = f (K, p,\hspace{0.1cm} _t)\)

General reflections: I think that one of the most important of Winston’s techniques discussed today is story. Winston’s argument for the importance of the story, in terms of memory and retention are certainly in line with my own experience. That said, I rarely plan for the use of story in my lectures - though I do use anecdotes quite often, they’re more often emergent events than planned ones. Hmm… Since most of my classes in the last two years have been asynchronous and recorded, I wonder if I am telling fewer stories… I suspect I probably am, but when I think of my teaching practice I still picture myself in a classroom, despite it having been nearly two years since that was the norm.

Perhaps there is something to be said for having a stock set of stories that could be deployed. Of course, it’s not good to repeat stories and for them to get stale… But perhaps I should be thinking about lecture in the same way that I might think of a pitch or a research talk, both of which should be stories in their own right. I guess because lecture is episodic one can get away with breaking at inopportune points, knowing that you can pick it up later. But of course, the best lecturers are those who tell a story with every lecture… Hmm. This is my ‘aha!’ moment of the day, but I’m not sure what to do with it…

(As an aside, a few years ago I actually tried to write a class as an ongoing story, where each lecture / week was a chapter. The idea was that I would write a cohesive and compelling narrative to present throughout the semester. It feel flat though, since what seemed like a good idea in the spring when I was writing, sounded forced, or perhaps disingenuous, when prepped in the fall.)

I think the question of nervousness is an interesting one. I’ve been teaching full-time since 2014, on average probably eight courses per year, so somewhere around 60 or 70 courses over the last eight or so years. I’d say that on average the level of nervousness that I feel prior to a class has lessened significantly over the years, but I think to some degree it is still sometimes there - especially prior to a first class in the semester when I’m meeting new students for the first time. It’s not the type of thing that typically lasts, though, and I would say that I have an overall different outlook on the subject of nervousness, compared to Winston’s position on it.

In reference to the formula above: I think the times that I have been most nervous have been those times when my K value has been too small. Especially in my early days of teaching, when I was working primarily as a sessional, I was willing to take any courses that were offered to me, even if they were pretty far outside my area of expertise. But I think my p value, in those cases, often made up for it. Of course, one might also argue that now that I am more established and teach fewer courses outside of my wheelhouse that a reduced nervousness is to be expected. Correlation, or causation?

That said, there must be an ‘upper bound’ on K, at least in the context of p and t. A large K cannot completely compensate for a completely negligible t. And I think there is a lot to be said about types of knowledge that might overlap with elements of t. In particular, I think that while Winston is absolutely right in what he presents in terms of how to speak in front of a class, and these types of lectures certainly have their place in the education system, there is increasing interest today in alternative pedagogical strategies that compliment lecture. Speaking certainly has it’s place there, but do all of Winston’s tools?

Lecture 5 (Jan. 24, 2022)

Class notes:

Lecture 6 (Jan. 27, 2022)

Class notes: The idea not to give it all away in the talk is a fantastic one - definitely the ‘aha!’ moment of the day. I mean, it feels a little hacky, or underhanded, but it makes an incredible amount of sense. From a lecture perspective it would be really useful as a tool to encourage discussion: leaving some low-hanging fruit out there means that students will be more willing to ask, and thus might be more engaged.

We were asked in class to describe the advice we would give the pioneers of our respective fields if we were able to travel back in time with all the knowledge that we have today. Difficult, since I’m not sure my immediate field is mature enough yet to warrant such consideration - I suppose there is always the “keep at it!” advice, since there are lots of naysayers when it comes to proof assistants. Anyway, I came up with the following:

Domain: Computational theory and proof assistants.
There is somewhat of a division today between developers and users of specific proof assistants (e.g. Lean, Coq, Isabelle/HOL, Agda). For the most part, this division isn’t antagonistic, but it is an undercurrent that runs throughout the discourse relating to which proof assistant is the ‘best’ tool for a particular application. This has the tendency to influence especially newer community members towards particular biases either for or against the use of specific proof assistants and can create some toxic interactions between different communities. I don’t think this is a problem specific to my domain, but perhaps it is all the more noticeable since in proof assistant development there is an incredible diversity of users with a broad spectrum of motivations and goals (i.e. from CS researchers interested in developing the language, to mathematician from many domains interested in using proof assistants for research purposes, to hobby users just interested in contributing in some way). My advice then is to consider how the various communities that form can be more inclusive and supportive of one another and their individual goals.

General reflections: I guess the idea of not giving it all away has some relationship to Winston’s rhetorical questions, so we’ve sort of talked about it before. Honestly, though, it seems so obvious to do this I don’t know why I haven’t thought of it before. Maybe I have used it in the past in class, but it seemed so unremarkable to do so that I’ve just forgotten… Maybe this is again a symptom of all of my classes in the last two years being asynchronous: I don’t really have to deal with student questions these days - expect by email or Slack.

Lecture 7 (Jan. 31, 2022)

Class notes:

General reflection: Although I’ve worked in industry previously, and have made efforts to follow industry leaders when it comes to social media and blogs, for some reason I haven’t really thought to do so when it comes to research. I’ve been paying attention to community forums in my field, but not to specific individuals. This is something I should change - definitely my ‘aha!’ moment for the day.

It’s funny that I haven’t thought to do this, since I do follow some publications that in some ways serve the same purpose, though that are typically more general audience focused. Quanta Magazine is by far one of my most favourite things, and I think everyone in CS and Math (and other sciences too, of course) should pay attention to it; it’s mandate of producing general audience content about recent and important scientific results is really quite admirable - especially since they do it so well! There should be more of this type of content available broadly.

(Or maybe I haven’t thought of following researchers directly since any time I’ve tried before I’ve had a hard time getting over the initial difficulty of reading content within a field that I don’t have a lot of experience in. Now that I’ve been working on this MSc (part-time) for more than a year, I feel like I have a better basis from which to attack this idea.)

Conferences are another good thing to think about: Bruce and I have discussed off and on, but haven’t identified quite the right one at this point. I’d like to go to a formal verification, or a Lean-specific conference / workshop. I expect there might be one at some point this year… Funny story: The last face-to-face conference that I was at was the SIGCSE conference in Portland in March 2019. Can you imagine how that ended? :) I arrived just as things were getting really bad in Oregon, attended one workshop the night of my arrival, and by the next morning the organizers had completely canceled the conference. I scrambled back to the airport and managed to catch and early flight home that day (though it had some long lay-overs). I was gone less than 36 hours, but because I crossed the boarder, for my trouble I got to stay home in self-isolation for two weeks…

Lecture 8 (Feb. 3, 2022)

Class notes: I wrote down the following points during class today. I think my ‘aha!’ moment was the idea that I should be able to identify the top three researchers (for me) in my field. Not necessarily the top three of all lists, but the top three that are most important to my particular project.

General reflection: In terms of those alleged negative reviews who review only to tear down others’ work: what I wonder is why that culture of “trashing papers” even exists. I guess there will always be people who want to take others down, but I wonder if there is something inherent to the culture that results in this type of behaviour? If editors and chairs are empowered to provide constructive criticism, then that should curtail some of this, but I guess there are instances where things slip under the radar for a variety of reasons.

This is difficult to articulate because I’m not inside at the moment - I’m looking from the outside based on what I’ve heard from others who have been part of peer review. Maybe the problem isn’t as bad as I assume it is, or that it appears to be from reports, but it certainly does seem prevalent in many disciplines. Taking a step back then, the question i: if checks and balances do exist (i.e. empowered editors), why is it that there always seems to be this undercurrent of the failures of peer review? Are we making a bigger deal of it than it is - because it’s not perfect? Are we expecting too much? Do we - as a culture - for some reason feel the need to apologize for the peer review process? Is this due to the well-publicized failures of the review process in recent years where bogus papers have been accepted, and bad science has had major societal influences (e.g. the anti-vaxx movement). Are the complaints we have about peer review actually sound, or do scientists complain because it seems like the right thing to do? are we a culture of apologists? O.o

If the complaints are valid (and I lean toward assuming they are), then I wonder - as stated above - if there is something about the process or about the academic culture that fosters this subversive behaviour (i.e. reviewers trashing papers) in terms of review. Or is it just the power of [supposed] anonymity that encourages this behaviour? If it is cultural, then should we be looking for an alternative to peer review? I don’t have any good suggestions, but I am certainly wary of historical institutions that resist change because of a priori logic.

Lecture 9 (Feb. 7, 2022)

Class notes:

Important review criteria:

Thoughts on my pitch:

General reflection: I’ve been thinking a bit about questions: There was lots of opportunity to ask questions during the pitch presentations, and they were all quite good, I think. At research talks, however, it’s not atypical to have those meandering type of questions that are often thought of as posturing. In, I think, Simon Peyton Jones’ talk he addressed the idea of the “aggressive” question and how to handle it during a talk, but I’m thinking about this from the other perspective: How to ask a good question.

The reason being that I think some of those rambling questions might less often be aggressive posturing, and perhaps more often a symptom of uncertainty. I think there is perhaps value in discussing the right way to ask a question.

Perhaps most important would be considering some basic skills:

But how to ask the question? Getting to the point is critical. Keeping the question short obviously keeps everyone happy, and respects the audience - taking longer to ask a question means less time for other questions. Name dropping (and related) should be avoided, unless actually relevant for some reason.

Lecture 10 (Feb. 10, 2022)

Video recording of lecture failed, and I was unfortunately able to attend class this day.

Lecture 11 (Feb. 14, 2022)

Class notes:

General reflections: I enjoyed this talk - lots of good advice and reminders on how best to approach and organize one’s work. I found the two slides linked to above to be the most interesting and valuable now, and potentially into the future (not that the ‘managing your time’ and ‘managing your supervisor’ bit wasn’t valuable - there was lots there to consider, but I’m less concerned with my personal time management at the moment). As mentioned above, I think that I’m most interested in better understanding my place within my discipline - and understanding what my discipline is.

For some context: I’ve been spending a lot of time learning about the Lean programming language, but most of that time has not been geared too directly toward the cryptographic applications that I’m moving toward now. Most of the Lean user base are mathematicians looking to improve the mathlib library, and most of them are uninterested in formal logic and the underlying type theory of Lean. I find this quite interesting, and would like to learn more, but I feel like I’ve only so far been exposed to just a small slice of that discipline.

Lecture 12 (Feb. 17, 2022)

Class notes: I took the following notes during research pitches. Notes were intended to consider some key points in an effort to generate useful questions.

Process for paper review:

  1. Score
  2. Summarize paper - especially point out contributions. This is valuable to the reviewers understanding, and demonstrates to the author that their work was understood (or not understood).
  3. Strengths and weaknesses - separate and itemize for clarity
  4. Comments to authors (many of the following criteria are typically defined in the CfP, so there is often some measure for reviewers. Not all of these need to be represented in terms of strengths and weaknesses):
    • Significance: It is the author’s responsibility to determine significance, but it should be clear to the reviewer
    • Soundness: Paper must answer the questions it poses with rigor. Probably the most difficult to asses.
    • Novelty: The most subjective?
    • Verifiability and transparency: The source of data and methodology of experiments should be clear to the reviewer. The data should also clearly apply to the research question. This is probably the second hardest criteria to asses.
    • Presentation: Adherence to ACM or IEEE formatting standards (as defined in CfP) is critical.
  5. Questions for authors to respond to - constructive and encouraging questions are best.

General notes: I felt pretty good about some of the questions I came up with, but I didn’t get to ask many of then - was a bit too difficult on Zoom. Nevertheless, I think the process I took above generate some good ideas. I didn’t note speaker names because that felt a bit too challenging with speed of speech and accents, etc.

The process listed for the paper review will, I think, be fairly challenging. The list of five criteria (significance, soundness, etc.) seem to overlap in some cases, but perhaps that’s just my perspective as a novice, or just my impression on first examination. Maybe a rubric would help? Perhaps too prescriptive?

Lecture 13 (Feb. 28, 2022)

Class notes: Slides for this class come from Writing a thesis or dissertation. The following notes are some best practices and key notes from the lecture:

General notes: In retrospect, in response to my last set of general notes, and after having completed most of the individual paper review, the five criteria of the review process are more distinct than I initially thought. This lecture was fairly useful in helping to provide some distinguishing features as well, especially in regards to contributions and how they relate to novelty, and in contrast to significance.

I’m still pretty far out from making writing my main activity, but I can definitely appreciate the advice provided today - write early! write often! Probably better advice has rarely been given; I’ve certainly handed it out to my own students on many occasions (and, on many occasions, have despaired to learn they didn’t take such excellent advice… haha). So in with that in mind, let’s give it a go - see Thesis introduction.

Lecture 14 (Mar. 3, 2022)

Class notes: Slides for this class come from Writing a thesis or dissertation. The following notes are some best practices and key notes from the lecture:

Lecture 15 (Mar. 7, 2022)

Class notes:

General notes: The last three classes have focused on writing a thesis and on literature review. Although there is extant documentation on the organization of the thesis, this has been quite valuable, as I had yet to read it fully. It’s also good to get some tips and ideas on what to avoid and what to capitalize on in terms of content and approach.

I think more than anything this topic has helped crystalize Dr. Muller’s advice in regards to knowing one’s field - top 3 researchers and top 3 papers, etc. Having a handle on this is clearly critical, and something that I’m missing at present - though I’ve started to make some improvements (i.e. have asked Bruce for some ideas, which he provided - see Literature review below).

This sentiment about having a clear grasp of field was echoed in the discussion on literature review as well - which makes sense, since the lit review is part of the dissertation. I think the lit review was a much more enlightening and instructive - or maybe it’s just a reflection of where I’m at in my own progress. I can definitely see where there is a considerable gap in my knowledge that needs to be addressed.

Actually, I’m surprised that the lit review wasn’t the first topic covered in the course. It would seem that having a lit review in place would be very helpful when developing an elevator pitch, but I can see how developing the pitch independent of the lit review is a good way to keep the focus on one’s own work.

Lecture 17 (Mar. 14, 2022)

Class notes: Today we are going through the conference papers from each group. Each group will cover the following:

Spoke briefly with Dr. Muller after class. He expressed interest in my ‘good questions’ presentation, so I’ll go through and make up some slides in prep for possibly as early as next Thursday. Dr. Muller also asked if I knew of Robert ‘Corky’ Cartwright, who is known for type theory and formal systems, I think. Will have to look him and his work up.

General notes: I think the presentation of our (my and Addie’s) review work went well. I admit to not feeling really well-prepared for it, but luckily I didn’t have to go first. :) I made a point of speaking slowly because I knew I would need to think through as I was presenting, but I think that worked well. About 2/3rds of the way through Derek Robinson (sp?) gave a huge sigh - I guess he had left his mic on - which sort of distracted me but kinda kicked me into high gear for the last little bit. I wasn’t offended by the sigh, though it was kind of a funny moment.

Lecture 18 (Mar. 17, 2022)

Class notes: Guest lecture by Dr. Alex Thomo - distrbuted computing, databases and knowledge-based systems. Large-scale data mining and formal languages and their applications to databases.

Finding a problem

Once you find a problem the challenge is in finding a good way to extend it - can it be reframed, made more accurate, make it faster, use different hardware, or different data types, can you explain why it works well or can you make it simpler?

One approach to finding a good problem is to reproduce a paper:

Solving a problem

Writing a paper

Additional questions

Lecture 19 (Mar. 21, 2022)

Didn’t take any notes, but listened to the remaining conference paper reviews.

Lecture 20 (Mar. 24, 2022)

Class notes: Guest lecture by Sowmya Somanath on HCI research methods.

Research methods

Additional questions

Lecture 21 (Mar. 28, 2022)

Class notes: Guest lecture by Dr. Kui Wu. Theoretical foundation of networks and other network related applications.

Reading papers

Writing papers

  1. Blueprint
    • Helps define the story you want to tell
    • Use LaTeX and vector-based drawing tools
    • A good title is important (consider filtering approach above)
    • Writing Perfect Papers talk
    • The intro needs to tell an attractive story
      • General background / context
      • Clearly define the problem (no tech definitions, no mathematic notations; an illustrative figure helps)
      • Why is the problem challenging?
      • Point to existing solutions and pitfalls
      • High-level intuitive explantion of solution idea
      • Highlights of contributions
      • (optional) structure and content of rest of paper
    • Related work is very important for drawing a clear line between existing work and new ideas. It should be thorough, yet brief.

Additional questions

Lecture 22 (Mar. 31, 2022)

Class notes: Today I presented my talk on asking good questions in a research setting. I think the talk went well; it felt pretty smooth to me (despite a few technical glitches) and I kept it to just over 15 minutes. I think the ideas probably came across pretty clearly (based on discussion after), and I think Johanna’s addition of the ‘clarification, expansion, transposition’ framework worked really well.

No one was really jumping up to discuss afterwards, although those who did contribute with Dr. Muller’s prompting did have some great things to say. I really appreciated Derek’s words about nerves, and I think there were some other great discussion points too. Unfortunately, I didn’t write them down, because I was trying to be 100% present. The one that I did right down was Dr. Muller’s addition of ‘pertinant’ as a key quality of a good question. I think that’s a good one, and could probably replace ‘respectful’ since I inteded ‘respectful’ to capture a whole host of similar things, but it’s somewhat redudant with the other qualities.

STAR method for interviews

Structured behavioural-based interiew method

Situation

Task

Action

Result

How to prepare for a behavioural interview

Lecture 22 (Apr. 4, 2022)

Class notes: Lecture on quantum computing by Dr. Hausi Muller. I didn’t take as many notes during this talk because I wanted to try to follow the content closely, knowning that it might be somewhat challenging material. I can’t say I followed everything completely, but it was a pretty fascinating talk. I haven’t really given a lot of thought to quantum computing - other than the science news articles I read now and then - and I didn’t even realize before the beginning of this course that quantum computing was so accessible. I’m definitely convinced now that it’s something I should learn about - if only I can find the time. :)

Lecture 23 (Apr. 7, 2022)

Class notes: Guest lecture on emperical methods by Dr. Margaret-Anne Storey

Who-What-How framework

Additional questions

Back to top

Research elevator pitch & discussion

The elements of a research pitch

The elements of a good research pitch performance

Assuming a non-specialist audience [5]:

Some notes from 3MT winning pitches

Azalia Muchransyah, 2021 winner

Sean Marrs, 2020 winner

Ben Rein, 2020 winner

Aha! moments that I’d like to incorporate

Some of these are replicated in the notes above, but these are moments that stood out either during lecture, or watching assigned videos:

Developing my research pitch

Why the research must be done

This is a difficult one for me to fully qualify. My research project is a small part of a larger effort within cryptography, but I have very little familiarity with cryptography. That said, my novice understanding could be something of a benefit here - I can provide some context, but don’t have the vocabulary to even approach being overwhelming.

From what I understand (and with reference to [6]), the overarching goal of the research is to identify structural properties of cryptographic primitives as they relate to typical cryptographic assumptions. The typical approach of building complex primitives from assumptions is in some senses limited; the research in [6] demonstrates that another way of demonstrating the “feasibility” of primitive construction is possible. This in turn opens up new questions relating to simpler constructions of important primitives, or relating to the development of new assumptions or primitives.

My part in this is in the development of proof assistant (i.e. Lean) tactics to prove the security of the constructions posed in [6]. As a relatively new, but highly popular proof assistant, Lean represents a new frontier in the development of formal verification tools. Additionally, it has perhaps less baggage than Coq, which historically has been the proof assistant of choice in any cryptographic applications. In that sense the work I do will be relatively novel, and we hope that the work will continue to have broad application in providing demonstrable security for newly developed cryptographic primitives.

Pitch structure

The beginning

The middle

The end

Complete script

Hi, my name is Ashley and I am a master’s student in the department of computer science. You might have heard of Alice and Bob; they’ve been trading secret messages back and forth as part of cryptographic research since 1978. Over those years Alice and Bob managed to continually improve their methods and keep their communication secure, despite major advances in computational power.

But the cryptographic tools Alice and Bob use are fallible - sometimes surprisingly so. Recent cryptograhpic research has demonstrated that it is possible to build new cryptographic tools from unexpected sources; but the proof that these techniques are secure - in a cryptographic sense - is a challenge best handled by formal verification.

In my research project I use the Lean programming language to demonstrate the security of these cryptographic techniques. Embedded in Lean is a trusted kernel that can be relied upon in the development of rigorous mathematical proof. These proofs are rigorous because Lean’s trusted kernel is sound, implying the validity of all logic and proof that stem from the kernel. My research also helps to push the boundaries of interactive proof systems and to advance the broader project of proof automation in mathematics.

To find out more about my work, please visit my GitHub page at ashandoak.github.io

The development of my pitch and reflections

Although Bruce and I have had a pretty good idea of which direction I’d be going in for my MSc project since mid-last semester, it’s really only been since some time in December that we’ve started to formalize it, and since I’ve met our project collaborators. So in that sense, I’m just getting up to speed with a lot of the ideas I’ve discussed above. The development of my pitch and the work I’ve had to do to wrap my head around the motivations of my research are not something I’d yet been able to make the time for - so this has been incredibly valuable! I now feel that I have a better handle on what I’m doing and why I’m doing it.

The loose pitch script above came about relatively easily, although some additional practice was needed to build something that sounded natural. the structural approach was particularly helpful, as I was able to easily identify where I could inject some story and where I could be more technical. Also, some of the questions for research pitches listed above (in the elements of a good pitch section) were helping in identifying how I am contributing to the broader field. Although I don’t think I’ve fully capitalized on that yet - there is more to be said down the line, for sure.

It took me a bunch of attempts to get it right - in the end I actually memorized the script. Funny story: I did about ten takes and finally got something I thought I was happy with, then realized that the mic had been off the whole time… Did another ten takes before I got it right with audio - so I have a good quantity of b-takes for a potential blooper reel. :)

My research elevator pitch video

To see my research elevator pitch follow this YouTube link.

Back to top

Notes & reflections on assigned homework

How to Give a Great Research Talk - Simon Peyton Jones

In summary: Jones presents best practices for developing, preparing for, and delivering a 20 minute talk. Jones argues that most talks that the audience will deliver throughout their career will be a 20 minute talk, as hour long talks are rare.

Creating effective slides: Design, Construction, and Use in Science - Jean-luc Dumount

In summary: Dumount presents compelling arguments for the use of slides from an accessibility and effective redundancy perspective, but stresses that a talk is not exclusively the talk. Dumount encourages proper talk presentation, including planning (i.e. audience, needs, motivation, room conditions, etc.) and the so what? (i.e. the motivation, key idea and insight that the talk provides). Dumount argues that one should do slides well or not at all, else they are counter productive.

Designing effective scientific presentations - Susan McConnell

In summary: In this highly practical talk McConnel presents a number of best practices in slide construction and slide / talk structure and organization to create an effective talk. I think the key insight is the idea of the key slide, which is intended as a representation of the key idea of the talk, which can be used as a guiding tool throughout the talk.

Back to top

Research talk best practices & discussion

Reflections on talks by Winston, Simon, Dumount and McConnell on research talks

Unquestionably, the common thread through all of these resources are the concepts of intuition, story, and humanist appeal. When it comes especially to shorter modes, such as the elevator pitch and the 20-minute research talk, the important of providing some meaningful intuition that can communicate your key ideas to the audience is critical. Good stories (or analogies, as with the traffic stop / brain activity 3MT talk above) can help cement these intuitions into longer-term memory, while appealing to humanist ideals helps to ground intuition in meaning real-world concerns.

From a practical talk / slide construction perspective I’m also pretty enamoured of effective redundancy and relentless pruning. I think the idea of effective redundancy speaks a bit to the question I tried to ask about “slides as props” during the second Winston class. I don’t think I clearly articulated what I was feeling, but there were something about the distinction between slides as communication tools and slides as performance that was sort of tickling my brain… I’m still not sure there is really anything there, but the idea that slides exist as a tool of universal design for learning feels right, whereas the idea of slides as a tool for motivation (as a more traditional slideshow might be) seems appealing, but less is clearly less accessible. I think, fundamentally, I dislike slides as they are typically conceived and I struggle to find value in them (from a teaching perspective) when I know that students often expect them.

I wish I had time for relentless pruning… But, as the Mark Twain quote above states, it’s not always easy to write a short letter. I guess when it comes to a really high stakes talk, I would almost certainly spend the time, but in terms of lectures I’m often constructing slides days, or at best a week before their use, and I just don’t have the time, along with my other responsibilities.

Interestingly, I thought that the talk by Susan McConnell, about slide best practices, would be the least valuable of the three. However, the day after I watched it I needed to make some slides for a class I’m teaching and her advice was on my mind. I decided to lay out the slides by first writing out a bulleted list with sub-bullets that covered the content I wanted to get to. Each bullet was a full sentence and would eventually become the slide heading/title, and each sub-bullet became a bulleted point on the slide (though some slides were just pictures).

In the course of this it came about very naturally that I ended up using a home slide to bring the focus back to the main motivation of the lecture. I didn’t intend to use one, but it became clear about half way through that it would be highly valuable as a tool to cycle back and reaffirm the goals of the lecture. In addition, the full sentence heading/title was remarkably effective at getting the point across. Like many, I think I’ve typically thought of the title as literally that, but why not use the largest emphasis on the screen to get the main point across? It made pruning the actual bullets much easier as I didn’t have to try to communicate both contextual information and the point all at one time.

I’ve only done this for one set of lectures, but at this point, I think I’m a convert…

Back to top

Proposed talk: How to pose good questions at a research talk

Motivation

After our in-class research elevator pitch presentations I was reflecting on the process of asking good questions of a researcher during a research talk. I believe that asking questions is an important step in the learning process: As an instructional faculty member of a teaching college I try to push my students to ask questions as often as possible because I think that asking questions, even simple simple questions, helps to establish concrete foundations for learning. Opening a dialogue can be the first step to a meta-cognitive approach to learning.

There are many good questions asked at research talks, but there are also many bad questions (e.g., aggressive, posturing or rambling questions). Those aggressive and posturing types of questions are, I think, outside of the scope of what we want to accomplish here (but maybe not? Can an aggressive question be a good question?); I’m most interested in identifying what makes a good question, maybe in order to avoid the rambling style of question that sometimes manifests at research talks.

I suppose in particular, since we’ve talked a lot about presentation skills it seemed to me that there was a gap in not looking at the other side of a talk or lecture. Answering questions is something that was addressed in some of the videos we watch (see Simon Peyton Jones’ response to aggressive questions, for example), but research is meant to be a dialog, and asking good questions is clearly a key part of that process.

The purpose of this talk

My goal here is for this talk to act as a starting point for an open discussion on how to formulate good questions. What I propose here isn’t meant to be prescriptive, and I’m not claiming to have all the answers. My recommendations provided here stem from what has worked for me (as a ‘questioner’) and what I hope will be true of those who would ask me questions if I was the one giving a talk. You may agree with some things I say, you may vehemently disagree with some things I say - I would be happy to hear both perspectives since I think the way forward is through open dialogue. I certainly hope that this is useful as well, though I wouldn’t be surprised to learn that many of you already have your own process for asking good questions; nevertheless, if this sparks some dialogue on the topic I hope that we can all stand to learn something.

The qualities of a good question

At the risk of being labeled a reductionist I’m going to propose a number of qualities that I think contribute to making a good question. Of course, every opportunity for questions is different and one cannot always hope to articulate the ‘perfect’ question; yet I do believe that aspiring to the core qualities below could serve to improve any potential questions. Good questions are:

I think that there are potentially other qualities of a good question that we could aspire to, but not necessarily always achieve. Some of these qualities speak directly to the types of questions you can ask yourself when you are formulating questions, which we’ll talk about in a minute. These could include qualities like:

Strategies

I’ll preface this by saying these are not science-backed recommendations. Most of these come from either experience, or from common wisdom. So many will likely be familiar to you already anyway.

Back to top

Paper review: Constructive Mathematics and Computer Programming - Per Martin-Lof [7]

Overall merit

  1. Accept

Reviewer expertise

Brief summary of the paper

Per Martin-Lof justifies the argument that constructive mathematics and the act of programming are one and the same via the presentation of a formal system of his design, known as intuitionistic type theory. Martin-Lof contends that his type theory is precise enough to serve as a programming language and demonstrates this through a direct comparison of his type theory to common mathematical conventions, including notation and the evaluation of expressions, and also by introduction of the foundational judgements of his type theory and the inference rules used for the construction of formal proof. These judgements and inference rules can be used equally well from a constructive mathematics standpoint, or from a computer programming standpoint. Martin-Lof further elaborates this connection by arguing for the uninhabited-ness of the law of the excluded middle, but conversely providing a sound construction for the axiom of choice via the inference rules of his type theory.

Contributions of the paper

What are the strengths of the paper?

What are the weaknesses of the paper?

Significance

Martin-Lof has clearly positioned this paper as significant with respect to other major developments in the field by referencing important work by Dijkstra, Hoare, and Brouwer, among others. Martin-Lof is clearly responding to significant discussions in the field at the time, specifically the divide between classical and constructivist approaches to mathematics and logic, and to the increasing relevance of computer programming as a tool of scientific inquiry.

Martin-Lof makes a clear statement of significance at the end of page 503 when he discusses his project for intuitionistic type theory: a new formal system within constructive mathematics that attempts to close the gap between the constructive approach and the practice of programming.

In particular, Martin-Lof makes the claim that in his type theory the axiom of choice doesn’t limit the development of rigorous mathematical proof, and furthermore provides a construction of the axiom of choice. Rather, Martin-Lof points to the law of the excluded middle as being the most critical concern for the formalization of reasoning in intuitionistic type theory.

Soundness

The bulk of the paper is concerned with the rules of inference for the typing judgements that Martin-Lof has developed for intuitionistic type theory. These are sound (in a theoretical sense), and Martin-Lof demonstrates this through the construction of a representation of the axiom of choice via his inference rules. The results themselves certainly support the main research questions relating to the use of a formal logic / system as a tool for both constructive mathematics and programming.

There is some work left to the reader, in that Martin-Lof does not clearly demonstrate the derivation of the rules of inference (see below), but he argues this this paper “is not the place for such detailed explanations”.

Novelty

As mentioned above, Martin-Lof is successful in grounding his work in the contemporary conversation relating to formal systems; and furthermore, as an extension of his own work, this paper demonstrates a continuation of previously novel results. The paper is successful in demonstrating novelty in terms of the presentation of the inference rules of intuitionistic type theory. Certainly there would be an audience for this paper, although relatively narrow in scope.

Verifiability & transparency

There is a certain degree to which this paper is clearly a product of its time and of the ongoing contemporary conversation around formal logic. Retrospectively, it is hard to understand at what stage of the development of formal logic Martin-Lof’s paper appeared (from an absolute sense, it is easy to understand; but from a relative sense - i.e. concerning what knowledge a contemporary of Martin-Lof’s might have - it is much more difficult to comprehend where Martin-Lof fits). That said, for anyone well-versed in the topic at the time I think there is enough here to go on.

However, the lack of clear justification of inference rules might be a limiting factor of the paper. Certainly, it’s not a very inclusive approach. Although the process of deriving and justifying inference rules is likely familiar to many readers, there may be those who lack the necessary background.

Presentation

The paper itself is a pleasure to read; I very much enjoyed Martin-Lof’s prose. I’ve had similar experiences reading other older papers and texts (i.e. by Dijkstra and others) and do wish that more papers were written with the same kind of appreciation for language today.

The organization of the paper is excellent. There is a clear lead-in to the key points and a deeper exploration of the key contributions. Although there is a long section that contains simply the inference rules, the paper is easy to navigate and to predict.

The only concern I have is with the section that describes the four main judgements, both assumption-free and with assumptions. I found that following the argumentation in this section (encompassing pages 508 through 510) was particularly challenging, in part due to notational choices, and in part due to typesetting choices. This is also somewhat of an issue in the inference rules section.

Questions for the author

Back to top

Thesis introduction

The following are some basic notes that I have compiled that may eventually become the introduction to my thesis. At this stage (Mar. 2022) I’m still sufficiently early in the process that there are many unknowns to deal with. Additionally, I’ve learned through the process of CSC 595 that I’m really missing out on having a solid foundation in some of the topics that my research encompasses. Therefore, once I’ve completed a meaningful literature review (scheduled for this spring and summer) I’ll be able to fill in the sections below with much more confidence.

Establish research topic

Problem statement

The cryptographic primitives presented in [6] depend on a proof of the left-over hash lemma. I aim to develop the necessary theory and tactics in Lean to provide a proof of the left-over hash lemma.

Motivation and background

Proof assistants and formal verification tools are becoming more prevalent in research mathematics. Cryptographic researchers have also increasingly been using formal verification tools (e.g. EasyCrypt, and various libraries of Coq). This is a developing field that has potential significance and societal benefit.

I think I’ll need to include something here about type theory and monadic programming, which to me seem to be the key components in developing the necessary Lean tactics. I don’t think I need to have significant cryptographic understanding (our research partners will handle that side of things), but understanding the game-based method of proof of security seems critical.

Define the scope of research

Show the work’s significance, relevance, impact

State research questions and objectives

It’s too difficult to articulate clear questions at this stage, but I could state some loose objectives:

State expected contributions

Give an overview of your structure

Back to top

Literature review & Bruce’s suggestions

I’ve been using Zotero for some years as a tool to collect papers that I use for teaching and for pedagogical research. It was an easy transition to continue using Zotero as part of my MSc work; however, I’ve realized that there are features of Zotero that I haven’t taken advantage of, including the proper use of BibTex, tags and paper-specific summary comments. I plan to take some time this spring to review the papers that I have current stored in Zotero and the organizational scheme that I’m using. At this point, I haven’t yet used Zotero to do anything other than store papers that may become part of my literature review, so this is good timing.

As a follow-up to Dr. Muller’s suggestions on the top papers in my field I Bruce’s opinion. Given that our project sits something at the intersection of formal verification, interactive theroem proving (ITP) and cryptography, but Bruce’s background is more cryptography, the papers tend to lean in that direction. I’ve listed them here because I haven’t yet stored copies of them in Zotero, but will do so when I start the review process this spring. Better to have them here than in a random email thread.

Bruce suggests that picking a top 10 in ITP would be hard, but a few would be:

I guess I’d add to this Martin-Lof. Maybe also some homotopy people?

Bruce suggests the following top papers in symbolig crypto:

Back to top

A consolidated discussion of CSC 595 guest lectures

General discussion

We had guest lectures by Dr. Alex Thomo, Dr. Sowmya Somanath, Dr. Kui Wu, and Dr. Margaret-Anne Storey. I found all of these lectures to be valuable. The speakers were diverse, articulate, well-prepared, and were clearly engaged by the students and their questions. I think they were all really well chosen, and I’m happy to have heard them all speak. I’m going to give a few brief sentences for each that identify what struck me the most from each of their talks.

Dr. Alex Thomo

I found Dr. Thomo’s talk to be very engaging and practical. Although Dr. Thomo’s interest clearly lies in the direction of data analysis as a key research method, I think many of the steps he suggested as part of developing good research questions are absolutely much more widely applicable. And the presentation of his method was highly compelling: I particularly liked the clear motivations that Dr. Thomo provided as part of his argument, as well as the clear line that he drew across works published by his research group in recent years that support the demonstration of his research process. Very cool. Additionally, many of his ‘writing a paper’ suggestions further confirmed what we learned earlier in the semester.

Dr. Sowmya Somanath

I found Dr. Somanath’s talk to be very interesting, but it wasn’t new to me, having taken her topics class in designing creativity support tools last year. The talk she gave was very much a high level summary of much of the content that we covered in greater detail last year (in fact, I think the talk was part of her opening/introductory lecture in the class). Nevertheless, it is a great talk, and I got a lot out of that course - it was really valuable, and very well put-together. I do really appreciate Dr. Somanath’s perspective on design solutions, which seem to be very much in line with Dr. Storey’s.

Dr. Kui Wu

I found Dr. Wu’s talk to be very practical as well. I think there were some great suggestions on how best to manage the flood of papers that are regularly produced. I still feel that I’m not entirely sure it’s humanly possible to even use Dr. Wu’s very precise method, but I think I still lack the breadth of foundational knowledge that would allow me to quickly filter especially on the second step, based on references and state-of-the-art. Obviously, that takes some considerable practice. I’ll probably put Dr. Wu’s method to good use this spring as I work to build out that breadth of knowledge.

Dr. Margaret-Anne Storey

I found Dr. Storey’s talk to be hands down the most fascinating of the four. I’ve previously taught a number of courses (prior to my current appointment) that took a primarily cultural studies look at technology and society. As part of this, and due in part to my background in mathematics and computing, I’ve spent quite a bit of time reading related research in the social sciences and cultural studies. In fact, this was in part the impetus for me to complete an MSc in computer science: I have a great interest in the socio-technical, especially from a computational standpoint, and I wanted to build on that knowledge.

I’ve actually sent Dr. Storey a rather lengthy email asking about her opinions on the socio-technical from a cultural studies perspective, and if she thinks there is value in that particular approach (rather than a pure social sciences approach). I haven’t heard back from her yet, but I’ll report back here when I do.

Some key take-aways

Aside from some of the points above, the following are the key points that I have distilled from these four talks:

Back to top

Learning outcomes

The learning outcomes of CSC595 as stated in the official syllabus are:

  1. prepare and evaluate research papers
  2. perform literature reviews
  3. give effective oral and written presentations
  4. contribute to a positive work environment
  5. develop a term portfolio Jupyter Notebook

I think all five were valuable in their own right, but I’m going to say that the first three were the most important. While I think there were elements of ‘contributing to a positive work environment’ (i.e. in group work and in supporting other students in their presentations), and while I did enjoy the process of writing this learning journey using Jupyter Notebooks, I feel both had relatively small parts to play in what and how I learned.

The most surprising thing I learned had to do with learning outcome #3: While I don’t think I would claim to be an expert when it comes to presentation skills, I’ve definitely spent a lot of time presenting material in the last eight years, and so certainly have a lot of experience. The same can certainly be said for the development of presentation materials. Coming into this class I recognized that refreshing knowledge in those domains would be a valuable exercise to help me improve my skills - but I didn’t expect to learn quite as much as I did. There was a lot of really great material and ideas covered in the early part of the course relating to this learning outcome; I’ve reflected on a lot of it, and some of it I’ve already adopted. Perhaps the focus on research is what made it different from other recommendations I’ve seen/read about presenting before; either way, it has been quite valuable.

The first two learning outcomes were anticipated elements of the course, and I’m happy to say I learned a lot about both. I think learning outcome #1 was especially eye opening to me, as I realized throughout the semester that I could certainly do more to help further develop my breadth of knowledge. Honestly, now I wish I had taken this course last year, since in fall 2021 I had more time available for research, and having had this course under my belt would have been useful. That said, I’m happy that I got a chance to take Dr. Somanath’s class this time last year, and I’m happy to have had Dr. Muller this year for CSC595.

I think I learned a lot about learning outcome #2 and literature review as well, especially as it relates to my current research practice. I think there is still a bit of a gap in regards to systematic literature review (although now that I think about it, I wonder if I just missed that class? There was one class I missed - lecture 10 Feb. 10 - because I couldn’t attend live that day and the recording failed. Maybe systematic literature review was covered then? It might make sense based on the content of lecture 11).

I guess too that I was a little surprised that we didn’t discuss specific research methods in detail this semester. I don’t think this is really a problem - there was lots of other highly valuable content that I was happy to consume that on the whole will probably be more practically useful to me than would knowing about a bunch of research methods that I’ll probably never use. And aside from that we did get little sneak peaks of various methods via the guest lectures at the end of the semester. And I’ve written down references (interspersed above) of papers and texts relating to research methods that I’d like to read. I’m looking forward to doing so, because I think it is an interesting topic, but really, I’m much more interested in the interdisciplinary approach hinted to by Dr. Storey in her talk at the end of the semester.

As for personal epiphanies and insights: I’ve written about many and they are interspersed throughout this document. Many of the most critical are already captured in these appendix sections added to after the class notes. I think this is a document that I will continue to refer to, and perhaps to revise, for many years to come. Thank you, Dr. Muller, for providing such a rich learning experience!

Back to top

References

[1] Inger Mewburn. 2010. How to Sell Your Thesis in 3 Minutes (or Less). The Thesis Whisperer. Retrieved January 27, 2022 from https://thesiswhisperer.com/2010/07/01/how-to-sell-your-thesis-in-3-minutes-or-less/.

[2] Elevator Pitch Practice. Manning College of Information & Computer Sciences. Retrieved January 18, 2022 from https://www.cics.umass.edu/careers/articles/elevator-pitch-practice/.

[3] Gretchen Busl and Matthew Capdevielle. The Elevator Pitch: Presenting Your Research in Conversation. University of Notre Dame. Retrieved January 18 2022 from https://graduateschool.nd.edu/assets/76988/elevator_pitch_8_28_2012.pdf/.

[4] MMU. One Minute Elevator Pitch. Retrieved January 18, 2022 from https://www.mmu.ac.uk/media/mmuacuk/content/documents/research/Impact-Tool—-Elevator-Pitch.pdf

[5] Simon Clews. 2013. Making the Most of Your Three Minutes. 3MT: The Three Minute Thesis. Retrieved January 27, 2022 from https://wp.auburn.edu/graduate_school/wp-content/uploads/files/3MT/Simon_Clews_3MT_guide.pdf

[6] Navid Alamati et al. 2019. Minicrypt Primitives with Algebraic Structure and Applications. In Annual International Conference on the Theory and Applications of Cryptographic Techniques 2019 May 19 (pp. 55-82). Springer, Cham. https://ia.cr/2019/108

[7] Martin-Lof, Per. 1984. Constructive mathematics and computer programming. In Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences 312, no. 1522 (pp. 501-518). https://doi.org/10.1098/rsta.1984.0073

Back to top

Useful resources

Back to top