David Kil is an active technology entrepreneur. For a new company that uses causal AI and machine learning to help businesses connect their actions to outcomes to improve results. He was the chief data scientist at Civitas Learning from 2021 to 2023, where he developed college student success analytics to improved retention and graduation. He has participated in about a dozen companies in similar ways, since he started as the manager for Intelligent Systems at Rockwell Science Center. David got his BS degree at the University of Illinois at Urbana -Champaign in Electronics Engineering and Chemistry. His MS degree in Electrical and Electronics engineering at NYU and an MBA degree and Management and Finance at Arizona State University. I met David when he worked at Civitas and several Provosts and I got a SUNY grant to tie the New York State Department of Labor salary database to the content of programs at a few SUNY universities, particularly in regard to how much that program utilized internships and other experiential workplace experiences.
Topics discussed in this Episode include:
Resources discussed in or related to this episode:
[Team 0.00] Welcome to the Experience Ed podcast. I am Jim Steller. I'm Mary Churchill. And I am Adrienne Dooley. We bring you this podcast on experiential education with educators and thought leaders from around the country and the world.
[JS 0.25) David Kiel is an active technology entrepreneur. For a new company that uses causal AI and machine learning to help businesses connect their actions to outcomes to improve results. He was the chief data scientist at Civitas Learning from 2021 to 2023, where he developed college student success analytics to improved retention and graduation.
He has participated in about a dozen companies in similar ways, since he started as the manager for Intelligent Systems at Rockwell Science Center. David got his BS degree at the University of Illinois at Urbana -Champaign in Electronics Engineering and Chemistry.
His MS degree in Electrical and Electronics engineering at NYU and an MBA degree and Management and Finance at Arizona State University. I met David when he worked at Civitas and several Provosts and I got a SUNY grant to tie the New York State Department of Labor salary database to the content of programs at a few SUNY universities, particularly in regard to how much that program utilized internships and other experiential workplace experiences.
So, David, you are well known to me as a data -driven engineer who works on human behavioral problems. How did you get interested in data in college and decide to apply it through an MBA degree?
[DK 2.00] Well, thanks Jim for inviting me to your podcast. I was a chemistry major in pre -med initially, but I soon became interested in digital signal processing and making sense of complex situations for human decision making.
So, I transferred to electrical engineering, but I also realized that I enjoyed chemistry, especially organic chem. As a result, I decided to double major in E.E. and chemistry. After graduation I worked in the defense industry processing all kinds of sensor data from satellite imagery to underwater sonar and everything in between.
I developed a lot of predictive models and causal Bayesian networks to understand what's happening, what is likely to happen, why and what actions to recommend for favorable mission outcomes for our forces.
But then something clicked. I became more curious about the people who have to process AI outputs and make decisions based on those outputs. So how do they behave and how did they make decisions, especially when AI output are based on probabilities in the world of uncertainties?
That's when I decided to take MBA courses at night while working full-time. And looking back, that's where I learned the importance of collaboration between AI and humans.
[JS 3.32] That is fascinating. Now, you and I first met at the City University of New York, or CUNY, and we talked about health care and education. And this was interesting to me, especially given my personal background in behavioral neuroscience at Harvard and your tenure at Humana, a big healthcare insurance company.
So tell me now a little bit about your healthcare experience and how that shaped your higher education AI journey.
[DK 3:59] Sure, after a few years at Rockwell Science Center in Palo Alto, California, I wanted to find an opportunity to apply data science to a domain more related to improving human well -being. I found that opportunity at Humana through connections.
I joined Humanna as the Head of Analytics because, interestingly, my boss, Dr. Jack Lord, viewed my lack of experience in healthcare as an asset. For the first few months, I just listened, trying to understand and digest the nuances of the healthcare sector.
Jack then said, hey, Dave, your job is to build a best predictive model. Simultaneously, he invested in a behavioral coaching program to get patients activated in managing their chronic conditions. He really wanted to understand how to apply behavioral science as well as predictive science to help improve patient outcomes.
I had a talented clinician named Marlene in my team, and she and I worked together to create a number of interesting derived features around disease conditions and disease progression. Soon, our model based on Ovation Regularization Deep Learning Network significantly outperformed all known predictive models at the time.
We then ran a pilot program on members struggling with diabetes. After a few months, I developed causal AI models using cell -based matching, as well as both predictive and propensity models to measure program impact.
To everyone's relief, I reported significantly positive outcomes, and everyone was very happy. we thought we had discovered our secret sauce. Encouraged by our CEO, then we scaled the behavior coaching program to encompass all 3 million commercial members at a time.
However, when I conducted the same follow -up impact analysis three months later, the results were unexpectedly negative.
[JS 6.20] Wow. So, can you tell us what happened? Of course.
[DK 6.27] In two words, population heterogeneity.
In retrospect, it was clear that there was no one-size-fits-all program for all high -risk members. Upon conducting a detailed drill -down integral analysis to understand why the program did not work, I found that many members with lifestyle -related chronic conditions such as diabetes, cardiovascular disease, and musculoskeletal conditions benefit a lot from the program.
On the other hand, members of complex comorbidities and HIV did not fail as well. So that was one interesting finding. But I did not stop there after conducting what I call human capital impact analysis to understand if there are highly effective coaches to learn from.
I also listened to conversations between our health coaches and patients to understand why some health coaches were so much more effective in activating patients and improving their outcomes. It turned out that nurses with a social work background were far more effective than those with ER, ICU, or research background.
And effective nurses were also good at building empathy, setting bite -sized goals for patients, and explaining the benefits of following their recommendation in simple language with patients because some patients had some cognitive difficulties.
Furthermore, I discovered that reaching out to patients, especially those with congestive heart failure within a week of their discharge from heart bypass operations, resulted in far better outcomes than contacting them more than a month later.
As you can imagine, people are far more receptive to advice on avoiding adverse events shortly after experiencing the discomfort of heart bypass operations rather than one system on the road.
[JS 8.37] Makes sense.
[DK 8.39] So, all these causal insights were crucial in improving patient care programs. We ended up creating a portfolio of patient care programs, each catering to the specific needs of different patient groups.
Furthermore, following the CEO's guidance, we established monthly cross -functional steering committee meetings. In those sessions we reviewed the most recent impact analysis findings and strategized on how to improve program operations continuously.
I greatly enjoyed this entire experience at Humana.
[JS 9.16] This is fascinating and probably deserves a whole blog itself or podcast itself. But I want to turn back to education. So, when we work together, you are at Civitas on the problem of how certain programs and universities that carry internship, something dear to this podcast, versus those that don't.
And we were trying to use the New York State Department of Labor data on salary 10 years after graduation to learn something about what those programs had as an impact. I know what interested me in that effort.
But tell me, what interest did you?
[DK 10.02] Well, I mean, it was you. So, I thoroughly enjoyed our conversation during my visit to New York City, where I had an opportunity to present at the NYU Center for Data Science. I've always been interested in career outcomes and how hired institutions can help students achieve greater success post-graduation.
So, it was really a match made in heaven in that it was you and the subject was so interesting. I jumped at the chance to work with you on your team on this causal integral analysis of experiential learning programs at SUNY.
[JS 10.39] It was an amazing experience for me as well. Let's shift the focus a little bit broader here and talk about how higher education is continuing to involve with the presence now of big data and even with artificial intelligence to digest for the parents and the families how universities should be viewed.
Do you think that someday this kind of employment data that we investigated will be standard in evaluating universities by families?
[DK 11.13] I certainly hope so in a thoughtful manner. And as you know, this is a very nuanced topic. On one hand, the integration of employment data and AI models will be crucial to understand what works for whom in career outcome so that evidence -informed decisions can be made at the policy level and evidence-based programs can help students at an individual level.
On the other hand, this is something I've seen so many times, decisions need to be made based on poor looking data and crucial skills needed in the future. So around five years ago, I was talking to a computer science professor in Austin about the static admission rates to the CS program despite the growing demand for software developers.
He said that the primary reason was the boom-and-bust cycle of developer job trends. And upon hearing that, I recall conversation with some parents advising their children against computer science majors because many developer jobs were being offshored at the time.
And now some experts are predicting that demand for CS and IT jobs may decline because of generative AI. And I also use generative AI to sort of speed up my development time. And another example is that DARPA is looking for some new ideas on how to increase AI skills in today's workforce, especially under business translation part.
And in terms of parents and families, there will be a greater demand for transparency and accountability regarding career outcomes because that's what they really care about. We need to show the pathways towards improved career outcome using causal insights.
Based on my experience, such causal insights are crucial in enacting evidence -based educational policies and programs. Agencies such as DARPA, or Institute of Education Sciences, or Foundations, can catalyze this kind of initiative adopting a star -small, scale -fast approach with a select cohort of institutions.
This would involve establishing a robust data framework that prioritizes privacy through mechanisms such a differential privacy. This kind of infrastructure would enable researchers to collaborate on various AI analytics to significantly enhance career outcomes for current and future students.
In essence, the Career of Questions centers on the types of lifelong learning, financial knowledge and life skills that help students adapt and thrive in rapidly changing environments. From this vantage point, the career outcomes data that I would like to process encompasses students' backgrounds, a concise and comprehensive summary of student information system, learning management system data throughout their college years, intervention data such as internships and e -portfolio and other types of experiential learning programs, and longitudinal career outcomes data that can be obtained through, let's say, the State Department of Labor or working with private vendors that collect such data sets.
So this approach will facilitate the ability to measure the influence of their college experiences and exposure to various interventions on their career path and social mobility, and such a data framework encompasses us to explore which programs and modifiable student characteristics contribute to social mobility career advancement and overall life satisfaction.
And by the way, I did the exact same work with consumer health data, and we learned so much through a three -year NIH grant, working with university researchers.
[JS 15.45] Fascinating. I think I want to return at the end to this idea about causal insights and have you talked more about it. But before we do that, given this wonderful setup, can you react to this question?
Will US News or some other comparable an entity hire one of your companies to figure out the rankings for them to provide to customers? What do you think about that idea?
[DK 16:10] Yeah, that'd be great. But seriously, I envision the application of causal AI in helping institutions see and act on their relative student metrics, rather than absolute rankings. And for example, I recall discussions with a few provosts over dinner in Manhattan.
They share their motivations for choosing to work at institutions that serve underprivileged students. Their primary drive was the profound impact they could have on improving their students' outcomes much more than at elite institutions they came from.
Similarly in health care, we evaluated hospitals and providers based on risk -adjusted patient outcomes. So, this information enabled us to establish an efficient provider network so we could nudge patients towards these top performing providers through targeted benefit programs for optimizing population health.
[JS 17.22] Excellent. So let me ask you this question personally. Would you want your kids going to a college or university based on these statistics? I tend to think that other stuff matters too, like critical thinking maturity, etc.
If so, how do we fit that into your analysis?
[DK 17.43] Yes, definitely. As long as there is transparency and trust in statistical measures of outcomes based on causal AI. And interestingly, my family members, they already select hospitals based their rankings on various specialties which are provided by the CMS or Centers for care and Medicaid services, and other reputable organizations.
As you know, we all want the best for our children, the best schools, best care, food, etc., but how do we define best? For me, is it by the alpha, also known as the excess return in clear outcomes beyond the expected?
Considering the diversity among some populations where access returns can differ significantly, should our focus be on identifying the most suitable skills for my child, especially tailored to their major and other nuanced factors such as, as you mentioned, critical thinking, maturity, self -efficacy, advocacy, academic and non -academic factors, and evidence based on success programs these schools offer.
I would love to build an application that shows which institutions would offer the best return on education for students like my child. This app would analyze the impact of each institution on career outcomes for various heterogeneous student groups and rank them based on excess returns for students like my child.
I would want my children to attend an institution that will help my child do the best and achieve the most possible career outcomes.
[JS 19.44] Well said, and I feel the same, although my children are older now and one of them is already a professor. She's come to the other side of the equation. So, as we come to the end of this podcast, I want to just give you an opportunity to say if there's something you want.
And after you do that, maybe you could talk a little bit about causal insights, because I'm hearing you talk about them as the great leveler of society where people who have perhaps not as much access to information about higher education could figure out their proper place to go critically important.
So, what do you want to say? And maybe we can talk about that idea of causal insights as the great leveler.
[DK 20.31] Yeah, thanks, Jim. So, the world is changing rapidly, as you know. This necessitates a new level of collaboration between humans and AI. And talking about causal AI, I mean, there's really a branch of AI that attempts to run causal relationships between actions and outcomes in the context of modifiable factors for students, patients, or consumers.
And my experience working with higher institutions, healthcare providers, fintech companies, consumers, and even HR departments shows that the most successful organizations are those that leverage causal insights to drive continuous process innovation.
And if you think about it, decision -making needs to shift from mirror correlations to understanding and acting on the underlying causality. So, our focus on causality empowers us to make evidence -based decisions that benefit the entire society.
For example, let's say you have risk prediction models that identify which students are at risk, but unless you act on those predictions based on evidence-based practices to improve outcomes and improve the entire process continuously.
What good are those predictions? So, from my perspective, there are all kinds of different analytics, but what integrates all of these different AI algorithms as well as humans is the causal AI that really helps you understand what the right evidence-based actions are to improve outcomes based on both exogenous and endogenous risk factors, and how to shape, let's say, consumer students' patients towards a more favorable trajectory.
[JS 22.49] This is fantastic. And you've taught me something that I didn't really know before about the power of causal insights, not just having the risk factor, but also knowing what to do with those risk factors as a university to help the students succeed, to find their right way, and to execute that so that the result is they profited from the time and the money they spent on their education.
That would make everybody happier. So, what a great conversation. I really appreciate it. Thank you for appearing here.
[DK 23.27] Thanks, Jim, for having me on your podcast. This has been a fun conversation and I hope we do it again. Excellent.
[JS 23.35] We will do that again.
[Team 23.38] Thank you for listening. We hope you will come back soon for the next installation of Experience Ed. As we continue to talk about the neuroscience and sociology of enhancing higher education, by combining direct experience with classical academic learning.