Scaling Voice AI at MasterClass with Effect & TypeScript
#3: Scaling Voice AI at MasterClass with Effect & TypeScript
In this episode Johannes Schickling had a conversation with David Golightly, Staff Engineer at MasterClass, to explore how his team built Cortex – a real-time voice AI orchestration layer that powers personalized conversations with celebrity instructors like Gordon Ramsay and Mark Cuban.
In this episode of Cause & Effect, Johannes Schickling had a conversation with David Golightly, Staff Engineer at MasterClass, to explore how his team built Cortex – a real-time voice AI orchestration layer that powers personalized conversations with celebrity instructors like Gordon Ramsay and Mark Cuban.
Song: Dosi & Aisake - Cruising [NCS Release]
Music provided by NoCopyrightSounds
Free Download/Stream: http://ncs.io/Cruising
Watch: http://ncs.lnk.to/CruisingAT/youtube
- (00:00) - Intro & David’s background
- (04:56) - Discovering Effect & early impressions
- (08:32) - Why RxJS wasn’t enough for Cortex
- (16:15) - MasterClass On Call
- (19:10) - Building the orchestration layer
- (25:30) - Incremental adoption of Effect at MasterClass
- (31:43) - Text-to-speech component
- (40:08) - Error handling, observability, open-telemetry
- (01:01:20) - Looking ahead: Effect 4.0 & the future
- (01:08:00) - Closing thoughts
Transcript
00:00When I started looking
00:01into Effect, I started seeing,
00:03Well, this has a lot of promise and
00:05potential if you use
00:07it in the right way.
00:08What I mean by that is you don't have to
00:10necessarily go from 0%
00:12to 100% overnight.
00:14You don't have to completely
00:16redesign the entire application
00:18to take advantage of some of the benefits
00:20that Effect has to offer.
00:22That's how I started, was incrementally
00:24adopting Effect in
00:26parts of the code base.
00:27And this was really
00:28what enabled it to work.
00:33Welcome to Cause & Effect,
00:35a podcast about the TypeScript library
00:37and ecosystem called Effect,
00:40helping engineers to build
00:41production-ready software.
00:43I'm your host, Johannes Schickling, and
00:44I've been building with
00:45Effect for over four years.
00:47With this podcast, I want to help others
00:49understand the powers and
00:51benefits of using Effect.
00:53In this episode, I'm talking to David
00:55Golightly, who's a Staff
00:56Engineer at MasterClass.
00:58In this conversation, we explore how
01:00David has built Cortex,
01:02MasterClass's AI voice chat system,
01:04leveraging Effect streams heavily
01:06to build a cutting-edge AI
01:08experience. Let's get into it.
01:10Hey David, so great to have you on the
01:12podcast. How are you doing?
01:14Doing great. Thanks for having me.
01:17I'm really excited. The two of us had the
01:19pleasure to now meet in person twice in
01:22the course of the last half a year.
01:24The first time we've met in person was at
01:26the Effect TypeScript
01:28Meetup in San Francisco.
01:30And then a couple of weeks ago, we had
01:33the pleasure to meet again in beautiful
01:35Italy in Livorno, where you've also given
01:38a talk about Cortex and what you're
01:41building at MasterClass.
01:43And it was such a pleasure to spend a
01:46bunch of time together and talk about all
01:48things TypeScript related beyond.
01:51You also have some beautiful music gear
01:53in the background. But today we're going
01:55to talk about how you're
01:57using Effect at MasterClass.
01:58So would you mind introducing yourself
02:01and sharing your background?
02:04Yeah, sure. My name is David Golightly. I
02:07am a Staff Engineer at MasterClass.
02:10And I've been in the industry building
02:12mostly web applications for
02:14about almost 18 years now.
02:18I've also dabbled in some mobile
02:20applications, some back end, some other
02:23embedded applications over the years.
02:25But my main focus has been first
02:27in the JavaScript world and then more
02:29recently in the TypeScript
02:30world for the last several years.
02:32My actual background, though, I don't
02:34have a Computer Science degree. My
02:36education is in music.
02:38And I grew up
02:40surrounded by pianos.
02:41My father is a piano technician and also
02:45a piano professor who plays
02:47classical, you know, everything you could
02:50think of classical music.
02:52He has played it. You know, he's played
02:54with orchestras and he's, you know,
02:56played solo concerts and chamber music.
02:58And so forth. So just steeped in
03:02classical music growing up in pianos and
03:04also in repairing pianos, which I found
03:07somehow translated
03:09into programming sort of.
03:12You're doing a lot of really repetitive
03:13tasks. You have 88 keys. You have 230
03:16some strings on a regular piano.
03:19So there's a lot of repetitive tasks and
03:21you quickly figure out ways to work more
03:24efficiently when
03:25you're doing piano repair.
03:27So later on, after I got my degree in
03:29music, I found that it wasn't well, well,
03:32music is a passion of mine and it's
03:34something that I still do.
03:35It's not something I want to try to make
03:37a career in. And so I quickly discovered
03:40that, well,
03:41programming is just right there.
03:43And I actually enjoy doing it somewhat.
03:45And so I devoted some energy into
03:47learning how to do that professionally.
03:50And, you know, all these
03:51years later, here I am.
03:52That is awesome. And such a funny
03:54parallel since we never talked about
03:57this, but my mother also
03:58happens to be a professor in piano.
04:01And so I likewise also grew up surrounded
04:05by pianos and played the piano for quite
04:09a while, but then also found myself more
04:12attractive to technology.
04:14But I think this is a fairly common
04:17overlap of people who had their start in
04:22music or a different kind of art and then
04:25gravitated also towards programming.
04:28And I think it's just a certain way of
04:30like how someone's brain works that
04:33really like there's a couple of people
04:36come to mind to have professional music
04:38background who are just like absolutely
04:40brilliant in the engineering field.
04:43So that's a very funny parallel. But
04:46keeping it a bit more focused on like
04:48Effect, what has brought you to Effect?
04:50How did you first learn about it? And
04:53what did you think about it when you
04:55saw it the first time?
04:56Yeah, so I was not necessarily going to
05:00adopt Effect for the work I was doing,
05:02say, maybe over a year ago,
05:05which was more React focused.
05:08It wasn't something that I was
05:08necessarily considering.
05:10You know, I think React is a
05:12great framework and it totally
05:13transformed how we do
05:16front end development.
05:17But it's not without its problems in
05:20terms of state management and so forth.
05:21But I was mostly happy with it for the
05:23front end and we try to
05:24keep things lightweight there.
05:26But then I got asked to work on this new
05:28project Cortex, which I'll
05:29talk about a little bit more.
05:31And that was going to be a server side
05:33application that was also not a
05:35conventional API server.
05:36Instead, it was managing a lot of async
05:38events, a lot of open WebSockets, more
05:42than one of like several open WebSocket
05:44WebSocket connections that
05:46it needed to then coordinate.
05:48So I was looking at a proof of
05:51concept that somebody else at our company
05:53had built that was really, you
05:55know, what we call async hell.
05:57That is just lots of callbacks, lots of
06:01event listeners passing
06:03around references to random things.
06:05And we're starting to try to build into
06:07it observability and error handling and,
06:11you know, more advanced interruption type
06:13features that involve coordinating the
06:15state of multiple WebSockets.
06:17And it just wasn't going to work.
06:18It was not going to scale.
06:20The implementation was just going to get
06:22increasingly tangled.
06:24And so that's when I started looking
06:26around for, well, what is
06:27the state of the art here?
06:28I had used RxJS in the past.
06:30And I think that, you know, for people
06:32who've used RxJS, when they see Effect,
06:34that's what their first thought is.
06:36Oh, this is RxJS.
06:37I've seen this before.
06:38You know, this is this is familiar to me.
06:39It's just another RxJS kind of knockoff.
06:42I didn't really find it to be the case,
06:44though, for several reasons.
06:46But I think that what really caught my
06:48eye with Effect was the
06:51really active community.
06:53Because when I'm looking to adopt a new
06:55framework that is going to form the
06:58backbone of how we build an application,
07:00I don't want to have to
07:01be inventing stuff out of whole cloth
07:03I don't want to have to
07:04resurrect things from a couple of
07:06examples that don't
07:07really apply to my application.
07:10I don't want to be in
07:12the dark and be on my own.
07:13And I especially don't want to ask my
07:15teammates to accept a framework where
07:18they don't have the proper
07:20documentation or the guidance.
07:22They don't know where to find help.
07:23And so really, I found that the community
07:25seems to be Effect's biggest asset.
07:28And just how helpful everybody is and how
07:30enthusiastic everybody is
07:32really ease the adoption process.
07:35That is awesome to hear.
07:36And that definitely also reflects my own
07:39perspective and my own reality.
07:41This is what drew me
07:42to Effect very early on.
07:44The community was much smaller at this
07:46point and I tried to play a positive role
07:49in like growing and
07:50forming that community.
07:52But it's also something I'm like super
07:54excited about and super
07:56proud of that we like together.
07:58Yeah, broad Effect to
08:00the point where it is today.
08:02And it attracts more
08:03like brilliant minds.
08:04And it's also interesting that Effect
08:08attracts a lot of experience engineers
08:10like yourself, but also new engineers
08:13that are drawn to
08:15building something great.
08:17And that community is such a compelling
08:19aspect of that as well.
08:21So maybe to linger on the RxJS part for a
08:25little bit since yes, quite a few people
08:28are kind of comparing Effect with RxJS.
08:33For folks who are familiar with RxJS but
08:35not yet with Effect, where would you say
08:38there are parallels and the commonalities
08:41and where would you say
08:43things are much more different?
08:45Well, in RxJS, the idea is
08:47it's very stream oriented.
08:50So you essentially are creating
08:52everything is an observable.
08:56It can emit events.
08:58And this is really useful actually.
09:00I think a lot of programming situations
09:02can be modeled in this way.
09:04You have especially the asynchronous
09:06stuff like really anything side-effectey
09:08is happening asynchronously.
09:11You can't necessarily expect it or
09:13predict it, but then you have to react to
09:15it and do something in response to it.
09:18You have user input,
09:21user loads the web page.
09:22You didn't know when or why
09:23they were going to do that.
09:24They click on a link or they decide to
09:27watch a video or fill out
09:28a form or what have you.
09:31This is ultimately the timing and
09:33coordination of these things
09:34and these inputs are driven by humans.
09:36And likewise, when you have systems where
09:38you have connections coming from third
09:40party services or you have other kinds of
09:43asynchronous behaviors where you have to
09:45connect users to each other.
09:49A lot of the hardest parts of programming
09:51happen when we have asynchronous behavior
09:55that we have to model.
09:56And this is why I think you look at
09:59paradigms like HTTP or more specifically
10:03like a model view controller that you see
10:05in Rails that is still so successful.
10:07And one of the things that that has
10:09largely done is allow us to architect
10:11applications around kind of one shot
10:13actions where you get essentially a
10:17function call via an HTTP request.
10:20And your application now has to implement
10:24the function call and respond with it.
10:27But it's everything that happens between
10:29the request and the response is largely
10:32typically serialized.
10:34It's sequential. It's
10:35not in parallel, right?
10:36Which is great. It makes things a lot
10:38more easy to reason about.
10:40You know, things happen in a sequence A,
10:42B, C, D, E and then you're done.
10:44But most of what we have to deal with,
10:47especially in building web interfaces or
10:50building applications like
10:51Cortex, is not like that.
10:53Because you have a lot of things that can
10:54happen and you might be in the middle of
10:56one thing and then something else happens
10:57that you didn't expect.
10:58And now you have to deal with that. And a
11:01lot of this also involves kind of keeping
11:03state updates going.
11:05This is why honestly, this is why React
11:07superseded jQuery is because they're
11:09really good at this.
11:10But getting back to your question about
11:12RxJS, RxJS is kind of an even more next
11:16level approach at this where you can
11:17model everything as an observable.
11:19You don't know when it's going to happen,
11:21but you know that you might receive one
11:23or more different kind of events.
11:25And then how do you combine events from
11:28one observable with events from another
11:30observable when they need to interact to
11:32produce some sort of output.
11:34And RxJS is really built
11:35around this sort of model.
11:38The thing is though, RxJS is also trying
11:41to be not just RxJS, but it's a reactive
11:45framework that is cross-platform.
11:48So you have the same APIs more or less
11:51that are designed for JavaScript or
11:56TypeScript that are then also ported over
11:58to Java or ported over to Swift or ported
12:01over to Kotlin or ported over to
12:03whatever other language or framework.
12:06And so I think there was an intentional
12:08desire on the part of the designers there
12:11to keep things language agnostic in their
12:14designs, which in my opinion is kind of a
12:17flaw because it means that it's
12:20yes, you can transfer that knowledge
12:21from one language to
12:23another pretty easily.
12:24If you know reactive programming, you can
12:26do it in any language.
12:27But it means that you're passing on a lot
12:30of the strengths of each language that
12:32you're in in order to make things as same
12:35as possible between every
12:38language that you're working in.
12:40Effect in my mind is taking kind of a
12:42very different approach, even though it's
12:44based on the ZIO, which
12:45is the Scala equivalent.
12:48It was the inspiration for Effect.
12:50My understanding is that now, Effect's
12:52design is pretty detached from needing to
12:55keep parity with ZIO.
12:57Effect is basically about TypeScript and
13:00getting really good at TypeScript, making
13:02TypeScript the best it can be.
13:05And there is no, as far as I know, I
13:08can't speak for the core team, but as far
13:11as I know, there's no attempt at saying,
13:13"Well, what if we make Effect
13:15for Java or Swift or Kotlin?"
13:17It's not about that. It's just about what
13:21is TypeScript good at and how can we be
13:23really good at that.
13:24So that's a really big point, I think,
13:26because with RxJS in TypeScript, it never
13:30felt like they really got how to make a
13:33TypeSafe reactive stream
13:35application really work.
13:37the types
13:37kind of sort of worked.
13:40As far as I recall, there wasn't really a
13:42great story around requirements
13:44management or even really error handling
13:47the way there is with Effect.
13:48And so it lacked a lot of the
13:50potential, it failed to take advantage
13:53of a lot of the potential that TypeScript
13:55has to offer in the way that Effect does.
13:58Does that answer your question?
14:00No, it does. And it
14:01reflects also my perspective.
14:03In fact, I've been using ReactiveCocoa
14:06back in the days, and I believe that was
14:09early on in Swift, and it was quite
14:12pleasant to use, but it was very
14:14deliberate for the purposes of streams.
14:19And there was sort
14:20of like a peak moment
14:22where it got really into it.
14:23But then I felt like a lot of impedance
14:26mismatch where everything wanted to be
14:29modeled as a stream, but
14:31not everything is a stream.
14:33Some things are just like a one-off
14:35thing, and you just do an HTTP request
14:39and you get it back.
14:41Ensure that might fail, and eventually
14:43you're just interested in one value, but
14:45you kind of model your tree
14:47tries, etc. as a stream, etc.
14:50So this was my first exposure
14:52really to this mindset, but it felt like
14:56the gravity was too hard on
14:59this very specific primitive.
15:02And this is where I would rather like
15:05create more distance between Effect and
15:07RxJS, because Effect is not trying to
15:11tell you, "Hey, everything is a stream."
15:13No. Effect gives you a stream abstraction
15:16that lets you use the stream abstraction
15:19when you have a stream.
15:20For example, when you use a WebSocket or
15:24when you use something else, maybe you
15:26want to read a file and use streams
15:29through the bytes, etc.
15:30This is where streams are a great use
15:32case, but if you just want to do a
15:35one-off thing that happens to be
15:37asynchronous, then you don't need to be
15:39forced that this is the
15:40stream mindset, but this is where Effect
15:43gives you different primitives.
15:44And I think this is kind of the huge leap
15:48beyond what RxJS gives you, and I think
15:51this was always like the big caveat for
15:53RxJS that not everyone buys into the
15:57"everything is a stream" mindset.
16:00And this is where Effect is much more
16:02applicable to any sort of circumstance
16:05where programming
16:06languages is applicable to.
16:08Very interesting to hear your perspective
16:11on this. You've been
16:12mentioning Cortex now a few times.
16:15So before we dive into how it's built
16:17specifically and how it leverages Effect,
16:20can you motivate what Cortex is and what
16:24role it plays within MasterClass?
16:27Right, yes. So Masterclass has been
16:30developing over the last year plus our
16:34own in-house voice AI chat that has got
16:38kind of a MasterClass twist to it,
16:39and that you're not talking with
16:41fictional, invented characters or
16:44anonymous service bots, but we got
16:48authorization from many of our
16:50instructors who are working closely with
16:52us to essentially clone
16:54them into an AI persona.
16:57This includes an LLM that is trained on
17:01their ideas, their writings, their public
17:05speaking, and it also
17:06includes a voice clone of them.
17:08And in some cases also, we have some
17:11chefs like Gordon Ramsay is being
17:14released soon, and we have a lot of his
17:16recipes that are included, and he can
17:18walk you through
17:18making one of these recipes.
17:20So it's a really interesting, interactive
17:23way that I think enables MasterClass
17:26instructors to make themselves really
17:29available to a much broader audience than
17:32they would be able to on
17:34their own as individuals.
17:36And so this has also presented an
17:38enormous technical challenge because I
17:40think when you are talking to an
17:43anonymous AI, well, you might
17:45be a little bit more forgiving.
17:47You know that it's a bot that you're
17:48speaking to on some level.
17:49But with these real people who are very
17:54invested in the integrity of their image
17:56and their public persona, the bar is much
18:00higher, I think, in terms
18:02of really representing that.
18:03And this is also like if you sign up and
18:05you try to interact with Gordon Ramsay,
18:06well, if you're a fan, how many times
18:08have you seen him on TV?
18:10You can watch the MasterClass courses,
18:12you know, it's also
18:12like more than an hour of content.
18:15You know what he sounds like, you
18:16know how he speaks, you know what he's
18:19passionate about, what he doesn't like.
18:21You know, this personality is something
18:24that you've probably learned already. And
18:25now we have to put that into an AI agent
18:28or an AI bot that you can talk to.
18:31And so I feel like the very similitude has
18:34to really be there. And so that's been
18:36really challenging just to kind of get
18:38through the uncanny valley stage of that.
18:41And so this is the background of what
18:43we're building. If you want to try it
18:45out, you can go to
18:46oncall.masterclass.com, all one word
18:51oncall, O-N-C-A-L-L, dot masterclass.com.
18:54And you can sign up and try it out.
18:57So this is the general product that we're
18:59building. And we have a whole LLM team,
19:01you know, a machine learning team that is
19:03working on getting the voice models and
19:05the LLM models really trained up to
19:08represent each instructor.
19:10And what I got brought on to do was
19:12essentially to create an orchestration
19:14layer because we have several different
19:17components in the pipeline in order to
19:21build this experience.
19:22First, we have a speech-to-text component
19:26where the end user's voice of microphone
19:30stream is sent to a speech-to-text
19:33service that then listens to the user's
19:36speech and derives transcripts
19:38from it of what they've said.
19:41And then at certain intervals when we
19:43believe that the user's done speaking and
19:45is ready to hear what the AI has to say,
19:47which is not a trivial problem, by the
19:49way, we then send that transcript to the
19:52LLM to generate a response.
19:55And then the LLM generates text response,
19:57and then we send that off to a text-to-speech
19:59service to generate the audio for
20:02that response in the instructor's voice
20:05and then send that back to the user.
20:06So we have several different
20:08asynchronous services, the
20:10speech-to-text and the text-to-speech
20:12components of which are WebSockets.
20:15And then the LLM is a HTTP streaming
20:19connection essentially wrapped in an
20:22async iterable in the JavaScript world.
20:25So this is coordinating
20:28between these different elements.
20:30But as you can see, there's kind of a
20:31step one, step two,
20:32step three to this process.
20:34What makes it kind of interesting is, as
20:36I mentioned, we don't know if
20:37the user is done talking yet.
20:39The user can kind of start
20:40talking again at any time.
20:42What we want to do is shut down anything
20:44the bot is doing at that point because
20:46the user is supreme.
20:48The user drives the conversation.
20:50We don't want the bot talking over the
20:52user when the user is
20:54trying to say something.
20:55And so it's really crucial that we are
20:58very responsive to that and stop whatever
21:01is in progress downstream, whether it's
21:04at the LLM stage or
21:05the text-to-speech stage.
21:06And so this is
21:07essentially what Cortex is doing.
21:08Cortex is a Node.js application that is
21:12connected to bi-WebSockets from our two
21:15clients that we have, Web and iOS
21:18clients, that connect
21:19to Cortex over WebSocket.
21:20And Cortex, in turn, provides WebSocket
21:24interface that abstracts these multiple
21:27services and receives the microphone
21:31audio stream from the user
21:32and emits back bot audio events.
21:35So from the client's perspective, the
21:38contract is simply send the user's
21:42microphone audio to Cortex and receive
21:44back the bot audio and then the clients
21:48have to actually play
21:49it through the speakers.
21:50And that's kind of the, you know, the
21:52whole the whole contract.
21:53There's a couple of other details in
21:55terms of, you know, if a tool call
21:58happens or, you know, if state change
22:01happens or something like that.
22:03But that's the main gist of what that
22:06WebSocket contract to
22:08Cortex looks like under the hood.
22:10However, Cortex has to orchestrate the
22:13various services that I talked about,
22:16execute tool calls and and do a whole
22:18bunch of other, you know, state
22:20management under the hood.
22:22And so that's what we built with
22:22Effect. It's 100 percent.
22:24I mean, 100 percent. But like it's it's
22:27pretty much the every piece of it is
22:29built in Effect or
22:30using Effect in some way.
22:32That's awesome. So you've mentioned that
22:34initially when you got into this project,
22:37you got a first like proof of concept and
22:39where it really like exposed you to all
22:41of the complexity that's for solving this
22:45actual product to
22:46implement this actual product.
22:48This complexity need to be tamed. Did you
22:51work off that initial proof of concept or
22:54did you rather roll
22:55something from scratch?
22:57And how did you go about that?
22:59That's an interesting question because
23:00I'm always trying to figure out how to.
23:04Like re-architect a
23:06piece of software, you know.
23:08It's something you don't want
23:09to have to do very often.
23:11Also, I don't know, actually, if it's
23:13me or just the stage of career I'm
23:16at and the kind of problems that I get
23:18brought in to help with.
23:19But I often find myself looking at a
23:21piece of software that is
23:22in need of re-architecting.
23:24I think it's largely because I was
23:26specifically asked to
23:28re-architect it in this situation.
23:30But the proof of concept was interesting
23:33because, you know, it was built by our
23:37project lead on call, who
23:39is now our VP of architecture, who's
23:42just a brilliant person who has no
23:44experience really with
23:45JavaScript or TypeScript.
23:47And asked chat GPT to help. I think it
23:50was chat GPT to help with building it and
23:53essentially just piece that all together
23:55from chat GPT prompts.
23:58There were a couple of things that came
24:00out of that that were, for example, a
24:02library that is like a private library
24:05that isn't really maintained.
24:06There wasn't really a
24:09top level architecture.
24:11you had individual pieces that
24:12were kind of worked in isolation, but
24:13then the glue between them became really
24:16convoluted and hard to follow.
24:19So so that that kind of became a problem.
24:21But it worked. You know, ultimately, this
24:23proof of concept kind
24:24of worked for a demo.
24:27It didn't handle interruptions very well.
24:30It didn't have great latency.
24:32I think in reality of like most
24:35production grade systems are actually let
24:39me take that back, probably not
24:40production grade, but applications,
24:43JavaScript applications that are running
24:44in production are probably
24:47That sort of state where is working well
24:51enough on the happy path, but all of like
24:54those higher level requirements that
24:57really make a end user experience superb
25:00that you can interrupt an AI model when
25:05when it's talking to you or that's more
25:07resilient to errors, etc.
25:09This is where you need to go beyond that
25:11convoluted everything
25:13kind of pieced together.
25:15And I think this is like a perfect point
25:17to to start with, like
25:20applying Effect to to go beyond that.
25:23That's sort of like a messy state.
25:27Yeah, this is why I wanted to
25:31tame the asynchronous
25:32behaviors in in Cortex.
25:35And I was looking for a package that
25:37would help me do that. And
25:38this is when I found Effect.
25:40And then at that point like I
25:42said, I had seen RxJS. and I had
25:45good experiences at
25:46first building RxJS
25:47But like kind of one of the common
25:49complaints is that, well, now, like you
25:52said, everything has to be a stream.
25:53Now you have to model
25:54everything in this way.
25:56And it really completely changes how you
25:58architect your software and you need a
26:01lot of buy in from your team if you're
26:02going to start doing this, because now
26:04everybody has to learn this.
26:05They can't just sit down and, you know,
26:07start coding on whatever part, you know,
26:09feature they're trying to do.
26:10They have to first learn this framework
26:11and that can be a big ask for folks.
26:15And so that's something that I wanted to
26:16be conscientious about.
26:18However, when I started
26:19looking into Effect, I started seeing,
26:22well, you know, this has a lot of promise
26:24and potential if you
26:25use it in the right way.
26:27And what I mean by that is like, you
26:29know, you don't have to necessarily go
26:32from zero percent to
26:33100 percent overnight.
26:34You don't have to completely redesign the
26:37entire application to take advantage of
26:39some of the benefits
26:41that Effect has to offer.
26:42That's kind of how I started, was
26:44incrementally adopting
26:45Effect in parts of the code base.
26:48And this was really what
26:49enabled it to work at all.
26:51If I had to say, hold on, I need to take
26:54this for a month and just completely
26:56build it from scratch and nothing's going
26:58to work and I can't deliver any new
26:59features until I'm done.
27:03That was not going to fly.
27:04I was in a position
27:05where I did not have to ask for
27:07permission to go live in a cave for a
27:10month or however long
27:11it was going to take.
27:12I always like avoid
27:13being in that situation.
27:15I don't think that's a good place for us
27:16to be in as software engineers.
27:18We need to be shipping like every day,
27:20shipping features, shipping bug fixes,
27:22shipping products constantly.
27:25And like we never have the luxury to just
27:28say, well, hold on, I'm just going to
27:30like spend a month not shipping, you
27:32know, to do this technical thing that I
27:34can't really explain
27:35why I have to do this.
27:36Right. Nobody, nobody likes that.
27:38So fortunately, Effect is not
27:41a framework that is like that.
27:43That's not what was
27:44asked of us to adopt Effect.
27:47Instead, it was you can start plugging in
27:50Effect judiciously here and there.
27:52And I started at the top level of the
27:54application around really around the
27:56WebSocket connection layer, around the
27:58HTTP layer and the application runtime.
28:02And that allowed us to kind of have a top
28:05level of error handling that immediately
28:09provided benefits for recovery and so on.
28:13But we didn't stop there.
28:15That was just the first step.
28:17We started kind of rebuilding different
28:20parts of the application.
28:21Some of them were like isolated behind a
28:24run promise or, you know, like a stream
28:26that was, yeah, emitting
28:28events into an event emitter.
28:30But it allowed us to kind of rebuild
28:32isolated parts of the application on
28:35their own where like selectively where we
28:37thought they had the most value to be
28:39converted into Effect.
28:40And often driven by product requirements.
28:44So when I wanted to implement
28:45interruptions, that was the reason to now
28:49rebuild the speech-to-text component in
28:52Effect, because it was going to be a lot
28:54more work to do that.
28:56If I was going to have to do it
28:58the old way, it was like, really, it
29:00wasn't like rebuild the whole thing
29:02in Effect, the entire application.
29:05Or just keep working the old way. It was
29:07really for each isolated piece.
29:09It was like, do I want to like add more
29:11spaghetti to this application in this
29:13component or can I just rebuild this
29:15piece in Effect and make it much more
29:18expressive and elegant while building the
29:20feature that I'm being asked to build.
29:22So it was much less of a refactoring
29:26process and more of a incrementally
29:29adopting it while also building and
29:31delivering features.
29:32I love that. And I think you've hit on a
29:34couple of really nice points here.
29:36One is that Effect, even though you get
29:40a lot of benefits once you have
29:43Effectified more and more of your
29:45application, you're going to find that
29:46you can delete a lot of code.
29:48Everything just fits
29:49nicely together in the same way.
29:51If you incrementally adopt React, let's
29:54say we're just in this transition of like
29:56transitioning a larger code base from
29:58jQuery, something else to React.
30:00You don't need to do that in one like
30:03long night, but you
30:04can do that step by step.
30:06And once you've reached a point where
30:08everything is in React, then you can
30:10delete a whole bunch of like glue code, a
30:13whole bunch of like things that bridge
30:15from the old way to the new way.
30:17But it is also totally fine that you go
30:20through this like transitionary phase.
30:22But the other thing that you've mentioned
30:24that I think is like a super great
30:26insight, which is like, how do you
30:28prioritize what to refactor with Effect
30:31when or when to rethink
30:33something and apply Effect?
30:35And this is so elegant that you can like
30:38go top down from like, hey, what is the
30:41thing that we want to improve for users?
30:43What is like the business outcome that we
30:46want to affect here?
30:48And then like looking at those, Effect
30:51typically has
30:52something in store for that.
30:54If you want to improve performance, if
30:56you want to improve reliability,
30:58resilience, error handling, whatever it
31:01might be, Effect typically
31:02has something in store for that.
31:05And then you can prioritize what to work
31:07on depending on your end user needs.
31:10And hearing here about the speech to text
31:13aspect and applying Effect for that,
31:17that sounds super interesting.
31:19So I want to hear a little bit more like
31:20zooming into from macro into a bit more
31:23micro into this component.
31:25Can you give a rough overview of like how
31:28that thing was built before?
31:31What you diagnosed the biggest things to
31:35improve where and which primitives of
31:38Effect did you use to
31:40ultimately implement it?
31:42Well, actually what I prefer to do is
31:44zoom in on the text-to-speech component.
31:47Oh, I'm sorry.
31:49I remember that wrong.
31:50But yes, no, it's okay.
31:51You remembered it right.
31:53It was you.
31:53I said speech-to-text earlier, but I feel
31:55like the text-to-speech component.
31:58This is the component that I also talked
31:59about in my Effect Days
32:01talk and did a deep dive on.
32:03I feel like the text-to-speech component
32:05was kind of like a real unlock, really an
32:09aha moment of like, wow, this is this is
32:12what Effect can do in
32:14terms of like saving
32:15a lot of complexity from your code base.
32:18As I mentioned, this is a
32:20component where like it's a WebSocket and
32:22we stream LLM thoughts to it.
32:26So an LLM response, as you might be
32:28familiar, it sends what are called chunks
32:32in a chat completion response.
32:34And the chunks are usually a token or two
32:36tokens and they're not
32:37complete words a lot of the time.
32:39So then we're like accumulating the
32:42chunks into basically what we call
32:44coherent thoughts and the coherent
32:47thoughts can then be sent to the text-to-
32:50speech component to
32:51generate the bot audio.
32:53However, if there's an interruption, we
32:55need to shut down the LLM and we also
32:57need to shut down the
32:58text-to-speech component so that we don't
33:01continue to generate more thoughts based
33:03on the previous thing that the user said
33:05before they continued talking.
33:07And now we want to start over and respond
33:09to the most recent thing
33:11that that user has said.
33:12So the text-to-speech component now it's
33:16a WebSocket connection.
33:17When you send it a
33:19coherent thought, that connection
33:22will then respond asynchronously with one
33:26or more different
33:28events that you might get.
33:30And we're basically just
33:31streaming those up to the client.
33:32But when there's
33:34an interruption, we need to actually shut
33:36down the WebSocket
33:36connection, close the connection.
33:38Abruptly, so we don't get any more
33:40messages from it and then reopen it.
33:43And then in that period of time, and it's
33:45not usually very long, it can just be
33:47like a hundred milliseconds or two
33:48hundred milliseconds where we're waiting
33:50for that WebSocket connection to open.
33:52We hope that
33:52we've created a connection, but we've not
33:54yet received the open
33:55event from the connection.
33:56And it's in that time that we were often
33:59getting the LLM was trying to send
34:02messages to it, but it was erroring
34:04because we were sending WebSocket
34:07messages out to a
34:08WebSocket that was not yet open.
34:10So we had to now queue those messages to
34:13wait for the open event from the
34:16WebSocket connections and then flush the
34:18queue when it was open.
34:20So as you can imagine, this created some
34:22code complexity in the pre-Effect
34:26implementation and it was something that
34:29Effect turned out to be actually very
34:31good at because these are the kinds of
34:32things that Effect has out of the box.
34:35In Effect we were able to replace the
34:39WebSocket handling code with the
34:41WebSocket utility from
34:43Effect from effect/platform APIs
34:47And that has kind of a magical property
34:51to it that you don't really ever have to
34:53think about when the WebSocket is closing
34:55and opening and you don't
34:57have to wait for an open event.
34:59What it essentially gives you is what is
35:02called in Effect a channel.
35:03And this became a primitive
35:05that I became curious about.
35:06It's something that I wish was a little
35:08bit more first class in the effect world.
35:11It's certainly used behind the scenes in
35:14a lot of things like stream and other
35:17APIs in the Effect world.
35:19But this is what you get essentially when
35:21you create a WebSocket connection using
35:23the effect/platform API.
35:24But then if you use the Effect stream
35:26operator pipe through channel, you now
35:28have a duplex stream, which is one where
35:32you can start listening to other streams.
35:35And then instead of doing like a run for
35:37each or run collector, whatever you're
35:40doing and you typically do it with a
35:41stream, you now are piping the events or
35:45the items in that stream out through the
35:48channel through the WebSocket connection.
35:50And then downstream from that pipe
35:52through channel, you are getting incoming
35:54events that are coming from that
35:56WebSocket connection that you can then
35:58emit further on down your application.
36:01So this is great. But this is also what
36:04it is also doing is this is abstracting
36:06the WebSocket lifecycle
36:08into the stream lifecycle.
36:09So if you emit an error
36:12upstream, it will close
36:14the WebSocket for you.
36:15And then if you have a stream.retry, it
36:18will reopen the WebSocket for you
36:19in the event of an error.
36:21And because the streams are pull based,
36:23you don't have to rely on queuing
36:26explicitly. You aren't going
36:28to miss any of your events.
36:30When the stream is reopened, it will
36:32start pulling those events
36:33and behaving as expected.
36:35So this really allowed us to
36:38abstract all of the sort of tangled like
36:39we had, I think, a promise that was a
36:42reference to a promise that was kept
36:44around and was
36:44awaited in different places.
36:45And it was it was a mess before. And now
36:49we had a single stream that was just a
36:52linear stream that where you had the
36:55stuff going out, going in the top of the
36:57stuff coming from the stream, coming out the bottom.
36:59And it became very easy to reason about
37:02what was happening. And you didn't really
37:03even have to think about the stream
37:05lifecycle at all. It was just you made an
37:07error when you want it to
37:09close and then just retry.
37:10The WebSocket
37:11connection is now handled by the stream
37:14lifecycle. So you can use
37:15the stream retry stream.
37:18will be shut down when the
37:21scope of the stream ends and the
37:24WebSocket will automatically closed.
37:26We also have a flush event that we send
37:30out to the text-to-speech
37:32service, which essentially says we're
37:34done sending you a new speech for now.
37:38So send us everything you got. And the
37:41peculiarity of this particular service is
37:44that they will then accept your flush
37:46event and they will
37:47promptly close the stream on you.
37:50That's not really what we wanted, but I
37:52don't know. They designed it this way.
37:53And I don't really have, you know,
37:55leverage to get them to redesign their
37:57entire API. I just have to
37:58work with what we have. Right.
37:59This is a lot of application development.
38:01You don't have the liberty to redesign
38:04every API that you're working with. You
38:07have to abstract it in some way. And so
38:09this is what we're having to do here. But
38:10the Effect stream
38:11primitives make it really easy.
38:14That sounds brilliant so far. And I think
38:16what is so nice about this is
38:18that it is A very clear and very
38:22intuitive from a user perspective what
38:25the system should do.
38:27And as users, we're like all familiar
38:30with when this goes wrong and how
38:33frustrating it is. Like if I'm talking to
38:36the AI, the AI goes off like in a wrong
38:39direction that I don't want.
38:41And I want to interrupt it.
38:42And it doesn't act on this interruption.
38:45You need to listen to it for another 20
38:48seconds until I finally need to repeat
38:50what I've just said.
38:52And all of those things, they need to be
38:54handled. And all of that is a lot of
38:56complexity. And if you leave that
38:59complexity unchecked, like you said, it
39:02was a mess. And I think that nicely
39:04reflects the majority of JavaScript
39:07applications that are out there or
39:09TypeScript applications.
39:10And I think it's really like this
39:13fine line to walk where you capture
39:17all of like the existential complexity
39:20that your use case requires and shaving
39:23off like all the accidental complexity
39:26and isolating this nicely.
39:28And even to a degree where you say like,
39:30okay, those services that you need to
39:32call, they're not in the ideal shape that
39:35you would like it to be. But then you can
39:37just like wrap it and create your own
39:40world that you're happy in and where you
39:43can model your application in a nice way.
39:45And yeah, I'm very happy to hear that
39:48effect streams and the various primitives
39:51that you've been using. You've been using
39:53the WebSocket abstraction that Effect
39:56gives you, and I
39:56suppose also queues, etc.
39:59That all of this has been so nicely
40:01fitting together to model your use case.
40:05So going a little bit beyond streams,
40:08which other aspects of Effect have you
40:11been using or which other kind of
40:13superpowers have you been getting out of
40:15Effect that have played a
40:17meaningful role in the application?
40:19Well, the error handling has been huge.
40:22Honestly, we modeled all of our possible
40:25errors. We have, I think, maybe up to 30
40:29or so errors that the system can emit
40:31that are domain specific tagged errors.
40:35And those are decorated with a specific
40:40error code and an error source, because
40:43one of the things that will often happen
40:46or that was happening originally, I
40:49think, was, oh, we got an error. The
40:51connection went down.
40:52Well, we got an error and something weird
40:55happened, and I don't know why, and now
40:57I'm in a weird state. Oh, we got an
40:59error, and the whole service just crashed
41:03or something like this, right?
41:04And even if you can just wrap everything
41:08in a try catch, best case scenario, you
41:11have some unrecoverable error, your
41:13connection goes down, and you don't know
41:16why. You just know, oops,
41:17bye, and then all of a sudden.
41:19And so it's frustrating for the user, and
41:23it's also frustrating for the rest of the
41:24team when they're trying to diagnose
41:26what's going on, which we spent a lot of
41:28time doing in our
41:29development workflow internally.
41:31And, you know, I'm a big fan of passing
41:35the buck, so I don't like things to be my
41:38fault. And I think
41:39we're all in that domain.
41:41I say this to joke, actually, I'm fine
41:44with taking responsibility if it's my
41:46fault, but I would rather things not go
41:48wrong because of the
41:49decisions that I made.
41:50A lot of times early on, it was like,
41:53oh, Cortex is down. Oh, Cortex emitted an
41:55error that I don't understand.
41:57And, you know, fair enough from a
41:58client's perspective or from a test
42:00engineer's perspective,
42:01that's what it seems like.
42:03But that doesn't really give you enough
42:05information to troubleshoot because most
42:08of the time it's not, you know, Cortex is
42:11just a hub. Cortex is just passing events
42:14from one service to another.
42:15It's not Cortex really as the source of
42:18the errors. Instead, what we see a lot of
42:22the time is, oh, one of our
42:23backend services went down.
42:25Oh, a backend service emitted something
42:26that we didn't expect. Oh, it's being
42:29slow or something like this.
42:32And now we're able to like create
42:34targeted pinpoint errors whenever a
42:37specific thing goes wrong
42:38somewhere in the system.
42:39And then those propagate up to our top
42:41level error handling. And so if we have
42:43something that's unrecoverable that
42:44happens, we can now close
42:47the connection like before.
42:48But now we can send up detailed
42:51information that allows our people to
42:54diagnose what's the
42:55problem. So it's not Cortex.
42:57It's like, oh, our speech-to-text service
43:00crashed or is out of memory or something.
43:03And so now we aren't able to create calls
43:07until we fix that
43:08piece of infrastructure.
43:09So that gives us a lot more information
43:11that kind of actually saves a lot of time
43:14debugging the system. It points you
43:16directly to where the source of the
43:18problem is instead of making
43:21you go on a debugging hunt.
43:23So that has been huge for us. The error
43:26handling has been extremely valuable.
43:28There are a lot of other errors that are
43:30recoverable, but we want
43:31it to log and report them.
43:33So the whole error handling story in
43:36Effect is fantastic, just surfacing when
43:40things can go wrong and
43:42forcing you to deal with it.
43:43It has also meant, interestingly, I feel
43:46like that, you know, like within Cortex,
43:50not every function is an effect.
43:51You know, not every single line of code
43:54is a yield star. There's a fair amount of
43:57just plain old, data manipulation
44:00that is happening throughout the code.
44:03It's data manipulation using
44:05functions that are
44:06synchronous and aren't going to throw.
44:11Right. You could have very high
44:12confidence that, you know, if you're
44:14trying to get an index of a thing from a
44:15string, you know, or
44:17whatever, you're not going to throw.
44:19you can do like a
44:20lot of just kind of conventional
44:22programming in areas that
44:23are kind of safe and sandboxed.
44:25And it doesn't mean that every single
44:27calculation needs to be done in an
44:30effect. It just gives you a safe place to
44:33do that kind of work without having to
44:36worry, you know, oh, if this throws or
44:38oh, if this does something asynchronous
44:39and I, you know, don't handle it, you
44:42know, don't await it or whatever.
44:44You know, usually those those kinds of
44:46cases are they get a lot of attention
44:48because we have to
44:50think about them so much.
44:52But that's not usually the most of the
44:55work that we're doing. Ideally, right?
44:57We're thinking about like, purely
44:59functional transformations of data from
45:02one state into another.
45:03taking the input from
45:05some kind of asynchronous effect and
45:07sending it out to
45:08some asynchronous effect.
45:09But like the actual program, the business
45:11logic is usually something that is like
45:13pretty, you know, step by step, you know,
45:16just just logic. Is usually when
45:18we're not interfacing with an external,
45:21you know, service or
45:22some kind of side effect.
45:24Then we can just write code like normal.
45:28You know, we don't have to model
45:29everything as a stream just to add up
45:32some numbers or something.
45:33Right. And I think that the super plain
45:36way how you put it just write code like
45:38normal. I think this is kind of the in a
45:42nutshell, the difference
45:43between Effect and RxJS
45:46Where in RxJS you need to do everything
45:49as a stream. And in Effect, you can write
45:52code like normal. And another aspect of
45:55writing code like normal
45:57is trading errors as values.
46:00This is we're all super used to just
46:04passing around and
46:05manipulating data. And somehow, we're
46:09kind of brainwashed into
46:11thinking that errors need to be like we
46:13need something we need.
46:14We're almost like paralyzed about like,
46:16how should we deal with errors? But if
46:19we're just trading errors as values as
46:22well, errors as data and passing them
46:24around and Effect just makes that easy by
46:26giving us a separate channel
46:28to deal with that error data.
46:30And then, like you say, like you don't
46:33want to you'd like to pass it along, then
46:35it's just data that you pass along. So I
46:38think that's just like code like it's
46:41normal. I think that is like one of
46:43Effect's superpowers.
46:45And closely related to errors is like
46:48having visibility into when errors happen
46:51when something doesn't go as expected. So
46:55and I think if I remember correctly, the
46:57telemetry part the observability part has
47:00also been a key aspect of
47:02building Cortex and operating it.
47:04So maybe can speak a little bit more to
47:07how you do observability and telemetry
47:10usually within MasterClass, particularly
47:14within JavaScript applications and how
47:17Effect has helped you to
47:20maybe improve that even further.
47:22Right. Yeah. So I sort of have to admit
47:25that we don't have an excellent story for
47:27the most part or before Cortex didn't
47:30have an excellent story about
47:31observability at MasterClass
47:33in the JavaScript world.
47:34We have a number of services.
47:36We have we have a Raygun. We have New
47:39Relic and we have Core Logics. We get our
47:42logs to we send our logs to and we have.
47:44So we have a bunch of observability
47:46services for things like video. We have a
47:48dedicated video monitoring service that
47:51we integrate and
47:53a few high value business.
47:55Applications like that. We want to keep
47:57an eye on, you know, error rates for
48:00people visiting our home page or things
48:01that are really
48:02indicate business traffic, business
48:05being impacted by
48:07some technical problem.
48:08However, usually that that amounts
48:11to like something
48:12that's easily reproducible.
48:13And easily fixable usually and there's a
48:16either some infrastructure that needs to
48:19be restarted or code change that needs to
48:21be rolled back or something like that.
48:23Cortex really represents a new level of
48:26complexity when it comes to understanding
48:28what's going on internally. And I think
48:29that a big reason for that is that it is
48:32not a one shot HTTP server type of
48:35application, but is instead
48:38You know, a stream of streams and is
48:41handling all of these asynchronous events
48:43that are passing through it. It's not
48:45directly doing much of any work. It's
48:47just literally in between all these
48:49services handing events
48:51from one service to another.
48:52So, as I mentioned before, when
48:54things go wrong, they're mostly not going
48:55wrong in Cortex. And likewise with
48:57observability
48:59where the system is spending time
49:01doing work, it's mostly not spending time
49:04inside of Cortex doing that work.
49:07Cortex is instead, waiting for
49:10events from other services. And so what
49:12we're interested in measuring is not
49:14really the conventional, I think,
49:16telemetry story when it comes
49:17to building tracing and spans.
49:20I think this is a story that is also
49:22baked into the default Effect telemetry
49:26story, right? When you have a Effect dot
49:29with span with a name, you know, and you
49:32and you and you wrap
49:33that around an Effect.
49:35Well, that's great. That ensures that
49:37that span or that Effect, the time that
49:40it executes is going to be represented by
49:43a span in your trace. And for most like
49:46one shot type of actions that you might
49:48perform, that works great,
49:51which is most of the time.
49:52If you're doing actual work within an
49:55effect, within a one shot effect, then
49:57that is the conventional way that you do
50:00telemetry. We're not really doing that at
50:02all in our telemetry
50:04implementation in Cortex.
50:05Instead, we're interested in measuring
50:07time between events that are coming from
50:10outside services. Cortex is the only
50:12place where we can really gather this
50:13information because it's the hub.
50:15But Cortex isn't sitting around. We don't
50:19have like an effect that is, you know,
50:21sleeping until it gets a certain
50:23notification from or an event from
50:25another stream.
50:26It wouldn't make any sense
50:27to build the application that way.
50:28And so it doesn't really make a lot of
50:30sense to build our telemetry that way.
50:33I suppose what we're doing with
50:35Open telemetry is a little unconventional
50:36in that a span doesn't represent Cortex
50:40doing that work. Instead, it represents
50:42usually really represents like the LLM is
50:44working or the text-to-speech
50:46service is working.
50:48And we're just waiting for that. But it's
50:49measured from Cortex, not from these
50:52other services. But because
50:53it's really all streams.
50:54What we have to go on isn't an effect
50:57that we're measuring. It is literally the
51:00time between two events in those streams
51:03that we're measuring. And so we really
51:05had to roll a lot of our
51:07own telemetry handling.
51:10But, you know, we were going to have to
51:11do this anyway,
51:12ultimately, because when you have.
51:14Let's say we're not using Effect. We're
51:16using the original approach, the
51:19non-Effect approach that is event
51:21emitters everywhere, you know, WebSocket
51:23event handlers and so forth.
51:25You get a transcription from the speech
51:28to text service and you want to start the
51:31span of time that it takes that you're
51:34measuring that it takes to.
51:36Generate a response
51:37to that transcription.
51:39Well, you can have a central hub where
51:42like maybe you're
51:42collecting all of these events.
51:44But you're starting that span in one
51:48event handler and then you're entering
51:49you're ending it in a different event
51:51handler for a different service.
51:53And so you need a place where you're
51:55holding on to those references might be a
51:57centralized location that is listening to
51:59all of these events.
52:00But then it becomes kind of tangled up
52:02because you're having to.
52:04Keep these references around and
52:06keep them alive, from one
52:08event handler to a
52:10completely different event handler.
52:11And this is an area where.
52:14Yeah, we had to roll some of our own code
52:15in Effect do it this way.
52:17But I feel like Effect kind of made it
52:19easier to do it this
52:20way anyway.
52:21Allowing us to have a kind of a
52:23connection level, long lived reference to
52:27these spans and then just manipulate
52:30spans in what is essentially a stream dot
52:34tap where we are listening to all of the
52:37incoming events and then just
52:38Starting and stopping them based on
52:40which events are occurring.
52:42It's not been perfect, honestly.
52:45It has been a little error prone
52:46sometimes and we've had to go in and kind
52:49of tweak things when we
52:53have unexpected behaviors.
52:55It's an area that has provided immense
52:57value for us, however.
52:59It's given us a lot of insight into what
53:01people are experiencing
53:02if people are experiencing.
53:04Really slow behaviors, slow responses.
53:08If people are experiencing the bot is
53:10talking over me or this sort of thing.
53:12If we have errors somewhere in the
53:14system, we can see exactly where and when
53:16that happened and in what service and in
53:18what part of that
53:19services work it happened.
53:20And we're able to trace, you know, what
53:24was the sequence of of chunks
53:26that were emitted by the LLM.
53:28You know, how long did it take
53:29for us to get that first
53:30chunk out of it out of the LLM.
53:32You know, comparing the user message in
53:35the bot response and you know, if the
53:37user interrupted the bot, how much of the
53:39bot speech did the user here?
53:42And so a lot of these questions that are
53:44really of interest to the business and
53:46also for us technically.
53:48When it comes to how on call is being
53:51used and how people are experiencing it
53:54are really answered by the telemetry that
53:57we've built using
53:58Opentelemetry and Effect.
54:00But,it's a very
54:02custom system that I I don't know that it
54:05has its optimal form yet.
54:08And I also don't know that is necessarily
54:10going to apply to
54:12everybody in the same way.
54:14I don't know. This is like I said, it's
54:15it's it's a very custom system that is
54:18built for our use case that will not
54:21apply in most conventional applications.
54:24But I think that's OK.
54:25There's always special cases.
54:28This makes sense. And, where
54:30Opentelemetry or the previous
54:32technologies that is based on Open
54:35Sensors and Open Tracing, I believe those
54:40were the two predecessor technologies
54:42that merged into Opentelemetry.
54:44Where they're historically coming from is
54:46like from distributed tracing. And that
54:49is typically in a
54:50microservice kind of architecture.
54:52We have one service request response
54:54style calling into another
54:56one, calling into another one.
54:58So I think where those systems or this
55:00technology shines historically, at least,
55:02is on a request response pattern where
55:06you just get that burned on charge that
55:08that you know from like a
55:09performance profiler in a single threaded
55:12environment or a multi-threaded
55:13environment, now we get it
55:14over like a network boundary.
55:16So this is where those shine. But going
55:19beyond that for different modalities,
55:21like for long running streams or I've
55:24been also experimenting with using
55:26OpenTelemetry in a front-end setting
55:28where like a front-end session, you don't
55:31have that request response, but you have
55:33a front-end session.
55:34For example, think about how you use
55:36social media. You might be doomscrolling
55:39for a very long time. So is your entire
55:42session, is that the trace that you have
55:45with possibly like thousands of spans?
55:49Or where do you make cut basically? How
55:51do you design your trace and your spans?
55:54I think that is still something that
55:55we're figuring out as an industry.
55:57And it's been cool to hear about your
56:01usage of it. And I think this also speaks
56:03to the flexibility of Effect. Yes,
56:06they're like default patterns make it
56:09really easy and kind of trivial to
56:11instrument your app out of the box.
56:13But if you want to instrument it in a way
56:15that's a little bit more specific to how
56:18you would like to give your trace
56:21meaning, that's possible as well.
56:23Maybe taking a step back here for folks
56:26who are curious about Opentelemetry and
56:29like generally see the value in
56:31observability, but maybe haven't taken
56:33that leap yet themselves instrumenting
56:36their app, which sort of guidance would
56:38you offer to people what to focus on,
56:41maybe what to initially leave out of the
56:44picture just to get going?
56:46Oh, that's a really good question. I feel
56:47like the answers are going to be really
56:49specific to your use case. And in the
56:53case of an application like Cortex,
56:55extremely custom. And we have spent a lot
56:58of time refining and iterating on our
57:02telemetry implementation. But most
57:04applications probably
57:05don't need that, honestly.
57:07I think, especially in the
57:09JavaScript world, there's both browser
57:11and Node based auto instrumentations that
57:15are available that do a lot out of the
57:17box. So I feel like a lot of what I would
57:21want to start with are the ends of the
57:23application when your code is calling out
57:26to another service or when you receive
57:28events from the user.
57:30Because that kind of determines the shape
57:32of that user session or that interaction
57:34that you might be
57:35interested in measuring.
57:37And then it will identify kind of
57:39hotspots of like, oh, the user waited a
57:42long time for this response or whatever,
57:44what's up with that? And then you can
57:46start to drill further.
57:48And then the other thing that I think is
57:49really super valuable is distributed
57:53tracing where you are propagating a trace
57:56across service boundaries. And sometimes
57:58this is just, you know, you're
58:00instrumenting in your browser application
58:02or in your iOS application.
58:04you're making calls out to your API service and
58:07you want to see what's going on in the
58:09API service as well during that time
58:11period. You're propagating the trace from
58:14your client to your server so that when
58:17you see them all together,
58:20you can kind of piece together.
58:22Oh, the client called out to the server
58:24and then the server made these database
58:25calls and you can see that all in one
58:27distributed trace. That's really super
58:29valuable. So just focusing on the ends,
58:33either incoming events or outgoing
58:36service calls, and then making sure you
58:39have your distributed trace propagation
58:41set up correctly. Those would be the top
58:43things I would recommend.
58:45Right. I agree. And I think the benefits
58:48of having a good observability story
58:50for your application and for your system
58:52is so manifold. Like it helps you with
58:56correctness to kind of understand like,
58:59oh, like something is not going well.
59:01That you're not just like
59:03completely in the dark and looking for
59:05the needle in the haystack, but that you
59:07actually have a great foundation to
59:10figure out what went wrong.
59:12That is the, I think the foundation where
59:14people start leaning on observability
59:17beyond just console log. But then also
59:21like doesn't just help you with making
59:22things correct or diagnosing when
59:26something was not correct,
59:27but also making it faster.
59:29Like otherwise you might just know like,
59:31okay, that API request has taken two
59:33seconds, but why? And sometimes there's
59:36like really counterintuitive situations.
59:38It's very simple and very
59:40easy to realize, oh,
59:42this is why it's so slow.
59:44And also speaking of AI, this will like
59:49be a perfect foundation to give an AI
59:52even more context what is going wrong and
59:55let the AI iterate further and help you
59:57make your app more
59:59reliable and more performant.
01:00:00One of the frontiers that we're going to
01:00:03be exploring, I think now that we've
01:00:05cracked the seal on observability is
01:00:08integrating it with our existing data
01:00:10analytics or our end user analytics that
01:00:14we are already collecting.
01:00:15In MasterClass we're a really good, robust data
01:00:17team that is, you know, while respecting
01:00:21like anonymity and user privacy is still
01:00:24really invested in
01:00:25understanding the user journey.
01:00:28What are people doing? You know, why are
01:00:30they there? What is enticing
01:00:31them? What's driving them away? And these
01:00:33sorts of questions that are really
01:00:35foundational to understanding the best
01:00:39way that we can
01:00:39deliver value to our users.
01:00:41Integrating this with a sort of Opentelemetry
01:00:43will give us even more insights
01:00:46of like, oh, did a user try to
01:00:49load a page, but then they bounced
01:00:52because it took too long to load and
01:00:54things like this that will give us an
01:00:57integration level between the sort of end user metrics
01:01:01that we've been using and also the technical
01:01:04implementations behind the scenes that
01:01:05are underpinning that user experience.
01:01:09I'm really looking forward to being able
01:01:11to explore that further. And I feel like
01:01:14it has tremendous
01:01:15potential to offer value.
01:01:17That sounds awesome. So speaking of a bit
01:01:20more forward looking perspectives
01:01:22I'm curious now that you've been part of
01:01:25the Effect community, I think at this
01:01:27point way beyond a year already and super
01:01:30impressive what you've been able to build
01:01:33with Effect in that short period of time.
01:01:35What are the things that you're looking
01:01:37forward most to when it
01:01:39comes to the Effect ecosystem?
01:01:42I'm really looking forward to seeing
01:01:43Effect 4.0 come out. That looks awesome.
01:01:47A lot of the improvements they've made to
01:01:49the bundle size and to the implementation
01:01:53complexity look really promising.
01:01:56And, you know, I really admire how
01:01:59responsive the Effect team has been to
01:02:03the community, to the needs of the
01:02:05community and really listening to
01:02:07feedback, incorporating it, iterating.
01:02:10That's really, I think, been vital to any
01:02:13platform like this, getting any traction.
01:02:16It's a very ambitious platform. It just
01:02:20has to be said, the design of it to
01:02:23encapsulate so much information about
01:02:26what's going on at an atomic level in
01:02:28your application, but then extending that
01:02:30out into really
01:02:31building whole frameworks.
01:02:34The HTTP API, the Schema, the tracing
01:02:38integration, the possibility that the
01:02:42database abstractions, the networking
01:02:45abstractions like WebSockets, file
01:02:47system, node runtime,
01:02:50there's React integrations.
01:02:52Really, you name it, there's just tons
01:02:55and tons of, and there's a whole bunch of
01:02:56stuff coming in down the pipeline,
01:02:58Cluster and AI integrations.
01:03:01The paradigm is really extensible and it
01:03:05has proven itself really robust and able
01:03:08to handle all these different scenarios.
01:03:10But now making all of those things all
01:03:13work, you know, that's
01:03:15an incredible challenge.
01:03:16I hope something that they
01:03:19succeed at. It's just so much.
01:03:22But I think that with this large and
01:03:25growing community, I think there will be
01:03:27people using every aspect of that.
01:03:30Probably not everybody is going to use
01:03:33every piece of the Effect
01:03:34ecosystem. I am not, certainly.
01:03:37And most people will only use a small
01:03:39sliver of it. And that's fine. That's all
01:03:41you really need to do.
01:03:43But when you have hundreds or thousands
01:03:45of engineers all building different kinds
01:03:48of applications with it, now you start to
01:03:50get a lot more signal on
01:03:52for this specific use case.
01:03:54Here's what I'm trying to do.
01:03:56Here's how I've approached it.
01:03:57And that information, feeding that
01:03:59information into the design of the
01:04:01platform is going to be tremendously
01:04:03valuable to making it more extensible.
01:04:07But anyway, I'm really interested in the
01:04:09Effect 4.0 developments that I think have
01:04:12come out of this exact kind of feedback
01:04:15integration and iteration.
01:04:17And also, I'm really excited about things
01:04:20like the, I think there was at the Effect
01:04:23Days, there was some, there was a demo
01:04:24that I think Mattia did.
01:04:26Mattia, who did about the error reporting
01:04:30and that was integrated into the editor
01:04:32when there's like type errors in Effect.
01:04:35Sometimes those can be cryptic and it's
01:04:38nice to see that they're working on
01:04:40making those a lot more
01:04:41human readable and friendly.
01:04:45It's nice to see the dev tools getting so
01:04:47much love. It's nice to see the
01:04:50documentation getting so much
01:04:51improvement, so often very thankless.
01:04:53Honestly, more than any specific
01:04:56technical feature of the API, I just love
01:05:00to see the developer
01:05:01experience getting some attention.
01:05:03And it makes things so much easier on
01:05:05everybody to start building really cool,
01:05:08ambitious things if you get really good,
01:05:13clear, understandable errors at build
01:05:16time or in your editor.
01:05:18And if you have really good debugging
01:05:21tools that let you understand what's
01:05:23happening, if you have really good
01:05:24documentation that has use cases and
01:05:26examples of all kinds of different types
01:05:30of patterns that you might
01:05:31want to take advantage of.
01:05:33This is the sort of, I think often unsexy
01:05:36work of building a framework like this
01:05:38that is so fundamental to people, like
01:05:41end users being able
01:05:42to get value out of it.
01:05:43And it seems like the Effect team is
01:05:45taking this seriously. And that to me,
01:05:48more than almost anything, is what gives
01:05:50me excitement and hope about
01:05:53the future of this framework.
01:05:54100%. I mean, I couldn't agree more.
01:05:57They're all super ambitious efforts on
01:06:01their own. And luckily, we have a small
01:06:05but super talented and absolutely
01:06:08brilliant core team of folks who are
01:06:11working on the various pieces.
01:06:13You've been mentioning the docs. We have
01:06:15Giulio working more or less full time on
01:06:18the docs as well as on the Effect Schema,
01:06:21which we could probably also fill an
01:06:23entire show on just talking about the
01:06:25Effect Schema, which I
01:06:26think you're also using.
01:06:27Oh, yeah. We're using Effect Schema all
01:06:29over the place. I didn't
01:06:30even talk about that. Yeah.
01:06:31Yeah. But then also the the dev tools.
01:06:35I'm super excited that
01:06:37Mattia will be joining.
01:06:39Well, at the time of this recording, a
01:06:42couple of days from now, but when the
01:06:43time comes and the recording airs, I
01:06:46think Mattia will have already started
01:06:48working full time on the dev tools and
01:06:50other pieces and improving
01:06:52the developer experience.
01:06:54And then at the core of all of it, Effect
01:06:564.0 and what it will enable. That's
01:06:59really been the distillation
01:07:01of all the feedback that we've gathered
01:07:03over the last couple of years.
01:07:04And I think really tackle some of the
01:07:07most common points of feedback or
01:07:09frustration head on. And I think it's
01:07:12really like a huge improvement over
01:07:15what's already really great and useful.
01:07:19So I'm fully on board with with
01:07:22everything you've you've shared. And
01:07:25yeah, I think I can't thank you enough
01:07:28for sharing all that what you've been
01:07:31building with Cortex at MasterClass.
01:07:34It's really, really impressive.
01:07:36What you've been able to build in such a
01:07:38short period of time speaks to
01:07:40experience, but also speaks to just like
01:07:43how capable the building blocks are. And
01:07:45like when like experience and ambition
01:07:48comes together with great materials.
01:07:50I think this is how we're going to get
01:07:51great experiences and great applications.
01:07:54So thank you so much for doing that work
01:07:57and sharing it.
01:07:58And thank you so much for coming on the show today.
01:08:01Well, thank you so much for having me.
01:08:03And I just you know, I have to say that
01:08:05so far, I think the on call product has
01:08:08been a huge success and Cortex has been
01:08:12very, very crucial part of that.
01:08:16And I feel like it would not have been
01:08:18possible without the Effect ecosystem and
01:08:21also the support of the Effect core team
01:08:23and the community to helping us get
01:08:27there. I think that has played a fundamental
01:08:29role. So thank you. I mean, to you,
01:08:32Johannes, and thank you to the broader
01:08:34Effect Team and the community for all of
01:08:37the support and assistance and enthusiasm
01:08:39and building this incredible framework.
01:08:40It has really been a game changer.
01:08:43That is awesome to
01:08:44hear. Thank you so much.
01:08:46Thank you.
01:08:48Thank you for listening to the
01:08:49Cause & Effect Podcast.
01:08:51If you've enjoyed this episode, please
01:08:53subscribe, leave a review
01:08:54and share it with your friends.
01:08:56If you haven't done so already, you can
01:08:58join our Discord community.
01:09:00And if you have any questions, feedback
01:09:02or suggestions about this episode or
01:09:04about Effect in general,
01:09:06don't hesitate to get in touch.
01:09:08See you in the next episode.