Managing Airflow at Scale using the Flowrs TUI

1 00:00:01,083 --> 00:00:04,958 You open up your laptop and it's time to check the overnight data pipelines.

2 00:00:04,958 --> 00:00:08,000 You open up a browser, you log in and check the pipelines.

3 00:00:08,000 --> 00:00:12,583 You open up another tab, you log in again, different environment, same thing.

4 00:00:12,875 --> 00:00:18,541 12 environments, dev production, that's 24 environments every single morning.

5 00:00:18,625 --> 00:00:28,125 My colleague Jan got tired of that, so he built a terminal user interface called Flowers that allows him to manage all of his environments at once, from his keyboard, no mouse, no

6 00:00:28,125 --> 00:00:30,208 browser, no nonsense.

7 00:00:30,250 --> 00:00:36,083 And in this episode, Jan is going to show us how that works and how you can build something like this yourself in Rust.

8 00:00:36,416 --> 00:00:41,208 I am Jonny, knowledge theory at Dataminded, and welcome to Technology Explorations.

9 00:00:49,708 --> 00:00:54,791 Hi everyone, today we'll have a look at text user interfaces or terminal user interfaces.

10 00:00:54,791 --> 00:00:56,166 For that, I've invited Jan.

11 00:00:56,166 --> 00:00:56,833 Welcome Jan.

12 00:00:56,833 --> 00:00:57,458 Thank you, Jonny.

13 00:00:57,458 --> 00:01:00,666 Could you tell us a bit more about yourself and your role at Dataminded?

14 00:01:00,666 --> 00:01:04,583 I'm working here at Dataminded as a data engineer.

15 00:01:04,833 --> 00:01:11,458 I'm a team lead currently at one of our clients and I'm also heading uh our academy.

16 00:01:11,500 --> 00:01:14,750 Okay Jan, so you have been building a terminal user interface.

17 00:01:14,750 --> 00:01:17,083 Maybe show us what you built

18 00:01:17,083 --> 00:01:17,458 Sure.

19 00:01:17,458 --> 00:01:23,458 I'm in my terminal and in my case the tui is called Flowrs and you can start Flowrs by just typing Flowrs.

20 00:01:23,875 --> 00:01:33,208 The name Flowrs is a pun on Apache airflow and flow Flowrs and Because it is written in rust a lot of rust projects.

21 00:01:33,208 --> 00:01:37,916 They have this RS suffix So that's where the name comes from Flowrs

22 00:01:37,916 --> 00:01:41,958 And so for our viewers, Airflow is an orchestrator for data engineers.

23 00:01:41,958 --> 00:01:44,875 Could we maybe have a look at what the UI looks like normally?

24 00:01:44,875 --> 00:01:45,333 Sure,

25 00:01:45,333 --> 00:01:47,250 So this is the Airflow UI.

26 00:01:47,250 --> 00:01:54,333 And in Airflow, you have your DAGs, your directed acyclic graphs, which represent your workflow, your data pipelines.

27 00:01:54,333 --> 00:01:58,083 And you can inspect them and see how they are running, what they are doing.

28 00:01:58,083 --> 00:02:01,291 So here you have an overview of all of your DAG runs.

29 00:02:01,291 --> 00:02:01,791 Each...

30 00:02:01,791 --> 00:02:03,791 vertical line is a DAG run.

31 00:02:03,791 --> 00:02:07,333 Each square here is a task your pipeline.

32 00:02:07,333 --> 00:02:15,208 You can also visualize it as this DAG, as this directed acyclic graph of all these operations that will get scheduled by Airflow.

33 00:02:15,208 --> 00:02:19,583 And so Airflow is basically the babysitter of these DAGs, of these pipelines.

34 00:02:19,583 --> 00:02:23,583 And as you can already see, to navigate in this UI,

35 00:02:23,583 --> 00:02:25,375 First of all, there's a lot going on.

36 00:02:25,375 --> 00:02:27,041 There's a lot of things that I can click.

37 00:02:27,041 --> 00:02:30,250 There's a lot of different buttons and tabs.

38 00:02:30,250 --> 00:02:39,500 What you've now seen was one Airflow environment, one Airflow instance, and that's still manageable if you need to navigate through the DAGs and through all of the pipelines to

39 00:02:39,500 --> 00:02:45,250 see after a night of batch runs, which pipelines have failed, where do you need to take action.

40 00:02:45,250 --> 00:02:53,541 But if you have 10 or tens of these environments, it quite quickly becomes a hassle to do this through the UI by just clicking.

41 00:02:53,541 --> 00:02:56,750 And I got sick and tired of doing that, and that's why I built the Tui.

42 00:02:57,166 --> 00:02:57,583 Yeah.

43 00:02:57,583 --> 00:03:03,500 So you hate clicking around and jumping across all these environments because it takes a lot of time.

44 00:03:03,875 --> 00:03:06,666 And as a data engineer, you manage multiple ones of these.

45 00:03:06,666 --> 00:03:13,500 Like in this case, we see 12 environments, and you have a development production, you need to check all of them.

46 00:03:13,500 --> 00:03:15,833 So that would require 12 tabs being opened.

47 00:03:15,833 --> 00:03:20,625 And then even in those environments, you manage multiple DAGs, lots of work.

48 00:03:20,625 --> 00:03:26,250 lots of clicking, of exactly, lots of digging deeper to figure out what happened.

49 00:03:26,250 --> 00:03:27,791 And I wanted to make this easier.

50 00:03:27,791 --> 00:03:32,750 Yeah, so you built your own terminal user interface called Flowrs to navigate it.

51 00:03:32,750 --> 00:03:36,916 So could you show us around a bit in the terminal user interface on how you use it?

52 00:03:36,916 --> 00:03:37,791 Yeah, sure.

53 00:03:38,000 --> 00:03:39,833 So let's drill down on an environment.

54 00:03:39,833 --> 00:03:44,250 So when you go into an environment, you have, first of all, the list of DAGs.

55 00:03:44,250 --> 00:03:51,458 So these are your pipelines, your workflows, and then you can navigate through the DAGs just using your keyboard.

56 00:03:51,458 --> 00:03:53,125 can use VIM key bindings.

57 00:03:53,125 --> 00:03:56,375 So meaning J is going down, K is going up.

58 00:03:56,375 --> 00:04:03,916 You can also jump to the top of the table by using gg, or you can go to the bottom with capital G.

59 00:04:03,916 --> 00:04:12,000 And you can also filter if you just quickly want to find a DAG that you're interested in by using the slash command and then you see a filter popping up.

60 00:04:12,000 --> 00:04:18,041 at the bottom and then you can look for your favorite DAG and in my case I want to have a look at this DAG.

61 00:04:18,041 --> 00:04:23,333 If you then select the DAG, you get to see all of the DAG runs of the DAG.

62 00:04:23,333 --> 00:04:29,208 So a DAG run is an instantiation of a DAG and DAG is this concept of a workflow or pipeline.

63 00:04:29,208 --> 00:04:31,125 DAG run is a run of this pipeline.

64 00:04:31,125 --> 00:04:33,000 You get some information about this DAG.

65 00:04:33,000 --> 00:04:33,791 When did it run?

66 00:04:33,791 --> 00:04:35,125 How long did it take?

67 00:04:35,125 --> 00:04:35,750 There's

68 00:04:35,750 --> 00:04:40,000 a line gauge to show how long it took in comparison to other DAG runs.

69 00:04:40,000 --> 00:04:48,250 And if you then drill down, you can see all of the individual tasks of your DAG run to see how long did they take, did they run successfully.

70 00:04:48,250 --> 00:04:56,333 And then even at the run or the task level, you can drill down and see all of the logs of that specific run.

71 00:04:56,333 --> 00:04:58,000 I also see a follow mode here.

72 00:04:58,000 --> 00:05:04,375 So you would be able to have a trail and follow along when the pod is running in this case, where the task is running.

73 00:05:04,375 --> 00:05:12,916 Exactly, So it basically continuously polls for new logs and then it will follow along so you don't have to do anything.

74 00:05:12,916 --> 00:05:22,916 while you did the drill down, I saw these tabs at Every time you went a bit deeper and you go from the top level where you have all your instances to your DAGs, to the runs, to the

75 00:05:22,916 --> 00:05:24,416 tasks, to the logs.

76 00:05:24,458 --> 00:05:28,375 And how do you effectively do use this in your day to day?

77 00:05:28,375 --> 00:05:36,083 In my case, what I typically do, there's a few workflows, but we typically get alerted of important pipelines when they fail.

78 00:05:36,083 --> 00:05:38,666 This is happening in our alerting system.

79 00:05:38,666 --> 00:05:49,208 It's pager duty, And then I just look at the name of the DAG in the alert and I navigate to the right environment and I start searching for that DAG to see uh it failed.

80 00:05:49,208 --> 00:05:52,083 And then I drill down to see what failed, why did it fail?

81 00:05:52,083 --> 00:05:57,250 And as I mentioned before, you get the application logs for a failed task instance.

82 00:05:57,250 --> 00:05:58,083 So I can easily

83 00:05:58,083 --> 00:06:00,083 figure out what went wrong and try to fix it.

84 00:06:00,083 --> 00:06:00,500 Yeah.

85 00:06:00,500 --> 00:06:03,375 your trigger is an error from a pipeline.

86 00:06:03,375 --> 00:06:05,291 You get an alert, there's something wrong.

87 00:06:05,291 --> 00:06:11,375 And this allows you to immediately dive deep into that error without opening a browser, opening the right environment.

88 00:06:11,375 --> 00:06:13,583 So it saves you a bit of time, I guess.

89 00:06:13,583 --> 00:06:14,541 Yeah, it does.

90 00:06:14,541 --> 00:06:16,625 can also, if I know which DAG it is,

91 00:06:16,625 --> 00:06:21,500 Here there's a bunch of successful tasks, but I know that there's also a couple of failed ones.

92 00:06:21,500 --> 00:06:23,833 And then I can also filter on the state.

93 00:06:23,833 --> 00:06:27,166 You also see hopefully that there's like this autocomplete.

94 00:06:27,166 --> 00:06:32,375 It will also cycle if I start typing failed, it already knows.

95 00:06:32,375 --> 00:06:36,125 And then I just have an overview of the failed tasks of my...

96 00:06:36,125 --> 00:06:44,000 DAG run in this case and then I can have a look at the logs and see okay here's some a dbt job is running and one of the tests actually failed

97 00:06:44,000 --> 00:06:44,416 Yeah.

98 00:06:44,416 --> 00:06:48,958 And also these things like this auto-complete, you built that yourself in this UI.

99 00:06:48,958 --> 00:06:53,291 yeah, the autocomplete is basically a state machine.

100 00:06:53,291 --> 00:06:59,208 It generates the possible attributes of a DAG run or of a task that you can filter on.

101 00:06:59,791 --> 00:07:00,416 very nice.

102 00:07:00,416 --> 00:07:03,750 I think this looks quite slick actually, this UI.

103 00:07:04,208 --> 00:07:06,166 Anything else you'd like to show

104 00:07:06,625 --> 00:07:11,500 Actually, yes, on the UI side, Jonny, I recently added something based on a request of a user.

105 00:07:11,500 --> 00:07:22,125 So we all know, and you also know that any self-respecting developer uses a dark mode terminal, but there are some people that have maybe some color issues, color blindness,

106 00:07:22,125 --> 00:07:26,500 and they sometimes prefer a light mode terminal.

107 00:07:26,500 --> 00:07:35,666 So when you would open Flowrs in a light mode terminal, looks at your terminal, tries to figure out are you in a light mode terminal or in a dark mode terminal.

108 00:07:35,666 --> 00:07:37,375 uh

109 00:07:37,375 --> 00:07:40,958 Because terminals, do they pass on the information in the environment?

110 00:07:40,958 --> 00:07:45,958 well, I cannot give you the full details because I used a dependency to do this.

111 00:07:45,958 --> 00:07:51,500 There is a crate called Terminal ColorSaurus and I like the name already, but that does it for you.

112 00:07:51,500 --> 00:08:01,083 And if you look at the readme of the project, it seems like it's people really knowing what they're doing and really knowing terminals inside out, looking at specific escape

113 00:08:01,083 --> 00:08:02,916 sequences and C codes.

114 00:08:02,916 --> 00:08:04,291 It just works in my case.

115 00:08:04,291 --> 00:08:07,416 It correctly can figure out that this is a light mode terminal.

116 00:08:07,416 --> 00:08:11,166 So the extra feature is light mode where we used to see dark modes.

117 00:08:11,166 --> 00:08:13,708 You have a light mode as a new feature.

118 00:08:13,708 --> 00:08:14,500 exactly.

119 00:08:14,500 --> 00:08:22,916 And not only light modes, you can also, because of course we have to over engineer everything, you can also select these Catppucin themes.

120 00:08:23,125 --> 00:08:24,708 like latte, frappe, macchiato.

121 00:08:24,708 --> 00:08:28,375 And they each have a slightly different color scheme.

122 00:08:28,375 --> 00:08:29,958 And you can also enable

123 00:08:30,000 --> 00:08:41,958 uh Flower supports both Airflow V2 and V3, which is something that was quite useful for us because we have many of these environments and we were migrating these environments from

124 00:08:41,958 --> 00:08:43,250 V2 to V3.

125 00:08:43,250 --> 00:08:46,125 But we still wanted to manage our DAGs in the same way, right?

126 00:08:46,125 --> 00:08:50,208 We still wanted to navigate through our DAGs, find the failed task instances.

127 00:08:50,208 --> 00:08:54,916 And it kind of looks the same for both V2 and V3 Airflow.

128 00:08:55,166 --> 00:08:55,416 So

129 00:08:55,416 --> 00:08:59,125 So you provide more UI stability than Airflow itself at this point.

130 00:08:59,208 --> 00:09:01,541 Yes, but also less features.

131 00:09:01,541 --> 00:09:04,500 In the Airflow UI, apparently now you can play Doom.

132 00:09:04,500 --> 00:09:06,625 That's something you cannot do with Flowrs.

133 00:09:06,625 --> 00:09:07,541 nice not yet at least.

134 00:09:07,541 --> 00:09:12,000 I wanted to show this escape hatch maybe not everything you want is visible

135 00:09:12,000 --> 00:09:13,375 within Flowrs.

136 00:09:13,458 --> 00:09:22,958 So you can press O on any object on the DAG, on the DAG run or on task instance, and it will open the Airflow UI of that environment.

137 00:09:22,958 --> 00:09:29,666 you can easily, if you need more features or more graphical features, you can easily dive in.

138 00:09:29,666 --> 00:09:30,166 Nice.

139 00:09:30,166 --> 00:09:33,791 if you press V, it shows you the code of the DAG and you can

140 00:09:33,791 --> 00:09:45,041 one last thing I would like to show, which I think is quite cool, is if you have a bunch of failed DAGs, for example, and you want to all mark them as successful, you typically

141 00:09:45,041 --> 00:09:48,416 would do this if there's nothing really actionable about your failed DAG.

142 00:09:48,416 --> 00:09:56,625 Maybe your DAG has failed because there was a timeout or something, but you know that it's fixed now, and so you just want to mark your DAG as successful.

143 00:09:56,916 --> 00:10:06,125 You can press shift V, you go into visual mode, just like in vim, you can navigate and that basically selects a bunch of these, in this case, DAG runs.

144 00:10:06,125 --> 00:10:11,916 And you can then press the button to mark them as success, failed I want to mark them as successful.

145 00:10:11,916 --> 00:10:18,250 So in terms of managing such a large workload or multiple environments, this is a lot faster than going to the UI.

146 00:10:18,666 --> 00:10:21,250 I see tuis popping up everywhere.

147 00:10:21,250 --> 00:10:22,833 Where is this coming from?

148 00:10:22,833 --> 00:10:25,250 that's a good question.

149 00:10:25,250 --> 00:10:30,458 terminal user interfaces were the OG interfaces of our computers, right?

150 00:10:30,458 --> 00:10:35,250 You used to have only a terminal and some text on it.

151 00:10:35,250 --> 00:10:38,791 This is a VT100, one of the first tele typewriters.

152 00:10:38,791 --> 00:10:40,958 That's what you got, black screen and some text.

153 00:10:40,958 --> 00:10:45,208 And that used to be more than enough for a lot of use cases.

154 00:10:45,208 --> 00:10:48,166 You also have an example of MS-DOS.

155 00:10:49,000 --> 00:10:57,041 Also the first games, I really like this kind of trivia, the first games were also just text-based games And I think as a developer,

156 00:10:57,041 --> 00:10:59,625 Yeah, you kind of like working in a terminal.

157 00:10:59,625 --> 00:11:02,291 You feel efficient, you feel fast.

158 00:11:02,291 --> 00:11:08,416 If you have to navigate through a file system, I think most developers would say they prefer doing that than clicking around

159 00:11:08,416 --> 00:11:11,583 You have this element of speed, of being efficient.

160 00:11:11,583 --> 00:11:12,875 it's a lot less.

161 00:11:12,875 --> 00:11:17,208 convoluted and a lot less busy than the typical web UI.

162 00:11:17,208 --> 00:11:22,583 Although of course, if people properly design web UIs, they can also be simple and focused.

163 00:11:22,958 --> 00:11:25,833 And then there's also these retro aesthetics.

164 00:11:25,833 --> 00:11:27,875 It's more of a personal thing, but I like it.

165 00:11:27,875 --> 00:11:30,833 and it's also available on many systems, right?

166 00:11:30,833 --> 00:11:35,208 Like there's always like a bash terminal available that you can use.

167 00:11:35,500 --> 00:11:36,083 Indeed.

168 00:11:36,083 --> 00:11:43,416 Where I also think it shines is in for these kinds of applications where you have a big surface and like Airflow is a good example.

169 00:11:43,416 --> 00:11:48,583 You have a lot of things you can do with Airflow with the UI.

170 00:11:48,583 --> 00:11:52,166 There's also an Airflow CLI, which is also quite a big CLI.

171 00:11:52,166 --> 00:11:53,166 You can do a lot of things.

172 00:11:53,166 --> 00:11:57,125 What is then the difference between CLIs and TUIs according to you?

173 00:11:57,125 --> 00:11:59,625 When would you opt for each of them?

174 00:11:59,625 --> 00:12:01,125 a very good question.

175 00:12:01,125 --> 00:12:07,041 And also it can be tied to this new wave of agentic AI and agents doing things for us.

176 00:12:07,041 --> 00:12:10,875 A tui is a very visual thing, right?

177 00:12:10,875 --> 00:12:14,500 We humans are very visual creatures, right?

178 00:12:14,500 --> 00:12:17,416 And we easily spot small changes in uh

179 00:12:17,416 --> 00:12:23,666 in a visual thing, reading a bunch of text, which is typically what comes out of a shell command that takes time for us.

180 00:12:23,666 --> 00:12:25,083 We don't read that fast.

181 00:12:25,083 --> 00:12:27,916 Now agents do read very fast, right?

182 00:12:27,916 --> 00:12:34,833 So you have all of these agent skills and a lot of agent skills wrap around the CLI, they use CLIs.

183 00:12:34,833 --> 00:12:37,958 And I really think that's a beautiful marriage between the two.

184 00:12:37,958 --> 00:12:42,041 Agents can read super fast, the CLIs just give text back.

185 00:12:42,041 --> 00:12:46,666 But as a human, you are, I think, more efficient by just having something visual.

186 00:12:46,666 --> 00:12:56,333 In the case of Airflow, it's way easier to spot one red dot on your screen to show that something has failed, and then read the output of one CLI command.

187 00:12:56,333 --> 00:13:01,833 Yeah, so your Flowrs TUI does it mean it's not as suited for Agents to use?

188 00:13:01,833 --> 00:13:03,625 Yeah, maybe it could be used.

189 00:13:03,625 --> 00:13:06,250 I'm not sure if it just will be very efficient.

190 00:13:07,083 --> 00:13:10,500 And so what is then the reason you designed Flowrs?

191 00:13:10,500 --> 00:13:14,875 Because now we have agents, they can talk to CLIs, you could just ask a question.

192 00:13:14,875 --> 00:13:20,083 Do you still have the need for a speed up that you cannot achieve with agents?

193 00:13:20,416 --> 00:13:31,000 Well, first of all, I started developing Flowrs before there were agents and it was born out of frustration with having to click in the UI and also curiosity.

194 00:13:31,000 --> 00:13:33,583 I like to know how things tick.

195 00:13:33,583 --> 00:13:35,583 Like how does it actually work?

196 00:13:35,583 --> 00:13:37,791 What does it do?

197 00:13:37,791 --> 00:13:42,458 So I started building this before the whole agentic revolution,

198 00:13:42,458 --> 00:13:46,333 agents already replacing Flowrs in my day-to-day workflow?

199 00:13:46,333 --> 00:13:51,458 Not yet, but I also don't really have a good reason or argumentation of why they couldn't.

200 00:13:51,458 --> 00:14:00,041 But what I think you're still faster when doing it on your keyboard because if you know your Flowrs then you can easily jump into things that you know.

201 00:14:00,041 --> 00:14:02,875 So it's like second nature to you, seems.

202 00:14:04,000 --> 00:14:06,291 You still win of the AI in this case.

203 00:14:06,375 --> 00:14:09,041 Still out competing the bots.

204 00:14:09,041 --> 00:14:09,916 Yes, that's true.

205 00:14:09,916 --> 00:14:17,666 But if cost is not of a concern, because in the end you can indeed tell an agent, hey, here's the airflow CLI, figure it out.

206 00:14:17,666 --> 00:14:22,708 And it might take half an hour to figure it out and burn a bunch of tokens to do it.

207 00:14:22,708 --> 00:14:24,958 that might be a trade off you're willing to make.

208 00:14:24,958 --> 00:14:26,458 how do you build such a system?

209 00:14:26,458 --> 00:14:27,875 Where do you start?

210 00:14:27,916 --> 00:14:34,458 Yeah, so when I started building Flowrs, I started first looking for frameworks that people are using to build these tuis.

211 00:14:34,458 --> 00:14:36,666 I quickly found these three.

212 00:14:36,666 --> 00:14:45,875 You have Bubble Tea, Textual and Ratatui and each are written in a different language which is also quite nice if you're very familiar with a specific language With a colleague

213 00:14:45,875 --> 00:14:47,958 we experimented a bit with Bubble Tea.

214 00:14:47,958 --> 00:14:48,958 It's written in Go.

215 00:14:48,958 --> 00:14:50,750 It's very easy to use.

216 00:14:50,750 --> 00:14:52,708 It's very elegant as well.

217 00:14:52,708 --> 00:14:59,833 You also, by default, get a nice UI if you just take the default templates of Bubble Tea.

218 00:14:59,833 --> 00:15:01,333 But I didn't choose Bubble Tea.

219 00:15:01,333 --> 00:15:04,333 I thought, well, let's do something more challenging.

220 00:15:04,333 --> 00:15:08,666 And I wanted also to improve my Rust knowledge So that's why I went with Ratatui.

221 00:15:08,666 --> 00:15:11,208 Textual, I think it's also quite good.

222 00:15:11,208 --> 00:15:19,250 despite Python having this connotation of not being a very performant language, I think textual UIs can be also very interactive.

223 00:15:19,250 --> 00:15:20,708 and very performant.

224 00:15:20,750 --> 00:15:24,291 and what is the reason for you that you're so hyped about Rust?

225 00:15:24,291 --> 00:15:25,250 I hear this a lot.

226 00:15:25,250 --> 00:15:26,625 I haven't looked into Rust myself.

227 00:15:26,625 --> 00:15:29,375 It's still somewhere on the backlog for me.

228 00:15:29,375 --> 00:15:37,000 I think one of the main reasons is that it's fun to get hyped about stuff, In the end, it's just a programming language, they're all Turing complete.

229 00:15:37,000 --> 00:15:39,500 But it's fun to get hyped about things.

230 00:15:39,500 --> 00:15:44,708 And the other reason is that I like the expressiveness of the type system.

231 00:15:44,708 --> 00:15:54,375 I come from a scientific background, physics and mathematics, and I always liked the rigidity of proofs of logic.

232 00:15:54,375 --> 00:15:57,458 And then with Python, it's kind of the opposite.

233 00:15:57,458 --> 00:16:00,666 You can write a lot of things and nobody will complain.

234 00:16:00,666 --> 00:16:08,458 And at some point they might crash and burn in production and you will look at the wreckage and then see, ah, yeah, okay, that went wrong.

235 00:16:08,458 --> 00:16:09,958 I need to fix this in this way.

236 00:16:09,958 --> 00:16:12,166 With Rust, you have

237 00:16:12,208 --> 00:16:13,458 kind of the opposite.

238 00:16:13,458 --> 00:16:21,125 You are developing your application and for the first, in my case it was many days, the damn thing doesn't even compile.

239 00:16:21,125 --> 00:16:23,250 Some people say the compiler shouts at you.

240 00:16:23,250 --> 00:16:31,625 I now look at it as the compiler teaches you why what you wrote doesn't make sense, is not fault tolerant, has some issues with it.

241 00:16:31,625 --> 00:16:36,333 So I like the expressiveness of the type system and the guarantees that it provides.

242 00:16:36,333 --> 00:16:42,083 Okay, so it's a static compiled language, I assume, and you get a lot more safety.

243 00:16:42,083 --> 00:16:42,416 yeah.

244 00:16:42,416 --> 00:16:46,083 And one of the selling points and the tagline is always blazingly fast.

245 00:16:46,083 --> 00:16:50,041 You could do a benchmark with some other tuis and you would probably find that it's faster.

246 00:16:50,041 --> 00:16:55,166 All right, so you have these different frameworks, but under the hood, they're all actually quite similar.

247 00:16:55,166 --> 00:16:58,541 In the end, it's just one big event loop, and it just...

248 00:16:58,541 --> 00:17:00,166 does three different things.

249 00:17:00,166 --> 00:17:08,416 It listens for input events, when you press a key on your keyboard, or you get a response from an API call.

250 00:17:08,416 --> 00:17:11,166 And then based on the event, typically updates some state.

251 00:17:11,166 --> 00:17:16,125 And then after the state has been updated, there's a render phase and you display them on the screen.

252 00:17:16,125 --> 00:17:18,750 And for the rendering phase, you often have...

253 00:17:18,750 --> 00:17:23,833 built-in widgets like a table, a bar graph, and that just repeats in an infinite loop.

254 00:17:23,833 --> 00:17:32,583 So most of these frameworks give you either built-in widgets that you can easily configure yourself or you can create your own widgets and they typically have a few functions that

255 00:17:32,583 --> 00:17:33,500 you need to implement.

256 00:17:33,500 --> 00:17:41,625 So an example of some of these widgets, this is like the demo or the showcase project of Ratatui.

257 00:17:41,625 --> 00:17:44,250 So you can create these kind of tabs with it.

258 00:17:44,250 --> 00:17:48,333 You can have maps, you can have sparklines, gauges, whatever you want.

259 00:17:48,333 --> 00:17:53,458 This is really good for in the movies to put on screen and show something really cool is happening, right?

260 00:17:53,458 --> 00:17:57,958 This is the basic gist of it.

261 00:17:57,958 --> 00:18:00,791 You have your application, which is basically a loop

262 00:18:00,791 --> 00:18:07,041 which first draws or renders to the terminal, and then it starts processing events.

263 00:18:07,041 --> 00:18:08,250 See if something has changed.

264 00:18:08,250 --> 00:18:12,791 In this case, the only thing that can happen is you press a key and then it will break.

265 00:18:12,791 --> 00:18:15,041 And then basically your app will shut down.

266 00:18:15,041 --> 00:18:22,125 But what you would do is here, put all of the logic, like if people press the J key or the down key, that means you

267 00:18:22,125 --> 00:18:23,916 to go to the next item in a table.

268 00:18:23,916 --> 00:18:38,250 em If there's a tick event, like this clock ticking every 200 milliseconds, for every 2000 milliseconds, make an API call to the Airflow REST API to get the latest DAGs.

269 00:18:38,250 --> 00:18:44,583 You update your state with that, and then the next loop starts basically, terminal.draw, and it will render.

270 00:18:44,833 --> 00:18:48,250 actually my main takeaway message is go forth and mulTUIply.

271 00:18:48,250 --> 00:18:51,583 It's very easy to build your own Tui and to wrap around it.

272 00:18:51,583 --> 00:18:54,000 And how long were you working on this?

273 00:18:54,000 --> 00:18:55,000 that's a good question.

274 00:18:55,000 --> 00:18:58,750 in the beginning, of course, I didn't know the language, so I had to learn Rust.

275 00:18:58,750 --> 00:19:01,083 I made a lot of stupid rookie mistakes

276 00:19:01,083 --> 00:19:02,250 So I think it took me.

277 00:19:02,250 --> 00:19:08,666 on and off about a year to get like a first UI where I could just see the DAGs and the DAG runs in the task instances.

278 00:19:08,666 --> 00:19:19,333 But then this filtering, for example, with the state machine, this theming stuff that became a lot easier because I feel comfortable with the language and we have agents to

279 00:19:19,333 --> 00:19:19,750 help us.

280 00:19:19,750 --> 00:19:20,458 Okay, nice.

281 00:19:20,458 --> 00:19:22,625 And so people can actually use this, right?

282 00:19:22,625 --> 00:19:25,250 Can they brew install Flowrs or how does that work?

283 00:19:25,250 --> 00:19:27,250 Actually that's also a nice thing.

284 00:19:27,250 --> 00:19:33,750 there was a contribution from somebody from the community last week and they added it to the brew core taps.

285 00:19:33,750 --> 00:19:37,125 So now anybody can just install brew install Flowrs.

286 00:19:37,125 --> 00:19:41,541 So when you open Flowrs for the first time, actually don't see that much, right?

287 00:19:41,541 --> 00:19:47,666 Because works with airflow instances and you need to tell Flowrs where to find those airflow instances.

288 00:19:47,666 --> 00:19:52,708 Airflow is an open source project, which means you can just host it yourself.

289 00:19:52,708 --> 00:19:57,875 can basically deploy it on any cloud environment on AWS, for example, with some EC2 machines.

290 00:19:57,875 --> 00:20:02,250 But you also have a bunch of managed services.

291 00:20:02,333 --> 00:20:05,708 So actually before you would even uh open Flowrs, you would do something

292 00:20:05,708 --> 00:20:11,916 like Flowrs, config enable -m for managed service and then you can type a managed service.

293 00:20:11,916 --> 00:20:15,041 There are a bunch of managed services as I already mentioned.

294 00:20:15,041 --> 00:20:16,833 There's conveyor which is

295 00:20:16,833 --> 00:20:25,708 Dataminded's own managed airflow offering, but you also have MWAA, which is managed Workflows for Apache Airflow on the AWS cloud.

296 00:20:25,708 --> 00:20:27,458 You have Google Cloud Composer.

297 00:20:27,458 --> 00:20:34,625 You have Astronomer, which we all know now because of the CEO news story.

298 00:20:34,625 --> 00:20:38,250 um managed service.

299 00:20:38,250 --> 00:20:40,750 have and then it generates a config for you.

300 00:20:40,750 --> 00:20:43,125 Exactly, so you enable it.

301 00:20:43,125 --> 00:20:55,041 In this case, it was already enabled and there is a config file If it doesn't exist, it will get created and it basically contains some configuration that the TUI manages for

302 00:20:55,041 --> 00:20:55,541 you.

303 00:20:55,541 --> 00:20:58,416 And then you can start Flowrs and it will actually.

304 00:20:58,416 --> 00:20:59,583 uh

305 00:20:59,625 --> 00:21:01,666 find all of the Airflow environments.

306 00:21:01,666 --> 00:21:05,250 If you don't want to use a managed service, you just have your own Airflow instance.

307 00:21:05,250 --> 00:21:08,750 You can also just provide a host name and credentials.

308 00:21:08,750 --> 00:21:17,958 If you use basic authentication like a username and a password or OAuth 2 with the JWT token, those are all also valid options.

309 00:21:17,958 --> 00:21:21,791 Also in this case you have conveyor, but conveyor manages multiple of these environments.

310 00:21:21,791 --> 00:21:24,666 So it also discovers even the environments that you have.

311 00:21:24,666 --> 00:21:27,000 Cause that depends on the managed service, right?

312 00:21:27,000 --> 00:21:27,541 Exactly.

313 00:21:27,541 --> 00:21:40,666 And on MWAA so on Amazon, it also auto-discovers the environments that you have within your region, On GCP, you can also select a region and a GCP project, and it will also

314 00:21:40,666 --> 00:21:44,416 auto-discover the composer environments that you have available.

315 00:21:44,416 --> 00:21:46,208 oh Sure.

316 00:21:46,208 --> 00:21:47,458 So here's the repo.

317 00:21:47,458 --> 00:21:48,250 Yeah, exactly.

318 00:21:48,250 --> 00:21:50,666 143 stars!

319 00:21:50,666 --> 00:21:52,666 Exactly and of course it's self-starred.

320 00:21:52,666 --> 00:22:01,958 So this already existed since 2023 and I was mainly using it myself, And I was constantly poking some colleagues like, hey, have you tried Flowrs already?

321 00:22:01,958 --> 00:22:04,291 I know that you have to work with 10 airflow environments.

322 00:22:04,291 --> 00:22:05,750 Maybe you want to try Flowrs.

323 00:22:05,750 --> 00:22:08,916 And then at some point I got an email

324 00:22:08,916 --> 00:22:13,083 Mentioning like, your TUI is the TUI of the week.

325 00:22:13,083 --> 00:22:19,333 And then also the maintainer of the Ratatui project also shared it on his LinkedIn, I think.

326 00:22:19,333 --> 00:22:21,083 And also some more people found it.

327 00:22:21,083 --> 00:22:25,166 And I recently posted it in the Apache Airflow community Slack.

328 00:22:25,166 --> 00:22:28,583 So there's some community interaction now, which I actually like.

329 00:22:28,583 --> 00:22:30,250 It's also a first for me.

330 00:22:30,291 --> 00:22:33,375 I built some stuff, but usually just for myself.

331 00:22:33,375 --> 00:22:36,375 What did you learn about this whole TUI process

332 00:22:36,375 --> 00:22:40,875 I learned that I really like Rust as a programming language.

333 00:22:40,875 --> 00:22:45,750 I also learned that it's fun to just sometimes take a peek under the cover.

334 00:22:45,750 --> 00:22:51,541 We all use these applications, figuring out how they tick is a fun and rewarding experience on its own.

335 00:22:51,541 --> 00:23:00,833 I've also learned that agentic AI, especially the last couple of months, can really speed up the frequency at which you can push out new features.

336 00:23:00,833 --> 00:23:10,833 And I also learned that you sometimes also need to push people, like you have to be the annoying marketeer in order for people to start using your app.

337 00:23:10,833 --> 00:23:12,291 Even if you think it's amazing.

338 00:23:12,291 --> 00:23:14,250 I mean, I use it on a daily basis.

339 00:23:14,250 --> 00:23:15,416 I like using it.

340 00:23:15,416 --> 00:23:18,833 Just building it and making it public on GitHub.

341 00:23:18,833 --> 00:23:22,458 That doesn't result in people finding it and actually using it.

342 00:23:22,458 --> 00:23:26,791 You have to push it out and go after your audience yourself a bit.

343 00:23:27,083 --> 00:23:27,875 Yeah, okay.

344 00:23:27,875 --> 00:23:35,416 Yeah, I think it's a very nice tool you showed us, So thanks a lot for sharing what TUIs are, how you build them, and put them in the market.

345 00:23:35,416 --> 00:23:37,291 if you're using Airflow, check out the tool of Jan.

346 00:23:37,291 --> 00:23:40,500 We'll share the link in the description and in the comments.

347 00:23:40,500 --> 00:23:43,166 So Jan, thanks a lot for explaining this.

348 00:23:43,166 --> 00:23:44,875 Thank you everybody for watching.

349 00:23:44,875 --> 00:23:47,083 Check out Jan's tool and we'll see you next time.

350 00:23:47,083 --> 00:23:47,750 Bye bye!

351 00:23:47,750 --> 00:23:48,375 Bye!