---
title: "Video Extraction on Steroids"
video_id: oqsJjtWBnSY
url: https://www.youtube.com/watch?v=oqsJjtWBnSY
workflow_page: https://workflows-of-ai.com/workflow/video-extraction-steroids
language: en
captions: generated
---

0:00 This video is sponsored by Freely give. If you need help with Drupal and AAI, get in touch with us. So, this is
0:07 another video of the examples I gave in Atlanta. So, here you're going to see how you actually set this up. So, the
0:14 idea is this. Gordon Ramsey comes and contacts us. He says, "I have 500 videos on YouTube. I need a recipe site
0:21 tomorrow. You need to have filled out all the content." Well, luckily for us, there is the umami website. So there is
0:29 a website with all the structure that you need for a recipe website, but filling out this content is going to
0:35 take forever. So I I actually tried with this video. The video is just four 4
0:40 minutes and 20 seconds, but filling out this manually and trying to figure out going back and forth in the video. Well,
0:47 it takes around 40 minutes to fill out one content and then you have 500. So you're going to need to hire a lot of
0:54 people. uh or you can let the AI do it. So, what
1:00 we have set up here is that instead of filling out all this, just need to fill out this and the YouTube link. Click
1:07 save and you're going to end up with something like this. So, this is actually going to be the AI looking at
1:13 the video trying to figure out uh how to create this dish from uh what it sees
1:20 and what it hears, well, what it transcribes to hear. and uh write out
1:26 directions, ingredients, all this uh metadata that we need and even take a screenshot from the video of the
1:33 finished dish. Uh so if you're interested in figuring out how this work, just follow
1:42 along. Right. So we have uh umami here. It's just uh vanilla umami. I've gotten
1:51 I've downloaded four different modules. admin toolbar. It just makes life easier. Token is required for the
1:58 automators. Then we have the AI module and the open AI provider. So you can actually use any vision module and you
2:06 can use any transcription model. Deepgram for instance is actually a
2:11 better transcription model for this. But um since many people use OpenAM, I'm
2:16 just going to uh assume those for this. So I turn on admin toolbars. I turn on
2:23 automators and oh yeah I should turn on open AI as well. I forot that but let's
2:30 do that a second step
2:39 here take some time and
2:45 uh the next step after that is that we have to set up open AI as
2:56 well. There we go.
3:11 So we have it set up but we also want the open AI
3:16 provider. So we're also going to set up that. While I'm doing that, I will also
3:25 uh log to the machine because this is also a requirement for the thing we're going to do. Since we're going to look
3:30 at video, we need to be able to actually have something that can decode video.
3:35 And what you need for that is uh ffmpeg. So, you have to have ffmpeg
3:43 running on on the server where this runs. But be aware, it's the coding. It's not encoding. So it's not going to
3:49 kill off you know all your CPU usage or anything like that. This is a pretty light operation that is
3:57 running. So uh let's see how it went here. Yeah. So
4:05 we go to AI provider settings open AI uh authentication. We have to create a
4:11 key. Um so you're adding it here. I'm not going to showcase this so you don't see
4:18 my key here. I'm instead gonna do that off screen.
4:23 Uh so let me just add one [Music]
4:31 here that I'm calling open AI key.
4:52 So I know this I mean the advantage you have you can forward in the video it's not maybe
4:59 that fun to listen to someone like coding but I want to showcase actually how long time it takes to set up
5:05 something. I don't want to add a lot of um steps where you know I cut or make it
5:11 look faster than it actually is. So
5:17 uh we now have the open AI
5:22 installed. We have xfmpeg installed. Uh so that is a good
5:30 start. So uh there is another library and installation we have to do in ddev.
5:36 So it's this drupal YouTubedl. And you might ask yourself what why is this not
5:42 on drupal.org? This is actually a module. So you can just use composer require this as normal. This is also my
5:50 um my repo even boring. So the reason for that is that YouTube DL that is the
5:57 underlying uh application running here is kind of um debated. Let's say it like
6:05 that. It's actually um have forbidden where I live in Germany. Germany have
6:10 very hard laws on privacy. You should never use pirate pay here. So because of that I you know GitHub I think actually
6:18 got into legal threats because of their hosting YouTube DL. So because of that
6:25 I'm uh keeping it off group.org. So if you want to use it just get it from here. Uh you can either use the YouTube
6:32 DL uh binary or you can use this uh YT DLP.
6:39 Doesn't really matter. It works with both. So, I'm just going to do the copy the instructions here to install the
6:46 binary. And I'm just going to uh run that
6:56 uh uh let's see
7:03 here. Oh, failure I think. Okay, let's do this. Maybe there
7:09 is some Uh well let's do this instead. Let's
7:15 store it
7:20 under user user bin and do
7:31 sudo. Uh all right we have to also write the name.
7:36 So, and then we need to set the chod on
7:42 it. So, we're just installing it and
7:47 oops, sudo and then we're going to just see. So, it's running. Yep. So, we have it
7:55 there. So, I have in gotten this module as well. So, I'm just going to go ahead
8:01 and install that as well. Let's close here. So no, it's not called
8:09 YouTube. It's called video downloader because YouTube even if it's called YouTube downloader, it can actually take
8:15 as you see here like thousand different vid video providers. So in this case you can create a workflow where you
8:22 basically uh have one uh video field and one file
8:28 field and it will just download this as a video.
8:34 So now we can go to recipe. Then uh what is important for us to be able
8:41 to try this out is that first of all we need to turn off a lot of required fields because we will actually want to
8:50 uh install those ourselves. Right? So we have preparation time. Uh we're going to unrequire that
9:03 There we go. What do we have? Number of
9:12 servings. So, this might be boring to look at, but I I want everything to be
9:19 uh realistic how long time this actually takes so people get an idea how long
9:24 time it would take to set up something similar. Of course, I have kind of thought through this beforehand. So
9:32 there's some time to think through what you want to solve as well. But hopefully all these
9:37 uh things that I'm doing gives you a good idea what you can
9:43 do. So I'm going to turn off this as a
9:48 required field summary directions. Let's do like this.
10:00 That should be it. So title of course that is required and that we're going to have to keep just
10:09 uh so so if we update here now hopefully it
10:16 looks a lot better. It's just the recipe name that we want to
10:22 use. So, we're going to create our first um we're going to create our first uh
10:30 chain here. So, we start uh with a link and we call this YouTube link.
10:38 We're going to use YouTube in this case, right? Um, and we we set this required because
10:45 this of course you have to to give. We don't need a link text and we don't need to enable the AI
10:52 automator because this is only a input field. It's not a field that actually does anything. This is the field that
10:59 doesn't something. So, we're going to call this YouTube
11:04 video and it's a file. Add it. Now, I have forgotten what
11:12 it creates by default if it's MP4 or
11:17 um maybe it could. Let's
11:23 see. Uh uh uh if it's
11:30 Matrosa how do I even forgot what that is. MKV could be to create MKV I don't
11:36 know. Uh and in this case we enable AI automator and there is video downloader
11:43 link to video uh file. We use our YouTube link as base mode. Uh we're
11:49 going to use patches because this is all going to take a long time. We set a very low weight because this is the first
11:55 thing that has to run. So, we're just going to save this. Uh, oops. What did I
12:01 do wrong here? Uh, is that not how you write
12:07 it? You just write like this.
12:12 Maybe this works better. Yep, that'll work better. And
12:17 then we want to make sure we pull this up so we can see it in the beginning of the form field. Uh you will see here it
12:25 created an automator status and that's just if you're using this API you can actually see the status of it
12:32 working. So we're going to put well we can put the video up there but the
12:38 important thing is the link.
12:45 So, and uh then we will test this
12:55 out. And the recipe we're going to use is this one. So, it is from uh it is
13:01 from Gordon Ramsey. It's not Gordon Ramsay himself cooking it. Um, and it is a chef basically
13:10 explaining how to do it. She says what she's adding into it. She talks about
13:15 it, but there isn't actually like a sign where it says exactly how the recipe is
13:20 done. So, this is going to be basically the uh AI looking at the video and
13:29 trying to figure out all of these different things that we need to uh do. But first of course we have to try. So
13:36 the downloader works. So we have to give a recipe name. So these are the two
13:41 inputs that we require. The rest we don't
13:47 require. And this will take some time because it's downloading a video now. Um
13:52 but I I I will not as I said forward here. I want this to be real time. So you get the idea how long time this
13:59 takes.
14:04 So, if we go to edit
14:10 now, you can see uh that we have a video here. I'm just going to Today we're
14:15 going to learn how to make our lobster. We have the video. So, first step done. So, uh for the second step, I have
14:24 to basically go back to the manage fields. We can close this again. Uh, and
14:31 I'm going to create a new field. But I created a new tab because I want to see everything I need to Well, well,
14:37 actually, let's do this. Let's go to our content and we're going to check here what we
14:44 need to fill in because what we're going to do now is that we're going to take a
14:50 uh should we take format? No, just take plain text
14:55 uh text plain long. And we call this video description. And this is where
15:01 basically the magic happens. So what this is going to do, it's going to look at the video based on our prompt and try
15:08 to extract all these things that we're actually wanting here. So there is video to text and
15:15 that's what we're going to use. So uh we're going to use the YouTube video which is the file field as our base
15:22 mode. Uh before I start writing this prompt, I will just set this to batch,
15:28 but it should be higher than 50 that we used before. Uh speech to text provider, we use OpenAI whisper one and we use the
15:35 default advanced JSON uh model here. Uh
15:40 we don't really have to give any extra prompts there. So you are
15:59 expert and turn them into recipes.
16:24 Um so preparation time try to
16:32 uh figure out approximate how long time this takes.
16:41 Also, this is a people meet online. Also note that this is a professional
16:49 shift that normal people probably need double the
16:59 time preparation time in minutes.
17:07 Uh cooking time in
17:18 minutes. This always answer
17:26 something. Uh
17:57 So
18:07 hard. Oh, this actually we can infer from the recipe later. So uh category
18:13 and tags we don't have set media image comes
18:18 later a summary of three four sentences of what this recipe is about.
18:34 all the ingredients with name, measure length
18:40 and amount uh
18:49 directions in uh uh
18:54 ordered ordered list format and that is what we need I guess.
19:02 Oh, let's try this. We might need to improve it. Let's
19:07 see. So, I'm just going to reload here now. Oh, no. I'm not just going to
19:13 reload. I'm going to make sure that this field is higher up. So, we see the
19:19 results right away. Now, the video. Now,
19:24 we can reload here. So we see we have video description and that is what we're going
19:31 to try to get the results for now. So we just going to click save here
19:39 now then. And this is also going to take some time. So uh just so you get an idea what it's
19:45 doing here. So it can't actually look at a video and because the only one that can do that right now is Gemini. So
19:52 there is a process going on here that I I think Drupal was almost the first with
19:58 uh doing and I even got you know I've gotten two people contacting me to
20:03 create startups around this idea. Uh but it is actually not very
20:09 uh complex in the end. So what it's doing it's transcribing the audio if time stamps exist it has time stamps as
20:16 well and then it's using ffmpeg uh to take out all the key frames.
20:22 So I used to work as a video engineer. This is why this is easy for me probably. But uh it's taking out the key
20:29 frames. And a key frame you can think of is like a scene. Something happens in the video that is so different that it
20:35 can't use the other frames to encode this frame. So it creates a completely
20:40 new frame. You can think of it as like a new image. Creates images and overlays
20:46 those with the actual time stamp so you know when it's happening in the video. And then you provide this to the
20:52 algorithm. So you prov it also puts that in a raster. So you don't feed it, you know, I don't know, this is a 4 minute
21:00 video. You don't want to feed it like 400 images, but instead it's it's like a raster of 16 images per image, I think.
21:08 Uh, and that is what it's looking for. So
21:13 you can see we have a lot of very unstructured information but it is
21:19 enough for it to actually start filling out the other fields. So that is what
21:24 we're going to do now and this is simpler. So we just go
21:29 back here click manage fields I guess uh and we're just going to go through
21:34 them one by one now then. So cooking time is an integer. So we just enable a
21:43 And now the actual chaining doesn't really matter anymore. Uh so we're just
21:49 going to use the video description that it yeah video
21:54 description which is the context here. Uh and try to find the part of the uh
22:04 text where it says cooking time. So that's supposed to be somewhere here,
22:10 right? uh and extract the value. Uh wait, we don't have to use the
22:18 word context. We just use the word text. I still do batch though. It looks nicer.
22:24 And we just try this first. So this actually works with filling out preparation time. And then I'm just going to go ahead and fill out all of
22:33 them. So let's save this and see. So it works. We click edit.
22:40 Scroll down. Oh, that didn't work. Why didn't that
22:45 work? Uh, try to find the part of the text where it says cooking extracted
22:52 uh integer value. I don't know if we need to specify it like this. Data
23:01 issue. Sometimes you might need to be exact. So, it might have been trying to save 30 minutes, which won't be savable
23:10 when I enter the uh No, now it works. Cooking time. 30 minutes. It was cooking time. Yeah.
23:17 Cool. Uh then, let's see here. We're going to
23:24 just add all of them now. So, difficulty. This is a list text. So,
23:30 it's a little bit uh little bit more things you have to add here. Actually,
23:37 something that's good here is to copy the uh placeholder. So, there's Scott Oyer
23:44 is working on um prompt module that we're going to
23:49 implement here. So, you can have example prompt so you don't have to copy from there. So
23:56 uh based on the text find the difficulty
24:06 uh and fill it
24:11 out possible difficult this
24:17 uh and yeah that's it batch
24:24 So we have difficulties directions I don't remember was
24:29 direction one field was it? Yeah, what field? I can see it there. One field. We
24:35 want the full HTML. Enable H AI automator. We use LM text. Simple. So
24:42 there's lm text. LM text is very good when you want to look at the text and maybe split this up into multiple text
24:48 like give me 10 suggestions on how you can do this. But the lm text simple is
24:53 just it's not going to try to structure anything. It just takes whatever it outputs. And this works a lot better
24:59 with simple models as well. Um, can you extract the part that is
25:08 instructions here and output that using
25:14 HTML? Be sure to use for the list. So uh
25:21 O L I and also
25:27 emphasize I have no clue to spell that but luckily enough uh AI is very good at
25:34 understand anyway. to emphasize uh words with strong em
25:46 uh yeah that's good I guess context is the
25:54 context and we just do batch and we say that we should always use full HTML
26:00 otherwise it will try to figure out from the chosen ones which one to use but it's better to do this. Oh, maybe I go
26:08 back. See, so I actually using HTML output is always good to write here. Uh
26:16 uh with HTML the limiters is even better to
26:23 write because if you're using this extracting it can actually extract just that part very
26:29 easily. Layout we don't have to I will come to media image later. That's the last thing we do. Uh number of servings
26:37 is that actually there did I ask for that? Yeah, it's
26:43 there. Uh so once again video description. Can you extract the number
26:51 oops number of servings from this recipe
26:57 from this text of our recipe and save the
27:03 entire view and context
27:11 context do batch and of course this I'm doing very simple prompts now that might fail from
27:18 time to time. When you're doing this professionally, you should really test the prompts out over and over until you
27:25 know you see that it working really well. So, in this case, I'm just going to copy from cooking time because this
27:31 is just the same more or less.
27:39 Um, preparation time,
27:46 What do we have left here? Uh, recipe category. Oh, I come to
27:52 recipe category and tags later. So,
28:00 summary. This is also HTML. So,
28:05 uh actually let's do this.
28:10 Uh a word where did you have directions was the
28:17 thing field is called instruction but the title is
28:23 directions anyway [Music] uh part that is a summary of the
28:31 recipe here and I'll put that
28:37 uh Oh, let's do this. Strong nm
28:47 uh that summary tags video description. We don't Oh, the rest are stuff we have
28:52 already, but we're missing something. It should be So, there's directions. There
28:58 should be some ingredients. I forgot that
29:04 problem. Yeah, forgot that. So,
29:10 uh, can you look at the text and,
29:20 uh, uh, extract the
29:25 ingredients. Make sure to extract one ingredient per value. So
29:34 this is where we're going to use oh we're going to use lm text not lm text simple. So this is a typical example
29:40 where we're taking one one text and this is a multiext as you can see uh uh
29:47 field. So we make sure to extract one ingredient per
29:53 value. Let's do this. So just if you're going to do as I just
30:00 reiterate this, you're going to do this in production, don't keep these prompts. Try to get better prompts, but they
30:06 should be kind of good enough now. So we're going to try this out
30:12 now. Uh before we continue with the categories, um so we just
30:19 save. And now it's going to be many batch jobs, I'm guessing. Let's see.
30:30 That doesn't sound like enough. Didn't change more than three. Let's see. I
30:36 might have missed something or I might have said something to not that mistake. See if there is something not
30:44 added there. We have preparation time cooking time, number of servings,
30:50 difficulty. We have a summary. Oh, did I forget to set forgot to set it before
30:57 HTML probably the word here. Uh, and it has filled out also the
31:04 ingredients there. Uh, oh, but there is one obvious
31:11 thing amount. Yeah, this is not great. So, I'm just going to I'm going to
31:17 showcase how you actually fix when something is not correctly set. pump. So
31:23 you can just remove everything because then it's going to rerun. So if we go to
31:28 ingredients now I also going to see set because it took very literally that we
31:35 just want ingredients including it's uh measurement
31:43 uh and uh measurement and amount.
31:49 So we do this, we save and then we can as soon as this is saved, we can resave.
31:56 So everything else seem to work but not that one. So that's a way how you can reiterate. You see that most of it is
32:02 working but this was not good. So we have to reiterate to make sure that this is
32:09 correct. Uh whoops. That still was not good. Let's see what was wrong now.
32:18 You look at the thing including make sure that you one ingredient one
32:25 set. So small words can make a huge difference here. Uh measure
32:32 measurement amount and ingredients. Let's see if that works
32:39 better. And yeah, even if this works now, it might not work on the next
32:45 ingredient. So, as I said, you will have to try it with a couple uh another video. I mean, you will have to try it
32:52 with a couple of ones until you find the correct prompts that basically creates the things you want. So, now it's
32:59 working well finally. Uh so, what we want to do now
33:07 is tags, I guess. Uh well, there was yeah, there was recipe category and
33:12 tags. That's two different things. And that's interesting because they do kind of different things in the automator. So
33:18 I'm going to enable the AI automator again. Uh we're going to use LM
33:24 tonomy. Uh and we're going to use uh in this. Let's try to no I think actually
33:33 we can just use the simple one. Uh is that a multiffield? Yep, it
33:40 is. So we can actually once again copy the placeholder.
33:46 Uh choose up to two categories from the category that fits the text. Just use
33:54 the value option comma and use the context context. Right? Uh we set this
34:00 to batch. Uh in this this doesn't mean anything in this case. We don't have to
34:07 do anything more. That's just saving this. And then for tags, it's more
34:13 interesting because I'm guessing this tags also has uh yeah create referenced entities if
34:19 they don't already exist. So in this case, the automator is also able to create, but it can also look at uh uh
34:27 taxonomies that exist. And this is quite interesting. So we can just paste this
34:34 again. Choose uh or create between one to three
34:41 category tags, we write from the tags context that fits the text or create
34:48 them if they do not exist. And it has the value options,
34:54 comma, it has the context here. But something that's interesting, you can actually remove this if you want. uh if
35:02 you have a lot of tags, it's going to cost a lot of money to do this. So, what you can actually do instead is uh do
35:10 this find similar tags. Uh and this will basically use the model to try to find
35:17 similar tags that exist. Uh and and
35:25 uh well, think of this example. you know, if you're saving this as
35:32 uh uh I don't know, rice dish, it adds a tag called rice dish, but there is
35:38 already a tag called dish of rice. You don't want those two tags to exist at the same time. So, this is what's making
35:45 sure that that doesn't happen basically. And then we can make sure maybe we want everything lowerase. I'm not sure if
35:51 that is the standard. I haven't looked actually how it is. Uh maybe I should do
35:56 that. Let me just know that I can see under structure texonomy tags how they
36:03 actually write this. No first uh oops what am I doing? I want to go
36:11 back here. It should be first character uppercase. And let's see if this works.
36:22 Save. Oh, I would I should go back and look. I probably forgot to set batch on
36:27 some of these. That's why it didn't actually count down here. Uh, and we check now if we have so
36:37 dinner party, seafood, seasonal, and it's a main course. So, it has set this.
36:43 So, last thing we want to do is the image.
36:49 So uh we go to media image, we enable AI automator, we use the lm video to media
36:56 image. So this can extract images from uh a video using a prompt and it has the
37:03 audio track as well. So you could actually ask it like cut out where he says this or something like that. So cut
37:09 out where the whole dish is seen and looking
37:16 sulfury. We're going to use as prompt uh cut out oh sorry one image that's
37:23 important you can of course set this to unlimited or something and it will create multiple medias for you if you
37:28 write something else there but we just want one image and you have to choose image uh as the media type under
37:36 advanced settings we also did batch job here then again and then we're just going to go
37:42 ahead and save and this is actually even more complex than the other uh video to
37:48 text uh thing. Uh so what this does is it's
37:55 doing the same thing. It's getting all a raster of all the key frames uh and it's
38:00 looking at the and looking for candidates based on the prompt you wrote. So right now it's looking for
38:07 okay it looks like the dish can probably be here the whole dish. And then it
38:12 takes those candidates and creates another raster from uh where this is
38:18 where it takes the average key frame distance in time and goes back that much
38:24 time and goes forward that much time. So that's usually around two to 10 seconds depending on how much compression you
38:30 want in a video and getting all the frames there and looking at those frames
38:35 and then taking a final decision. And the reason for that is because when you're looking at a video that's zooming
38:42 in or zooming out, you don't see that it's a lot of artifacts and blur and and
38:47 things like this. But there might actually just be two, three frames in such a scene where uh the dish is fully
38:55 in focus and the rest is just zooming in or being out of focus and so on. So it's important to get the actual right frame
39:02 where it looks kind of uh uh in focus at at least. So that's why we're doing
39:08 this. So let's see here if it found a good image. Yeah, that looks okay, I guess.
39:17 Um, and yeah, that's it actually. So what can have what we do now, of course,
39:23 is we set this to published. This you could also use the automator for. There is actually a workflows automator that
39:30 could look at all the results and see does this does this sound like a recipe. So it's not, you know, saying that you
39:36 should put gasoline in the food or something like that. And when it's allowing it, it can auto publish. But
39:42 right now, we want to do it ourselves. So, we save it. And then, let's see here. We have our results. So, we have
39:49 the Gordon Ranc's Hell's Kitchen Butternut Squash Lobster is a recipe. That's a main course. Seafood seasonal
39:57 dinner party with a nice little uh introduction here. It has set number of
40:03 servings, cooking time, preparation time, difficulty. It has set the ingredients.
40:10 Um, not all of these are said in the video. So, it's trying to estimate actually. But that is probably similar
40:17 to how uh someone that looked at the video and would have to write it down would do as well, unless you actually go
40:24 ahead then and do the recipe. That's that's of course what you really should be doing. But if you if you have this
40:31 scenario where you have 500 videos and you maybe have a couple of you know days
40:36 to finish this then that's not going to work. You can just do all these stitches
40:42 and it has given uh directions as well.
40:48 Um and that is it. Thank you for watching.