---
title: "Contextualize using files"
video_id: a-EK-j8zdTg
url: https://www.youtube.com/watch?v=a-EK-j8zdTg
workflow_page: https://workflows-of-ai.com/workflow/contextualize-using-files
language: en
captions: generated
---

0:00 hi there everyone Marcus here today I'm
0:02 going to Showcase a new member of the AI
0:05 interpolator family called
0:07 unstructured This is a helper module to
0:11 contextualize files basically so
0:14 unstructured IO is a service that you
0:19 can either use their software as a
0:21 service they have a free tier so you can
0:24 try it out for I think it's 1,000
0:26 requests per month uh but it's also
0:29 available then as a open source server
0:32 that you can host yourself which makes
0:34 it very very powerful so what it is is a
0:38 server that takes unstructured files uh
0:41 you can see some examples here HTML PDF
0:43 CSV it does Excel for instance as well
0:46 uh images PowerPoints markdown uh and
0:50 some other
0:51 stuff um and structures this then into a
0:56 way uh where AI can basically
1:00 uh use it for for whatever different
1:04 request you might think of
1:06 right um so the module exists here uh
1:11 it's called
1:13 unstructured uh it's not necessarily
1:16 just for AI interpolator there is a subm
1:18 module for AI interpolator the idea is
1:21 later that this is of course extremely
1:23 powerful for search API where you can
1:26 index you know PowerPoints index uh
1:30 uh images and so on based on text inside
1:34 images and stuff like this uh so if
1:38 someone wants to co- maintain and help
1:40 out with that please let me know I will
1:43 eventually come to it later but I don't
1:46 have all the time in the world um so in
1:49 this case I'm just going to showcase the
1:51 AI interpolator part of it then so I'm
1:54 going to Showcase it in three different
1:57 ways I'm not going to set up chains by
1:59 the way in this case I think you can
2:02 realize how powerful this is in terms of
2:05 chains because setting up something as
2:08 simple as um you might have seen the
2:11 example where I uploaded a video and it
2:13 generates a web page take screenshots of
2:15 that right doing the same with a PDF or
2:18 a PowerPoint is extremely simple with
2:21 this and the open AI
2:23 plugin uh so this opens up for a lot of
2:27 uh powerful
2:30 uh workflows but in this case I'm just
2:32 going to show how to plainly use the
2:35 module uh and I think you understand if
2:38 you look at the other videos how you can
2:40 actually change this into really
2:42 powerful things
2:43 then I'm going to Showcase it in three
2:46 different ways uh
2:49 because uh there's a lot of ways how you
2:52 can use this uh first of all the actual
2:56 module itself it has um it has three
3:00 different rules but within those rules
3:02 there are a lot of settings as well I
3:05 will just show you here so for you if
3:07 you don't know about it the AI
3:09 interpolator is fully documented how to
3:12 get started how to write your own
3:14 plugins for it all the available plugins
3:17 but also all the available rules so I
3:19 think we're up to like 80 rules or
3:22 something now uh and if you go then to
3:26 any of the plugins and in this case
3:28 unstructured you can see the rules
3:30 available uh so I'm going to Showcase
3:32 two of those in this case the one I'm
3:35 not going to Showcase is that you can
3:37 also take a PDF or a image that has
3:41 multiple images inside it so so to speak
3:44 so if you have an image of a poster
3:45 where it's text and some small
3:48 images uh you can basically upload this
3:51 file and get all those images out into
3:54 image Fields then uh but this I'm not
3:57 going to Showcase I'm going to showcase
3:59 file to string and file to
4:02 tables uh so the first showcase I'm
4:05 going to do is take this
4:07 PDF and this I'm not going to make ready
4:10 for AI in this case what I want to do
4:12 instead is I want to take this uh file
4:15 that has that is fairly structured right
4:18 it has like header one header two you
4:20 know paragraphs uh text uh links a table
4:26 and a image and I want to generate
4:29 actual HTML from this so we're going to
4:31 take a formatted field and generate HTML
4:34 from this um the other two examples then
4:39 is I'm going to take a a very simple
4:41 Excel file this can be a lot more
4:43 complex of course doesn't really matter
4:46 uh and I'm going to use this files to
4:49 tables and this takes the table field
4:52 module that uh is a third party uh field
4:57 type that is a structured way of sh
5:00 tables it's awesome module if you want
5:01 to show tables right so this
5:04 structuralize things as well this is
5:06 also probably not as useful for
5:09 AI uh and then then the third thing I'm
5:12 going to show is I'm going to take a
5:14 image of a receipt and this is a a photo
5:17 of our receipt right so it's as you can
5:20 see it's not perfectly
5:22 structured uh the text inside the
5:24 receipt is of course not uh written in
5:29 text text format in the jpeg uh as it is
5:32 in a PDF for Excel file then so this is
5:35 a lot harder and in this case it's going
5:38 to use AI modules to extract this uh
5:42 just so you know this can also run
5:44 normal OCR mod models which is AI it's
5:48 deep learning right but they have a
5:50 generative AI model that's a lot more
5:52 complex than I'm going to
5:54 Showcase uh in this case as
5:58 well uh um so before I start showcasing
6:02 how to set up this I just a friendly
6:04 reminder there is workflows Das of-
6:07 ai.com uh where you can find all the
6:10 videos I've produced so far uh in a
6:13 searchable and filterable format of
6:15 course you can use YouTube you can
6:17 subscribe to YouTube if you like the
6:19 videos uh but if you prefer to actually
6:21 find them this website exists so let's
6:25 get started well let's go to the
6:28 module uh um so we're just going to do
6:33 ddev in this case because I'm using ddev
6:35 as the local environment and composer
6:38 require same with the table field
6:41 here so because um this is not the
6:48 um uh just a plugin for a interpolator
6:51 the the composer file in this case will
6:54 not download AI interpolator by default
6:56 the core module so if you don't have it
6:59 and want to do this example just click
7:01 here on AI interpolator right uh and
7:05 then you just click
7:08 here and you just get this as well
7:12 right
7:14 so that is getting the modules we just
7:17 log into our um Clean Slate Drupal in
7:21 this case as always the only thing I
7:23 have done is install the admin toolbar
7:26 we just oh sorry not appearance extend I
7:30 just need to uh turn on the different
7:34 modules now so we need unstructured and
7:36 we need table field in this case
7:39 right so table field is for this
7:42 specific Excel example but the first
7:45 example we're going to start with is uh
7:48 PDF to
7:50 HTML I just disable a lot of stuff here
7:53 I don't care
7:55 about uh and then we're going to remove
8:00 the default body field going to create a
8:02 new field and this is going to be the
8:04 file we upload that we're going to use
8:07 as the file that we want context from
8:09 well in this case HTML then and this is
8:12 a file that we're going to call the PDF
8:16 file uh and we allow the extension PDF
8:20 of course here then you could make a
8:23 workflow where you take a lot of
8:26 different types of unstructured filed
8:28 and it always fills in a in this case
8:32 formatted field type uh field right so
8:36 that's what we're going to create now a
8:38 format the text text format long uh that
8:41 we called HTML representation so this
8:43 going to be the HTML representation of
8:46 this PDF we use full
8:50 HTML uh and then we enable the aor
8:53 that's the important thing right we're
8:55 going to use this rule unstructured API
8:57 file to string we're going to use base
9:00 mode uh and we're going to base it on
9:02 the PDF file right then under advanced
9:06 settings what is important then if we
9:08 want to actually create HTML there is
9:11 this output format where you can say you
9:14 want pure text you want markdown or you
9:16 want HTML pure text and markdown is very
9:18 good for AI context HTML is not but if
9:23 you actually want like in this case you
9:26 don't care about the AI uh perspective
9:28 of it you just want a representation of
9:30 the PDF in HTML you just choose
9:35 HTML these are the different types that
9:38 uh unstructured uh splits up text into
9:41 so you can actually choose that uh oh we
9:44 just want images uh uh for instance
9:48 that's possible if if you would just
9:50 want that right uh then you can also
9:53 split it into multiple Fields so if you
9:56 set uh up here then to Unlimited
10:00 uh so you can create multiple text
10:02 fields and you want to create one text
10:05 field per page you can split it up per
10:07 page or even per element this creates a
10:10 lot of fields yes so you know so this is
10:12 every every paragraph every header and
10:15 so on becomes its own field in that case
10:18 uh then you can set strategy AO is very
10:21 good if you have structured uh uh I mean
10:25 if you have digital Fields so not image
10:28 representation of a receipt like we're
10:30 going to do later then AO is usually
10:33 good enough but what you have here then
10:36 is you have fast which is just trying to
10:38 do it as fast as possible you can do OCR
10:41 only as far as I know they use TCT in
10:44 the background there uh or you can
10:46 choose hress and then you can choose one
10:47 of the models I'm going to come to this
10:50 later uh we're going to have checked uh
10:53 extract images so this means that if
10:56 there are images in the PDF or image
11:00 uh and this doesn't work for word for
11:02 some reason just so you know that so
11:03 it's not going to extract images from
11:05 word then you need to convert it into a
11:07 PDF if you want images it's going to
11:09 extract that and put it into the Y wig
11:12 field in this case so CK editor right uh
11:15 there is this checkbox that's just to
11:17 save API credential so bypass on perfect
11:20 match that means if if this file that I
11:23 upload a Wasa HTML file there's no
11:26 reason for me to send it to their API to
11:28 create HTML it is already HTML same for
11:31 Mark Time same for text so we're just
11:34 going to save this I just realized I
11:37 forgot something so we need to go to
11:39 configuration to set up unstructured as
11:41 well under web services unstructured
11:43 config so here there are two ways to do
11:46 it then so you can use the uh software
11:49 as a service then you need to get the
11:50 API key uh and that's what I'm going to
11:53 do in this case then
11:56 um so I'm just G to
11:59 copy and paste my API key here but of
12:01 course if you self host it you can just
12:04 reference your self host if you use API
12:06 key there of course you can H use it as
12:09 well so that's it then we're going to
12:12 try this PDF to HTML we just call the
12:16 title PDF um we have this under
12:20 examples this PDF was what what we
12:23 wanted to do and this is going to fill
12:24 out the HTML representation now I we
12:26 quick save that
12:32 and there we go I'm just going to open
12:35 this file as well so you actually can
12:38 see it side by side
12:41 here uh so you see here this is a
12:45 title the title is not centered in CK
12:48 editor so this doesn't look the same but
12:51 then there's a header one to get started
12:54 paragraph header two header three right
12:58 uh and then a tap able for you and in
13:01 this case oh actually you know what
13:05 we're going to do here you see an
13:06 example that I used auto that actually
13:08 created the fastest version in the
13:11 fastest version it doesn't try to figure
13:13 out how a table works it just cuts it
13:15 out as a image so you can see here this
13:17 is actually image which is okay for HTML
13:21 I guess if you're not going to send it
13:22 through to AI it doesn't actually matter
13:26 but let's just change this let's just
13:29 just retry this so we're going to go to
13:31 our PDF to HTML and just change this
13:33 setting
13:35 then and we're instead going to use
13:39 hris uh and we use the default hus model
13:42 I will come to the models later we just
13:44 go back to our content so with AI
13:46 interpolator it basically works that uh
13:50 if nothing is filled out it will run
13:52 again so we're just going to remove this
13:54 now so it runs uh reruns uh this job and
14:00 hopefully we're going to get a table now
14:02 that is a actual yeah an actual table
14:06 right so here you can see then it has
14:10 basically generated A visual
14:12 representation of the PDF but in CK
14:15 editor and since it is CK editor you can
14:18 also click edit here and start you know
14:20 editing it if you want to do that as
14:24 well um so that was the first example
14:28 then I'm just going to show you how
14:29 table Field Works
14:31 then so we take uh and I'm this is Excel
14:36 file then so table Fields by itself can
14:38 import CSV files but in this case you
14:40 can use Excel files or you could even
14:42 use um stuff like um oh let's call this
14:47 Excel to table field uh you could use
14:50 stuff like someone wrote a table on an
14:54 nain because you got a smart idea during
14:56 dinner or something like that you you
14:59 can import this into an actual
15:01 structured table right so once again we
15:05 need a file in this case
15:09 XL uh and this I it X LS X I think going
15:16 to create another field and in this case
15:18 then uh we're going to do a table field
15:21 so we do table
15:23 [Music]
15:25 representation
15:26 continue enable AI Lor in this case we
15:31 use this rule unstructured API file to
15:34 table uh usace mode used Excel as the
15:37 base field and we use strategy Hest I I
15:43 should probably put this as default on
15:45 table actually because you really need
15:47 it uh and save the settings then we just
15:50 go to add content Excel to table field
15:54 call it Excel and we just need to upload
15:59 wait let me see if that was the right
16:01 one yeah test table
16:04 xlsx we just save this and this table
16:07 field hopefully now will be filled out
16:11 uh it even takes the name of the actual
16:13 table then so if you have multiple
16:15 fields in Excel and you set up that you
16:17 can add multiple table Fields it would
16:19 create this as
16:21 well um and then let's go to the third
16:25 example so we're going to uh do if image
16:29 to text in this casage
16:34 right
16:37 and just do this
16:45 again so this actually in this case
16:49 doesn't take a image field I might add
16:51 that rule later but in this case you
16:53 have to use the fire field because this
16:55 is just a base field used as context
16:58 right but yeah uh if you need it to be
17:02 able to create from image field add that
17:04 ticket if it's in a hurry for you
17:05 otherwise I will come to that sooner or
17:07 later right uh so this we call the image
17:12 source uh and we allow jpeg PNG
17:17 GI jpeg like that I guess as well save
17:21 that and we create a new field that we
17:24 take a plain text field in this case I
17:27 think uh and we're going to do a
17:30 markdown representation in this case
17:33 right so we take enable AI interpolator
17:36 on structure file to string again base
17:38 mode image search Source right so that's
17:40 our source uh we're going to take
17:43 markdown as output format and we're
17:46 going to use strategy
17:48 hris and there are different models then
17:51 for doing the hris strategy uh by
17:55 default it uses detectron Tu uh with o X
17:59 so this is a Facebook mod model that is
18:02 very fast so if you're in a hurry uh
18:06 this is usually good enough when you
18:09 have a a actual digital file but in this
18:13 case where there's a image
18:15 representation usually to be honest OCR
18:19 works even better than detect
18:22 from uh then there is Yolo uh that is a
18:27 lot slower it's fairly good and there's
18:30 a quantized version that's a little bit
18:33 better but then there is also chipper
18:35 and chipper is a beta model that comes
18:38 from uh unstructured that I would say is
18:41 superior to all of them because it's not
18:44 super slow it takes
18:46 like depending on image size it takes 20
18:50 30 seconds if you use their software as
18:52 a
18:53 service uh if you host it locally it
18:55 depends on your machine basically right
18:58 uh so we're going to use shop shoer
19:00 shipper in this case uh to try to get
19:05 all the information then from this uh
19:09 receipt so we're just going to upload
19:12 this and receip the jpeg this is now
19:17 going to take a little bit of time
19:20 because uh as I said it's a little bit
19:22 more complex let's just open this
19:25 receipt so we have it here uh so the
19:28 import important thing if you're going
19:29 to use this now in a uh AI capacity is
19:33 that all the data is there uh also
19:38 maybe understand that merchant ID is
19:41 connected to this number terminal ID to
19:44 this number and so on or
19:46 code um so let's see
19:49 here so we get this and you can see here
19:52 it's using markdown even to create that
19:55 markdown table in this case right so
19:57 merchant ID is going to this terminal ID
20:00 is connected to this uh number is
20:02 connected to this and so on this is a
20:05 ag3 in this case purchase becomes right
20:10 so uh this for me is awesome I worked
20:14 with OCR before but it it doesn't become
20:18 perfectly structured so I would say this
20:20 is really awesome for this use case uh
20:24 they have done an incredible job here
20:26 and of course yeah a might have
20:29 mentioned markdown is something open AI
20:31 knows about something that the
20:35 biggest uh generative llms knows about
20:38 so you can do really really powerful
20:41 things when you have gotten this
20:43 structured um yeah so that's it I hope I
20:49 even if I didn't show any AI chains here
20:52 I hope I got some uh ideas into your
20:55 brain how you can use it on your uh
20:58 project and also it goes without saying
21:00 then since you can host this
21:03 yourself uh and you could use a large
21:06 language model that you host yourself as
21:08 well you could use this for internal
21:10 documents without you know sending it to
21:12 a third party as
21:14 well so yeah thank you for watching