---
title: "Deep Crawling"
video_id: 5bcdfsNkaVs
url: https://www.youtube.com/watch?v=5bcdfsNkaVs
workflow_page: https://workflows-of-ai.com/workflow/deep-crawling
language: en
captions: generated
---

0:01 hey guys Marcus here um so I'm going to
0:05 Showcase a workflow today that's like
0:07 workflow Inception basically what we're
0:09 going to do is we're going to uh scrape
0:13 our whole web page so that's something I
0:15 added to the scraping but you can
0:16 actually scrape a whole web page where
0:18 it starts going down deep down into the
0:20 web page and get
0:22 everything uh and use that as a context
0:24 and what I thought I would do today is
0:26 basically to take uh the whole
0:28 documentation of the AI inator all the
0:31 rules scrape that and then create a a
0:34 form where you can ask hey I want to
0:37 create this workflow is that possible
0:39 using the AI inator and it hopefully
0:42 will answer like yeah you can set it up
0:44 using these
0:46 configurations um but before I start
0:50 that I would just take a couple of
0:51 minutes and talk about uh stuff that
0:55 happened in my personal and uh my uh
0:58 professional life
1:00 so I have actually been very busy even
1:03 if you haven't seen videos in a while
1:05 now you can see here in May so the last
1:08 couple of weeks I've been pushing a lot
1:11 there's a lot of new features uh in AI
1:14 interpolator um I've been looking into
1:17 other AI stuff as well been pushing
1:19 stuff
1:20 um and I I just didn't have time to
1:23 actually create videos for this yet so
1:25 I'm going to create videos for it um
1:27 it's still a lot of it is still lacking
1:30 in documentation as well so this is
1:32 coming as
1:33 well uh but I just want to talk about
1:36 the reason why I can be so productive
1:39 now basically is these guys um so I
1:45 joined a company called freely give uh
1:48 and as you can see here they are
1:50 basically a company that specializes in
1:53 the
1:54 combination Drupal and CRM so they have
1:58 like a a native Drupal CRM him uh
2:00 they're very focused on that and they've
2:03 been building that for uh the longest
2:06 time uh it's also interesting because
2:09 they they like they found Drupal as the
2:14 natural platform for what they were
2:17 doing uh because of its extremely
2:19 flexible nature in terms of what you can
2:22 do with Fields databases structure and
2:24 so on uh which is very similar to why I
2:27 took a decision in my old company that
2:30 we should use druple for uh the AI
2:35 chaining because it was pretty obvious
2:38 pretty fast even if Lang chain and Lang
2:40 chain JS existed for Python and
2:43 OJs building the whole structure around
2:46 something that you can do no code so you
2:50 can be a site Builder and set up uh
2:53 chains that would take a lot of time
2:55 even in using something like Jango
2:58 together with Lang chain would just take
2:59 a lot lot of time for me to build so
3:02 drupa was actually the Natural Choice
3:05 there uh so that's very interesting
3:08 basically they they saw the same thing
3:11 and they have also been looking heavily
3:12 into AI you can see on the the web page
3:15 you're looking at that they are focusing
3:17 a lot on
3:18 AI um so so I met Jamie in the slack
3:22 channels drup of slack channels for AI
3:25 so that's basically how I met them Jamie
3:27 is one of the directors then at uh
3:30 freely
3:31 give uh and they gave me this fantastic
3:35 opportunity basically to uh help them
3:38 with AI but also while doing this help
3:42 the community in large so they are True
3:45 Believers in open source like I am I you
3:48 know open source is really important for
3:50 me
3:52 um they they have worked a lot with AI
3:56 so if you used like the search AP
4:00 AI module that's from freely
4:03 give uh or partially at least from
4:06 freely give um and they have done a lot
4:09 of you know Rag chatbots and stuff like
4:11 this so they have already looked in a
4:14 lot into the AI uh part and they have
4:18 customers then that are uh some of the
4:21 bigger like UK train companies um and
4:25 they have a case management system there
4:28 so you can read more about the here uh
4:31 called Journey call uh which is yeah
4:36 it's partly a delay repay system so you
4:38 know the train was late you want money
4:40 back and parts of this
4:43 actually um
4:45 there uh they they try to build AI into
4:51 it so it actually can take decisions or
4:53 at least give a a certain help for the
4:56 person taking the decision if money
4:59 should be paid feedback and so on uh so
5:02 they are actually doing this AI in real
5:05 life production environment which is
5:08 extremely exciting
5:11 um
5:13 so yeah I I just wanted to say I'm
5:16 extremely grateful for uh Jamie uh and
5:20 the other director Andrew for giving me
5:23 uh this
5:24 opportunity uh it's I I think it's good
5:28 for me personally hopefully I can
5:30 contribute back to the company so so um
5:33 they have use for me outside of just
5:36 contributing to the open source
5:39 Community um and I will hopefully be
5:43 able to contribute a lot more to the
5:45 open source Community not just AI
5:46 interpolator but in general a in AI uh
5:51 this opens up a lot of
5:53 opportunities so if you're interesting
5:56 interested in that you know just contact
5:59 me or or Jamie would be the the person
6:01 to contact if you if you want to talk
6:04 about actual
6:06 business uh and this also opens up the
6:09 credibility then maybe for the AI
6:11 interpolator that there is a company
6:14 behind it that you can also pay for
6:17 consultancy which means that it's not me
6:19 doing it on my free time and me having
6:21 to figure out administrative how I can
6:24 you know uh work for someone for a
6:27 couple of days with all the things with
6:30 bills and stuff because I never liked
6:32 being a freelancer or or or having my
6:35 own company so this is a possibility now
6:37 as well if you need help with some
6:39 specific workflow if you need help with
6:41 uh uh uh building some specific rules
6:44 into AI or you know AI consultation in
6:49 general because there's a lot of smart
6:51 people in this company uh and not just
6:54 in in druple PHP but in other languages
6:57 that works with AI uh you know please
7:00 contact Jamie just look for freely give
7:02 and you will find his contact data there
7:06 [Music]
7:07 um so yeah that's it once again I'm very
7:13 uh grateful for this uh opportunity and
7:17 let's start our
7:20 workflow so before I show the example
7:23 just there is workflows of ai.com where
7:25 you can filter search for all the videos
7:27 I've done so far uh um so I started with
7:32 doing what I said in the intro that I
7:34 would do to scrape the workflows of AI
7:36 documentation that's on drupal.org
7:38 drupal.org has scraping protection in
7:42 line the modules I've created can bypass
7:45 this I'm not going to show you how to do
7:47 this uh if you ever want to do this you
7:50 have to figure it out yourself I would
7:52 not recommend it the the protection is
7:55 there for a reason so I'm not going to
7:57 scrape drupal.org in this case
8:00 uh what I'm going to scrape instead is I
8:03 just took a simple example here the
8:04 guzel documentation right and we're
8:06 going to ask a question about
8:08 guzzle uh and why is it important to
8:11 scrape here by the way and why is it
8:13 important to have the guzzle
8:14 documentation before we ask the llm
8:16 something so you would think with a
8:19 model like open AI GPT 40 you know the
8:22 latest coolest version it would be able
8:26 to answer any question about guzzle
8:29 uh and this it can do in
8:34 99.99% of the
8:37 cases but the problem in the end is an
8:40 llm an LM is not an answering machine LM
8:46 is a text generator which means that it
8:49 might
8:51 be not so credible in this case so as
8:54 far as I know GLE doesn't exist as a
8:57 product or as an even a name somewhere
9:01 else but it might be in the token space
9:04 when when it's generating uh your
9:06 sentences that you want as answers it
9:08 might actually have a high temperature
9:11 have high creativity and actually go
9:13 somewhere
9:14 else uh in this case it also makes sense
9:17 that you know as you can see we're on
9:19 the latest stable version we we just
9:21 want the latest stable version of the
9:23 documentation so it doesn't answer
9:26 something for uh gusle
9:29 or GLE 5 or gusle 4 we wanted to answer
9:32 something about gasel version 7 and I'm
9:35 sure anyone that's in druple might have
9:38 noticed this with chat GPT that
9:40 sometimes uh it gives back functions
9:44 from Drupal 7 even if you're asking it
9:46 like hey in Drupal 10 how do you do this
9:48 and this is the same thing it's going to
9:50 a token space where maybe the word
9:53 druple and the combination of whatever
9:56 you're asking is is weighing more than
9:59 the druple 10 token um and that's why
10:03 you have to in that case you know
10:05 reminded oh this doesn't exist in Drupal
10:08 7 I'm uh sorry this doesn't exist in
10:10 Drupal 10 I meant Drupal 10 can you
10:12 please answer how you do it there and
10:14 then maybe it
10:16 answers um and because of this always if
10:21 you want a truthful answer you need
10:23 context so this uh the fancy word for
10:26 this is rag retrieval augmented
10:28 Generations where you you know if you
10:30 create a chat but you can have
10:31 embeddings which is a complex way of
10:34 searching with uh
10:37 intent uh but this is a simple way of
10:40 doing rag where you basically fetch a
10:41 context and answer it uh and of course I
10:46 will also mention if you're doing like a
10:48 chatbot or answering machine or
10:50 something you you only need to get it
10:52 one time usually I'm just setting this
10:55 up so you can see shaning basically um
10:59 so so what we're going to use in this
11:00 case we're going to use Simple crawler
11:02 there's two modules there's the scraping
11:04 button and there's the simple
11:06 crawler uh so you get some idea what the
11:09 difference between these are both of
11:12 them are for fetching web
11:15 content uh scraping bot has the
11:19 advantage first of all scraping bot is a
11:21 service so it's actually calling API
11:23 this means that if you want to run this
11:26 on some kind of data uh where you have
11:29 privacy privacy concerns you should not
11:31 use scraping butt first of all because
11:33 it goes through their service I'm pretty
11:35 sure it says in the terms and condition
11:37 that they are not spying on this but you
11:39 know who knows you can't control that uh
11:43 but what they have that's a huge
11:44 Advantage is that they're running inside
11:46 a
11:47 browser so it opens Chrome uh and can
11:53 actually go to websites that you know
11:55 are JavaScript based or headless or
11:58 anything that that's not server side
12:00 rendered it can actually render this it
12:03 also has two other main advantages I
12:06 would say one thing you can run it from
12:08 almost any country uh any country that
12:12 matters for testing I would say on in
12:16 terms of uh economical ma markets except
12:20 China China has you know special rules
12:23 um and thirdly uh also there it can have
12:28 a lot of different IPS right so so you
12:30 can you can
12:32 bypass yeah what I was talking about
12:35 before basically if if you have rate
12:37 limiting to an IP drupal.org has more
12:40 complex than this I've seen but if you
12:43 just have rate limiting to IP can
12:45 connect from multiple IPS um third and
12:49 last Advantage when it comes to scraping
12:51 bot is that they have premium Proxes as
12:53 well uh and to explain this Netflix is
12:57 probably the best
13:00 uh explanation here even if you're not
13:02 going to scrape Netflix but uh I'm in
13:06 Germany sometimes I want to watch
13:07 America Netflix show so I want to
13:10 pretend I'm American right if you just
13:12 get any VPN for this it's going to
13:15 fail uh and the reason behind that is
13:18 that almost all of the vpns runs on a
13:22 Cloud Server so that's how they how they
13:25 can scale how they work right and the
13:28 cloud servers have very defined IP
13:31 ranges that they own which means that if
13:35 some VPN service set up like openvpn or
13:38 some other service in USA I pay for this
13:41 I go via this and it's set up in AVS or
13:45 aure cloud or something right then my IP
13:49 will look like I'm coming from there and
13:51 Netflix would see okay hey that's a
13:53 server IP you're not allowed to watch
13:55 that you're trying to you're trying to
13:57 bypass our rules right
14:01 uh in the case with scraping but this
14:03 might be interesting when it comes to
14:05 LinkedIn for instance you want to scrape
14:07 LinkedIn uh
14:08 Twitter some certain other uh uh
14:12 Services what scraping bot has they have
14:16 something called premium proxy it's
14:17 actually 10 times as expensive or you
14:19 use 10 times more credits but sometimes
14:21 it might be worth it what it can do then
14:24 is basically scrape from a what looks
14:27 like a home internet so it doesn't look
14:29 like a server internet so that's three
14:34 uh reasons to use it if you're going to
14:35 use it you have this code here as well
14:37 that helps me a little bit because I get
14:40 a free testing account from them so I
14:41 can run automated tests basically um so
14:45 you and you get 10% of the first month
14:47 if you use this the other thing that
14:49 exist then is the simple crawler uh and
14:52 this is very simple it's just using
14:54 GLE uh it has a lot of options though so
14:58 it has some some other advantages that
15:00 you can do with scraping Bol so one
15:02 thing is it can byass bypass basic goth
15:06 I guess scraping B can bypass basic go
15:09 if you put it in the URL in theory but
15:11 you don't want to do that of course uh
15:14 if you care about privacy or security
15:16 because this scraping bot would see so
15:19 you can do uh basic o you can put your
15:21 own cookies in there you can pretend to
15:23 be any user agent um and some other
15:26 settings you have in there uh that are
15:28 pretty cool so
15:32 uh what it can do then that's super
15:35 important is it can and can only scrape
15:38 server site rendered web pages with
15:40 which this GLE documentation is so
15:42 that's what we're going to use in this
15:44 case um lastly that might be an
15:48 advantage and a disadvantage this runs
15:50 from the server of course which means
15:52 that if there's IP protection you know
15:54 you're not allowed to spam with from the
15:55 same
15:56 IP this is bad on the other hand if you
15:59 want to migrate something locally you
16:02 can actually do this because this runs
16:04 from your computer in that case so uh I
16:07 you can see here I'm on a d local
16:09 development environment if I had some
16:12 old let's say I want to migrate from a
16:14 WordPress site or a LEL site that is not
16:18 online but the project exists there uh
16:21 so I can run it locally I can of course
16:23 use this to scrape that
16:25 website uh and then we're using open AI
16:28 then so I I'm just going to start by
16:31 logging in
16:32 here um everything is set up I have
16:36 installed simple crawler I've installed
16:38 the admin toolbar as you can see I've
16:40 installed the token module and I have
16:42 installed open AI open AI have set up a
16:46 a key under open AI settings as well so
16:48 we're just going to start here by
16:50 creating a content type uh that we call
16:54 the guzzle I have it there guzzle
16:56 answering M machine uh I'm just clicking
16:59 around here if you don't know droo you
17:02 will have to look for information what
17:04 I'm doing here somewhere else uh but
17:07 soon I I will probably add a uh custom
17:10 entity for workflows a disposable one so
17:13 more information comes about that
17:15 later uh we're just going to start by
17:18 adding The Source
17:19 link so this is the link we're putting
17:22 in right this is the one that we want to
17:24 scrape from external and
17:26 disabled uh we save that so oh we should
17:30 set that required as well and then we
17:33 also want to ask a question right uh we
17:36 do that takes plain long and we call
17:39 that uh guzzle
17:45 question and set that required so these
17:48 are the two things we have to no let's
17:51 do this because the title is going to
17:53 show let's use the title instead we just
17:55 call this guzzle question well it's 255
17:59 characters but that should be enough for
18:02 my my
18:04 questions then we create a format text
18:07 text format long which are the HTML
18:11 Pages uh this is super important that
18:14 you set unlimited here or set a high
18:17 value at least what you don't want to do
18:19 is that you're
18:23 oh sorry I haven't explained yet what
18:26 I'm what I have added here so
18:29 just so you know all these videos I'm
18:31 not putting any production values into
18:33 it I should probably do that but I don't
18:35 have the time to like coding more it's
18:38 basically this is one part of me testing
18:40 the actual module so I'm just turning on
18:43 OBS and talking while I'm doing that and
18:45 that's that's that's how these videos
18:48 comes to be uh but I think it's
18:51 helpful at least in some degree to
18:55 understand how how everything works so
18:58 what I forgot to mention is there are
19:00 new rules to both these so before they
19:04 could you know take a link and scrape
19:06 the header or the main image or the web
19:09 page what it can do now is that it can
19:12 crawl so you can basically say hey I
19:15 want to craw the whole website and put
19:17 this in in a lot of text Fields right so
19:20 we have the entire website as um as uh
19:26 the the the context because of course
19:29 with the guzzle I just think they have
19:31 like 10 15 P pages in the end uh so it's
19:36 maybe possible you know I copy here
19:38 click that copy there click there I
19:39 could do that but if you have thousands
19:42 of pages that you want to use for
19:44 context this sucks of course and because
19:46 of this you have to put it unlimited
19:48 this thing I added of course also adds a
19:51 lot of context that you can put into the
19:54 search API AI which you can do rag
19:57 chatbots and really cool stuff also
20:00 there's another uh depth link rule that
20:05 does the same it just doesn't store the
20:08 actual content of the website instead it
20:10 looks at other Links of specific
20:13 extensions that you can configure so say
20:16 that you want to go to a web page and
20:18 find all the PDFs that's on that web
20:21 page and get the links to those and with
20:24 those links you can use the extractor to
20:26 download them later so AI interpolator
20:28 extractor
20:29 um that is possible as well but now
20:32 we're use we're going to use uh this
20:35 rule I'm going to show you so we turn on
20:37 full HTML and you understand how why
20:38 unlimited is important here and we're
20:41 going to use the simple depth crawler it
20:44 would be scraping bot depth crawler than
20:46 if you're using scraping bot uh oh one
20:49 more thing uh simple crawler has a
20:51 release now with this scraping but it's
20:55 still in depth because I need to fix
20:57 some other stuff so if you want want to
20:58 test it today in in scraping but you
21:00 have to get the de
21:02 version advanced settings we're going to
21:04 go we say depth two will be enough so
21:09 this means that it's going to click link
21:11 here and it finds another link well in
21:14 this case it's a internal link so that
21:15 doesn't count but uh if it finds another
21:18 link even deeper it's going to go two
21:21 steps in this case uh don't set this
21:25 very high unless you really want to
21:27 scrape a lot right include Source URL
21:29 means that it gets this actual page as
21:32 well so the first page not just the page
21:34 that it scrapes host only means that if
21:38 the host is docs. Cle php. org and it
21:41 links to something else like our cookie
21:44 policy.com or something we're not going
21:46 to start scraping that keep that checked
21:49 unless you know what you're doing I'm
21:51 going to come later you can add ring
21:53 expense stuff because there are valid
21:57 reasons why not to do this because of
21:59 course if you want to scrap PDFs like I
22:01 talked about before and the same
22:03 settings exist in this link stuff then
22:06 uh those might be on a blob storage that
22:09 has a completely separate uh uh host
22:13 name right so you can do that URL on top
22:17 just puts the URL on top this is
22:19 important if you're doing rag or
22:21 something and you want to to reference
22:23 the URL where you got the answer
22:25 from
22:27 uh I we can keep that actually because
22:30 then we can maybe ask it even to
22:32 reference it in the prompt so we can see
22:34 it's correct uh body only means
22:38 that it this is not talking about what
22:40 it's going to scrape so the content it's
22:42 going to get get it's rather that you're
22:45 saying that only look for links within
22:47 the body tags so if there happens to be
22:49 a link in the head tag it's not going to
22:52 use that here is just your comma
22:54 separated exclude pages and these are
22:56 some normal stuff that you don't care
22:58 about like terms and condition cookie
23:00 policies and stuff and then you have
23:02 include patterns and exclude patterns
23:05 and these are reg then that you can put
23:07 in so in our case if you click here
23:10 there are actually links to 6.5 5.3
23:14 version five PDFs and stuff that we
23:16 don't care about but it follows a
23:19 pattern here where the latest version is
23:21 under stable and we only want the
23:23 English version as well right so we can
23:26 just copy that this should be some in
23:28 the path right uh and this is then regex
23:32 so I have to escape here so you need to
23:35 know regex to use
23:37 this um you can leave empty if you want
23:40 to include all a cool down means that
23:43 it's basically waiting uh in between
23:46 each uh each time it uh scrapes uh be
23:50 nice there keep a cool down like half a
23:53 second or something you don't want to
23:54 take down website by spamming them
23:56 that's really not nice and uh if you
23:58 check your logs for when Bots hit your
24:01 website or non-malicious Bots you will
24:03 see that they do this they don't just
24:05 spam you with if you see Google bot for
24:08 instance unless you're a very big
24:10 website they're not going to spam you
24:12 with you know thousands of requests in a
24:14 minute or something like
24:16 that uh if you want to pretend to be a
24:19 user agent this is usually good because
24:22 this is what many websites actually uh
24:25 checks for when it comes to um is this
24:30 website uh or is this request from a
24:33 browser or not right so we're just going
24:35 to add the Chrome there here you see
24:38 basic off custom headers I can add
24:40 custom cookies we don't care about that
24:42 then we have the r crawler mode where we
24:45 can say either we get the whole dump of
24:47 the web page so we would get
24:51 everything or or we can use HTML
24:54 selector which is similar to jQuery
24:56 we're just looking for ID or classroom
24:58 or something like that but in this case
24:59 we use article
25:01 segmentation uh which comes from the
25:03 readability Library so the readability
25:05 Library comes from Milla Foundation uh
25:08 you see it in the Firefox browser where
25:10 you can basically go to our website and
25:12 say hey I only want to look at the
25:14 article I don't care about all the ads
25:16 and other menus and stuff I want to read
25:18 the article cleaned out it's using the
25:21 same thing so in this case it's just
25:23 going to get this while these links and
25:26 these links are menus it recognizes say
25:28 this and doesn't scrape
25:31 that uh then we're going to use a weight
25:35 of
25:36 95 so in this case direct could perhaps
25:40 work in theory I think it actually might
25:43 finish before 30 seconds uh because it's
25:48 like I said it's like 15 pages or
25:50 something but of course if you're going
25:52 to do some deeper scrape this will never
25:54 work so instead what we're having is uh
25:59 you can choose between these two qron
26:01 this is the most optimal one because
26:04 there's no usually no timeout you might
26:06 have PHP timeout but usually on Rush you
26:08 don't have it uh batch mode it's a
26:11 little bit special because what this
26:13 does it doesn't put all of the scraping
26:15 into one batch it hooks into the AI
26:19 interpolators batch and adds all these
26:22 jobs basically where they're supposed to
26:24 be so for each web page it basically has
26:27 whatever PHP your sorry web server
26:30 timeout you have so we're going to try
26:32 batch now because it's also very
26:34 visible uh and then we're just going to
26:37 have an
26:38 answer so we just call that
26:42 answer uh and here is where open AI come
26:45 in then so we're going to use open AI
26:47 string long in this case uh we're going
26:50 to use advanced mode because we have to
26:52 take two different uh two different
26:56 context and one thing that's important
26:58 for you to know with base mode as well
27:01 so if we were to use context of
27:04 the HTML
27:07 Pages this means that actually it would
27:11 split up each of them and run this
27:13 prompt for each of the pages which in
27:16 this case would make no sense right so
27:19 that's always how the base Field Works
27:21 because that's how it have to work if
27:23 you have four links before what we were
27:25 talking about scraping it should scrape
27:27 all those four links L separately right
27:29 so this is always how it works so when
27:31 you want to say that I want to take all
27:33 the HTML Pages as one big blobe instead
27:36 you always have to use the advanced mode
27:38 and
27:39 token so let's write our prompt so uh
27:44 you are a guzzle
27:48 expert
27:50 and have the guzzle
27:54 documentation in the context
27:58 the user has a question about guzzle you
28:04 will answer this by using the
28:10 context the answer should be
28:15 between two to four sentences long um
28:21 please reference the
28:24 [Music]
28:25 URL that the
28:28 that you used for
28:32 the uh
28:34 answer all the URLs are visible at the
28:40 top of the page uh top of each uh uh
28:47 document
28:49 documentation
28:51 page question so this is the question
28:54 then we get from
28:56 node uh what do we call this we called
28:59 it Godel question I guess no
29:06 question uh uh I'm completely lost now
29:10 let's see if I go back
29:14 here oh I didn't sorry I didn't add the
29:19 question wait is this this is answer oh
29:23 man I I have a Misha so I'm going to add
29:28 had the question here I thought I did
29:32 guzzle question it's even in my history
29:35 let's see required field
29:40 save okay that's weird so let's reload
29:43 here I have copied it luckily
29:46 my um prompt uh so token question
29:53 notes guzzle question that's what we're
29:55 looking for let's also put it like this
29:58 this so it understand where this starts
30:00 and when this
30:02 ends H context GLE
30:08 documentation and we do the same here
30:12 start
30:17 stop yeah so regarding by the way I do
30:20 test stuff of course uh
30:23 before uh I'm showcasing this but these
30:27 are usually the first time I uh test it
30:29 on complete clean environment that's why
30:31 this is running and recording these
30:34 workflows are very good actually for
30:37 testing as well so I H get the field
30:40 HTML pages I set this to batch mode as
30:43 well we used the new cool GPT 40 model
30:47 that was released recently this weight
30:49 has to be higher than the other I think
30:51 I lowered it but anyway it's at 100 five
30:53 just to be
30:55 sure and let's see well one thing I have
30:58 to do now because I didn't add the
31:01 question correct I want
31:04 to
31:06 ah now I remember we wanted to use the
31:11 title let's re sorry I can't even keep
31:16 track of myself we remove this again we
31:19 go back to this uh prompt uh sorry not
31:23 HTML Pages I'm so utterly confused now
31:26 answer with we go to the prompt and the
31:29 question here is going to come from the
31:32 title that's what we said
31:36 man it's not easy sometimes so uh we
31:40 take the question here save
31:42 it let's see now if we can add a gun
31:45 guzzle answering machine because then we
31:47 only need to add two things here which
31:49 is good so I'm just going to copy this
31:52 URL and then I will go to some page here
31:55 and figure out something
31:58 H how do I access headers that's a good
32:01 question how do I access the
32:07 headers and I might have to you will see
32:11 if if it cuts now it's because it took
32:13 too long time let's see so I will talk
32:16 through this I don't think it will take
32:19 long actually uh it will go through all
32:22 the Links at the first level then the
32:24 now it's the uh second level basically
32:27 and then it goes one deep as well
32:29 because the first is counted as zero if
32:32 it finds something it would now create
32:34 another batch job here because of course
32:37 it can't know beforehand how many jobs
32:40 is in the second level at the beginning
32:44 so that's why it might seem a little bit
32:47 funny because it goes back to zero and
32:48 then starts again and goes back to zero
32:50 and starts again but that's the way I
32:52 could solve it now if someone has some
32:54 good idea how to actually make this look
32:56 nicer please contacted me and what's
32:59 happening now I think because it takes a
33:01 long time it's actually connecting to
33:03 the AI so it goes back to basically the
33:06 normal AI interpolator batch job queue
33:10 after it's finished getting all the the
33:13 data when you're using batch by the way
33:16 it's saves during the time so if it were
33:20 to fail because batch is a little bit
33:23 frail right you you're based on
33:24 JavaScript your internet can die and
33:26 then it dies in The Middle and so on at
33:28 least it will have stored everything up
33:30 until that time then so uh oh a 404 not
33:35 found let's try that
33:37 actually uh yeah this is a 44 not found
33:40 so that's fine uh let's find the next
33:45 one so we can see here Source it this is
33:49 what it adds then yeah here's the next
33:51 one
33:52 overview uh quick start and quick start
33:56 is where I think we had the head stuff
33:58 anyway there's a lot of content here
34:00 let's do this if I go back here and go
34:03 to manage displays I don't want to see
34:06 the HTML Pages here I just want to see
34:07 my answer right so if I reload
34:11 now uh okay you can retrieve header from
34:14 the respond using has header Geth header
34:16 and get headers for example response get
34:19 headers but it wasn't very good because
34:22 it didn't actually answer the question
34:24 it might be I'm just going to try a
34:27 thing now so this this is something I've
34:28 seen in general with uh gp4 do gp4 o
34:34 it's actually not great it's not as good
34:38 as gp4
34:39 turbo uh and it might be connected to my
34:43 prompt as well it it's a lot faster but
34:46 it's not super great actually it might
34:48 be also please reference the URL you use
34:52 for the
34:53 URL the URL should be inside
34:58 the two to four sentences so this can
35:01 also be uh that it actually created two
35:04 ansers so if I would have put unlimited
35:06 it was in the second there uh because
35:09 basically if you're if you're asking it
35:11 to generate many paragraphs it will
35:13 generate those in new text
35:15 Fields uh and this how it has to work
35:18 because if you're asking for instance a
35:20 image generator to uh generate three
35:23 images it of course has to be in three
35:25 images you can't concatenate that with
35:27 text there is a version where you can
35:29 always concatenate there's going to be
35:31 option for this soon so this is uh a
35:34 ticket I have uh to fix um or actually
35:37 there's another guy that hopefully will
35:39 fix it I will I will try to teach him
35:41 how everything works and maybe he can he
35:43 can join and help in in the effort with
35:46 the interpolator but
35:48 um for now this is how it is so I'm just
35:51 going to save and how it always works if
35:54 you're not happy with the answer because
35:56 the prompt was not perfect you can just
35:58 remove it and then it runs that uh
36:01 particular job again so now it's not
36:03 going to fetch the website again it has
36:04 fetched everything it's just going to
36:06 run the AI job again so now we're trying
36:09 with a little bit improved prompt
36:11 engineering better model and see if we
36:14 get the URL as
36:17 well no you can access the header from
36:20 the geter meod respon if you need to
36:23 check uh okay I'm going to go even
36:26 harder at this because
36:28 uh it's obviously not understanding the
36:31 UR should be inside the two four
36:35 sentences uh the output should look like
36:41 this uh uh
36:46 sence
36:47 one about the answer sentence two about
36:52 the
36:53 [Music]
36:56 answer the URL wait we have to do fit
37:00 sentences it should of course be
37:04 dots uh the URL to the answer from the
37:13 documentation let's try this maybe this
37:17 is a dead cause maybe it's uh oh no
37:20 let's also do this I just reload top of
37:24 each documentation page with the title
37:29 source so it does that connection in the
37:32 token space as well because it's a
37:34 source uh Source not URL
37:37 right uh as you can see when this load
37:40 it says like this Source
37:43 col uh remove and try last time and if
37:47 this doesn't work I give up but you can
37:49 see that the answer at least is correct
37:51 and this is to get this correct this is
37:54 just prompt engineering but I don't want
37:55 to B bore you with that to too much
37:58 sometimes for the production stuff I do
38:00 I sit um hours to get the right
38:05 prompt yeah
38:07 finally
38:09 uh part of the pi or seven respond
38:13 object yeah but this was actually not
38:15 correct why is well let's see maybe it's
38:18 there as well
38:20 get get
38:23 header okay you can retrieve okay okay
38:25 no no no this is fine then so it's
38:28 actually there as well so it could be in
38:31 quick start or it wasn't maybe in quick
38:35 start yeah it's explained there as well
38:38 but I guess it's actually better
38:40 explained here complex headers accessing
38:43 headers okay sorry the AI is right I'm
38:47 wrong I just skimmed through the web
38:49 page so you see AI is super good to have
38:52 uh when you don't have time to actually
38:54 read something
38:55 yourself uh so it did
39:00 work well it gave back too much
39:03 sentences actually but maybe it felt it
39:06 needed to do
39:08 that
39:10 uh so yeah that's the workflow sorry
39:13 about the missteps
39:16 but as you can see I'm just doing this
39:18 as I'm trying out stuff uh and it's
39:21 probably good anyway to see that it's
39:24 not perfect that you will you know
39:27 follow into these problems yourself when
39:30 you're setting it up probably so yeah
39:33 thank you for watching