1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
|
#+title: Work for robots
#+PROPERTY: header-args:sqlite :db /scratch/titan/apache2/cgit-logs.sqlite :colnames yes
#+PROPERTY: header-args :exports both :cache yes :eval no-export
#+HTML_HEAD: <script src="http://127.0.0.1:8095/skewer"></script>
#+HTML_HEAD: <link rel="stylesheet" href="static/uPlot.min.css" />
#+HTML_HEAD_EXTRA: <script src="static/uPlot.iife.min.js"></script>
#+HTML_HEAD_EXTRA: <script src="plots.js"></script>
I self-host some of my git repositories to keep sovereignty and independence
from large Internet corporations. The public facing repositories are for
everybody, and today that means for robots. With the =AI-hype= on coding
assistants, I wanted to have a look at what are those AI companies taking. It is
worse than everything, it is idiotically everything. They can't recognize that
they are parsing git repositories and use the appropriate way of downloading
them.
#+begin_src sqlite :exports none
SELECT
min(date),
min(datetime (date, 'unixepoch')),
min(datetime (date, 'unixepoch', 'localtime')),
max(date),
max(datetime (date, 'unixepoch')),
max(datetime (date, 'unixepoch', 'localtime'))
FROM
logs
#+end_src
#+RESULTS[8f773b86167f2d36db335568f344063f13838a11]:
| min(date) | min(datetime (date, 'unixepoch')) | min(datetime (date, 'unixepoch', 'localtime')) | max(date) | max(datetime (date, 'unixepoch')) | max(datetime (date, 'unixepoch', 'localtime')) |
|------------+-----------------------------------+------------------------------------------------+------------+-----------------------------------+------------------------------------------------|
| 1735686035 | 2024-12-31 23:00:35 | 2025-01-01 00:00:35 | 1745109504 | 2025-04-20 00:38:24 | 2025-04-20 02:38:24 |
* Who is visiting
I analyzed the =Apache= log files of my =cgit= service in the period from
=2025-01-01= till =2025-04-20=. Table [[top-users]] shows the top /users/ of public
facing git repository. The leading AI companies =OpenAI= and =Anthropic= with
their respective bots =GPTBot= and =ClaudeBot= simply dominate. I found it
unbelievable that they could extract about =≈7GiB= of data each. That is a lot
of Bandwidth out of my server for a few git repositories and in a lightweight
web interphase.
#+begin_src sqlite :exports results
--SELECT
--count(*) AS Requests,
--round(total (length) / 1024 / 1024, 1) "Tx MiB",
----0,
--'Everybody else' AS "User Agent"
--FROM
--logs
--WHERE
--agentid NOT IN (143, 1, 19, 6, 4602, 3, 2, 4, 10306, 9)
--AND path NOT LIKE '/ingrid/%'
--UNION
SELECT
count(*) AS Requests,
round(total (length) / 1024 / 1024, 1) "Tx MiB",
--agentid,
user_agent AS "User Agent"
FROM
logs
JOIN agent ON agent.id = logs.agentid
WHERE
path NOT LIKE '/ingrid/%'
GROUP BY
agentid
ORDER BY
2 DESC
LIMIT 10
#+end_src
#+name: top-users
#+caption: Top 10 /users/ as self-identified by their /User Agent/ to the server
#+RESULTS[36d7b647efa39c3af86581279748a2bb53d034f3]:
| Requests | Tx MiB | User Agent |
|----------+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3572480 | 8819.6 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) |
| 1617262 | 6766.3 | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| 273968 | 721.4 | Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) |
| 80159 | 498.3 | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 |
| 207771 | 475.8 | Scrapy/2.11.2 (+https://scrapy.org) |
| 69697 | 466.1 | Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot) |
| 59832 | 416.4 | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) |
| 14142 | 83.3 | Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) |
| 2500 | 53.7 | Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com) |
| 3578 | 30.9 | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.7049.52 Mobile Safari/537.36 (compatible; GoogleOther) |
That is the total. What does it look like as a function of time? Figure
[[fig:agent-traffic]] shows the load on CGit frontend service by each visiting
agent. Hover over the plot to read the exact value for each agent at a given
time on the legend. You can highlight a specific curve by hovering over it or
its legend.
#+begin_src sqlite :results value file :file top_agent_traffic.csv :exports none
SELECT
date / 14400 * 14400 AS time, -- 4h bin
count(*) AS requests,
total (length) FILTER (WHERE agentid = 143) AS "OpenAI-GPTBot",
total (length) FILTER (WHERE agentid = 1) AS "ClaudeBot",
total (length) FILTER (WHERE agentid = 19) AS "Barkrowler",
total (length) FILTER (WHERE agentid = 6) AS "Macintosh",
total (length) FILTER (WHERE agentid = 4602) AS "Scrapy",
total (length) FILTER (WHERE agentid = 3) AS "PetalBot",
total (length) FILTER (WHERE agentid = 2) AS "AhrefsBot",
total (length) FILTER (WHERE agentid = 4) AS "Bytespider",
total (length) FILTER (WHERE agentid = 10306) AS "SeekportBot",
total (length) FILTER (WHERE agentid = 9) AS "Google",
total (length) FILTER (WHERE agentid not in (143, 1, 19, 6, 4602, 3, 2, 4, 10306, 9)) AS "Rest"
FROM
logs
WHERE
path NOT LIKE '/ingrid/%'
AND date NOT NULL
GROUP BY
time
#+end_src
#+RESULTS[87f78b2de4d43785f81e682925c6b6542d794883]:
[[file:top_agent_traffic.csv]]
#+attr_html: :id agent-traffic
#+CAPTION: Load on CGit frontend service by each visiting agent. The first black dashed line shows the total request at the server and uses the right axis scale. All other solid-filled lines, use the left axis and represent the bandwidth usage.
#+NAME: fig:agent-traffic
[[./jsfill.png]]
You can see how aggressively the =ClaudeBot= scrapes pages, consuming a lot of
bandwidth is a /short/ time. =OpenAI-GPTBot= works show its rate-limitation, and
performs its scraping over a /longer/ period of time. However, as seen in table
[[top-users]], it performs more than twice the amount of request consumes =30%= more
bandwidth.
The rest of the visitors are bots too. =Barkrowler= is a regular visitor
gathering metrics for online marketing. =AhrefsBot= is of the same type, yet
started crawling in March. =Macintosh= is certainly a bot hiding itself as a
browser and constantly probing. =Scrappy= also unknown, came at the start of the
year and never came back.
=PetalBot= is for a search engine with AI recommendation by =Huawei=, it lingers
and slowly scrapes everything. =Seekport= is a search engine, it came all of a
sudden, took as much as it found useful, =<1%= of what the big AI bot take, and
swiftly left again.
=Bytespider= is almost background noise, but it is also to train an LLM, this
time for ByteDance, the Chinese owner of TikTok.
The last one =Google= doesn't even seem to be the bot for indexing its search
engine, but rather one to test its chrome browser and how it renders pages.
=Rest= is all the remaining /robots/ or /users/. The have consumed around
=≈400MiB=, placing them in aggregate in a behavior like =Macintosh, Scrapy,
PetalBot & AhrefsBot=.
* How should they visit?
This is a collection of =git= repositories. Yet, you can browse some of my code,
some files, that is it. Yet, when you want *everything*, the correct use of this
service is through the =git= client, downloading my publicly available software.
That makes the data a lot more useful, even for those AI companies as the data
cleanup would be easier. They, themselves should use their AI to recognize what
kind of page they are vising and act accordingly instead of stupidly scraping
everything.
How have the good citizens behaved? That is on table [[git-users]]. The =Software
Heritage= keeps mirrors of git repositories. It thus watches for updates and the
downloads. There are other people besides them that downloaded, but in total, in
the same time they only downloaded =≈21MiB=. That is =0.3%= compared to
=ClaudeBot=.
#+begin_src sqlite
SELECT
--agent.id,
user_agent as "User Agent",
count(*) AS hits,
round(total (length) / 1024, 1) AS "tx KiB"
FROM
logs
JOIN agent ON logs.agentid = agent.id
WHERE
user_agent LIKE '%git%'
and id != 10897
GROUP BY
user_agent
order by total(length) desc
#+end_src
#+name: git-users
#+caption: Git users
#+RESULTS[333fcbc738819c497f14b4445a3b45f391f0db7e]:
| User Agent | hits | tx KiB |
|----------------------------------------------------------------------------------+------+---------|
| git/2.40.3 | 1075 | 12821.3 |
| git/2.34.1 | 1149 | 3687.1 |
| Software Heritage dumb Git loader | 337 | 2533.6 |
| git/2.48.1 | 115 | 1908.6 |
| Software Heritage cgit lister v6.9.3 (+https://www.softwareheritage.org/contact) | 8 | 21.3 |
| Software Heritage cgit lister v6.9.2 (+https://www.softwareheritage.org/contact) | 8 | 21.0 |
| git/dulwich/0.22.7 | 2 | 8.5 |
| git/dulwich/0.22.6 | 1 | 4.2 |
|----------------------------------------------------------------------------------+------+---------|
| Total | 2695 | 21005.6 |
* What are they looking at?
The web front end of git repositories of course, but it there a pattern?
#+begin_src sqlite
#+end_src
#+RESULTS[5c52f5ea84c4e903cbbbf8dd8f73139532de16dc]:
#+begin_src sqlite
select count(*), total(length), status
from logs
where agentid = 4602
group by status
#+end_src
#+RESULTS[91700bd4c58675dde008a24f8e07cfcea3966f58]:
| count(*) | total(length) | status |
|----------+---------------+--------|
| 207240 | 497513096.0 | 200 |
| 348 | 1195624.0 | 404 |
| 183 | 186683.0 | 500 |
#+begin_src sqlite
select count(distinct ipid)
, count(distinct agentid)
from logs
where
agentid not in (143, 1, 19, 6, 4602, 3, 2, 4, 10306, 9)
#+end_src
#+RESULTS[efe7c2c9c481b475baccee8d9ed48d814003b23e]:
| count(distinct ipid) | count(distinct agentid) |
|----------------------+-------------------------|
| 35144 | 20648 |
#+begin_src sqlite
select count(distinct ipid),
max(date),
max(datetime (date, 'unixepoch'))
from logs
where agentid = 4602
#+end_src
#+RESULTS[49b56e0faa552ea7df0d2515250f8f16365a7a40]:
| count(distinct ipid) | max(date) | max(datetime (date, 'unixepoch')) |
|----------------------+------------+-----------------------------------|
| 7 | 1742216884 | 2025-03-17 13:08:04 |
#+begin_src sqlite
SELECT
round(total (length) / 1024, 2) "tx KiB",
count(*) hits,
agentid,
user_agent
FROM
logs
JOIN agent ON agentid = id
--where agentid in (
--select id from agent
WHERE
user_agent LIKE '%google%'
--)
GROUP BY
id
ORDER BY
1 DESC
#+end_src
#+begin_src sqlite
SELECT
status,
count(*) as hits,
round(total (length) / 1024 / 1024, 2) as "tx MB"
-- round(total (total (length)) OVER (PARTITION BY status / 100) / 1024 / 1024, 2) AS "total tx MB"
FROM
logs
GROUP BY
"status"
order by "tx MB" desc
#+end_src
#+RESULTS[6e8a6986b881fbf55f5e1328d505eb280c73f3b0]:
| status | hits | tx MB |
|--------+---------+----------|
| 200 | 4255831 | 13463.72 |
| 404 | 422650 | 1617.18 |
| 400 | 3087 | 3.89 |
| 500 | 790 | 1.37 |
| 302 | 202 | 0.5 |
| 304 | 240 | 0.3 |
| 206 | 297 | 0.26 |
| 403 | 29 | 0.11 |
| 301 | 23 | 0.08 |
| | 90 | 0.06 |
| 421 | 7 | 0.03 |
| 408 | 3 | 0.01 |
#+begin_src sqlite
SELECT
count(*),
count(distinct ipid),
count(distinct agentid),
total (length),
request_method,
-- status,
substr(path, 0, 50)
FROM
logs
WHERE
path NOT LIKE '/ingrid/%' AND
status = 200
GROUP BY
path
ORDER BY
1 DESC
LIMIT 15
#+end_src
#+RESULTS[8469c661009196d66d76364873866923b89c8205]:
| count(*) | count(distinct ipid) | count(distinct agentid) | total (length) | request_method | substr(path, 0, 50) |
|----------+----------------------+-------------------------+----------------+----------------+---------------------------------|
| 3540 | 1779 | 425 | 22903047.0 | GET | / |
| 2171 | 1563 | 48 | 9744532.0 | GET | /robots.txt |
| 855 | 518 | 50 | 4234481.0 | GET | /favicon.ico |
| 334 | 262 | 31 | 1304042.0 | GET | /cgit.css |
| 150 | 116 | 59 | 1030405.0 | GET | /hugo-minimalist-theme/about/ |
| 144 | 1 | 1 | 854165.0 | GET | /?XDEBUG_SESSION_START=phpstorm |
| 119 | 69 | 52 | 421470.0 | GET | /cgit.png |
| 98 | 36 | 14 | 330051.0 | GET | /hugo-minimalist-theme/ |
| 46 | 31 | 15 | 165446.0 | GET | /homepage/ |
| 42 | 32 | 13 | 157112.0 | GET | /dotfiles/ |
| 39 | 34 | 14 | 255204.0 | GET | /hugo-minimalist-theme/log/ |
| 37 | 30 | 13 | 132913.0 | GET | /hugo-minimalist-theme/tree/ |
| 37 | 33 | 11 | 103541.0 | GET | /?s=name |
| 36 | 33 | 12 | 242095.0 | GET | /dotfiles/log/ |
| 35 | 32 | 14 | 124829.0 | GET | /dotfiles/tree/ |
#+begin_src sqlite
SELECT
count(*) as hits,
sum(length),
agentid,
path
--substr(path, 0, 50)
FROM
logs
WHERE
path NOT LIKE '/ingrid/%'
AND status = 404
GROUP BY
path
ORDER BY
hits desc
limit 10
#+end_src
#+RESULTS[3dbf9422fb49ecb2cae63c67615825ef1b1c30a9]:
| hits | sum(length) | agentid | path | |
|------+-------------+---------+-------------------------------------------------------------------------------------+--------------------|
| 2815 | 8768239 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.RelPermalink%20%7D%7D | |
| 1643 | 5653820 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | |
| 1564 | 5049557 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.%20%7D%7D | |
| 1218 | 4488059 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$href%20%7D%7D | |
| 1077 | 3930371 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.Permalink%20%7D%7D | |
| 924 | 3522501 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Next.URL%20%7D%7D | |
| 919 | 3500841 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Prev.URL%20%7D%7D | |
| 818 | 3088624 | 143 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20if%20ne%20.MediaType.SubType | |
| 807 | 2530428 | 143 | /hugo-minimalist-theme/plain/layouts/%7B%7B%20.Type%20%7D%7D | |
| 750 | 3000020 | 143 | /hugo-minimalist-theme/plain/layouts/taxonomy/%7B%7B%20.Name%20 | %20urlize%20%7D%7D |
#+begin_src sqlite :results value file :file fails.csv
SELECT
count(*) as hits,
sum(length),
path
--substr(path, 0, 50)
FROM
logs
WHERE
path NOT LIKE '/ingrid/%'
AND status = 404
GROUP BY
path
ORDER BY
hits desc, path
limit 550
#+end_src
#+RESULTS[28ba3f6de596e93fd80baf9516e5ae32a8239c72]:
[[file:fails.csv]]
#+begin_src sqlite
select count(*), sum(length), path,user_agent from logs
join agent on agentid=id
WHERE
path NOT LIKE '/ingrid/%'
AND status = 404
group by path, agentid
ORDER BY
sum(count(*)) over (partition by path) desc, 1 desc, path, agentid
limit 50
#+end_src
#+RESULTS[cb18bd88215d0873626e54f3a56ea4b6851c4138]:
| count(*) | sum(length) | path | user_agent | |
|----------+-------------+-------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------|
| 2805 | 8732358 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.RelPermalink%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 7 | 18923 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.RelPermalink%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 3 | 16958 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.RelPermalink%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 1629 | 5597913 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 7 | 25024 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 3 | 8467 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com) | |
| 1 | 2655 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 1 | 6609 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Mozilla/5.0 (Macintosh; PPC Mac OS X 10_7_9 rv:3.0; mt-MT) AppleWebKit/532.12.1 (KHTML, like Gecko) Version/5.0.3 Safari/532.12.1 | |
| 1 | 6609 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Mozilla/5.0 (X11; Linux i686; rv:1.9.6.20) Gecko/8855-02-04 23:24:49.455231 Firefox/3.6.10 | |
| 1 | 6543 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.URL%20%7D%7D | Opera/9.30.(X11; Linux x86_64; tn-ZA) Presto/2.9.175 Version/11.00 | |
| 1559 | 5018667 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 4 | 23869 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 1 | 7021 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 1209 | 4437078 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$href%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 5 | 27298 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$href%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 4 | 23683 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$href%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 1060 | 3867072 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.Permalink%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 12 | 41170 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.Permalink%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 3 | 8087 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.Permalink%20%7D%7D | AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com) | |
| 2 | 14042 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.Permalink%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 916 | 3490491 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Next.URL%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 7 | 29355 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Next.URL%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 1 | 2655 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Next.URL%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 912 | 3471446 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Prev.URL%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 7 | 29395 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20$pag.Prev.URL%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 817 | 3081643 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20if%20ne%20.MediaType.SubType | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 1 | 6981 | /hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20if%20ne%20.MediaType.SubType | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 798 | 2487903 | /hugo-minimalist-theme/plain/layouts/%7B%7B%20.Type%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) | |
| 5 | 26298 | /hugo-minimalist-theme/plain/layouts/%7B%7B%20.Type%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) | |
| 4 | 16227 | /hugo-minimalist-theme/plain/layouts/%7B%7B%20.Type%20%7D%7D | Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) | |
| 745 | 2977378 | /hugo-minimalist-theme/plain/layouts/taxonomy/%7B%7B%20.Name%20 | %20urlize%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) |
| 5 | 22642 | /hugo-minimalist-theme/plain/layouts/taxonomy/%7B%7B%20.Name%20 | %20urlize%20%7D%7D | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) |
| 65 | 348406 | /.env | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0 | |
| 63 | 310359 | /.env | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36 | |
| 58 | 310754 | /.env | Mozilla/5.0 (Linux; U; Android 4.4.2; en-US; HM NOTE 1W Build/KOT49H) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 UCBrowser/11.0.5.850 U3/0.8.0 Mobile Safari/534.30 | |
| 52 | 301380 | /.env | l9explore/1.2.2 | |
| 22 | 111334 | /.env | Go-http-client/1.1 | |
| 18 | 38352 | /.env | - | |
| 16 | 85808 | /.env | python-requests/2.32.3 | |
| 16 | 36940 | /.env | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 | |
| 9 | 52245 | /.env | Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr | |
| 9 | 48190 | /.env | Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50 | |
| 8 | 42840 | /.env | python-requests/2.26.0 | |
| 6 | 32086 | /.env | python-requests/2.31.0 | |
| 5 | 26773 | /.env | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3 | |
| 3 | 8719 | /.env | Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 | |
| 3 | 16043 | /.env | python-httpx/0.28.1 | |
| 3 | 17375 | /.env | Mozilla/5.0 (Linux; Android 9; Redmi Note 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.89 Mobile Safari/537.36 | |
| 3 | 17375 | /.env | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0 | |
| 3 | 16139 | /.env | python-requests/2.27.1 | |
#+begin_src sqlite :results value file :file leadfail.csv
SELECT
datetime (date, 'unixepoch'),
status,
count(*) as hits,
round(sum(length) / 1024, 2) as KB,
user_agent
FROM
logs
JOIN agent ON agent.id = logs.agentid
WHERE
path = '/hugo-minimalist-theme/plain/layouts/partials/%7B%7B%20.RelPermalink%20%7D%7D'
group by agentid
ORDER BY
referrer
#+end_src
#+RESULTS[e3501152f1ec3e7ff822a6713736ff35fb0dcb46]:
[[file:leadfail.csv]]
|
