Looking for some selfhosted webscanner/scraper

hi, i'm kinda out of ideas -maybe you know some selfhostable solution.

I want to monitor some of my pages for gdpr compliance and have reports on that.

So far i tried selenium container with k6, i tried selenium locally with some Python automation and even gave phantomJS with webXray a try. In any case sth. is missing...like selenium wont grab the thirdparty cookies, webxray code has outdated Python and so on.

You guys have any suggestion on how i could selfhost my own automated toolset to check entire webpages for all cookies and thirdpartyscripts? Ppl make changes which might have impact on the gdpr compliance and i dont want to check manually. How would you do it selfhosted?

26 points · 7 comments · view on lemmy.world

7 Comments

morras@links.hackliberty.org · 6 pts · 2y (2 replies)

The EDPS open-sourced a cookie inspection tool:

https://edps.europa.eu/edps-inspection-software_en

yournamehere@lemm.ee · 2 pts · 2y (1 reply)

oh wow...how did Google not suggest this to me? thanks mate.

edit: got i running...works like a charm. just great. thanks so much.

morras@links.hackliberty.org · 3 pts · 2y

You are welcome :)

steersman2484@sh.itjust.works · 4 pts · 2y (1 reply)

You could try Puppeteer, but I don't know if you can check third party cookies.

Also I can recommend using Apache Airflow for automation. It is like a fancy crontab with a webui and email notifications, and lots of other stuff.

yournamehere@lemm.ee · 2 pts · 2y

thanks. was thinking about airflow or n8n for automation but i still cant find a way to check the thirdpartycookies. someone wrote firefoxwebdriver could do it but you'd need to extract from the profile an stuff...nothing really worked.

0xD@infosec.pub · 2 pts · 2y (1 reply)

Check out nuclei or maybe greenbone.

yournamehere@lemm.ee · 2 pts · 2y

those look exciting! thank you very much.