Python GPT Crawler

A GPT knowledge file web crawler in base python. Inspired by Builder IO’s GPT Crawler and the Python interpretation by A-baoYang, I decided to create my own version which is a bit lighter on prerequisites. Despite the prior works being very accessible, installing any combination of node, chromium, and playwright can be a pain if your hosting provider does not support installing these dependencies. Therefore, I rewrote the project using only Python’s os, requests, and beautifulsoup4 libraries.