Message from @Spicy
Discord ID: 453299600678453268
```
cat /var/log/pornspider/videos.log | grep 'forcing girl to install gentoo' | grep 'Os.GNULinux' | wc -l
1
```
<:blobhyperthink:427568506905559040>
By the way I've been meaning to ask but do you incorporate any C# in your pornspider?
yes, the codebase is roughly
50 % C++
25 % C#
15 % Haskell
10 % HTML, CSS, Javascript
<:googlethink:328536447219138561>
How do you combine them
Not asking about last part but C# and C++?
Pornspider is mostly a unix approach
A ton of independent services
And they share a data source
Like, there's a shared library which has a database connection and a basic mapping layer
So wait, C++ is doing the scraping or what?
Exactly, C++ is mostly scraping the websites and also partially building search indexes
Huh, I thought you were scraping in C#, and was asking since I tried making a scraper before in C#
what exactly is this pornspider doing
But aside from linkin shit I didn't exactly manage how to get data to work properly
@Spicy it indexes a lot of porn websites and provides a cross site search over all of them
I made that while I was temporarily jobless because I really had nothing to do
how much effort did it take to not get banned everywhere
probably just regular logs
Well that's the Haskell part. My girlfriend is doing a lot of data analysis in haskell. Removes illegal content, keeps the huge amount of data consistent
Just search logs, yes. I don't even log IPs
and its just a web interface?
Site interface?
web
Ah the only GUI is web
pretty neat
But the web interface is pretty basic
I'm not that much of a web dev
fuck it
Time to try and make my scrape shit work
Why? You can just contribute to mine and we can make pornspider great
Why make your own?
Because I want to scrape other things
Mainly Boorus and sometime 4chin
Imagine a desktop client for 4chin
Since too many times I see few nice images in a thread
Save it, only for it to pop up to 500 and go into archive at fireden
And browsing fireden images is reaaaaaaally ***fun***