c/technology · by HelloHotel@lemmy.world · 3y

I easly got the reddit r/place data into a 4.3GIB sqlite database, where can I post it?

cross-posted to: https://lemmy.world/post/2499861

As I said, I made a lossy reformat of the database and a lossless one for 6.0 Gib (6,477,905,920). compared to ~26GIB from Reddit, where fields are almost intentionally anti-compressed to take up more room.

If there is somewhere I can host it, let me know.

also, I couldn't figure this out, do sqlite databses store any information on the creator or editor of a document?

::: spoiler why it's lossy It's missing a large table of base64 urandom technically required to recreate the document fully :::

151 points · 8 comments · view on lemmy.world

8 Comments

BitSound@lemmy.world · 60 pts · 3y (3 replies)

!datahoarder@lemmy.ml looks active and seems like a good place for it

HelloHotel@lemmy.world · 8 pts · 3y (2 replies)

thanks, how do I crosspost/ move this one?

Skyhighatrist@lemmy.ca · 15 pts · 3y (1 reply)

Using the web-ui, on this post there is an icon made up of two squares. It's right next to the star for saving the post. That's the cross post button.

HelloHotel@lemmy.world · 5 pts · 3y

thanks, made updates to the post

inspxtr@lemmy.world · 21 pts · 3y

here are a few options that I see but never actually use.

contact https://wiki.archiveteam.org/; they have some reddit related archival activities going on
more academic type and will be very helpful for researchers:
- https://zenodo.org/
- https://figshare.com/ (this I’ve used and very easy)
- https://datadryad.org/
- https://academictorrents.com/ (pushshift archives are also on here)
- https://socialmediaarchive.org/

Your data don’t seem to be massive compared to the types of data people store on there. So I don’t think it’s gonna be an issue. Plus, if you deposit your data in 1 archivist place + 1 research place, the data may be used by more people. Don’t forget about licenses btw.

EDIT: added https://socialmediaarchive.org/ to the list, just found out about that.

fiat_lux@kbin.social · 3 pts · 3y (1 reply)

Is this derived directly from the data reddit stored/created or is it a reconstruction of some kind from observing the r/place output? I'm tempted to look at the table structures but not tempted enough to download 4 gigs of it just yet.

HelloHotel@lemmy.world · 4 pts · 3y

rebuilt from reddit's offitial sources, still messing with optomizations, is adding a color definitions table worth it?

edit, YES, only 32 unique colors ever

Kalcifer@lemmy.world · 1 pts · 3y

[ removed ]

InfiniteHench@kbin.social · -9 pts · 3y

Last I heard with large projects like this, people usually upload them to DEEZ