Archiving tweets to Dreamwidth?

[ursamajor]

Yeah, so earlier this summer, I got a bug up my butt about "given how the wider internet is going to hell, how about I centralize my archives in one place?"

So I'm looking at Twitter and trying to figure out how to actually get my tweet archive into Dreamwidth, and what I want that to look like, and what tools actually exist to help with that right now. I think, ideally:

* I have something like 22,000 tweets. I would *really* like to not create 22,000 Dreamwidth entries, even if I can automate this import somehow. I will absolutely not be manually creating 22,000 DW entries!

* It would be lovely if these imported tweet posts could be posted private, or access-only.

* Ideally, the import tool would make one post per day, including multiple tweets in a post if they exist.

* Uber-ideally, the import tool knows how to handle uploading images to Dreamwidth :) and in the 5 <1% (I just looked at the media folder in my tweet archive; out of 1120 media files, 29 are .mp4s and 59 are .pngs; everything else is a .jpg) of cases where the media is an .mp4, figure out a way to fail kindly until I can get the movie files uploaded somewhere at least linkable, if not necessarily embeddable.

Have any of you done this before? Or know of any tools that do? ... or is the real answer going to be "guess what, Bear, you get to figure out how you want to translate the tweets.js format into a Dreamwidth-importable format!"? LOL. (And, um, what does that latter bit look like?) Though it looks like currently, the importer only works directly with other LJ-based sites, so I can't fling an appropriately-formatted file at it from my desktop.

Comments

[kareila]

If I were doing this (I'm not) I'd write a JSON to emailpost translation script. When posting by email, you can control access level and include image attachments.

[ursamajor]

Ooooh, emailpost is a good point, thanks for the reminder! It's how I was bringing my Instagram posts over for awhile, until that connection died because of whatever Insta did.

At least with a static offline archive, I don't have time as a pressure element anymore, and can figure this out at a reasonable pace.

[ursamajor]

... though I've thought of another complication; you can't backdate an email post on initial posting, IIRC? (Have perused the FAQ but am not seeing anything referencing this being possible.)

Which means if I do some sort of feeding-to-emailpost, I should make sure it posts privately, because: 1. I'll have to manually go in and edit the post date to reflect its correct temporal location; 2. I'll need to tick off "don't show on reading pages;" 3. Depending on what the actual post content is, I may want to adjust the security level to access or public; 4. I should also create and use a specific tag for these so that I can filter to them for the loads of manual work ahead.

(Reminder to self: I *can* set the default privacy for email posts to private for the time it takes to do the import; doing that will be easier than adding one more complication to individual posts. :) )

[kareila]

That's true, I didn't consider the backdating aspect of the problem. But at least editing dates and tags is relatively painless compared to the challenge of importing the content in the first place.

Good luck :)

[ursamajor]

Yeah, and I can keep that even lower bandwidth by just making the post-time always the same each date. Though I was hoping there was a post-date or post-time stamp keyword I’d missed ;) Ah well, it’s manageable. And thank you! :)

[kareila]

Belatedly I realize... I think it might use the date stamp from the email?

Although sending emails that appear to be from the distant past might get you added to some ISP blocklists...

[ursamajor]

Makes sense! (Though the premise in isolation also sounds like the beginning of a modern YA magical realism story, even though it is also obvious spamcraft.) And I have zero desire to do anything harmful or server-hammery, just, like, tucking about 3 million text characters and ~1000 image files (probably half of which are under 100 KB, 99% under half a meg) safely into bed/archiving at whatever pace is reasonable. :)

[cesy]

alexwlchan has done a lot of this kind of archiving, it may be worth asking them or looking at their GitHub.

[ursamajor]

Sweet! Thanks for the point in their direction :)

[pauamma]

If that doesn't pan out, I remember afuna and azurelunatic both doing something of the kind several years ago.

[ursamajor]

Thank you! I will poke them as well :)