My Social Portal

Kyanite Facebook Archive Browser v1.0.0 Release

2022-04-09T16:00:00+00:00

This is the first 1.0 release of the Kyanite Facebook Archive Browser. There were some minor bugfixes since the last beta release. You can find the latest release at the Downloads Page and the source code at this GitLab repository.

Fediverse Archive Browser v1.0.0 Release

2022-04-08T16:00:00+00:00

Earlier this year I released the first version of the “Friendica Archive Browser”. In the intervening months there has been some churn on some big Diaspora instances. While I haven’t used Diaspora much since 2019 I do federate to Diaspora instances from Friendica. I also still have a soft place in my heart for my time developing on the Diaspora code base. Because of that I decided to make the “Friendica Archive Browser” the “Fediverse Archive Browser”. It now supports reading in the archives generated by my Friendica Archiver tool for Friendica or the archives generated from Diaspora’s built-in data exporter tool. They share identical features between both archive versions.

Those features are:

A timeline view of all posts with any federated comments that it knows about
Post’s are rendered with HTML and any known likes/dislikes
Any photos that are attached to the posts are rendered with the posts
Links in the post will generate OpenGraph previews and can be clicked on to open in the browser
Can filter the timeline based on:
- Text in thep ost
- Date ranges
- Only with comments
- Only with images
- Only with videos
Shows “orphaned comments” which are comments you made to posts which aren’t known in your archive (likely other user’s posts or comments)
Can click on the link to go to the original post on the server, if available
Can copy the entire post content to the clipboard for use in another program
Can export post photos to the disk in a more convenient location
Shows the statistics of your posts including :frequency per day of week, month of year, or year, a heatmap per year, and who posts most. Posts statistics can also be filtered by the same terms as the timeline

You can find the latest release at the Downloads Page and the source code at this GitLab repository.

Friendica Archiver v1.2.0 Update

2022-04-04T02:00:00+00:00

There is a small tweak to the Friendica Archiver to fix the timeline pulling REST service call. Although it worked at one point it has stopped working in a recent release. Looking at the original code I’m not sure how it worked in the first place but it has now been fixed. You can find the latest release at the Downloads Page and the source code at this GitLab repository.

Friendica Archive Building and Browsing

2022-01-24T02:00:00+00:00

Due to various circumstances I got sidetracked from getting the Kyanite Facebook Archive Browser out the door this month as intended. Part of that was because I found out that my Friendica instance was shutting down at the end of the month. Friendica is one of many projects on “The Fediverse” alternatives to the big-tech owned platforms (Facebook, Twitter, Instagram, etc.). While it has many advantages one of the disadvantages is that most of the servers that you would join through are run by one or two people. That means when they decide to pull up stakes you have to figure out what to do with your fediverse digital existence too. That’s where I found myself at the beginning of the month. My answer was to take my new found skills with Dart and Flutter and the Kyanite code base to create an archival and archive viewing system for Friendica. The archiver and browser can be found at this link

Generating an Archive

First lets talk about generating the archive. The archive generator is a command line tool that uses the Friendica API to query for your timeline. You provide your username and password credentials (sorry OpenID logins don’t work right now), the name of your server, and the directory the archive will be put into. It then goes about archiving your entire timeline from the most recent to very first post and comment. So it will be something like:

./friendica_archiver --username user1 --server-name myfriendicaserver.com --archive-folder ~/Downloads/FriendicaArchive

Along with the posts and comments it will also by default download any attached images for the post as well. That way you have a complete archive of all of your personal content from the server. The default also executes until it runs to the end of the timeline or until it hits 20 billion post/comments. The user has flags they can use to change that behavior, including being able to restart it mid-way through.

For an active account I recommend running it with the standard defaults and watching for when you start seeing duplicate posts coming up in the prompts. Because Friendica allows changing the most recent ones may not be literal duplicates but updates. However if you know you didn’t edit posts or comments back far enough you know you are just doing unnecessary duplicate queries. In the future the logic may get more complicated to automatically stop when it “hit the already archived zone”.

When this is completed you will now have a folder with a structure of:

├── images
├── images.json
└── postsAndComments.json

The images folder contains all of the archived images. The images.json file maps the image file names, which are UUIDs, to URLs that they came from. The postsAndComments.json is an archive of the raw JSON responses from the server. This top level folder is what you will point your archive viewer to for browsing.

Browsing your Archive

With your archive ready to go you can download the Browser. On initial startup it asks you to choose the folder to load, which it will remember on startup the next time. You can jump to your posts view:

This shows your post text, actually the Friendica-generated HTML data which will include previews, some of the tweet structure etc. For posts or comments with photos there is also a photo gallery option so you can cycle through them. In most cases the HTML preview has the images as well which creates a duplicate image view unfortunately. It will also include how many likes/dislikes it has. If you hover over the like or dislike icon you can see who liked/disliked it as well. It will also show any comments you made on that post as well, although it will not show comments by other users. Lastly there is a copy button for copying the text and data to the clipboard and a link button to launch a browser with the original post or comment location. Like with Kyanite there is a full search capability to allow you to easily find content. Below is a screenshot showing such a search and what embedded comments look like:

For comments that are on posts other than your own or on comments on your post there is an “Orphaned Comments” view. All of the controls are the same as with posts and comments on the comments page, including the ability to launch the browser to the original comment on your server:

One thing I like to do is look at my posting habits over time. Since there is also data about how people interact with your posts I also added the ability to see who the top interacting accounts are. Below are two screenshots showing the gamut of statistics available to you. In the case of the “top interactors” clicking on the name launches their profile page:

Conclusion

With these two tools it is possible to archive your post and comment data from your Friendica instance for your personal use. With the browser it is also possible to have searching capabilities which are far better than what one is capable of doing with server side search and browsing.

Kyanite Beta 3: Now With Link Previews, Stats, and More!

2021-12-20T14:00:00+00:00

Time flies especially as holidays are coming through, doesn’t it? A couple weeks ago I posted about the Kyanite Beta 2 release. It was about a month ago that I introduced the first beta of Kyanite. Today the third beta release. This is the one that I consider to be feature complete for a 1.0 release. Coming out of the second beta the two big features I wanted that were missing were link previews and the ability to look at posting statistics. There are some other features, tweaks, and bug fixes in this release as well. You can get the latest version for download here.

Link Preview

In the previous two releases links in posts were simply rendered as the URL of the link. This is obviously different than what we expect on modern user interfaces. What we expect nowadays is one of those little tiles with a preview image if it’s available, a headline, and maybe some body text. We then click on that which would bring us to the link. The magic of that most of the time is a standard called OpenGraph. This is some additional information that a website puts at the top of the HTML file that the link points to that gives hints to how an application could build a preview for its purposes. In this release Kyanite now reaches out to the server to generate that query and build that tile if it is available. If it isn’t available then it simply shows the URL like before. This will happen for links to other social media system posts like Twitter tweets because they actually don’t have a means for generating that data any longer. How do those tweets or Instagram links render in websites then? There is a whole JavaScript library that those sites link against to have it dynamically build that. Kyanite doesn’t have any of that so can’t use it. For links to regular websites though it works well. Below is a screenshot showing what the link preview looks like both when it can’t find it (the Twitter link) and when it can (the foodie link):

Saved Links/Items Screen

Facebook has a mechanism for saving posts or links that you like for future browsing. That data can be exported with your archive. It provides basic information about the link, when you saved it, and the hyperlink if it is to an external system. Most of the data is lost when the link is to another Facebook post or media item. So unfortunately for those things the best you’ll get back is something like “You saved John Smith’s post” and the time that you did. For external links though you’ll get more information and using the new link preview feature a standard preview if it is available too:

Stats Screen

By far one of my favorite things is to quantify things in my life. It’s an engineer thing. I really wanted to do something like that with my Facebook data long before I got onto the whole “Delete Facebook” thing. The new Stats Screen will let you do just that. The stat screen has three components. The first component is a bar graph which displays the count of all of your posts by either the day of the week, the month of the year, or the year. This lets you see longer trends like how your posts per year have changed over time or if you more often posted on the weekends, for example. For example below is the graph of all of my Facebook posts per year and by day. As you can see I was pretty consistent with the days of the week I posted, but slightly less on the weekends. You can also see how exponential my posting frequency got every year, until I decided to try to start getting off Facebook in 2018. I still look forward to seeing my 2022 stats which will hopefully be nearly zero (just enough to post Kyanite and alternate social network posts).

The second component is a “heat map graph” which divides the year into 365 (366 in leap year) tiles laid out vertically by day of the week and horizontally by month. I don’t know if GitHub invented the concept but they certainly popularized it. It is a way to quickly see when you are most active over time. The color of each block is determined by how many of whatever item you are looking at statistics on you have. The darker the color the more active you are. If you hover over one of the tiles it will show the number of items on that date. I posted two of my heat maps below, one for 2019 and one for 2021:

Heading into 2019 I was going to try to double down on getting off of Facebook, which I had some success on in bouts through 2018. You can see that by the unshaded early part of the the graph. However I quickly got back on and was often doing at least 1-5 posts per day. Compare that to 2021. In 2020 I again had some success with getting off of Facebook for larger stretches of time but the beginning of 2021 I got back on with a vengeance. In the beginning of January I have many days with more than 20 posts per day! On my busiest day I had 26 posts! That was across groups and my timeline but still that’s very verbose. Then it died down a bit until by July I was off it for good again.

The third component is a “top words” table. This filters out common useless words (like “a”, “the”, “I”, “you”, etc) and then tabulates over all of the text what the top words are. It defaults to the top ten but it’s possible for the user to select up to the top 100 posts. For example below are the top words in my posts for all of 2011:

Notice I said those were my top words for 2011. By using the same filter controls that exist on the other screens you can use the stats tool for getting in-depth information about your data.

Bug Fixes

There were some bugs or less than ideal behaviors that were tweaked in this release as well. One problem, which was made a lot worse by trying to load link previews, was the drawing of every item in the timeline when a user dragged from the top to the bottom of the timeline. While there was some jumping going on it was causing a lot of work and janky behavior. I implemented something to defer loading when you blast by a row but it was still creating bad behavior. I also tried limiting how fast one could drag the scrollbar cursor but it just didn’t cut it down. So scrollbars are turned off now and scrolling is now done through flicking, dragging, or scroll wheels only.

On the filter panels the date start/time fields looked like they were editable directly. They were not. Editing needs to be done inside the date control itself. Changes made in the text field never made any changes. The field is now properly read only.

The navigation bar on the left was also starting to get too long as we add more screens. Rather than make the application minimum size ever larger that is now a scrollable view.

As always if you have bugs or feedback you’d like to make on the project, feel free to use the Issue Tracker or send an email to this email address.

Kyanite Beta 2: Now With Maps and More!

2021-12-07T19:30:00+00:00

A few weeks ago I introduced the first beta of the Kyanite Facebook archive viewer. Today I’m announcing the second beta release complete with a bunch of bug fixes and new features. My favorite new feature is the map view for browsing/filtering posts that have geospatial information. You can get the latest version for download here.

Map View

One of the things I often did with Facebook was try to remember a place I visited to eat or visit. For example a friend asked me for a restaurant recommendation in a major city I went to many times. Unfortunately the name of the restaurant was a bit too generic to triangulate it via a regular cold internet search. But I remember that I took a picture at that restaurant and posted it to Facebook…at some point. Facebook search is useless for trying to find it via text search but they do have a feature that let me go to a map and start zooming in and around until the data jumped out at me. While Kyanite has much better text search than Facebook it would still be a very cumbersome process. That’s why I’m so excited for the new Map View tab:

The map view screen is divided into two parts: the map on the top and a list of selected posts on the bottom. There is a slider that that lets the user decide how much of the view they’d like to make the map use at the expense of the posts timeline. The posts timeline itself is identical to the one found on the main Posts tab except that it is only showing the posts that correspond with the summary billboards on the map. The initial view is zoomed out globally but centered on about the average mid point of all your geographic data. For world travelers this will be less interesting but for those that are mostly in one area, like in this example dataset where the posts are all in the United States, it will center the map approximately where you are, as shown below:

Billboards are put on the map geographically located and with all of the posts which are covered by it. As you pan and zoom the map it is being continually re-evaluated. If at any point you want to see the actual posts that are in a billboard you simply left-click on it. Right clicking anywhere else on the map clears that selection. Back to my example above, I can start zooming into the are and slowly see how the posts evolved. Because this is backed by standard map systems the map will zoom down with details all the way down to street view.

Image Carousel

A common use case is that you open a post that has multiple images, or you are in a photo album, and you want to see the full sized image. What happens when you want to see the previous or the next image? In Beta 1 the user would have to go back a screen and click on the next image. In Beta 2 we now have forward/backward arrows that can be used for cycling through the images in the post/album:

The user therefore can cycle through the images seamlessly and like before if they find one they like they can export it with the save button. Along with the on-screen buttons the carousel is also usable with the forward/backward arrow keys. Likewise hitting the Escape key will dismiss the window like the go-back arrow on the header bar.

Post/Comments/Conversations Copy to Clipboard

In Beta 1 users could select text to copy/paste it to the clipboard. However that would be cumbersome to get all of the post details. In Beta 2 we added a “Copy” button to each post/comment/conversation line. This button generates a human readable text only version of the post that is added to the system’s clipboard. In there will be things like the title, the post/comment/conversation text, the local path to the image files, the hyperlinks, etc. When the text is finished being copied to the clipboard Kyanite puts up a snackbar status to let you know it is completed, which will be nearly instantaneous. Below is an example of the plain text that is copied to the clipboard from a post:

Title: 
Creation At: December 31 2015 5:11 PM
Text:
"We must not say every mistake is a foolish one." --Cicero


Photos and Videos:
/home/hankg/tmp/test_archive/posts/media/TimelinePhotos_aoeu/69f5010332334ba6a5773e0ced547a1b.jpg
/home/hankg/tmp/test_archive/posts/media/TimelinePhotos_aoeu/423fb91b4e664192b179a0163aa85c92.jpg
/home/hankg/tmp/test_archive/posts/media/TimelinePhotos_aoeu/ed2cfdc34cfd425e9b5ce7d94a0da38c.jpg
Name: Jackson, Tennessee
Latitude: 35.6145169
Longitude: -88.81394689999999

Bug Fixes

During early testing of Beta 1 some bugs were found that have been squashed. There was a pretty serious memory leak that has been resolved. There was also a major bug where on certain platforms the default video player was being set to nothing thus causing an exception that would crash the app. The setting it to the wrong setting is fixed and the error handling is more pervasive to prevent it from crashing and to report it to a log. Lastly there were some capitalization inconsistencies in the user interface which have been standardized. As always if you have bugs or feedback you’d like to make on the project, feel free to use the Issue Tracker or send an email to this email address.

Notes

The datasets used for the screenshots are created by a random generation program with three sources of data. Cities data is from this GitHub Gist. The quotes which are used for text body are from this Database Quotes JSON Project. Image data is from the Indoor Scene Recognition ML training dataset by MIT.

Fixing Facebook’s Borked File Encoding

2021-11-23T15:30:00+00:00

While working on the Kryanite Facebook archive reader I ran into a frequent problem of garbled text. Ninety-nine percent of the time everything would be fine but then I’d see a string of weirdly out of place characters. I had assumed that perhaps the default encoding picked by the Dart file reader was to blame. Perhaps it should have been UTF-16 instead of UTF-8, or something along those lines. Experimentation didn’t help the problem. It turns out the problem is one of Facebook not properly encoding their files. With the help of another blogger who discovered the problem and fixed it I was able to create Dart code which fixes it for my uses as well. Here is the snippet. Read below the fold for more details.

As luck would have it, <sarcasm>or maybe my superior DDG search skills</sarcasm>, I found another developer who ran into the exact same problem. Paweł Krawczyk had attempted to process his Facebook archive file which had lots of Polish diacritical characters. All of them were coming up wrong. In this blog post he documents exactly where Facebook engineers went wrong:

The story here seems to be that Facebook programmers mixed up the concepts of Unicode encoding and escape sequences, probably while implementing their own ad-hoc serializer. What Facebook outputs is binary representation of UTF-8 encoded Unicode character U+105 (LATIN SMALL LETTER A WITH OGONEK) but confusingly prefixed with \u.

The prefix is confusing, because it implies it’s an escape sequence referring to a single Unicode character U+C4 (LATIN CAPITAL LETTER A WITH DIAERESIS) followed by another single Unicode character U+85 (NEXT LINE).

This kind of “Unicode characters pretending to be bytes” is possible for characters U+0000 up to U+00FF, or simply the basic ASCII range, because their UTF-8 encoding is identical to their ASCII counterparts. Nonetheless, it’s used against its purpose here and clearly confusing for both humans and JSON decoders.

Algorithm

Along with diagnosing the problem he also posted Python code which read the Facebook JSON files as raw bytes, properly encoded the mis-encoded Unicode elements, and returned the proper file. I needed this in Dart for my Flutter application. I therefore decided to create a Dart implementation.

The premise of the algorithm he presented is pretty straight forward:

Load the text file up as raw bytes not as Unicode characters
Read in the data byte for byte looking for a character sequence \u
- If there is no \u sequence simply write the character to the buffer
- Else read the next four characters which have encoded the Unicode character position index and store in a byte buffer
- Continue reading and parsing the next six characters until you no longer have a \u sequence, dumping each result into the byte buffer
- When you’ve reached the end of the sequence then you convert the byte buffer into Unicode character(s) and write that to the string buffer.

Practically speaking what sorts of codes are we looking at? Well Krawczyk’s example from his post was the Polish diacritical “ą” which was written in the file by Facebook as \u00c4\u0085. In my own archive I ran into a heart Unicode character “❤” written as \u00e2\u009d\u00a4\u00ef\u00b8\u008f. You can see why it is important to not assume that there are just two \u sequences defining the character, it’s just not the case. A string of emojis are similarly better interpreted into Unicode by just reading the sequence in entirely. Therefore five star emojis one right after the other “★★★★★” was written by Facebook as \u00e2\u0098\u0085\u00e2\u0098\u0085\u00e2\u0098\u0085\u00e2\u0098\u0085\u00e2\u0098\u0085.

Dart Implementation

My first implementation used the String.fromCharCode to read each of the numbers after the \u escape sequence. Once those were built up into the index string I could then use the int.parse method with the radix: 16 option to convert that into a byte value for that element which could then be added to the byte array that I was building up. That felt like a lot of byte-to-string-to-byte conversions to be doing. It worked but it felt like it could be a lot faster if I just stayed in byte space. The final implementation directly converts their ASCII/Unicode string index into the proper byte index. For example ASCII/Unicode “character 0” is index 48. Therefore if I get a byte value of 48 I want to return 0 by subtracting 48. It needs some additional logic to encode the A-F hex values but the premise is the same. With all four hex values for each element of the index we can shift them into the proper place to build up the real byte value of the Unicode index that is captured by the \u sequence. That is then stored in a byte array we keep building until the end of the \u sequences which are then properly encoded by the standard UTF decoder. Below shows the conversion process for each \u sequence (where data is where we are in the original data byte array):

final chars = data
  .sublist(i + 2, i + 6)
  .map((e) => e < 97 ? e - 48 : e - 87)
  .toList(growable: false);
final byte =
  (chars[0] << 12) + (chars[1] << 8) + (chars[2] << 4) + (chars[3]);
byteBuffer.add(byte);

…and this is what the whole loop over the data looks like:

while (i < data.length - 1) {
  if (data[i] == leadingSlash && data[i + 1] == leadingU) {
	final byteBuffer = <int>[];
	while(i < data.length -1 && data[i] == leadingSlash && data[i + 1] == leadingU) {
	  final chars = data
		  .sublist(i + 2, i + 6)
		  .map((e) => e < 97 ? e - 48 : e - 87)
		  .toList(growable: false);
	  final byte =
		  (chars[0] << 12) + (chars[1] << 8) + (chars[2] << 4) + (chars[3]);
	  byteBuffer.add(byte);
	  i += 6;
	}
	final unicodeChar = utf8.decode(byteBuffer);
	buffer.write(unicodeChar);
  } else {
	buffer.writeCharCode(data[i]);
	i++;
  }
}

Performance

All of this reading does create some performance hit but it is not egregious. On smaller files, less than 100KB, the difference is literally in the noise but on larger files it can be up to 10% slower than a straight file read. The original version with the string building/converting of each byte segment was 2-3 times slower than the present byte-only version though. Because reading even large JSON files is still far less than a second it’s technically not that big of a deal but I still appreciate the extra efficiency enough to keep this version.

As posted above the full code with an example driver main program can be found in this GitLab snippet. In that incarnation reading a Facebook archive file with proper encoding is as simple as:

final data = await File(path).readFacebookEncodedFileAsString();

Harnessing Your Facebook Data With the the Kyanite Facebook Archive Viewer

2021-11-17T17:00:00+00:00

It is no secret that Facebook has been expertly doing the vendor lock play to expand and retain their user base. If all of your family and friend photos, event information, thoughts on current events, etc. are solely in their database then you have no place else to go once you finally join and start using it. It becomes more reinforcing the longer you are there and the more friends that join and do the same thing. In recent years though due to a combination of legislative action, public pressure, and fear of regulation they started allowing users to export all of their data from their system. It’s kind of hidden and can be a little confusing but once it’s done you can get a giant ZIP file archive of all your data. While it meets the letter of what was requested the fact is that that data is not digestible to the average user. To anyone that wants to actually use their Facebook data they still are stuck doing it through their system.

This post is about a preview release of a new tool I am working on to actually be able to use your Facebook Archive, which is called Kyanite. This tool is available for Windows, Mac, and Linux desktops. It will allow you to view, search, and extract data from your archive in a human digestible way. In same cases, like with search, it’s even more effective than the built in tools Facebook provides. Below is a quick tour of the product and how you can start using it. You can go right to the product page to start using it today. Also check out this post on how to get your Facebook Archive. Now on to the details of the Kyanite…

Viewing Content

A Facebook archive contains lots of information that you expect from your years of using Facebook. You have all the posts that you wrote. You have Facebook Messenger conversations you’ve interacted with. You have all the photos and videos you’ve shared. You also have your list of friends. There are literally dozens of categories of data available to download, with some being more useful to others. Below is a screenshot of the application when it first starts up pointed to your archive folder:

In the main part of the window we have a latest-first timeline view of all of your posts, just like one would within the Facebook app or website. If there are images or videos associated with the post those are rendered in there as well. We can also see that when there are links those are available too. If you click on the link that will open it up in a new web browser window/tab on your computer. If you click on an image it brings up a detailed view of the image, complete with an option to export this to another location of your choice on the computer:

In the main view to the left you have a toolbar which gives us access to the rest of the data available from your archive. The types of data available currently fall into the following categories:

Posts, which are all your posts stored in this archive.
Chats, which are the archive of all of your Facebook Messenger chats
Photos, which are all of the images you’ve ever posted into albums or posts from this account
Videos, which are all of the videos you’ve ever posted into albums or posts from this account
Events, which are all of the events that you created, attended, or were invited to
Friends, a list of all of the friendship connections, invitations, etc. you received
Comments, all of the comments you’ve ever made from this account.
Settings, which are configurations for this application including the archive folder that you are viewing

By clicking on the icons in the toolbar you will toggle through the various kinds of data in your system. For example here is what the view of the photo galleries looks like:

By clicking on any one of these gallery icons you are then brought into the gallery where you can explore further:

Just like with images in posts (or anywhere else) clicking on an image brings up the detailed view for the image which allows saving. Looking at the conversations view, we can see the archive of all of the Facebook Messenger conversations in your archive:

Clicking on any of the conversations on the left brings up the conversations panel with all of the media content, links, etc. as well as timestamp information on hover.

Searching and Extracting Content

One of the hardest things about getting Facebook data, both inside and out, is finding it in the first place. When I was a regular user I’d spend a substantial amount of time trying to find the posts I was looking for. Each of the screens has a search panel that allows for easily pairing down what is visible in the timeline/gallery/etc. Below is a screenshot of that in use:

Toggling the visibility of the search fields is done by clicking the spyglass button on the lower right. This brings up the floating control panel where we can select filtering criteria that includes date ranges, free text search, and/or whether it has images or videos. In the above screenshot I’ve paired down posts to any that have images and include “bread” in the post text. At the bottom of the search bar it shows how many of the items, in this case posts, are visible versus the total number.

Once you’ve found the posts now you want to get the data out for that post. As stated previous clicking through to the image details allows one to save that out to a new folder of your choosing. Clicking on a link brings it up in a new browser where you can save it as needed. Videos open in your a local media player (like Windows Media Player, QuickTime, or VLC) where you can work with as you wish. Lastly all the text is of the posts, comments, etc. is selectable. You can easily highlight then right-click to copy it to the clipboard for use elsewhere. In this way you not only have an easy way to view and find your posts but to easily extract the data as well.

How To Install & Use

Currently the application is still in active development. That means I’m working on adding new features and fixing bugs that are coming up the more it is being used. Because of this I am not putting it into app stores quite yet. Instead I’m providing direct downloads from the GitLab project site. The instructions for how to download and run the program can be found here under the Installation section of the page.

In order to use this you’ll need a copy of your Facebook archive. While Facebook hides it a bit the process is pretty straight forward. Here is a link to step by step instructions showing how to get all your data out of Facebook. Be warned that it can take up to several days for the process to complete if you’ve been on the site for a long time.

How Can You Help

If you are still reading this post then hopefully that means you are still interested in this tool. If so you may be asking how you can help. By far the best thing right now is for more people to run the software. This will allow the software to really be put through its paces. Currently I’m doing testing with my own data on the main development platform, Linux, as well as lighter testing on Mac and Windows. However There are bound to be cases that my data set is not finding. Also I’ve been writing it so far with how I want to use it. Having feedback on how other people are or wanting to use it can be very helpful. It would be a huge help if as you use the tool you can submit feedback on the project’s GitLab Issues Page. Bug reports and feature requests are of course very helpful but even just casual observations would be very helpful. Anyone with a GitLab.com login can submit issues. If you don’t have an account, GitLab supports single sign on services with Google, GitHub, Twitter, BitBucket, and SalesForce. It also allows people to create free accounts with an email address. If you don’t want to create a GitLab account or connect it into an SSO provider to create issues you can instead send it via email at this email address. All users can browse the issue tracker without a GitLab account though.

Future Steps

There are a few more major features that I want to incorporate into the tool before I consider it 1.0 status. Some of them are little tweaks while others are larger usability things. Getting those implemented are the biggest short term goal until bug reports/feature requests/feedback start flowing in. After that shake out period I intend to publish the application in the various platform app stores while still making it available for direct download on the GitLab page. The timeline for these things is of course unknowable but the hope is that I could get a 1.0 status app store version out by the end of the year or beginning of next year.

Getting Your Facebook Archive

2021-11-17T16:45:00+00:00

Facebook actually has a mechanism for users to download their entire history from their site. It’s not exactly on the front page but not entirely hidden. If you don’t download it correctly it’s hopelessly difficult to make it useful. When you do do it correctly it is still difficult to use but with a tool like my Kyanite archive viewer it is possible to finally get access to your Facebook data in a meaningful way. This post is about how one can properly generate this archive within Facebook to use with this tool.

What follows is the step by step process process for requesting your archive through the browser and then downloading it and preparing it for use in Kyanite.

On the main Facebook screen there is a menu bar up top. To the far right is a circular button with a down triangle which calls up the “Account” menu. Click on that to drop it down. In there you will want to select the “Settings & Privacy” menu option. That brings up the “Settings & Privacy” sub-menu over top of it, shown below.

Click on the “Settings” option which brings us to the setting screen. This screen is where all of your account settings are configured. We want to click the “Your Facebook Information” button near the top left of the Settings menu options which brings up that screen. A screenshot of the “Your Facebook Information” settings is shown below.

Near the middle is the “Download Your Information” option. To get to it click the “View” hyperlink at the far right of that row. This brings up the Downloads Screen, as shown:

By default all of “Your Activity Across Facebook” will be downloaded. Unless you are really sure you don’t want parts of that being exported you should leave these defaults. The settings at the top however we are going to want to configure. For the “Format” field we will want to toggle it to JSON. The JSON format is the one that Kyanite uses. For the “Media Quality” field you probably want this to be high as well. This will ensure you get the highest quality versions of the images and videos that you’ve uploaded to Facebook over the years. Lastly we look at the date selection. A screenshot of the date selector expanded, with the quality and format already configured, is below:

Like with with the other parameters if what you are looking for is to get all of your data out of Facebook you should select the “All Time” option. Then click “Update”. You should see the date ranges update with a selection that comprises the entire time you have been on Facebook. If you are looking for only a subset of data you can of course choose other options. Once you have configured these things the “Create File” button underneath should then be enabled. Click on it and it will begin generating your archive file. You should receive an email from Facebook confirming that you have started the archive process. The download page also contains information about the status of your download. A screenshot of what it looks like as that process unfolds is below:

This blue information banner in the middle stating that “A copy of your information is being created” will persist until your archive is generated. Depending on how long you’ve been on Facebook and how much data you’ve uploaded this process can take from several hours to several days. This can be many many GB of data. If it is over a certain size, approximately 4 GB, Facebook starts breaking them up into mulitple ZIP files. Let’s look at my account as an example. It took three days to generate the archive of all my data which goes back to 2009. It generated two ZIP files of 3.5 GB and 2.7 GB for a total of 6.2 GB. zipped with almost 9000 posts, 4000 photos, 13 movies, and almost 2000 Facebook Messenger Conversations. More detailed information as well as a history of available copies that you’ve previously generated can be found by clicking the “Available Copies” tab. A screenshot of that is below before your archives are generated:

As we can see it provides us with the same information banner about it creating an archive. It also shows some summary data about the time frame the archive covers, the format type, media quality, etc. Once your archive is ready you should receive an email stating that it is ready to be downloaded and will be available for a few days. Below is a screenshot of what my “Available Copies” screen looks like with both small and large archives that I’ve downloaded for testing:

Because this has a lot of your private data Facebook only keeps the archive around for about a week or so. If you miss the window to download it you’ll have to generate it again. Once it is ready the “Available Copies” screen looks like below.

From here click on the “Download” button for your archive to begin the download process. Again this archive could be very very large depending on how many photos and videos you’ve posted to posts, comments, conversations, pages you manage, etc. When selecting a download location you should make sure you have enough room to store it and the unzipped copy from the next step. You will need to do this for each file if your archive is broken up into multiple files.

Once your download has finished you now have ZIP file(s) with all of the Facebook data that you can possible extract. The Kryanite tool is looking for the unzipped directory version of this file. Simply unzip the archive like you would any other zip file on your operating system. On macOS this is as simple as double-clicking on the archive to unzip it in place. On Windows you can right-click on the ZIP archive icon and choose the “Extract All..” option. On Linux you can similarly choose the right-click option and choose “Extract Here” or “Extract To…”. If you have multiple ZIP files you’ll need to unzip it into a common sub-directory with only those archives in it. Then you can point Kryanite at that top level directory. An example of what my two archive folder structure looks like is in the screenshot below:

When I run Kyranite I point it to the “facebook_archive” top folder and it does the rest. If you only have one ZIP file and thus one uncompressed folder you can just point it to that directly.

See this post for more details on Kyanite.