A Raspberry Pi 4, top, and Pi 5, bottom.

NixOS on a Pi

Byrichard
Reading Time: 2 minutes

For a few weeks I have tried to install NixOS on a raspberry Pi without much success. I have finally managed to get NixOS to work with a GUI/Desktop environment. I kept getting stuck at the command prompt but in the end I found a blog post that helped me.

What I Struggled With

The first thing I struggled with was finding a version of NixOS that played nicely with Pi’s processor. With some distributions you download it, install and it’s easy. With NixOS it took some searching to find the right ISO image that I also had to download a tool to unzip.

The Command Prompt That Updates

I think I could have got NixOS to work much sooner but I was confused by the command prompt. Every few seconds it gives messages about bluetooth devices and more. It made me think that the install had failed and that there was an error. I also had to learn to use the passwd command at the first prompt to set the password for the nixos user before moving forward.

The Other Challenge

When you install NixOS on an HP laptop, or other device you can download the standard ISO, make a bootable USB stick, and then install NixOS from NixOS. With the Pi you can’t do that. You have to do some things in the command line. The key step is to set the password, find the IP address and then SSH from xour usual machine. Once that is done it is easy to experiment with setting up NixOS.

And Finally

Now that I have NixOS up and running on a Pi I can experiment with the OS. When I update the configuration file I can keep a copy of it. Any time I install Nix on a system I can re-use the config file and replicate a setup within minutes, rather than hours. I set it up on a Pi4 with 2gb of ram but I could move it to a Pi4 with 4 or 8gb of ram, when I see that it requires more ram. Now that I am at this point I can experiment with more flexibility.

Similar Posts

Playing with Hard Drives

Playing with Hard Drives

Byrichard

One of the easiest things to do is to buy a hard drive, and over a period of time fill it, and then get another drive, and a third, and a fourth, and a fifth. People will go from a small drive, to a larger drive, and a larger drive after that and backup plenty of the files across three to five drives. The result is that you have terabytes of storage, and some files are “backed up” on every drive, whereas others are precariously stored on just one drive. The result is that if you do want to consolidate that data to a single volume you need a larger and larger drive.

Over the years I have worked as a Media Asset Manager and I have a technique for finding duplicates, and freeing HD space, whilst at the same time ensuring that data is still backed up to at least two volumes. if Not more. As I deal with video I do this by organising every project by year-month-day-country-subject-individual where “individual” is the person working on that project.

Consolidate Media Assets onto a Single Volume

The idea is that as you copy files and folders from a collection of external hard drives you consolidate everything onto a NAS or other form of storage server. If the date is not provided I use the file names and project title and related information to situate it in with time.

Iterate

At first the process is easy, because every project and set of files are unique. The beauty of the year-month-day folder structure is that as you progress you spot that two or three folders have the same name, and that’s when you go and check file dates to see which is the most recent.

Fine tuning

One nuance is that if a file is named final final, and another is named I hope this is the final, and a third is named final final final final, then you change the name. Each file is renamed according to it’s creation date. This allows you to see within seconds which file was last exported, for example, and helps order chaos.

Backing up the Consolidated Files

In an ideal world you would have as many terabytes or petabytes of storage for the NAS but this isn’t always possible. What I did instead was to consolidate the data from the smaller external drives onto the central storage and then store them, as they were, as a backup in case the NAS fails. This isn’t ideal but it’s a good compromise when budget is limited.

Imagine that you have three or four one terabyte drives and they’re all getting full. If you have a two or three terabyte file you may be able to backup one or two drives, but run out of space for the two others. The solution is to get a four or five terabyte hard disk. With this you can dump the four smaller drives, and then start sorting projects and media assets by year-month-day but also by project name. With this work flow you can identify duplicates with ease.

Reducing the Number of Drives Plugged In

The aim of dumping drive A, B, C and D to drive E is that you get rid of the need to have four drives plugged in at once. You can have a projects folder for 2024. Each drive is in its own folder. You open two finder windows and then you can start organising your projects by year-month-day-project-name. You move the folder for the project from the Drive A folder to the year folder. When you finish bringing in drive A you repeat the process with drive B, Drive C and Drive D. When you spot duplicates you delete the duplicate files.

Removing Duplicatew

As you progress you go from having four terabytes of files to having three terabyte, down to two and a half, and so on. As duplicates are eliminated you regain that space so your four terabyte drive is not filled instantly. Ideally you would have the four terabyte drive mirrored on another four terabyte drive. If space is tight then you can re-use the small drives. 2024 can then be split across two or three drives. Now you have a primary storage solution, and several secondary drives. If the four terabyte drive fails you either fail over to the clone, or you fail over to the re-used smaller capacity drives.

If the four terabyte volume fails then you get a new four terabyte drive, and copy the data from the four backup drives and within a short amount of time you have recovered.

Shifting to Exfat

For a long time I was using MacOS so I had drives that were formatted with APFS or the Journaled versions. As I shifted from MacOS to Windows and Linux I found the need to have drives that could be read by all three systems. By consolidating media I gave myself the room I needed to backup a drive from one volume to another, convert it to exfat, and then move data back.

If the windows machine fails, or the Linux machine fails, or the Windows machine fails, then I do not want to have to replace that machine, just because of a file system. By having Exfat I have the freedom to slide from machine to machine, with fluidity.

And Finally

It can be overwhelming to see that you have ten to twenty hard drives that may contain duplicates. By getting a larger drive you can go from having a dozen drives, and a dozen places where things are stored to having a single place where everything is centralised. Once everything is centralised you can order files in folders that are organised by year, and then by year-month-day. In so doing you don’t need to know what you’re archiving intimately. You just need to be able to organise things by year-month-day.

I know that I repeat this point a lot but there is a reason. If you know when something was photographed, filmed or worked on then you can find files within seconds, rather than hours, and without the CMS, should the CMS fail. The other reason is that a project might have two or three names, depending on who worked on the project. If it’s organised well, then going upstream is easier.

This should be an itterative process. Start by the task that is easiest, and then itterate and fine tune the files and folders until everything is well organised, duplicates are detected and consolidated, or deleted, dependant on context. If you do this well then it can be relaxing.

I’ve been consolidating files and data to free up drives to experiment with Immich, PhotoPrism and Nextcloud in parallel.

ExFat Stability Issues
|

ExFat Stability Issues

Byrichard

Yesterday a drive failed to mount so Photoprism and Audiobookshelf failed to work. The server was up and running but the files were not accessible. For this reason two of my services have been unusable for several hours. I believe that the issue came from using an ExFAT drive rather than EXT4 or another journalled file system.

The reason for which journalling is important is that Pis and other systems crash, and when they crash, read write cycles are not completed. The consequence of this is that the drive that had been fine before the crash becomes corrupted after the crash and needs to be fixed. Most of the time this is quite easy. I use disk utility two to three times, and eventually the problem is fixed long enough to backup the data.

Why This Matters

The rational thing to do is to think, “I’m using these drives for something that NAS drives are designed for so it’s normal that external drives would fail.” The reality is that the file system has a bigger role to play than the drive type. If the drive keeps track of what it was doing, and what was interrupted then it can quickly resume from where it left off. Without a journal the drive just sees missing data and asks for repairs to be done. By “asks for repairs to be done” I mean that it fails to mount until the issue is resolved.

Looking Forward

From now on I will use ExFAT as a temporary solution for when I move data from one OS to another but once the move is finished I will use APFS for mac, EXT4 for Linux systems and NTSF or equivalent for windows when flexibility is not needed. It’s easy to format a drive to a more resilient file system.

And Finally

Once I have backed up the data I will reformat the drive to EXT4 and copy the data back. This should make the drive more resilient.

Rock Climbing in Virtual Reality
| |

Rock Climbing in Virtual Reality

Byrichard

Rock Climbing in Virtual Reality has an interesting future because some people are afraid of heights, others don’t have the muscle tone to climb and yet more live too far away from climbing locations to enjoy the sport. Virtual reality is a great way of enabling people to get some of the sensations of rock climbing without the safety concerns.

As I watched this video I was curious to see whether it was a passive experience and then noticed the playstation controller. That’s a nice enough way to go “rock climbing but it it was up to me then I would pair this with the HTC Vive and it’s related controllers. Ideally I would go a step further. I would develop a conveyor belt system, place it vertically and program it to put handholds where the climber has somewhere to grip. As he or she climbs they would place their weight on the hand hold and it would slide down as the person climbs.

We have seen demonstrations of equipment that can emulate 3d shapes and textures. The challenge would be to take this from being flat on a table to being vertical and strong enough to hold someone’s weight. Tests should be carried out on near vertical and vertical “climbing routes” before being rotated to allow for the climbing of overhanging routes. You would definitely get a workout from such a configuration. Gamers would go from their current physiognomy to being toned and healthy. The next generation of gamers are going to be fit thanks to Virtual reality workouts.

This could also be attempted for sports like Via Ferrata, via cordata and others. Via Ferrata is a simplified form of rock climbing and via cordata is walking around a nice landscape where safety gear is available to keep you safe in case of a slip or fall whilst hiking.

I really look forward to when this gaming/fitness training experience comes of age.

Dormant Social Media Life While Sorting Through Drives

Dormant Social Media Life While Sorting Through Drives

Byrichard

Recently my Social Media Life has become dormant. I do visit Facebook every so often but I ignore Instagram, barely touch Mastodon or the fediverse, and in general have stopped looking at social media for a social life. It’s not that my life offline has become vibrant. It’s that online is empty of meaningful engagement, especially in winter.

From the nineties right up to around 2018 or so social media was a place to meet and be social. It’s during the pandemic that social media seemed to die. I think that social media relies on meeting people in the physical world to have value. People on the social web use it when they’re on the toilet, or waiting for something else to happen. They’re just filling small gaps in their schedule.

Plenty of Potential Storage

The other reason is more positive. I have terabytes of storage spread across twenty two drives, or more and I am re-organising everything in order to see how much space I have free. I have at least twenty terabytes of data storage. I might have as much as fourty two terabytes of storage but due to file duplications I don’t have much space that is free.

Required for Video Projects

That’s frustrating, especially if you want to take video and can generate up to 64 gigabytes of data at a time. 64 gigabytes, because my Sxs cards have that amount of storage. The drone could have up to 512 gigabytes of storage if I put the right SD card into the camera.

Freeing Cloud Space

I can’t delete data from cloud storage solutions because I haven’t consolidated all of my photo and video files from iCloud, Flickr, Google Photos and one or two other services. If all of my files are organised chronologically then I can migrate from cloud storage solutions without worrying about losing images that might have been backed up only to iCloud, or Google Photos, or another solution.

By consolidating the data offline, I can manage data in the cloud with ease.

Shrodinger’s Storage Cat

For data to be safe you need to have two local copies, and one offsite backup. If you have a dozen, or two dozen drives then that data is like shrodinger’s cat. You don’t know whether it’s backed up (living) or a single copy (dead). Delete the wrong file and you might end up losing a few days, or a few months of data. By having a centralised main storage your small satellite drives become working drives. You use them while you’re working on a project, and once the project is over you move it to the main storage solution and either wipe and reuse that drive, or keep it as a backup. Shrodinger’s cat has left the storage device.

Looking Forward

Years ago I bought an eight terabyte drive because I planned to consolidate my personal video and photo files but I never got around to it. This morning I finished moving the junk I had on that drive to other drives and I have now started to backup the video and photo data that I had temporarily kept on a five terabyte drive. I realised that I have more data than would fit on a five terabyte drive, but it also failed to mount at least once.

For years I had the same data on five to six drives but in my move towards centralising, and then backing up my data I made myself unsafe. I was left with just one copy of data. Now that I am backing up the four terabytes of data I have from the five terabyte drive to the 8 terabyte drive I have a little margin of safety.

Consolidation

When moving files from the 5tb drive to the 8tb drive the process is simple. Move the video folde to the video folder, photo to photo, and documents to documents. It’s when I start moving the secondary drives to the main drive to consolidate my photos and videos that the value is generated because this is when I detect duplicate folders, videos and photos. This is when the value comes in.

Moving four terabytes of data takes hours, but once that data is moved, and as I consolidate data from six or seven other drives I will copy only the files that do not exist on the main volume. I will then move the files that I have checked into a zz-backed-up folder.

Low Value

When I was trying to free space on drives I deleted the files from the drive as soon as they were copied over. Now I am moving them to zz-backed-up as a scruffy backup. The aim is to be able to recover files if the 8tb volume fails, but these are a stop gap. The next step is to backup the 8 tb volume.

And Finally

Nothing is backed up until you have at least two copies locally, and a third off-site copy. The next step is to copy the files from an older volume to a newer volume. Old drives fail, so having files on older volumes is a risk. When I finish consolidating files to the eight terabyte volume I will then duplicate it to a newer 8TB volume.

As a side project, once I have two or more drives that are free of data I could experiment with setting up a raid system.

| | | |

Social networking and freebase

Byrichard

As you’re painfully aware by now there are hundreds of social networking websites but none of them have a communal database. If you’re on orkut your data stays there, if you’re on yahoo communities your data is there. All these social networking websites are very similar in what they ask of you but different in how they link you. Loudmouthman got me thinking about how you could use a database like freebase to share this data between networks, sort of like openid but with more data.

Freebase is a database for volunteers, similar to a wiki but whereas the wiki is a collection of articles this is a database where you write the front end and implement it according to your needs. If you’re interested in video production and television for example you could take the television section of that database and make it accessible within that site. If you encouraged your users to create their profiles within that communal database then details which are not so critical to your persona (whilst being careful not to make phishing to easy)  could be used so that you input your data only once.

By having a communal database the migration from one social networking website to another would be far smoother, more transparent. You’d have the same user profile in a number of places and you would have more freedom to concentrate on what you feel is important. If one website goes down then that is not as critical since your presence is spread in a number of places therefore there is less opportunity to be in trouble should your main social website go down, as was the case with Facebook and Skype within the past few weeks.

RSS feeds are already helping to spread your presence across a number of websites and sites like Jaiku and tumblr help aggregate your daily output to a number of locations.  As a result of such practices. In effect you’d have built in redundancy. If one node goes down then five others can be used whilst waiting for your preferred  social network to come back online.

In summary since we are members of more than one community there is a demand for a communal database from which the sharing of certain types of data would promote the spread of online presence between more than one community at a time. By facilitating the process of setting up a number of communities there is less opportunity for us to be bogged down in content we are uninterested in. Communities, rather than forums could be far more specific to our needs.

I would welcome your views on this topic.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.