Pi-Holes and Cloud Syncing

Pi-Holes and Cloud Syncing

Byrichard

Two evenings ago I was trying to sync files from Kdrive to the local drive and it kept getting blocked. I wasn’t clear as to why this was happening until I saw that Pi-Hole had throttled the IP address of the computer that was attempting to sync from Kdrive. It did this one in the morning, and the second time in the evening.

I suspected that for some reason the computer might go to sleep when it isn’t used, but a Pi doesn’t sleep, so that wasn’t it.

Temporarily Disable Pi-Hole

When I disabled blocking by the Pi-Hole and tried to sync once again it worked flawlessly. It took hours but the data was transferred from the remote machine to the local machine over the next twelve hours, or so.

Rate Limiting by Pi Hole

Pi-Holes will tolerate up to 1000 requests per minute from a device under normal circumstances. Usually a system would make a few dozen requests at a time depending on the website being visited. When you’re synching gigabytes of files via a cloud client it might send several thousand requests as it receives requests, and provides status updates on whether packets are received and when to send the next one. This is a generalisation. I could not find specifics for Kdrive.

The Wrong Assumption

Aside from considering a computer going to sleep and stopping data transfer I also considered that the cloud provider would throttle download queries, and speed. When I tried to download data from iCloud I did encounter a similar problem and when I tried from one or two other providers I seemed to have the same problem.

 Interesting to Know

When I tried to download several 2 gigabyte files manually with the Raspberry Pi I found that it sometimes saturated its 8 gigabytes of ram before slowing down to a crawl. It doesn’t like downloading large files in bulk. I came across this bug several times.

And Finally

In normal circumstances cloud synching is a few files at a time, rather than several hundred gigabytes at once, so you don’t run into this issue. If this was a normal situation then I would modify Pi-Hole to have a higher rate limit, but for now I will leave it as it is.

Kdrive and PhotoPrism

Kdrive and PhotoPrism

Byrichard

Yesterday I configured PhotoPrism to work with my iPhone photo album that was being synced to Infomaniak’s Kdrive, before then being synced to a drive that I could access via the Photoprism docker-compose config file. I then used No-ip to make that PhotoPrism instance available to the world wide web.

For several years I have had an Infomaniak Kdrive account but did not use it much, until I noticed that what costs 100 CHF with Google Drive costs 67 CHF with Kdrive. That’s a 34 CHF franc saving.

Migrating Data from Google to Infomaniak

This is interesting because of two things. The first one is that you can easily migrate your Google Photos Albums to Kdrive, and from Kdrive to your local machine without having to spend hours doing so manually. I am in the process of migrating from Kdrive to my local drive, for my Google Takeout album but this is slow.

What worked well was telling Kdrive to download the iPhone photo album from the Kdrive cloud to my local machine. I then edited the docker-compose config file but it took some trial and error before I understood how it works. It’s simple.

Telling PhotoPrism Where to Look

In my mind it should be “label: destination” but this is wrong. With Photoprism config files it is destination: label. To explain this more clearly: 

volumes:
 - "/mnt/photos:/photoprism/originals"
 - "/mnt/videos:/photoprism/originals/videos"

“mnt/photos” is the folder location and “photoprism/originals” is the label that docker and photoprism recognise. This is important because when you understand this you can set an external drive to be the photo gallery main drive, run index, and catalogue everything.

Ingesting Photos and Videos

If you export your Google Photos albums via Google Takeout, you can unzip the 2 gigabyte files, and as they unzip the files and their related JSON files are all brought together. Tell PhotoPrism that the Google Takeout folder is the import folder. After this mark “move files once ingested” and press import. PhotoPrism will then ingest all those files, delete what has already been ingested, catalogue everything, and leave you with an organised folder of photos and videos.

The indexing stage is very fast but can still take time, depending on album size.

The Next Step

There are two possible next steps. The first is to access this gallery from anywhere using tailscale, or using No-IP to access your photo album remotely. I recommend tailscale for this stage, because I haven’t seen much information about how secure PhotoPrism is so I’d rather not risk it.

I setup no-ip before telling my router about it and within a few minutes I had access to PhotoPrism and Nextcloud via No-Ip. I will write about that experiment tomorrow.

And Finally

When you’re playing with linux, and experimentating with projects like PhotoPrism you need to take time to understand how they work, to adapt them to your use case. I did this. Now I can get PhotoPrism to behave as I want it to. Now I can use external hard drives, and index photos, without having to import them. Within minutes I can plug in a drive, tell PhotoPrism where to find it, index it, and then use it.

If it had two factor authentication and more security measures then I would consider having it on the cloud, but for now I’m happy to use it via Tailscale’s VPN, as this is more secure.

Playing with NextCloud

Playing with NextCloud

Byrichard

Google, Apple and Microsoft have cloud storage solutions. So does Evernote, Kdrive and other products. The issue with all of these solutions is that they are owned by corporations. They are simple and convenient to use but at the cost of being locked in to an OS in some cases, and to corporate interests.


NextCloud is an open source alternative with hosting solutions that offer people with the choice of choosing between hosting solutions that are as local, or as remote as they want. I could choose between a German and a Dutch hosting solution. I chose [tab.cloud](https://tab.digital/) simply because they offer 8gb rather than 2gb like another solution. You can upgrade to 128 gigabytes, or to a maximum of over 500 gigabytes in the paying tiers.


You can install Nextcloud locally too. I installed it on a raspberry pi but I haven’t configured it to make it usable yet but I also installed it on a windows machine via Docker. That was quick and easy to do. I was thinking of playing with NextCloud on Linode but will do some local experimentation first. It seems intuitive and simple to use after several minutes of experimentation. That’s how quick and easy it is to get up and running.


The price for Google One is 100 CHF per year for two terabytes, and 120 CHF per year for iCloud with the same amount of storage. The price for theGoodCloud is 240 for one terabyte of data shared with five people. With Linode/Akamai the estimated cost could be 206.88 Dollars per year. The cheapest solution is Kdrive at about 60 CHF per year for two terabytes.


And Finally


Buying two terabytes of storage to keep at home costs from 63 CHF upwards. You could easily setup NextCloud on a Raspberry Pi and plug an external drive to that computer and keep it on 24/7. When you get home you could back up all the photos from that day. You would save on storage costs. One of my biggest frustrations with online storage solutions is that if you want to retrieve your data you need to spend several days downloading the data.


With a solution like NextCloud and the mobile phone app you could setup a local machine to backup all your photos. You would have the tertiary backup online but keep the pictures on your phone, the primary backup in your own home and the tertiary backup online.


I will setup a NextCloud system locally.

KDrive – A Viable alternative to Google One and iCloud

KDrive – A Viable alternative to Google One and iCloud

Byrichard

KDrive peaks my interest because instead of cost over 100 dollars per year it costs around 64 if you buy directly from their website rather than The Apple App Store, but also because once you send your photos up to the cloud, you can get them down more easily.


With Google One you can store all of your images to the cloud quite easily but because apps like Picasa and others no longer exist, you cannot get them back without spending hours downloading them manually. iCloud is not quite as bad but they are still not ideal. You can upload images to their service, but if you do so, your image gallery must be on a drive with enough storage to take the gallery offline. On mobile phones and laptops this is complicated. In effect your images are stuck until you buy a higher capacity laptop or phone. I know an HD would also work but the issue is that when your image gallery is on another drive you have to keep it plugged in, or sync regularly for it to be efficient.


Simple Synchronisation


With KDrive you have a folder that is synched automatically from your device to the cloud, and from the cloud to another device. If you decide to move your images from one device to another you can do so by requesting that the images are downloaded, and eventually they will be synced. This is a key selling point.


Google One


With Google One you have two terabytes, which are shared between photos, file folders, e-mail and more. They are however, separated. You can access all of the files that you uploaded as files, but you cannot access all of your photos and download them easily. In the past I tried to download images from Google Drive and I found that I was blocked by how many hours it takes to download zip file after zip file for days at a time. This is not a good solution despite being cheaper than iCloud.


iCloud and Price


Aside from the size of the HD you need on your mac laptop or iOS device to download your galleries you also need to pay 120 CHF per year, in perpetuity. With Apple device you pay a premium for the phone, for the laptop but then you pay a premium for the apps, for the services and more. Apple wants 120 CHF per year to keep your data safe, when drives of that capacity are going down in price every single year. I object to paying a tax of sorts, every year, when I have already payed a premium for the products.


KDrive


KDrive, by Infomaniak, based in Switzerland has a number of advantages. The first is that the company is local, so it feels good to support a local effort rather than the giants. The second reason is that their price point is half as much as their competition, especially if you commit to two or three years. The third selling point is that all the files are accessible as if they were already on your local machine. This means that within a short period of time you can recover the files you want to recover, or backup the files that you want to backup.


Features


In the settings you have photo backup, and within this you can enable automatic backup but what makes this one different is that you can choose where to save the images, including which folder. It also creates directory by month and year. This makes it easy for you to find images when you are looking for them.. I like that you can select to upload photos, videos, screenshots and even delete photos once they are backed up, although this is in beta.


I like these features because I don’t want to backup videos because they’re heavy and take time to load, but also because they are not relevant to my photos. I prefer to take care of them separately. No other service offers the option to exclude videos.


Another great feature is that you can choose whether to sync your photos from “now” or all. That is useful. If you’re on a trip and you just want to backup recent pictures then that’s a useful feature. If you have all the time in the world, and enough battery life, then you can sync your entire image library. The fact that you can exclude video initially speeds up the process considerably.


For more information about KDrive. The Prices.


And Finally


Google Drive and iCloud complicate rather than simplify your life, when you are dealing with photos. KDrive simplifies it. If you can migrate your photos away from Google Drive and iCloud to a solution that is more user friendly then you can also reduce the amount of space you need on your devices, as you have offloaded them to the cloud, but then the files that were offloaded to the cloud can be synced your local machine seamlessly. As a media asset manager I greatly appreciate this.


KDrive is now a speedy and efficient solution for the sharing of files, with some intelligent features for the backing up of your phones’ photo galleries. I am in the process of doing that now. I hesitated with other services in the past, but to me this is a clear winner.