qlyoung's wiki
Page Tools
User Tools

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
tiered_storage [2025/10/15 15:54] – qlyoungtiered_storage [2025/10/22 15:33] (current) – wording qlyoung
Line 5: Line 5:
 For a long time I was firmly in the first school. Files live in one big beautiful cluster that you can access from any device. I see the appeal. You can focus all your efforts making that cluster as big and beautiful as you want, as resilient as you want. There's only one copy of data, so you never have inconsistencies. But after lots of time using a big beautiful cluster, I kept having the experience that I wanted some file and I just couldn't get to it. And when I could get to it the experience of browsing around, searching, transferring files all over a network just...sucks. After you buy into the paradigm, after a while you get used to it and forget that it doesn't have to be like that. For a long time I was firmly in the first school. Files live in one big beautiful cluster that you can access from any device. I see the appeal. You can focus all your efforts making that cluster as big and beautiful as you want, as resilient as you want. There's only one copy of data, so you never have inconsistencies. But after lots of time using a big beautiful cluster, I kept having the experience that I wanted some file and I just couldn't get to it. And when I could get to it the experience of browsing around, searching, transferring files all over a network just...sucks. After you buy into the paradigm, after a while you get used to it and forget that it doesn't have to be like that.
  
-In the past my laptop didn't have enough space to store all my data. Small form factor drives just weren't big enough, and then they got big enough but they were so expensive that it only made sense to put them in a workstation (desktop). In 2025, annoyed by the downsides of network storage, I started thinking about storage and how to make my experience better. I realized I have two tiers of storage.+In the past my laptop didn't have enough space to store all my data. Small form factor drives just weren't big enough, and then they got big enough but they were so expensive that it only made sense to put them in a workstation (desktop). In 2025, annoyed by the downsides of network storage, I started thinking about how to improve the experience. I realized I have two tiers of data.
  
 === working set === === working set ===
  
 I have a set of files I frequently access; my music, photos, documents, and projects. This is what I think of as my "working set". It doesn't include stuff that falls in the bucket of "huge binary files" - movies, 300gb DNxHD Resolve projects, that sort of thing. Right now my working set is ~4tb. I have a set of files I frequently access; my music, photos, documents, and projects. This is what I think of as my "working set". It doesn't include stuff that falls in the bucket of "huge binary files" - movies, 300gb DNxHD Resolve projects, that sort of thing. Right now my working set is ~4tb.
- 
-This data is [[syncing|replicated]] on all devices. 
  
 === other stuff === === other stuff ===
Line 25: Line 23:
   - which means I can now have my working set completely stored on my laptop   - which means I can now have my working set completely stored on my laptop
  
-Due for a laptop upgrade anyway, I got 8tb of storage on my new one and then set about collating my working set into a portable structure. After that I installed syncthing on every device and started a massive synchronization campaign that took about a week to settle out. With careful preparation there were almost no sync conflicts.+My desktop already had 10tb of storage and my home server 12tb. Due for a laptop upgrade anyway, I got 8tb of storage on my new machine and then set about collating my working set into an organized directory tree to prepare it for [[syncing]]. I then engaged in a massive synchronization campaign that took about a week to settle out. With careful preparation there were almost no sync conflicts and my working set is now replicated on all devices.
  
 At this point in time I exist in storage nirvana. I work on my desktop. Grab my laptop and catch an airplane; all the data I was working on is on my laptop and I pick up where I left off. No need to connect to airplane wifi. Get where I'm going, connect my laptop; it's syncing everything to my home server and desktop in the background. At this point in time I exist in storage nirvana. I work on my desktop. Grab my laptop and catch an airplane; all the data I was working on is on my laptop and I pick up where I left off. No need to connect to airplane wifi. Get where I'm going, connect my laptop; it's syncing everything to my home server and desktop in the background.
Panorama theme by desbest
tiered_storage.1760543641.txt.gz · Last modified: by qlyoung
CC Attribution-Noncommercial-Share Alike 4.0 International Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International