Avoid sync folders and files in dropbox or google drive as git with .gitignore
Hello everybody,
I bet everybody has his own cloud solution for syncing it’s folders from home to work, viceversa and between devices.
Before today i always relied on a Dropbox / Google Drive combo depending on the project:
- Dropbox is great with Heroku which has a sync plugin (super-sweet!!) and its client is fast and not memory wasting
- Google Drive client is super slow and takes ages to init but i tend to use a lot the whole google suite on the web and it’s really integrated well (history, revisions, etc)
BUT as any other web developer i have this big, nasty problem of not wanting to sync:
- Git repositories (.git folders)
- NPM modules (node_modules folders) or Bower components (bower_components folders)
- Other trash folders left by the various IDE (nbproject folder) or OS (Thumbs.db files)
Everybody have tried a npm install with devastating consequences for poor cloud clients and it’s a known fact that in the long run both dropbox/drive will corrupt a git repo.
So i searched a bit and found a couple of way to try to do it with the dropbox protocol that includes using external scripts, and totally no way with google drive except using symlinks. Yet i was not convinced, i didn’t want to spent time configuring on each of my machine scripts in order to avoid dropbox to sync trash.
There is no easy way. You have to spend time, configure, on each machine.
Oh come on, there should be a solution, we are in the cloud era, every developer has this problem!!
Right, i agree and i discovered that:
- Bittorrent Sync has a way to ignore files using a IgnoreList
- MEGA (yes, Megaupload) has a way built-in in his client to ignore folders/files with regexp (exactly the same as .gitignore) in Preferences – Advanced – Exclude file names
- Seafile (thx Hugo) also allows a ‘seafile-ignore.txt’ file with file and folder patterns to be excluded. Works great with ‘.git/’ and ‘node_modules/’
Both Bittorrent Sync and MEGA seemed nice, the first as a private approach, you probably have to get a VPS or a Raspberry Pi in order to have an always on server. MEGA on the other hand offer 50gb in the free tier (sweet) and has a dropbox-like client, easy to install and configure, with a classic web interface.
I tried both, Bittorrent Sync is impressive but i think i found my solution using MEGA:
- Dropbox like user experience : webinterface, multi-platform client, auto-sync from a folder, everything you expect from a cloud service nowadays
- 50gb free tier is a lot, even too much for my dev needs, but it comes with a “bandwidth restriction”, i spent a little trying to figure out what does it means and i found on cloudwars the explanation: “The bandwidth has a limit of 10GB, and once it’s used up users have to wait 30 minutes for a refill to continue.”. LOL, i think that will not be a problem.
- The ignore mechanism works on both folders and files, already tested, even if they are in subfolders : /MEGA/Projects/myproject/node_modules/.. is completely ignored with just a “node_modules” insertion (but you have to configure it on every machine)
Speed and stability are fine:
- Client is quicker than drive (with 64k+ files / 10k folders it takes less than 15 seconds to start) which seems acceptable to me, google drive took me ages to start with that amount of files
- Client crashed a couple of times while delivering all the files on MAC (or just strucks other in uploading..) but after the crash it auto-restarted and lose no files, i’ll close an eye on this problem that can easily be a minor bug
- Upload speed is acceptable
- Sync with the web interface is almost immediate (websockets rocks) and i find it nice
So in the end, if you don’t have privacy issues or big deals i suggest trying out MEGA, it was fatty and quick enough for me. Otherwise your only fallback is Bittorrent Sync, which is a Sir Feedback in my opinion, but it still needs to be configured on a “always on” machine to be effective as MEGA.
MEGA has no referral mechanism, 50gb is all you can ever get from the free user tier, so this post is born from pure love.
If you find other services that offer the possibility of ignoring folders and files please leave a comment.
I’ll try and add them to the topic!
Great tip, I have been looking at solving this problem forever. Mega will be great for storing dev files. Thanks!!
No problems! I have been using MEGA since 2 months ago and i’m still satisfied by their service so I still approve the article. 😀
SeaFile also allows a ‘seafile-ignore.txt’ file with file and folder patterns to be excluded. Works great with ‘.git/’ and ‘node_modules/’
Thanks Hugo!
I have added it to the article as an alternative!
The title of the document is misleading. You’re not providing a solution for Dropbox or Google Drive. Good article otherwise but you really should change the tiltle.. to maybe Alternatives to GoogleDr or DrobB blah blah…
I also struggled with this dilemma.
I went for mega, like you did, but the website is flaky as anything (icons aren’t even displaying – why does every page have an animation on load?) and the sync app feels iffy, I don’t think my ignore patterns are being applied. There’s just something I distrust about Mega: perhaps the flakiness of the website, the fact the sync app isn’t behaving as I expect it, that certain things on their dev todo list have allegedly been there untouched for months, perhaps Kim Dotcom’s background, or maybe t is all good but I don’t trust current powers to leave it be and not just pull the plug on it.
I’ve just tried pcloud (i.e. downloaded the desktop app) and it lets you specify ignore patterns which it applies of files and folders. You can use * and ? wildcards, but it’s not regex or gitignore. But it means you can put “node_modules” and “.git” and it will correctly ignore them.
Dude, just reaching out to say thank you.
I had been struggling with all the thousands and thousands of dependencies in my projects and the more projects I had, the more annoying it was getting waiting for sycs between my PC/Mac at the office and home. Dropbox is surely dropping the ball on this one, I went with Megasync and so far the experience has been outstanding, I’ve been trying them for a couple months now and I thin I am ready to jump to the paid subscriptions just to support them a little since 50gb is enough for my needs.
Thank you, this article supported my decision to eventually (begrudgingly; this post is a form of forcing myself) switch from Dropbox to MEGA. It’s starting to become a little tedious to have to sync node_modules, and the FREE tier’s storage space is very nice to have.
How do you deal with syncing multiple computers with Mega and git repos?
I have the same setup as you: ignore .git directory, etc., and sync the rest with megasync. But then I work on my desktop, commit changes to git, and if I switch to my laptop it syncs with Mega. Now git thinks I have changes in the working directory, so doing `git pull` wants me to re-commit and do a merge. So I have to `git fetch` and `git reset origin/branch,` which seems kind of kludgy.