Open Source Continuous File Synchronization

Overview

Syncthing


Latest Linux & Cross Build Latest Windows Build Latest Mac Build MPLv2 License CII Best Practices Go Report Card

Goals

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important one being the first. This is the summary version of the goal list - for more commentary, see the full Goals document.

Syncthing should be:

  1. Safe From Data Loss

    Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.

  2. Secure Against Attackers

    Again, protecting the user's data is paramount. Regardless of our other goals we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.

  3. Easy to Use

    Syncthing should be approachable, understandable and inclusive.

  4. Automatic

    User interaction should be required only when absolutely necessary.

  5. Universally Available

    Syncthing should run on every common computer. We are mindful that the latest technology is not always available to any given individual.

  6. For Individuals

    Syncthing is primarily about empowering the individual user with safe, secure and easy to use file synchronization.

  7. Everything Else

    There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.

Getting Started

Take a look at the getting started guide.

There are a few examples for keeping Syncthing running in the background on your system in the etc directory. There are also several GUI implementations for Windows, Mac and Linux.

Docker

To run Syncthing in Docker, see the Docker README.

Vote on features/bugs

We'd like to encourage you to vote on issues that matter to you. This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.

Getting in Touch

The first and best point of contact is the Forum. There is also an IRC channel, #syncthing on freenode (with a web client), for talking directly to developers and users. If you've found something that is clearly a bug, feel free to report it in the GitHub issue tracker.

Building

Building Syncthing from source is easy, and there's a guide that describes it for both Unix and Windows systems.

Signed Releases

As of v0.10.15 and onwards release binaries are GPG signed with the key D26E6ED000654A3E, available from https://syncthing.net/security.html and most key servers.

There is also a built in automatic upgrade mechanism (disabled in some distribution channels) which uses a compiled in ECDSA signature. macOS binaries are also properly code signed.

Documentation

Please see the Syncthing documentation site.

All code is licensed under the MPLv2 License.

Issues
  • Filesystem Notification - Continuation

    Filesystem Notification - Continuation

    This is the continuation of plouj's PR #2807 on a branch in the syncthing repo for ease of maintenance. Refer to that PR for general information what this is about and commits previous to https://github.com/syncthing/syncthing/commit/1d424d84609a40d2a11ac8e06cb87aac6b130629.

    frozen-due-to-age pr-merged 
    opened by imsodin 184
  • Support for file encryption (e.g. non-trusted servers)

    Support for file encryption (e.g. non-trusted servers)

    So I have had a look at BitTorrent sync, syncthing and alternatives and what I always wondered about was the possibility to not only sync between resources I own and trust, but also external resources/servers which I do NOT trust with my data, up to a certain extent.

    One way to do this is using ecryptfs or encfs, but this has many obvious downsides: it is not an interoperable solution (only works on Linux), the files are actually stored in encrypted form on the disk (even if the resource is trusted and this is not necessary, for instance because of the file system being encrypted already), etc.

    What I propose is somehow configuring nodes which are only sent the files in an encrypted format, with all file contents (and potentially file/directory names as well; or even permissions) being encrypted. This way, if I want to store my private files on a fast server in a datacenter to access them from anywhere, I could do this with syncthing without essentially giving up ownership of those files. I could also prevent that particular sync node from being allowed/able to make any changes to the files without me noticing.

    I realize that this requires a LOT of additional effort, but it would be a killer feature that seems to not be available in any other "private cloud" solution so far. What are your thoughts on this feature?

    EDIT: BitTorrent sync mentions a feature like this in their API docs: "Encryption secret API users can generate folder secrets with encrypted peer support. Encryption secrets are read-only. They make Sync data encrypted on the receiver’s side. Recipients can sync files, but they can’t see file content, and they can’t modify the files. Encryption secrets come in handy if you need to sync to an untrusted location." (from http://www.bittorrent.com/intl/de/sync/developers/api)

    enhancement 
    opened by Natanji 143
  • Inotify support

    Inotify support

    To notice changes more quickly.

    enhancement 
    opened by jpjp 116
  • all: Store pending devices and folders in database (fixes #7178)

    all: Store pending devices and folders in database (fixes #7178)

    Purpose

    As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.

    This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in ClusterConfig messages. That is deliberately saved for another PR though.

    Testing

    All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the POST /rest/system/config REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.

    ~~Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is.~~ Unit tests included, with a few basic test cases.

    Documentation

    The envisioned API for this stuff is described in https://github.com/syncthing/docs/pull/498.

    The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.

    Breaking Changes For The User

    Existing <pendingDevice> / <pendingFolder> entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae24612397fea6182cfb3d6dcce592c78c6df5d until then.

    opened by acolomb 109
  • Option to not sync mtime

    Option to not sync mtime

    Trying to sync my android device and my NAS(armv5) both using syncthing v0.10.0:

    On the ntfs mount of the NAS the file /my/directory/.syncthing.example.test is created then I get:

    puller: final: chtimes /my/directory/.syncthing.example.test: invalid argument

    File /my/directory/example.test is not created. Syncthing continues with the next file but the same error happens for all the files in the directory.

    enhancement frozen-due-to-age 
    opened by otrag 108
  • GUI display of global changes

    GUI display of global changes

    Purpose

    This PR adds a button to the device section of the GUI that the user can click on to see all filesystem changes that originated/occurred on that machine. That is to say, not changed copied from the network. This should help people discover more easily what computer made a certain change that was propagated in larger P2P swarms.

    Screenshots

    image

    frozen-due-to-age 
    opened by nrm21 107
  • Folder isn't making progress

    Folder isn't making progress

    All nodes are on v0.10.2 GUI says

    9:35:58: Folder "default" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    9:36:41: Folder "PIA" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    

    In the log I see the "hash mismatch" info but the file is not changed during pull. It was probably the whole night in that state but i don't have more logs, I should probably increase the size... logs from the pulling node: http://alex-graf.de/public/st/no-progress.tar.gz

    Restarting also does not fix it

    opened by alex2108 104
  • gui: add advance config port mapping to gui

    gui: add advance config port mapping to gui

    Purpose

    https://github.com/syncthing/syncthing/issues/4824

    Gives the user the opportunity to display a link to web gui remote machines. The user can choose the port and set if the link will display.

    Testing

    I just check to see if the link was shown depending on the setting.

    Screenshots

    Screen Shot 2020-09-29 at 11 13 53 PM Screen Shot 2020-09-29 at 11 34 37 PM

    Documentation

    https://github.com/syncthing/docs/pull/573

    Authorship

    Your name and email will be added automatically to the AUTHORS file based on the commit metadata.

    opened by rjpruitt16 91
  • lib/connections: Add KCP support (fixes #804)

    lib/connections: Add KCP support (fixes #804)

    This is mostly for benchmarks and testing.

    Seems to connect, and not crash immediately, which is good news.

    Known issues:

    1. No timeouts as part of the protocol, so we usually end up with a dead connection for 5-7.5 minutes until it times out. This could be addressed by always using the new connection that we get, as long as the priority is right
    2. Has some weirdness around closing connections. See https://github.com/xtaci/kcp-go/issues/18

    The diff is massive due to deps, so I suggest you pull the PR locally and diff inside the lib and cmd/syncthing dirs.

    frozen-due-to-age pr-merged 
    opened by AudriusButkevicius 91
  • Memory and CPU usage is prohibitively high

    Memory and CPU usage is prohibitively high

    I've been testing syncthing across 3 machines, a laptop with 8GB of RAM and two NAS-style servers with 1GB and 2GB of RAM. My repositories have the following sizes:

    • 19 items, 536 KiB
    • 83471 items, 16.2 GiB
    • 181482 items, 387 GiB

    To sync these three repositories syncthing 0.9.0 uses a bit over 700MB of RAM and while syncing continuously pegs the CPU at 150% on all nodes.

    While the CPU usage I could manage during the initial sync the memory usage is simply too high. A typical NAS server like the two I have holds >4TB of storage. At the current level of usage that would require ~8GB of memory just for syncthing.

    Without looking at the code I assume an index is being kept in memory for all the repository contents. 700MB is 2.6kb per item on disk, which seems way too high. The index should only really need to store filename, parent, permissions, size and timestamp. On average (depends on filename sizes) that should only be 50-60 bytes per item which would only be 13MB. Moving that index to disk would also make a lot more sense.

    I assume the CPU usage is hashing lots of files. There I assume using librsync might be a better option.

    enhancement frozen-due-to-age 
    opened by pedrocr 86
  • lib/db: Use a more concurrent GC (fixes #7722)

    lib/db: Use a more concurrent GC (fixes #7722)

    This changes the GC mechanism so that the first pass (which reads all FileInfos to populate bloom filters with block & version hashes) can happen concurrently with normal database operations.

    The big gcMut still exists, and we grab it temporarily to block all other modifications while we set up the bloom filters. We then release the lock and let other things happen, with those other things also updating the bloom filters as required. Once the first phase is done we again grab the gcMut, knowing that we are the sole modifier of the database, and do the cleanup.

    I also removed the final compaction step.

    opened by calmh 2
  • Show discovery and listener status when not failed

    Show discovery and listener status when not failed

    In the GUI, the listener and discovery status is displayed as a count of active mechanisms. In case of a failure, details can be seen in a modal on click. The modal should be available always and show non-error status details as well.

    enhancement 
    opened by acolomb 0
  • Out-of-sync state of zero-byte files is not reflected on peers

    Out-of-sync state of zero-byte files is not reflected on peers

    From https://forum.syncthing.net/t/remote-stays-on-up-to-date-but-is-really-out-of-sync-when-looking-at-its-instance-directly/16852

    I have two peers (A and B) in sync with each other, whereby one of the peers (B) has a filename length limit. If a zero-byte file with a filename that surpasses this limit is created on peer A, peer B naturally can't sync it and displays 'Out of sync' in its gui. However, peer A will keep showing 'Up to date' for the remote device in its gui. If now a byte is added to the problematic file, peer A will show 'Syncing (99%, 1 B)' as expected in its gui.

    The expected behaviour for me would be that peer A reflects the out-of-sync state of peer B even when syncing zero-byte files, for example, as suggested by imsodin, by displaying 'Syncing (99%, 0 B)'.

    Even though I discovered the behaviour due to a filename length limit on one peer, I'd expect it to occur with any issue that prevents the file from being synced on a remote. I used 1.16.1 and also 1.17, both on Alpine and Windows, but imsodin was able to confirm the problem looking at the code, so I don't think this is relevant.

    bug ui 
    opened by James-Mat 0
  • Incorrect local state when using negated patterns inside ignored parent folder on both sides

    Incorrect local state when using negated patterns inside ignored parent folder on both sides

    I have finally managed to reproduce https://forum.syncthing.net/t/folder-stuck-in-sync-and-non-matching-local-and-global-states/16431.

    1. On Device 1, create a folder called test1.

    2. In the folder, create a folder test, and then inside it another folder test.

    3. The folder structure now looks like test1\test\test.

    4. Add the test1 folder to Syncthing with the following ignore patterns.

      !/*/
      *
      

      As expected, both the subfolder and its parent folder have been indexed.

      image

    5. Share the folder with Device 2, and accept it there with the same ignore patterns as above.

    6. The folder state on Device 2 is correct, but the local state on Device 1 has changed. The parent folder has been ignored, despite its child being unignored.

      image image

    Logs with model and db:

    syncthing1.log syncthing2.log

    One thing to note is that the folder states are correct when not using the same ignore patterns on the other side. It seems that it is only when the same negate/ignore patterns are applied to both sides that the issue occurs.

    bug needs-triage 
    opened by tomasz1986 0
  • Add a way to switch between folders or devices to the edit modal dialog

    Add a way to switch between folders or devices to the edit modal dialog

    Currently if you have many devices and/or folders and you want to edit their settings in succession it requires a lot of clicks - expand item, open modal, close modal, repeat.

    It would be nice if the edit modal dialog provided a way to switch or to navigate between items. For example navigation buttons (previous and next button) or a dropdown menu in the header.

    enhancement needs-triage ui 
    opened by GithubUser5462 2
  • pfilter: panic slice bounds out of range

    pfilter: panic slice bounds out of range

    https://sentry.syncthing.net/share/issue/d99cbc5edf044001ab6fa53385407270/

    panic: runtime error: slice bounds out of range [:423] with capacity 0
    
      File "github.com/AudriusButkevicius/[email protected]/conn.go", line 88, in ReadFrom
      File "github.com/lucas-clemente/[email protected]/conn.go", line 50, in ReadPacket
      File "github.com/lucas-clemente/[email protected]/packet_handler_map.go", line 294, in listen
    

    I had a look but don't have any idea yet how it can happen that the number of read bytes is not zero, but the packet buffer has zero length/cap.

    bug needs-triage 
    opened by imsodin 1
  • TypeError: Cannot read property ‘urAccepted’ of undefined

    TypeError: Cannot read property ‘urAccepted’ of undefined

    logbar.js:11 TypeError: Cannot read property 'urAccepted' of undefined
        at syncthingController.js:135
        at angular.js:9452
        at processQueue (angular.js:13337)
        at angular.js:13353
        at Scope.$eval (angular.js:14589)
        at Scope.$digest (angular.js:14405)
        at Scope.$apply (angular.js:14694)
        at angular.js:14984
        at completeOutstandingRequest (angular.js:4959)
        at angular.js:5347
    console.<computed> @ logbar.js:11
    

    Ref: https://forum.syncthing.net/t/typeerror-cannot-read-property-uraccepted-of-undefined/16844

    bug ui 
    opened by tomasz1986 1
  • Scan due to watcher can not found local change while modify file to empty

    Scan due to watcher can not found local change while modify file to empty

    Does your log mention database corruption?

    [SJTYM] 2021/05/31 13:51:50.195841 folder.go:197: DEBUG: sendonly/[email protected] Scan due to watcher
    [SJTYM] 2021/05/31 13:51:50.196699 verboseservice.go:41: VERBOSE: Folder "nh-1" is now scan-waiting
    [SJTYM] 2021/05/31 13:51:50.196782 verboseservice.go:41: VERBOSE: Folder "nh-1" is now scanning
    [SJTYM] 2021/05/31 13:51:50.198567 verboseservice.go:41: VERBOSE: Local change detected in folder "nh-1": modified file .idea/workspace.xml
    [SJTYM] 2021/05/31 13:51:50.198605 folder.go:197: DEBUG: sendonly/[email protected] Scan due to watcher
    [SJTYM] 2021/05/31 13:51:50.198635 indexsender.go:126: DEBUG: [email protected] for nh-1 to  at 127.0.0.1:64825-127.0.0.1:57905/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256: Sending 1 files (<160 bytes)
    [SJTYM] 2021/05/31 13:51:50.198681 verboseservice.go:41: VERBOSE: Folder "nh-1" is now idle
    [SJTYM] 2021/05/31 13:51:50.198837 verboseservice.go:41: VERBOSE: Folder "nh-1" is now scan-waiting
    [SJTYM] 2021/05/31 13:51:50.198865 verboseservice.go:41: VERBOSE: Folder "nh-1" is now scanning
    [SJTYM] 2021/05/31 13:51:50.199165 verboseservice.go:41: VERBOSE: Folder "nh-1" is now idle
    [SJTYM] 2021/05/31 13:51:50.241943 model.go:1442: DEBUG: [email protected] REQ(in): MDPJNTF-OSPJC65-LZNCQGD-3AWRUW6-BYJULSS-GOCA2TU-5DWWBNC-TKM4VQ5: "nh-1" / ".idea/workspace.xml" o=0 s=20286 t=false
    [SJTYM] 2021/05/31 13:51:50.393486 model.go:947: DEBUG: Index update (in): MDPJNTF-OSPJC65-LZNCQGD-3AWRUW6-BYJULSS-GOCA2TU-5DWWBNC-TKM4VQ5 / "nh-1": 1 files
    [SJTYM] 2021/05/31 13:51:50.393657 verboseservice.go:41: VERBOSE: Device MDPJNTF-OSPJC65-LZNCQGD-3AWRUW6-BYJULSS-GOCA2TU-5DWWBNC-TKM4VQ5 sent an index update for "nh-1" with 1 items
    [SJTYM] 2021/05/31 13:51:50.996695 progressemitter.go:296: DEBUG: progress emitter: bytes completed for nh-1: 0
    [SJTYM] 2021/05/31 13:51:51.010055 model.go:839: DEBUG: [email protected] Completion(MDPJNTF-OSPJC65-LZNCQGD-3AWRUW6-BYJULSS-GOCA2TU-5DWWBNC-TKM4VQ5, "nh-1"): map[completion:100 globalBytes:243445017 globalItems:9619 needBytes:0 needDeletes:0 needItems:0 sequence:9890]
    [SJTYM] 2021/05/31 13:51:51.010114 verboseservice.go:41: VERBOSE: Summary for folder "nh-1" is map[errors:0 globalBytes:243445017 globalDeleted:25 globalDirectories:1944 globalFiles:7675 globalSymlinks:0 globalTotalItems:9644 inSyncBytes:243445017 inSyncFiles:7675 localBytes:169992395 localDeleted:19 localDirectories:1943 localFiles:7674 localSymlinks:0 localTotalItems:9636 needBytes:0 needDeletes:0 needDirectories:0 needFiles:0 needSymlinks:0 needTotalItems:0 pullErrors:0 sequence:15250 state:idle version:15250]
    [SJTYM] 2021/05/31 13:51:51.010148 verboseservice.go:41: VERBOSE: Completion for folder "nh-1" on device MDPJNTF-OSPJC65-LZNCQGD-3AWRUW6-BYJULSS-GOCA2TU-5DWWBNC-TKM4VQ5 is 100%
    

    Include required information

    Please be sure to include at least:

    • which version of Syncthing and what operating system you are using

    v1.11.1 (The latest main branch still has this problem)

    • what happened,

    When a file that modified to empty, FSEvent will not be received, which will cause the modification to not be synchronized to other devices in time: syncthing/[email protected]/watcher_fsevents.go:121

    And the file will be mark as modified when next scanning due to timer.

    • what you expected to happen instead, and

    When a file that modified to empty, local change detected immediately.

    • any steps to reproduce the problem.
    1. start syncthing
    2. sync any file that not empty
    3. modified the file to empty
    4. no local change found

    There is a pr, and the test cases in it perfectly reproduce this bug: #7732

    bug needs-triage 
    opened by anurnomeru 1
  • gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715)

    gui, lib: Fix tracking deleted locally-changed on encrypted (fixes #7715)

    This PR removes the ugly hack explained in #7715:

    The ugly hack is to ignore all items that have the locally-changed flag set and are deleted. That's fine for actually spurious files, that were removed, but obviously not for files that should be there.

    Instead it adds the non-ugly, but somewhat scary ability to remove file infos from the database. We already do that, but only in bulk when a device changes index-id/is removed. It isn't rocket science, it's just the half of updateLocalFiles that removes the old file. The scary bit is just that the code it touches is already scary.
    An alternative to this approach would be adding a new flag (e.g. FlagLocalReceiveEncryptedDeleted) and mark locally-changed, deleted items in receive-encrypted folders with that flag. This sidesteps the database changes, but instead of getting rid of useless file-infos, it keeps them with a new flag. That's why I'd prefer the db variant.

    To facilitate removing as well as updating files while scanning, I added a scanBatch type with Update and Remove methods, that replaces the batchAppendFunction that was passed around as a function argument before.

    Testing

    Added one new test that ensures that spurious files added to either receive-only or receive-encrypted folders, that are subsequently deleted, don't show up as locally-changed. And extended an existing test to check the same thing.

    opened by imsodin 4
  • Database GC can block for a long time

    Database GC can block for a long time

    https://forum.syncthing.net/t/syncthing-stuck-gui-seemingly-responsive-but-nothing-is-happening/16828

    Syncthing can become completely unresponsive when GC happens. Ideally that doesn't take long (e.g. ~10s for a largish setup with DB on SSD for me), but if DB is on slow storage it takes a long time. Since https://github.com/syncthing/syncthing/pull/7707 there's now at least an indication in the log that GC is happening and might render Syncthing unresponsive until it's finished.

    Options to improve the situation:

    • Don't do db compaction. Might be a good idea anyway, considering the comment on the method at goleveldb that pretty much advises against using it.
    • Do most of the GC without holding the global mutex, and only acquire that for the final part when actually deleting the entries.
    enhancement 
    opened by imsodin 0
Releases(v1.17.0)
Easy automated syncing between your computers and your MEGA Cloud Drive

MEGA Sync Client Easy automated syncing between your computers and your MEGA cloud drive. This repository contains all the development history of the

Mega Limited 1k Jun 5, 2021
aria2 is a lightweight multi-protocol & multi-source, cross platform download utility operated in command-line. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink.

aria2 - The ultra fast download utility Disclaimer This program comes with no warranty. You must use this program at your own risk. Introduction aria2

aria2 22.9k Jun 7, 2021
High performance file syncing and sharing, with also Markdown WYSIWYG editing, Wiki, file label and other knowledge management features.

Introduction Seafile is an open source cloud storage system with privacy protection and teamwork features. Collections of files are called libraries.

null 8.7k Jun 5, 2021
Easily and securely send things from one computer to another :crocodile: :package:

This project is supported by: croc is a tool that allows any two computers to simply and securely transfer files and folders. AFAIK, croc is the only

Zack 13.6k Jun 6, 2021
🧲 A feature rich cross platform Transmission BitTorrent client. Faster and has more functionality than the built-in web GUI.

Transmission Remote GUI Table of Contents Introduction Installation Linux Easy way (recommended) Harder way Windows Portable zip tarball (recommended)

Transmission Remote GUI working group 2.3k Jun 4, 2021
The missing Desktop application for Pushbullet.

PB for Desktop PB for Desktop is a lightweight open-source Desktop app for PushBullet. Receive native push notifications on macOS, Windows and Linux.

sidneys.github.io 462 Jun 3, 2021
Network file transfer application for Windows, OS X, & Linux

Warning: the master branch is very unstable at the moment. If you want stable builds, please visit the releases page or checkout the 0.3.4 branch. Nit

NitroShare 1.2k Jun 3, 2021
Securely and anonymously share files, host websites, and chat with friends using the Tor network

OnionShare OnionShare is an open source tool that lets you securely and anonymously share files, host websites, and chat with friends using the Tor ne

Micah Lee 4.6k Jun 6, 2021
qBittorrent BitTorrent client

qBittorrent - A BitTorrent client in Qt Description: qBittorrent is a bittorrent client programmed in C++ / Qt that uses libtorrent (sometimes called

qBittorrent project 12.2k Jun 6, 2021
get things from one computer to another, safely

Magic Wormhole Get things from one computer to another, safely. This package provides a library and a command-line tool named wormhole, which makes it

null 12.8k Jun 7, 2021
❤️ Streaming torrent app for Mac, Windows, and Linux

WebTorrent Desktop The streaming torrent app. For Mac, Windows, and Linux. Install Recommended Install Download the latest version of WebTorrent Deskt

WebTorrent 8.3k Jun 5, 2021