FileSystemWatcher on Linux

[skywalkerisnull]

Hello,

I have the following scenario that works on Windows in dotnet 6.0:

1. Monitor directory is C:/Monitor
2. Have an existing directory such as C:/Monitor/Folder A, (a number of files are already present in this directory and no additional ones will be added)
3. Start the application
4. See all the existing files in Folder A added to the queue (separate thread)
5. Create a new folder C:/Monitor/Folder B
6. Place new images into Folder B
7. See these images placed in the queue only once

This works as expected with the following code:

        private void MonitorDirectory(string directoryToMonitor)
        {
            try
            {
                _logger.LogInformation(directoryToMonitor);
                var watcher = new FileSystemWatcher(directoryToMonitor, "*");

                watcher.Created += OnCreated;
                watcher.Renamed += OnRenamed;

                watcher.IncludeSubdirectories = true;
                watcher.EnableRaisingEvents = true;

                // var physicalFileWatcher = new PhysicalFilesWatcher(directoryToMonitor, watcher, false);
                // var changeToken = physicalFileWatcher.CreateFileChangeToken;
            }
            catch (Exception e)
            {
                _logger.LogError($"{e}");
            }
        }

        private void OnCreated(object sender, FileSystemEventArgs e)
        {
            if (Directory.Exists(e.FullPath))
            {
                return;
            }
            else
            {
                if (_fileTypes.Any<string>(e.FullPath.Contains))
                {
                    _fileTransferQueue.AddToTransferToCoordinator(e.FullPath);
                    _logger.LogTrace($"Added {e.FullPath} to the queue");
                }
            }
        }

        private void OnRenamed(object sender, FileSystemEventArgs e)
        {
            _logger.LogTrace($"File {e.FullPath} was renamed");
            if (Directory.Exists(e.FullPath))
            {
                return;
            }
            else
            {
                if (_fileTypes.Any<string>(e.FullPath.Contains))
                {
                    _fileTransferQueue.AddToTransferToCoordinator(e.FullPath);
                    _logger.LogTrace($"Added {e.FullPath} to the queue");
                }
            }
            return;
        }

How ever if we deploy this code to Ubuntu 20.04

1. Monitor directory is /data/Monitor
2. Have an existing directory such as /data/Monitor/Folder A (a number of files are already present in this directory and no additional ones will be added)
3. Start the application
4. See all the existing files in Folder A added to the queue (separate thread)
5. Create a new folder /data/Monitor/Folder B
6. Place new images into Folder B
7. ???

We do not see the files created or changed being added to the queue. Under the hood it looks like dotnet is using inotify which is what the python version of the above code was also using (watchdog uses inotify as well). The python version of the code will detect the newly created files and add them to the queue.

Any ideas on how I could debug this to figure out why the C# code is not detecting the files being added.

[skywalkerisnull]

By using the PhysicalFileProvider we are able to get most of what we are needing, in our case we needed to know when new files were added to the disk and send them to a queue. The code below achieves this for us and may be of use for others.

using FileStreamClient.Queues;
using Microsoft.Extensions.FileProviders;
using Microsoft.Extensions.Primitives;

namespace Project.Services
{
        private readonly ILogger<FileMonitor> _logger;
        private readonly IFileTransferQueue _fileTransferQueue;
        private readonly IServiceProvider _serviceProvider;
        private readonly IConfiguration _configuration;

        private readonly string _directoryToMonitor;
        private readonly List<string> _fileTypes;

        private FileSystemWatcher _fileSystemWatcher;

        private static PhysicalFileProvider _fileProvider;
        private static IChangeToken _fileChangeToken;

        private Dictionary<string, DateTime> _previousStateOfFiles = new();

    public class FileMonitor
    {
        public FileMonitor(ILogger<FileMonitor> logger, IServiceProvider serviceProvider, IConfiguration configuration)
        {
            _logger = logger;
            _serviceProvider = serviceProvider;
            _configuration = configuration;
            _fileTransferQueue = _serviceProvider.GetService<IFileTransferQueue>();

            _directoryToMonitor = _configuration.GetValue<string>("DirectoryToMonitor");
            _fileTypes = _configuration.GetSection("FileTypes").Get<List<string>>();

            for (int i = 0; i < _fileTypes.Count; i++)
            {
                _fileTypes[i] = _fileTypes[i].Replace("*", "");
            }
            StartMonitoringPhysicalDirectory();
        }

        private void StartMonitoringPhysicalDirectory()
        {
            _fileProvider = new PhysicalFileProvider(_directoryToMonitor);
            WatchForFileChanges();
        }

        private void Notify(object state)
        {
            WatchForFileChanges();
        }

        private void WatchForFileChanges()
        {
            ConcurrentDictionary<string, DateTime> currentStateOfFiles = new();
            
            IEnumerable<string> currentFiles= Directory.EnumerateFiles(_directoryToMonitor, "*.*", SearchOption.AllDirectories);

            foreach (string file in currentFiles)
            {
                // This will be either OnCreated, OnChanged, OnRenamed
                if (File.Exists(file))
                {
                    if (_previousStateOfFiles.TryGetValue(file, out DateTime previousWriteTime))
                    {
                        var currentWriteTime = File.GetLastWriteTime(file);
                        if (previousWriteTime == currentWriteTime)
                        {
                            // file has not been changed. It is being found again.
                            currentStateOfFiles.TryAdd(file, currentWriteTime);
                        }
                        else
                        {
                            // file has been Changed
                            currentStateOfFiles.TryAdd(file, currentWriteTime);
                            // OnChanged(file);
                        }
                    }
                    else
                    {
                        // file has been created or renamed
                        currentStateOfFiles.TryAdd(file, File.GetLastWriteTime(file));
                        OnCreated(file);
                    }
                }
                else
                {
                    _logger.LogTrace($"{file }has been deleted"); 
                    // OnDeleted();
                }

            }
            _fileChangeToken = _fileProvider.Watch("**/*.*");
            _fileChangeToken.RegisterChangeCallback(Notify, default);

            _previousStateOfFiles = currentStateOfFiles;
        }

        private void MonitorDirectory(string directoryToMonitor)
        {
            try
            {
                _logger.LogInformation(directoryToMonitor);
                _fileSystemWatcher = new FileSystemWatcher(directoryToMonitor, "*");

                _fileSystemWatcher.Created += OnCreated;
                _fileSystemWatcher.Renamed += OnRenamed;

                _fileSystemWatcher.IncludeSubdirectories = true;
                _fileSystemWatcher.EnableRaisingEvents = true;

            }
            catch (Exception e)
            {
                _logger.LogError($"{e}");
            }

        }

        private void OnCreated(object sender, FileSystemEventArgs e)
        {
            var fullPath = e.FullPath;
            OnCreated(fullPath);

        }

        private void OnCreated(string fullPath)
        {
            if (Directory.Exists(fullPath))
            {
                return;
            }
            else
            {
                if (_fileTypes.Any<string>(fullPath.Contains))
                {
                    _fileTransferQueue.AddToTransferToCoordinator(fullPath);
                    _logger.LogTrace($"Added {fullPath} to the queue");
                }
            }
        }

This also had a by product of grabbing all the existing files from disk as well.

[maxisam]

This seems like it will take a lot of memory and not efficient when there are lots of files. Feels like this is not a real solution. It is a pretty cool workaround tho.

[skywalkerisnull]

I agree that it is a work around.

For our situation, the files are processed and moved on very quickly, usually before another file appears, so the queue and dictionary are very small. But it would need a different approach if the number of files already on disk is large, or the ones being added are being added at a rate higher than they are being removed.

This also may start eating lots of clock cycles as we are looping over a potentially large enumerable.

[jozkee]

@skywalkerisnull you basically coded your own polling mechanism. PhysicalFileProvider already has that, and you can turn it on with these two properties.

_fileProvider = new PhysicalFileProvider(_directoryToMonitor)
{
    UsePollingFileWatcher = true,
    UseActivePolling = true
};

From the docs:

By default, PhysicalFileProvider uses FileSystemWatcher to listen to file change events for Watch(String). FileSystemWatcher is ineffective in some scenarios such as mounted drives. Polling is required to effectively watch for file changes.

You can find more examples on how to use polling by looking at the tests.

[amorenojr]

I am trying to implement a FileSystemWatcher in a .NET 8 C# console application and its not working when used on Linux (works great on Windows). Was curious if anyone knows what might be the issue. Has this issue been resolved or might be resolved in the upcoming releases of .NET?