Monitoring files in Azure Blob Storage

We recently had the need to synchronize (local) in-memory cache items between 2 web applications running in Windows Azure. A change made in the admin application should update the cache of our public (read-only) web application.

Azure Blob Storage was a good candidate for storing our cache dependency files as it can be accessed concurrently by multiple roles (and role instances).

As we were using the System.Runtime.Caching.MemoryCache we looked at using the built in FileChangeMonitor. Unfortunately this is tied to the local file system, so was not much use to us.

So we had no choice but to create our own monitor. Like most of our Azure developments we didn’t want to be coupled to a specific environment (so we can use it with both local and Azure file systems).

Let’s start with the IFileChangeMonitor interface:

public interface IFileChangeMonitor
{
    void Start();
    void Stop();
    void MonitorFile(string path);
    event Action<FileChangedEventArgs> OnFileChanged;
}

The OnFileChanged event is raised when a file changes. FileChangedEventArgs contains a reference to the monitored file:

public class FileChangedEventArgs
{
    public string Path { get; protected set; }

    public FileChangedEventArgs(string path)
    {
        this.Path = path;
    }
}

Now lets look at our implementation of IFileChangeMonitor. The StorageProviderFileMonitor can use any instance of IStorageProvider (our file system abstraction).

public class StorageProviderFileMonitor : IFileChangeMonitor
{
    private const int MonitorIntervalInSeconds = 10;

    private static readonly object syncLock = new object();

    private readonly IStorageProvider storageProvider;
    private readonly ILogger logger;
    private readonly ICollection<FileState> filesToMonitor = new List<FileState>();
    private readonly CancellationTokenSource tokenSource;

    private bool isRunning;

    public StorageProviderFileMonitor(IStorageProvider storageProvider, ILogger logger)
    {
       this.storageProvider = storageProvider;
        this.logger = logger;
        
        tokenSource = new CancellationTokenSource();
    }

    public event Action<FileChangedEventArgs> OnFileChanged;

    public void Start()
    {
        if (isRunning)
        {
            throw new InvalidOperationException("The monitor is already running.");
        }

        var ct = tokenSource.Token;

        var task = new Task(() =>
        {
            Sleep();

            while (true)
            {
                this.Cycle();

                if (ct.IsCancellationRequested)
                {
                    return;
                }

                Sleep();
            }

        },
        ct,
        TaskCreationOptions.LongRunning);

        task.Start();
        isRunning = true;
    }

    public void Stop()
    {
        if (!isRunning)
        {
            throw new InvalidOperationException("The monitor is not running.");
        }

        tokenSource.Cancel();
        isRunning = false;
    }

    public void MonitorFile(string path)
    {
        if (string.IsNullOrEmpty(path))
        {
            throw new ArgumentNullException("path");
        }

        if (!filesToMonitor.Any(f => f.Path.Equals(path, StringComparison.InvariantCultureIgnoreCase)))
        {
            lock (syncLock)
            {
                filesToMonitor.Add(new FileState(path));
            }
        }
    }

    private void Cycle()
    {
        // check each file in parallel
        Parallel.ForEach(filesToMonitor, fs => CheckFile(fs));
    }

    private void CheckFile(FileState fileState)
    {
        try
        {
            var file = GetFile(fileState.Path);
            var lastUpdated = file.GetLastUpdated();

            if (fileState.LastChanged != lastUpdated)
            {
                fileState.SetLastChanged(lastUpdated);
                OnFileChanged(new FileChangedEventArgs(fileState.Path));
            }
        }
        catch (Exception ex)
        {
            logger.Error(ex);
        }
    }

    private IStorageFile GetFile(string path)
    {
        return storageProvider.GetFile(path);
    }

    protected void Sleep()
    {
        Thread.Sleep(TimeSpan.FromSeconds(MonitorIntervalInSeconds));
    }

    private class FileState
    {
        public string Path { get; protected set; }
        public DateTime LastChanged { get; protected set; }

        public FileState(string path)
        {
            this.Path = path;
            LastChanged = DateTime.UtcNow;
        }

        public void SetLastChanged(DateTime lastChanged)
        {
            LastChanged = lastChanged;
        }
    }
}

When the monitor starts we kick off a long running Task that checks the monitored files every 10 seconds (MonitorIntervalInSeconds). The monitor stores a FileState instance for each file we are monitoring.

On each cycle we check all the monitored files for changes. Each file is retrieved using the storage provider and its modifed date compared to the LastChanged date of the matching FileState. If a change is detected we update the FileState instance and raise the OnFileChanged event.

When our application starts we register our event handler and start the monitor:

public void Execute()
{	
	// start file change monitor
	monitor.OnFileChanged += monitor_OnFileChanged;
	monitor.Start();
}

void monitor_OnFileChanged(FileChangedEventArgs args)
{
	var cache = MemoryCache.Default; 
	cache.Set(
        Path.GetFileName(args.Path), 
        Guid.NewGuid().ToString(), 
        DateTimeOffset.Now.AddMinutes(SiteCacheProvider.CacheDependencyDurationInMinutes)
    );
}

In the event handler we update our in-memory cache, using the file name as the cache item key (I’ll cover exactly how this works in the next post). Of course you could do anything you wanted in this handler.

We can start monitoring a file at any time by getting our instance of IFileChangeMonitor (scoped as a singleton by our DI tool) and call MonitorFile():

// create cache dependency file
var file = storageProvider.CreateOrGetFile(GetCacheKey()); // file name same as cachekey
// start monitoring file
monitor.MonitorFile(file.GetPath());

When a change is made in our admin web application we “touch” our cache dependency file to update it. This is done by opening up a stream to the blob and disposing it:

public void InvalidateCache(Guid siteId)
{
    var cacheFile = string.Format(CacheFileFormat, siteId);

    if (storageProvider.FileExists(cacheFile))
    {
        storageProvider.GetFile(cacheFile).OpenWrite().Dispose();
    }
}

The monitor has many uses, for example, monitoring changes to configuration files.

I’ll try and get the full source uploaded in the next few days.