denoiSplit: a method for joint microscopy image splitting and unsupervised denoising

Ashesh¹, Florian Jug¹

¹Human Technopole, Italy
ECCV 2024

Summary

For Computer Vision enthusiasts, we present one of the first approaches to tackle joint image decomposition with unsupervised denoising. We work with HVAE inspired architecture. We integrate Noise models into our approach for better unsupervised denoising.
From microscopists' perspective, the idea is to image two different structures (like Actin and Tubulin) into a single channel. The obtained image stack will therefore contain superimposed structures which will often be noisy. With our machine learning based approach, the goal is to decompose the noisy superimposed image into two channels, each containing denoised version of one type of structure.

The task: Decompose the noisy input patch (left) into two denoised output patches (right). Images taken from notMNIST dataset.

Technical Overview

In this work, the task is to decompose a noisy input patch into two constituent output patches. We build our work on our previous work µSplit. We make multiple changes to the original µSplit architecture to make it suitable for the task of joint image decomposition and unsupervised denoising. Firstly, we revert back to the classical KL divergence loss formulation for the unsupervised denoising task. Secondly, we integrate noise models into our approach for better unsupervised denoising. Since the architecture of denoiSplit is inspired from hierarchical VAEs, we inherit sampling: the ability to sample multiple predictions from the same input patch. We use this ability to add calibration to our network wherein we can get a pixelwise estimate of error just by sampling multiple predictions, without the need of any ground truth.

Qualitative results on the ER vs MT task. We show examples of noisy inputs, individual noisy channel training data (GT), and predictions by one of the baselines (μSplit) and our own results obtained with denoiSplit. We show high SNR channel images (not used during training) and show PSNR values w.r.t. these images. Additionally, we plot histograms of various panels for comparison (see legend on the right). The bottom cell in the first column of each panel shows the used noise models. The superimposed plots (green) show the distribution of noisy observations for two clean signal intensities.

Network architecture: In this work we use a variational encoder-decoder network to jointly solve an usupervised denoising and image splitting task. We make use of two noise models, one for each output channel. The noise models capture the noise distribution of the respective output channels and help to provide better likelihood estimate.

Sampling & Calibration: Here we show sampling for ER vs MT splitting task. We show cropped input (col 1), corresponding noisy ground trutn (col 2), two predicted samples (col 3 & 4), the difference between the two samples (S₁ - S₂) (col 5), the MMSE prediction (col 6), and otherwise unused high SNR microscopy image (col 7). All image patches are of size 256x256. We also show PSNR w.r.t high SNR patches. The dot plots in the first column show is calibration plot, showcasing that the error estimate we propose (x-axis) is close to the actual error (y-axis).

Paper video.

Video Presentation

BibTeX


        @InProceedings{ashesh_2024_ECCV,
          author       = {Ashesh and
                          Jug, Florian},
          title        = {denoiSplit: a method for joint microscopy image splitting and unsupervised denoising},
          month        = {October},
          year         = {2024},
          booktitle = {European Conference on Computer Vision (ECCV) 2024},
        }