Speaker
Description
In recent years, Super-Resolution Microscopy (SRM) techniques have pushed the resolution of fluorescence microscopy down to the nanoscale, enabling the observation of in vivo cellular processes. Some of these techniques, such as Stochastic Optical Reconstruction Microscopy (STORM), achieve resolutions below 20 nm. However, they are less suitable for live imaging due to acquisition times of up to 90 minutes for a single image and present significant challenges for multicolor imaging.
To address these limitations, we employed a deep learning model, the Enhanced Super-Resolution Generative Adversarial Network (ESRGAN), to generate super-resolution images from diffraction-limited ones. This approach allows us to massively accelerate STORM acquisition and overcome the complexities associated with multicolor imaging. Specifically, we trained the model to produce STORM-like images from diffraction-limited widefield (WF) images, which can be acquired in seconds using a conventional fluorescence microscope.
We used a dataset comprising 76 pairs of High-Resolution (HR) and Low-Resolution (LR) images of cellular microtubules (STORM and WF images, respectively) to fine-tune a pre-trained ESRGAN model on this new image domain. Specifically, there is an upscaling factor of 4 between LR and HR images, with the HR images having dimensions of $2048 \times 2048$ pixels and file size ranging from 1 to 3 MB. Due to the large dimensions and file sizes, processing these images is computationally intensive.
The model, which consists of nearly $4 \cdot 10^{7}$ parameters, was trained over several sessions, each consisting of $4 \cdot 10^{5}$ iterations with different hyperparameter settings. During each iteration, the network minimizes a complex four-term perceptual-driven loss function to progressively generate images that closely resemble the original STORM images to a human observer. The performance of the model during training were validated using Peak-to-Signal Noise Ratio (PSNR) and Structure Similarity Index (SSIM) metrics every $10^{4}$ iterations and finally assessed on indipendent test sets.
Overall, all these operations constitute a computationally demanding process. To perform it, we employed the resources provided by the Computing Center of the Pisa Section of INFN. Our setup included 10 $\times$ 10-core Intel Xeon E5-2640v4 @2.40 GHz processors, 1 NVIDIA Tesla V100 with 32 GB VRAM, and 64 GB RAM, achieving an average execution time of 36 hours.
Additionally, we evaluated the network's performance on a benchmark dataset and obtained results comparable to other state-of-the-art models, though with a higher resolution scaling factor. Once trained, the model generates super-resolution images in a few seconds per image. A selected trained model has already been applied to directly observe molecular motor movement as with dual-color STORM technique, greatly facilitating and accelerating the procedure, thereby enhancing the study of cellular in vivo processes and enabling faster quantitative analysis.
Acknowledgments
The authors acknowledge European Union by the Next Generation EU through the Italian Ministry of University and Research under PNRR - M4C2-I1.5 ECS00000017 “Ecosistema dell’innovazione” Tuscany Health Ecosystem - THE project (Spoke 1 “Advanced Radiotherapies and Diagnostics in Oncology” and Spoke 4 "Nanotechnologies for diagnosis and therapy"), CUP I53C21000350006, CUP I53C22000780001; INFN-CSN5 under the Minibeam Radiotherapy (MIRO) project; Fondazione Pisa prog. n. 134/202; Center for Instrument Sharing of the University of Pisa (CISUP) is acknowledged for the access to the Fluorescence Super-Resolution Microscopy facility.
