{ "cells": [ { "cell_type": "markdown", "id": "e3b95d40-f2a3-4d3b-82cc-c5204b1efd98", "metadata": {}, "source": [ "# Notebook showing all the spectral normalization methods implemented in pyFRESCO" ] }, { "cell_type": "code", "execution_count": 1, "id": "64fa6b08-95d6-4089-914a-62a29f99dfde", "metadata": {}, "outputs": [], "source": [ "import pyfresco" ] }, { "cell_type": "markdown", "id": "4b61083f-0ce3-4983-b702-8c228640b33f", "metadata": {}, "source": [ "## Data import\n", "\n", "In the following code cell we define the paths pointing to the img and header files of the proper hyperspectral datacube and of the summary parameters datacube.\n", "These two datacubes are the ones needed by pyFRESCO to work properly." ] }, { "cell_type": "code", "execution_count": 2, "id": "fade6b0e-6d27-41ad-a2a7-d3e547802a00", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['R770', 'RBR', 'BD530_2', 'SH600_2', 'SH770', 'BD640_2', 'BD860_2', 'BD920_2', 'RPEAK1', 'BDI1000VIS', 'R440', 'IRR1', 'BDI1000IR', 'OLINDEX3', 'R1330', 'BD1300', 'LCPINDEX2', 'HCPINDEX2', 'VAR', 'ISLOPE1', 'BD1400', 'BD1435', 'BD1500_2', 'ICER1_2', 'BD1750_2', 'BD1900_2', 'BD1900R2', 'BDI2000', 'BD2100_2', 'BD2165', 'BD2190', 'MIN2200', 'BD2210_2', 'D2200', 'BD2230', 'BD2250', 'MIN2250', 'BD2265', 'BD2290', 'D2300', 'BD2355', 'SINDEX2', 'ICER2_2', 'MIN2295_2480', 'MIN2345_2537', 'BD2500_2', 'BD3000', 'BD3100', 'BD3200', 'BD3400_2', 'CINDEX2', 'BD2600', 'IRR2', 'IRR3', 'R530', 'R600', 'R1080', 'R1506', 'R2529', 'R3920']\n" ] } ], "source": [ "path_if_mtrdr = \"FRT00009b5a/frt00009b5a_07_if165j_mtr3.img\" # Path to the proper hyperspectral datacube\n", "head_if_mtrdr = \"FRT00009b5a/frt00009b5a_07_if165j_mtr3.hdr\" # Path to the header of the proper hyperspectral datacube\n", "\n", "path_sr_mtrdr = \"FRT00009b5a/frt00009b5a_07_sr165j_mtr3.img\" # Path to the summary parameters datacube\n", "head_sr_mtrdr = \"FRT00009b5a/frt00009b5a_07_sr165j_mtr3.hdr\" # Path to the header of the summary parameters datacube\n", "\n", "img , img_sr , wavelength , sr_names = pyfresco.open_raw(path_if_mtrdr , head_if_mtrdr , path_sr_mtrdr , head_sr_mtrdr) # Open the two datacubes\n", "\n", "Nbands = len(wavelength)\n", "print(sr_names)" ] }, { "cell_type": "markdown", "id": "dbd58a61-6cd6-4e2d-b144-f5d2874be3e0", "metadata": {}, "source": [ "## Target spectrum extraction\n", "\n", "Before applying any normalization method we need a target ROI spectrum.\n", "Here we repeat the polygon extraction used in the first and second tutorial notebooks." ] }, { "cell_type": "code", "execution_count": 3, "id": "0ca73990-0e80-49a4-9d09-3015eab0f5a0", "metadata": {}, "outputs": [], "source": [ "SE = pyfresco.SpectraExtract(img , Nbands , wavelength , 750 , 2600) # Definition of the object for spectral extraction\n", "\n", "RGB = SE.upload_map('PHY' , folder = 'FRT00009b5a') # Upload an RGB map for the ROI selection" ] }, { "cell_type": "code", "execution_count": 4, "id": "b3c9658f-3086-4deb-8eed-26018281ade8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of points inside the drewn polygon: 297\n" ] } ], "source": [ "polygon_spectra , L , mask = SE.polygon_spectra(save_pixel = True ,\n", " folder = 'FRT00009b5a' ,\n", " name = 'target_polygon_coordinates') # Select the target ROI\n", "\n", "polygon_spectrum , polygon_error = SE.final_spectra(mean = True , c = 'blue') # Mean and standard deviation target spectrum" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![Point selection](notebook3_images/polygon_target.png)\n", "![Point selection2](notebook3_images/target_mean_spectrum.png)" ] }, { "cell_type": "markdown", "id": "b920b5a0-d173-461f-85e9-40a51d9a8b75", "metadata": {}, "source": [ "## Definition of the normalization object\n", "\n", "The same `SpectraNorm` object can be reused to test different neutral-spectrum choices.\n", "Each method overwrites the currently stored neutral spectrum, so the normalization is computed immediately after each neutral selection." ] }, { "cell_type": "code", "execution_count": 5, "id": "9ec50f1a-eb7b-4f22-9204-a659849f33e5", "metadata": {}, "outputs": [], "source": [ "SN = pyfresco.SpectraNorm(RGB , Nbands , img , img_sr , wavelength , # Definition of the object for spectral normalization\n", " polygon_spectrum , polygon_error , polygon_spectra ,\n", " 750 , 2600)" ] }, { "cell_type": "markdown", "id": "dc7bf7a6-5718-4d78-9412-d6d15321e237", "metadata": {}, "source": [ "## Neutral Polygon Method\n", "\n", "The first normalization method uses a manually selected neutral polygon.\n", "The median spectrum of this neutral ROI is used as denominator for the target spectrum." ] }, { "cell_type": "code", "execution_count": 6, "id": "06285e30-0dc1-408f-9ec5-7f4385c86030", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of points inside the drewn polygon: 2150\n", "9.386281588447654 % of the errors, set to zero for simplicity, are in reality NaN values.\n" ] } ], "source": [ "neutral_polygon_spectra , neutral_polygon , neutral_polygon_error , L_neutral , mask_neutral = SN.neutral_polygon_spectra(\n", " save_pixel = True ,\n", " folder = 'FRT00009b5a' ,\n", " name = 'neutral_polygon_coordinates') # Draw a neutral ROI\n", "\n", "SN.plot_together() # Plot target and neutral spectra\n", "\n", "norm_polygon , error_polygon = SN.norm_spectra() # Classical ratio and propagated error\n", "SN.normplot()\n", "\n", "norm_polygon_boot , error_polygon_boot = SN.bootstrapnorm(convexhull = False , N = 5000) # Bootstrap normalization\n", "SN.normplot()" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![Neutral polygon selection](notebook3_images/neutral_ROI.png)\n", "![Spectra together](notebook3_images/neutral_target_together_polygon_norm.png)\n", "![Spectra classical norm](notebook3_images/polygon_norm_classical_norm.png)\n", "![Spectra bootstrap norm](notebook3_images/polygon_norm_bootstrap_norm.png)" ] }, { "cell_type": "markdown", "id": "53326e6e-1b9d-4998-a1bf-2d8c52255953", "metadata": {}, "source": [ "## Convex Hull Method\n", "\n", "The convex hull method uses the continuum of the target spectrum as neutral spectrum.\n", "This is useful when an external neutral ROI is not available." ] }, { "cell_type": "code", "execution_count": 7, "id": "172e98c4-34cc-4201-a477-9b94fe8c18ad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "100.0 % of the errors, set to zero for simplicity, are in reality NaN values.\n" ] } ], "source": [ "neutral_hull , hull_spectrum , hull_error = SN.neutral_convex_hull(interp = 'linear') # Compute the convex hull neutral spectrum\n", "\n", "SN.plot_together(convex_hull = True)\n", "\n", "norm_hull , error_hull = SN.norm_spectra(convex_hull = True) # Normalize using the convex hull\n", "SN.normplot(convex_hull = True)\n", "\n", "norm_hull_boot , error_hull_boot = SN.bootstrapnorm(convexhull = True , N = 5000) # Bootstrap version\n", "SN.normplot(convex_hull = True)" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![convex hull](notebook3_images/convex_hull_method.png)\n", "![classic norm](notebook3_images/convex_hull_norm_classical.png)\n", "![bootstrap norm](notebook3_images/convex_hull_norm_bootstrap.png)" ] }, { "cell_type": "markdown", "id": "feb27db5-4ec4-472b-9fda-7482f5ceed09", "metadata": {}, "source": [ "## Single All-Map Method\n", "\n", "The single all-map method searches for pixels with low values in one RGB map.\n", "Those pixels are used to build the neutral spectrum." ] }, { "cell_type": "code", "execution_count": 9, "id": "9bf16ec2-f36f-4527-b22c-749d93beb9f2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0 % of the errors, set to zero for simplicity, are in reality NaN values.\n" ] } ], "source": [ "neutral_single , neutral_single_median , neutral_single_mad , x_single , y_single = SN.neutral_all_map_single(threshold = 0) # Search low-valued pixels in the current RGB map\n", "\n", "SN.plot_together()\n", "\n", "norm_single , error_single = SN.norm_spectra()\n", "SN.normplot()\n", "\n", "norm_single_boot , error_single_boot = SN.bootstrapnorm(convexhull = False , N = 5000) # Bootstrap version\n", "SN.normplot(convex_hull = True)" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![zero px](notebook3_images/PHY_zeroed_pixels.png)\n", "![together sam](notebook3_images/singleallmap_together.png)\n", "![classic norm2](notebook3_images/singleallmap_classical_norm.png)\n", "![bootstrap norm2](notebook3_images/singleallmap_bootstrap_norm.png)" ] }, { "cell_type": "markdown", "id": "71f028a6-878b-4a24-8991-d304f98ad798", "metadata": {}, "source": [ "## Multiple All-Map Method\n", "\n", "The multiple all-map method repeats the same low-value pixel search on several RGB maps.\n", "Only pixels satisfying the requested overlap condition are retained." ] }, { "cell_type": "code", "execution_count": 10, "id": "6e80a677-4bd2-44fe-9067-653439762ede", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.0 % of the errors, set to zero for simplicity, are in reality NaN values.\n" ] } ], "source": [ "RGBs_names = ['HYD' , 'PHY' , 'MAF'] # RGB maps previously saved with RGBImageManipulator.savemap()\n", "RGB_path = 'FRT00009b5a/'\n", "\n", "neutral_multiple , neutral_multiple_median , neutral_multiple_mad , superimposed , zero_map , xx , yy = SN.neutral_all_map_multiple(\n", " RGBs_names = RGBs_names ,\n", " RGB_path = RGB_path ,\n", " p = len(RGBs_names) ,\n", " threshold = 0 ,\n", " names = RGBs_names)\n", "\n", "SN.plot_together()\n", "\n", "norm_multiple , error_multiple = SN.norm_spectra()\n", "SN.normplot()\n", "\n", "norm_multiple_boot , error_multiple_boot = SN.bootstrapnorm(convexhull = False , N = 5000) # Bootstrap version\n", "SN.normplot(convex_hull = True)" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![zero px multi](notebook3_images/multi_zero_pixels.png)\n", "![zero px multi total](notebook3_images/multi_zero_pixels_total.png)\n", "![together mam](notebook3_images/multiallmap_together.png)\n", "![classic norm3](notebook3_images/multiallmap_classical_norm.png)\n", "![bootstrap norm3](notebook3_images/multiallmap_bootstrap_norm.png)" ] }, { "cell_type": "markdown", "id": "698ea4ee-60ef-4531-8c50-ba12aae04f3f", "metadata": {}, "source": [ "## Mineral Mask Method\n", "\n", "The mineral mask method uses spectral-parameter thresholds to select the neutral pixels.\n", "Reflectance parameters are treated differently from band-depth parameters." ] }, { "cell_type": "code", "execution_count": 13, "id": "9a0a0f0e-338f-48e5-a5b2-f14925bceeab", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "constraints_mask.size=581064\n", "constraints_mask.sum()=np.int64(13961)\n", "0.0 % of the errors, set to zero for simplicity, are in reality NaN values.\n" ] } ], "source": [ "band_names = ['R770' , 'BDI1000IR' , 'OLINDEX3' , 'BD1300' , 'LCPINDEX2' , 'HCPINDEX2',\n", " 'D2200' , 'BD2290' , 'D2300' , 'BD2500_2' , 'SINDEX2' , 'BD1900_2']\n", "band_values = [0.23,0.035,0.09,0,0,0,0.001,0.001,0.001,0,0.0045,0.012]\n", "\n", "neutral_mask , neutral_mask_median , neutral_mask_mad = SN.mineral_mask(\n", " b = 500 ,\n", " band_n = band_names ,\n", " band_v = band_values)\n", "\n", "SN.plot_together()\n", "\n", "norm_mask , error_mask = SN.norm_spectra()\n", "SN.normplot()\n", "\n", "norm_mineralmask_boot , error_mineralmask_boot = SN.bootstrapnorm(convexhull = False , N = 5000) # Bootstrap version\n", "SN.normplot(convex_hull = True)" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![mmdist](notebook3_images/mineral_mask_distributions.png)\n", "![zero px multi total mm](notebook3_images/mienral_mask_pixels.png)\n", "![together mmm](notebook3_images/mineral_mask_together.png)\n", "![classic norm4](notebook3_images/classical_norm_mineralmask.png)\n", "![bootstrap norm4](notebook3_images/bootstrap_norm_mineralmask.png)" ] }, { "cell_type": "markdown", "id": "0f0707a5-a600-40b0-a437-d15a055510c9", "metadata": {}, "source": [ "## Smoothing of the normalized spectrum\n", "\n", "After any normalization, the normalized spectrum can be smoothed with either a moving average or a Savitzky-Golay filter." ] }, { "cell_type": "code", "execution_count": 14, "id": "a43484c9-005f-441e-b481-07e2f3a2e4ec", "metadata": {}, "outputs": [], "source": [ "smooth_movmean = SN.moving_average(window_size = 5) # Moving average smoothing\n", "\n", "smooth_savgol = SN.savgol(window = 7 , order = 2) # Savitzky-Golay smoothing" ] }, { "cell_type": "markdown", "id": "1da1a63e-93aa-44a2-97b2-30583669aaee", "metadata": {}, "source": [ "The output of the above cell should look like this: \n", "![movingmean](notebook3_images/movingmean_smooth.png)\n", "![savgol](notebook3_images/savgol_smooth.png)" ] }, { "cell_type": "markdown", "id": "72c87ab5-5915-4b3c-becf-4515935893f5", "metadata": {}, "source": [ "## Saving normalized spectra\n", "\n", "The normalized spectrum and associated neutral spectrum can be saved for later analysis notebooks." ] }, { "cell_type": "code", "execution_count": null, "id": "f6b6d7ba-a545-4bc9-b9d4-bca757eb716d", "metadata": {}, "outputs": [], "source": [ "SN.save_spectrum(name = 'target_polygon' ,\n", " folder = 'FRT00009b5a/' ,\n", " method = 'min' ,\n", " normalized = True)" ] }, { "cell_type": "code", "execution_count": null, "id": "6ee44a04-f05e-4373-a84f-7e9a254d24d6", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.20" }, "nbsphinx": { "execute": "never" } }, "nbformat": 4, "nbformat_minor": 5 }