Interactive Dashboard | Carlos Mendez

Regional dynamics of DMSP-like nighttime lights 1992-2019

Fri, 14 Mar 2025 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

Title Slide

A Harmonized Global Nighttime Light Dataset (1992–2018)
Authors: Xuecao Li, Yuyu Zhou, Min Zhao, & Xia Zhao
Published in: Scientific Data (2020)
DOI: https://doi.org/10.1038/s41597-020-0510-y

🌍 Introduction

Nighttime light (NTL) data provide insights into human activity, urbanization, and economic development.
Two primary sources: DMSP/OLS (1992–2013) & VIIRS (2012–2018).
Challenge: Significant inconsistency between DMSP and VIIRS data.
Objective: Develop a harmonized global NTL dataset for long-term analysis.

👩‍💻 Data Collection

DMSP/OLS NTL Data (1992–2013):
- Downloaded from the Payne Institute for Public Policy.
- Digital number (DN) values range from 0 to 63.
- Spatial resolution: 30 arc-seconds.
VIIRS/DNB Data (2012–2018):
- Higher spatial & radiometric resolution.
- Monthly composites were processed into annual data.
- Spatial resolution: 15 arc-seconds.

🔄 Methodology

Three-step harmonization process:
1. Annual Composition of VIIRS Data:
  - Used cloud-free coverage data as a weighting factor.
  - Removed noise from aurora, fires, and temporary sources using thresholding techniques.
  - Applied a weighted averaging approach to generate annual composite images from monthly VIIRS data.
2. Conversion of VIIRS to DMSP-like Data:
  - Kernel Density (KD) Approach:
    - Aggregated VIIRS radiance data (15 arc-seconds) to match DMSP resolution (30 arc-seconds).
    - Used a Gaussian point-spread function to reduce differences in radiance distribution.
  - Logarithmic Transformation:
    - Applied logarithmic transformation to adjust radiance variations in urban, suburban, and rural areas.
    - Reduced differences in brightness levels between high and low radiance pixels.
  - Sigmoid Function Conversion:
    - Developed a sigmoid function based on 2013 data to map transformed VIIRS data to DMSP-like DN values.
    - Parameters of the function were optimized at a global scale and validated at continental and national levels.
3. Integration of DMSP & VIIRS Data:
  - Inter-calibrated DMSP data (1992–2013) using a stepwise calibration approach.
  - Applied derived sigmoid function to convert VIIRS data (2014–2018) into DMSP-like DN values.
  - Merged both datasets to create a consistent 27-year global NTL dataset.

🌍 Technical Validation

Histogram Comparison:
- Compared DN distributions of inter-calibrated DMSP and VIIRS-derived DMSP-like data.
- Verified similarity in data distributions for overlapping years (2012–2013).
- Identified a slight increase in high DN values (>60) due to DMSP saturation effects.
Temporal Consistency (1992–2018):
- Assessed trends in total nighttime light (NTL) intensity and number of lit pixels.
- Conducted analysis using different DN thresholds (7, 20, 30) to minimize low-luminance noise.
- Observed a stable and continuous trend in high-luminance areas (DN > 20).
Spatial Validation:
- Evaluated spatial accuracy using major metropolitan areas (e.g., Beijing, New York).
- Compared observed DMSP, raw VIIRS radiance, and DMSP-like VIIRS data.
- Verified agreement in urban spatial patterns, indicating robustness of the integration approach.
Independent Socioeconomic Correlations:
- Compared trends with external socioeconomic indicators (e.g., GDP, electricity consumption).
- Strong correlations between harmonized NTL dataset and economic development patterns.
- Ensures reliability of dataset for studies on urbanization and economic growth.

🏰 Applications of the Dataset

Urban expansion analysis (e.g., Beijing-Tianjin region).
Socioeconomic studies (e.g., GDP estimation, electricity consumption).
Environmental monitoring (e.g., light pollution, carbon emissions).
Disaster impact assessments (e.g., conflict zones, power outages).

📊 Key Findings & Conclusion

The harmonized NTL dataset enables long-term analysis (1992–2018).
Overcomes DMSP-VIIRS inconsistencies using a systematic integration approach.
Provides a valuable resource for urbanization, economics, and environmental studies.
Dataset Access: Original data repository
GEE dataset Access: Awesomme GEE community catalog
Exploratory Tool: GEE web app by Carlos Mendez

See web app in full screen HERE

Regional dynamics of luminosity-based GDP 1992-2019

Fri, 14 Mar 2025 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

📊 Global 1 km × 1 km Gridded Revised Real GDP and Electricity Consumption (1992–2019) 🌍

📌 Introduction

This study presents a high-resolution (1 km × 1 km) global dataset of real GDP and electricity consumption from 1992 to 2019.
The dataset is based on nighttime light data, calibrated using a novel Particle Swarm Optimization-Back Propagation (PSO-BP) algorithm.
The aim is to provide a more accurate and continuous measurement of economic activity worldwide.
Citation: Jiandong Chen, Ming Gao, Shulei Cheng, Wenxuan Hou, Malin Song, Xin Liu & Yu Liu (2022). Nature Scientific Data

💡 Background & Significance

📈 GDP and ⚡ electricity consumption are key indicators of economic development.
Traditional economic statistics often suffer from inconsistencies, especially in developing countries.
🛰️ Nighttime light data from satellites has been widely used to estimate economic output, but previous approaches had limitations in accuracy and continuity.

🗂️ Methodology

📚 Data Sources

🛰️ Nighttime Light Data:
- Defense Meteorological Satellite Program’s Operational Linescan System (DMSP/OLS)
- National Polar-orbiting Partnership’s Visible Infrared Imaging Radiometer Suite (NPP/VIIRS)
📊 GDP Data: Official GDP statistics from 175 countries, revised using nighttime light data.
⚡ Electricity Consumption Data: Collected for 134 countries.

⚙️ Data Processing & Calibration

🖥️ Image Unification:
- Applied PSO-BP algorithm to standardize DMSP/OLS and NPP/VIIRS data.
- Adjusted for sensor inconsistencies and temporal discontinuities.
📍 Grid-Level Estimation:
- GDP and electricity consumption distributed using a top-down approach.
- Revised real GDP growth based on a weighted combination of official statistics and nightlight-derived estimates.
🛠️ Correction Mechanisms:
- Eliminated biases in nighttime light intensity.
- Accounted for regional heterogeneity in economic activities.
- Applied inter-annual continuous series correction to ensure temporal consistency in nighttime light data.

🔍 PSO-BP Algorithm for Data Calibration

🔄 Training Process:
- Used artificial neural networks to train a model mapping relationships between GDP, electricity consumption, and nighttime light intensity.
- Divided the data into training (60%) and testing (40%) samples.
- Applied Particle Swarm Optimization (PSO) to optimize the Back Propagation (BP) neural network.
- Iterated 50 times with 20 population size to refine model accuracy.
📉 Data Matching Across Sensors:
- Addressed discrepancies between DMSP/OLS (1992–2013) and NPP/VIIRS (2012–2019) by:
  - Applying pixel-level calibration.
  - Ensuring consistency in spatial patterns by matching high/low DN values.
  - Normalizing DN values and applying machine learning for seamless integration.
📊 Estimation of GDP and Electricity Consumption:
- Derived GDP growth rate as a function of official GDP and nighttime light data.
- Applied weights (ρ = 0.94 for developed countries, ρ = 0.66 for developing countries) to adjust official GDP growth.
- Estimated electricity consumption growth using a combined function of GDP and light intensity growth.

🔬 Technical Validation

✔️ Validity Testing for Nighttime Light Data
- 🏙️ Urban Built-up Areas Validation: Compared estimated urban built-up areas with official MCD12Q1 land cover data, showing high accuracy.
- 🌎 Cross-sectional Analysis: Strong correlation (R² ~ 0.87) between sum of DN values and national GDP/electricity consumption.
- Validated temporal consistency of corrected light data across years.
🤖 Validation of PSO-BP Algorithm
- Trained the PSO-BP model using national GDP, electricity consumption, and nighttime light data.
- Achieved an R² > 0.99 in training and testing datasets, confirming model robustness.
- Outperformed previous models with improved spatiotemporal consistency.
- Compared simulated GDP/electricity consumption with external datasets, showing strong alignment.

📊 Key Findings

📈 Improved GDP Estimation:
- The revised GDP dataset offers better accuracy than official statistics, particularly for developing nations.
- Provides a more granular view of economic activities at a local level.
⚡ Electricity Consumption Trends:
- The dataset captures industrial and residential electricity use trends.
- Highlights regional disparities in energy access and usage.
📊 Validation Results:
- High correlation (R² > 0.96) between estimated and actual GDP/electricity consumption values.
- Comparison with external data sources shows significant improvement over previous models.

🌎 Applications & Implications

📊 Economic Research:
- Enables detailed studies on economic growth patterns.
- Useful for policy-making in regional development.
⚡ Energy Policy & Planning:
- Helps in assessing energy demand and infrastructure needs.
- Supports sustainable energy policy formulation.
🌪️ Disaster Impact Analysis:
- Can be used to evaluate economic impacts of natural disasters.
- Provides data for rapid response planning.

✅ Conclusion & Takeaways

This dataset provides a valuable tool for researchers, economists, and policymakers.
The methodology ensures high accuracy and continuity over nearly three decades, offering new insights into global economic trends.
The dataset enables micro-level analysis, particularly for regions with poor economic statistics.
The integration of satellite-derived economic indicators with official statistics enhances data reliability.
Future improvements may include:
- Integration with additional socioeconomic indicators to enhance model robustness.
- Refinements in machine learning techniques to further reduce errors in estimation.
- Expanding coverage to additional datasets that improve understanding of regional economic disparities.

📖 References

Full dataset and methodology details are available at Nature Scientific Data.
GEE dataset Access: Awesomme GEE community catalog
Exploratory Tool: GEE web app by Carlos Mendez

See app in full screen HERE

Regional dynamics of VIIRS-like nighttime lights 1992-2023

Fri, 14 Mar 2025 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

🌐 A Global Annual Simulated VIIRS Nighttime Light Dataset (1992-2023)

Authors: Xiuxiu Chen, Zeyu Wang, Feng Zhang, Guoqiang Shen, Qiuxiao Chen
Published in: Scientific Data (2024)
DOI: https://doi.org/10.1038/s41597-024-04228-6

🔬 Background & Summary

Nighttime light (NTL) data is widely used to measure human activity, urbanization, and socioeconomic trends.
Existing NTL datasets (DMSP-OLS & NPP-VIIRS) have limited temporal coverage and inconsistencies.
The study presents a new dataset, SVNL (Simulated VIIRS NTL), using deep learning to provide a continuous, high-resolution (500m) dataset from 1992-2023.
SVNL allows for long-term monitoring of human activity and urbanization trends.

📚 Data Collection

DMSP-OLS Stable NTL (1992-2013): Oldest available nighttime light dataset.
NPP-VIIRS Annual VNL V2 (2012-2023): Higher resolution and more accurate than DMSP.
Landsat NDVI (1992-2013): Used to improve calibration and reduce saturation.
Other datasets: Extended NTL datasets (ChenVNL, LiDNL), GDP data, and administrative boundaries.

🎯 Research Framework

Step 1: Preprocess and calibrate DMSP-OLS NTL data for consistency.
Step 2: Develop and train a U-Net super-resolution network (NTLSRU-Net) for cross-sensor calibration.
Step 3: Apply the trained model to convert DMSP NTL into VIIRS-like data (1992-2011).
Step 4: Merge simulated VIIRS data (1992-2011) with real VIIRS data (2012-2023) to create SVNL dataset.

🤖 U-Net Super-Resolution Model

The model enhances spatial resolution and corrects inconsistencies between DMSP & VIIRS.
Modifications:
- Removed pooling layers to preserve spatial details.
- Used transposed convolutions for up-sampling.
- Integrated Landsat NDVI data to correct for saturation.
Model trained using DMSP & VIIRS data from 2012-2013 and then applied for historical reconstruction.

🌍 Evaluation & Validation

Accuracy Assessment:
- Histogram and scatter plot comparisons between SVNL & real VIIRS data (2012-2013).
- High correlation observed at pixel, city, province, and national levels.
Spatial Pattern Validation:
- SVNL data closely matches real VIIRS data, avoiding saturation issues in urban areas.
Temporal Trend Validation:
- SVNL aligns well with economic indicators (GDP growth) and urban expansion patterns.

🔄 Key Findings

SVNL dataset provides a high-resolution, long-term global record of nighttime lights.
Outperforms previous datasets by maintaining spatial and temporal consistency.
Enables more accurate studies on urbanization, socioeconomic trends, and environmental monitoring.
Publicly accessible for researchers and policymakers.

💡 Conclusion

The SVNL dataset fills a crucial gap in long-term nighttime light data.
Facilitates detailed analysis of human activities from 1992-2023.
Future work includes further refinements using additional remote sensing data.
Dataset Access: Original data repository
GEE dataset Access: Awesomme GEE community catalog
Exploratory Tool: GEE web app by Carlos Mendez

See web app in full screen HERE

Space-time dynamics of nighttime lights: VIIRS-annual data

Mon, 01 Apr 2024 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

See app in full screen HERE

Space-time dynamics of nighttime lights: VIIRS-like data

Mon, 01 Apr 2024 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

See app in full screen HERE

Space-time dynamics of nighttime lights: DMSP-corrected data

Fri, 01 Mar 2024 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

See app in full screen HERE

About the data: https://developers.google.com/earth-engine/datasets/catalog/NOAA_VIIRS_DNB_ANNUAL_V21#description

Space-time dynamics of nighttime lights: DMSP-extended data

Fri, 01 Mar 2024 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

See app in full screen HERE

Space-time dynamics of nighttime lights: DMSP-like data

Fri, 01 Mar 2024 00:00:00 +0000

When the sun goes down and the lights turn on, there’s still a lot to explore.
Let’s study regional development from outer space!

See app in full screen HERE