What is ArcGIS notebook?
ArcGIS notebook is ESRI's Jupyter notebook environment, which provides the option to create python notebooks as part of the ArcGIS project and apply interactive spatial data analysis using python, spatial map, and data visualization. ArcGIS notebook was introduced in ArcGIS Pro in Version 2.5 in 2019 (The latest version by August 2021 is version 2.8). It is also available in ArcGIS online with the on-demand request.
Python environment setup is important in ArcGIS notebook to manage and install required python libraries for data analysis.
1. Go to the main interface of ArcGIS pro and click Settings to access the setting panel.
2. Under the setting panel on the left of ArcGIS Pro, click Python Then Manage Environments to create, edit, or remove python environments in ArcGIS Pro.
3. The default environment of ArcGIS notebook, "arcgispro-py3", cannot be modified. To install third-party python libraries, we need to create a cloned environment by clicking the Clone Default button or click the Clone button of the selected environment. Then active the cloned environment by selecting the cloned environment and click the OK button. After changing the active environment, you may need to restart ArcGIS Pro to reinitialize Python and use the new environment.
4. With a cloned environment, you can install available third-party libraries listed in Python Package Manager, or check, updated and remove installed libraries. Here is the link for detailed instruction.
Note: Different from Jupyter notebook or some other online python notebook like Google colab, ArcGIS notebook doesn't support library package installation by writing installation code in the notebook (like pip install or conda install). Invalid library installations may disconnect your notebook from the ArcGIS server / disable notebook functions in ArcGIS Pro until the environment is restored.
Pandas ---data wrangling, manipulation, and visualization; https://github.com/pandas-dev/pandas
Numpy ---scientific computing and performing basic and advanced array operations; https://github.com/numpy/numpy
Scipy ---linear algebra, integration, optimization, and statistics; https://github.com/scipy/scipy
Matplotlib ---the foundation of Python data visualization, standard data visualization like Matlab; https://github.com/matplotlib/matplotlib
Seaborn --- A wrapper of Matplotlib with modern colors and styles and support complex visualizations; https://github.com/mwaskom/seaborn
Bokeh --- interactive and scalable visualizations inside browsers using JavaScript widgets; https://github.com/bokeh/bokeh
Scikit-learn --- Includes the most common machine learning algorithms; https://github.com/scikit-learn/scikit-learn
Keras --- neural networks and modeling; https://github.com/keras-team/keras
PyTorch --- A popular Python framework for neural network and deep learning with GPU acceleration; https://github.com/pytorch/pytorch
Tensorflow --- Another popular Python framework for neural network and deep learning with GPU and TPU; https://github.com/tensorflow/tensorflow
Arcpy --- geographic data analysis, data conversion, data management, and map automation; https://pro.arcgis.com/en/pro-app/latest/arcpy/get-started/what-is-arcpy-.htm
ArcGIS Python API --- Support web-based geospatial analysis, deep learning and big data analysis: https://github.com/Esri/arcgis-python-api
Geopandas --- add support for geographic data to pandas objects; https://github.com/geopandas/geopandas
GDAL --- translator library for raster and vector geospatial data formats; https://github.com/Esri/gdal
Rasterio --- access and visualize geospatial raster data; https://github.com/mapbox/rasterio/blob/master/docs/index.rst
Folium --- leverage leaflet (https://github.com/Leaflet/Leaflet) to build interactive web maps; https://github.com/python-visualization/folium
Note: ArcGIS Notebook is available in ArcGIS Pro Version 2.5 or later.
1. Open ArcGIS Pro (ESRI account login is required to access ArcGIS Pro. Organization login is available to students, staff, and faculty at Purdue University by using BiolerKey).
For users with a personal ESRI account, select ArcGIS login, then input your username and password to log in.
For users with organization access, select Your ArcGIS organization's URL then input your organizational link to ArcGIS Pro. It will direct the user to the organization account login portal. Then input your organizational username and password to log in. (Purdue's ArcGIS link is "purdueuniversity.maps.arcgis.com", then it will direct the users to the Purdue Login)
2. Create an ArcGIS project by clicking the Map icon, then name the project ("Notebook Demo" as the example in the figure below)
3. On the project interface, select Insert, then click New Notebook to create a blank ArcGIS notebook.
4. Now a new notebook is created in the project. A new folder named Notebooks is also created in the Catalog, where you can access, rename, or remove notebooks in the current project.
ArcGIS Hands-on practice: Regional climate data wrangling and visualization
Introduction:
In this exercise, we will work on the WHIN station dataset (https://whin.org/about/) for data science practices from data wrangling to data analysis to data visualization. The WHIN stations capture daily weather data in ten counties (including Tippecanoe county) in Northern Indiana. While the daily weather records usually have some missing data from certain stations. In this exercise, we will apply two interpolation algorithms to fill those missing data in the database. After interpolation, we will compare the interpolated temperatures to the observed temperature (‘temperature’ column in the table) using python data visualizations and select the better algorithm to fill the missing weather data. We will also visualize the temperature data at the county level and discuss the temperature variation.
Data:
Data 1: WHIN_current_02232021.csv
Data 2: WHIN_stations.csv
Data 3: WHIN_weather_0223_IDW_EBK_Counties.csv
Note: Data 1 and 2 are for data wrangling and temperature interpolation practices; Data 3 is for data visualization practice.
Source code:
ArcGIS Notebook Code: ArcGIS_notebook_hands_on_regional_climate_data_analysis_2021.ipynb
Python libraries:
Arcpy (installed in default environment)
Numpy (installed in default environment)
Pandas (installed in default environment)
Matplotlib (installed in default environment)
Seaborn (installation needed)
There are multiple ways to access python programming in ArcGIS Pro. The users can write interactive python code (.ipynb files) and regular python script (.py files) through different platforms and functions.
Ways to access python:
Interactive python code (.ipynb files):
1. ArcGIS Notebook
2. Jupyter Notebook under ArcGIS Pro
Py scripts (.py files):
1. Python command line
2. IDLE (Python GUI)
3. Third-party platforms (Spyder and PyCharm)
How to access interactive python code (.ipynb files):
1: ArcGIS notebook as demonstrated in this tutorial
Image source: https://www.esri.com/arcgis-blog/products/arcgis-pro/analytics/introducing-arcgis-notebooks-in-arcgis-pro/
2. Jupyter notebook in ArcGIS Pro (Note: If you are using an organizational account for ESRI login, be sure to access the Jupyter notebook under ArcGIS to access Arcpy and ArcGIS API, Not other jupyter notebook environments like anaconda)
How to access python script (.py files)
1. Python command line: users can write python directly using the command line and execute .py files
2. IDLE (Python GUI): The editor is to write the py scripts and the shell is to execute the py scripts.
3. Access py scripts through third-party platforms like Spyder and PyCharm. First, install Spyder / PyCharm using the Python Package Manager, then active Spyder / PyCharm using Python Command Prompt.
Python versions: ArcGIS Map vs ArcGIS Pro
ArcGIS Pro uses Python 3.0+, while ArcMap uses Python 2.7. Python 3. and Python 2. are different languages and many functions are not compatible with each other. For ArcMap users, please check out here (list of ArcMap tools not available in ArcGIS pro: https://pro.arcgis.com/en/pro-app/latest/tool-reference/appendices/unavailable-tools.htm ) to find the functions that are not supported in ArcGIS pro.
Learn more about python in ArcGIS Pro: https://pro.arcgis.com/en/pro-app/latest/arcpy/get-started/installing-python-for-arcgis-pro.htm
Learn more about the differences between Python 3 and Python 2: https://www.guru99.com/python-2-vs-python-3.html
This online tutorial development is supported by IndianaView Minigrant 2021 (https://www.indianaview.org/). The data used in the hands-on exercise is provided by Wabash Heartland Innovation Network (WHIN) (https://whin.org/). Thanks to supports from Purdue Libraries' GIS services (https://www.lib.purdue.edu/gis) and D-VELoP lab (https://www.lib.purdue.edu/d-velop).
