ArcGIS

New Demo Page! by Robert Walker

Happy to announce that along I've started a demos page!  I'll be adding new demos occasionally that highlight some neat things you can do with JavaScript. 

Along with this blog, which features many demonstrations and how-to's, I think this demonstration page will help some users thing of new possibilities!

I'll try to keep as many of these demos to this site URL as possible, but may link externally for Node.js, ArcGIS Online, or other reasons I need server-side production. 

AGOL Migration Script by Robert Walker

ESRI recently released their ArcGIS Python API v1.3.  In the release post ESRI highlights the clone_items() method of the ContentManager class.  Given that we recently merged several accounts and had a new central ArcGIS Online (AGOL) domain I found this method super helpful.  We have dozens of apps, maps, and feature layers to move.  This single method made it a one-script process!  Just feed it the ID(s) of an App or Map and it will migrate all of the content required to make it work.

Hopefully to save you some time, you can copy the script below.  Just fill in the "blanks" and run it.

"""
Script to migrate AGOL items to another AGOL.  Requires ArcGIS Python API 1.3+

Author: Dylan Walker
Date: 12/25/2017

"""
import json

from arcgis import gis

FOLDER_TO_PUT = 'THE FOLDER TO SAVE ITEMS TO (OPTIONAL)'  # Folder on TO_GIS to put items into, optional
FROM_GIS_URL = 'THE *FROM* AGOL OR PORTAL URL'  # URL of the GIS Portal/AGOL to copy FROM
TO_GIS_URL = 'THE *TO* AGOL OR PORAL URL'  # URL of the GIS Portal/AGOL to copy TO

# Passwords JSON file - make this your own; username is the key, password is the value 
# (optional, you can hard-code the passwords if you want)
# Example: { "username1": "password1", "username2": "password2" }
with open('pw.json') as pwf:
    pws = json.load(pwf)

# Setup Usernames and passwords for the TO/FROM GIS
FROM_UN = 'USER NAME OF THE FROM GIS'
FROM_PW = pws[FROM_UN]
TO_UN = 'USER NAME OF THE TO GIS'
TO_PW = pws[TO_UN]

# GIS Objects with the items to copy and GIS to put items into
try:
    FROM_GIS = gis.GIS(FROM_GIS_URL, FROM_UN, FROM_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)

try:
    TO_GIS = gis.GIS(TO_GIS_URL, TO_UN, TO_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)


def migrate_items(items):
    """
    Builds a list of items to migrate from a list of ids passed in
    :param items: list of item ids that will be migrated
    :return: list of items
    """
    migrate_list = []
    for item in items:
        try:
            migrate_list.append(gis.Item(FROM_GIS, item))
        except RuntimeError as err:
            print("An error occurred: ", err)
    return migrate_list


# Build items to migrate; list of item ids
items_to_migrate = migrate_items([
    # LIST YOUR APP/MAP/LAYER IDs HERE
])

# Content Manager to call clone_items method on
content_manager = gis.ContentManager(TO_GIS)

# Clone Items
try:
    content_manager.clone_items(items_to_migrate, folder=FOLDER_TO_PUT)
except Exception as error:
    print("Error cloning data: ", error)

Example will the "blanks" filled in for my own use (user names removed for privacy) and no pw.json file included (duh).

"""
Script to migrate GeosyntecMD items to Geosyntec AGOL.  Requires ArcGIS Python API 1.3+

Author: Dylan Walker
Date: 12/25/2017

"""
import json

from arcgis import gis

FOLDER_TO_PUT = 'some project folder'  # Folder on TO_GIS to put items into, optional
FROM_GIS_URL = 'https://geosyntecmd.maps.arcgis.com'  # URL of the GIS Portal/AGOL to copy FROM
TO_GIS_URL = 'https://geosyntec.maps.arcgis.com'  # URL of the GIS Portal/AGOL to copy TO

# Passwords JSON file - make this your own; username is the key, password is the value
# Example: { "username1": "password1", "username2": "password2" }
with open('pw.json') as pwf:
    pws = json.load(pwf)

# Setup Usernames and passwords for the TO/FROM GIS
FROM_UN = "some user name here"
FROM_PW = pws[FROM_UN]
TO_UN = "another user name here"
TO_PW = pws[TO_UN]

# GIS Objects with the items to copy and GIS to put items into
try:
    FROM_GIS = gis.GIS(FROM_GIS_URL, FROM_UN, FROM_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)

try:
    TO_GIS = gis.GIS(TO_GIS_URL, TO_UN, TO_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)


def migrate_items(items):
    """
    Builds a list of items to migrate from a list of ids passed in
    :param items: list of item ids that will be migrated
    :return: list of items
    """
    migrate_list = []
    for item in items:
        try:
            migrate_list.append(gis.Item(FROM_GIS, item))
        except RuntimeError as err:
            print("An error occurred: ", err)
    return migrate_list


# Build items to migrate; list of item ids
items_to_migrate = migrate_items([
    'f165836f45934d11a94e7550a9ca7c97',    
    '21dc60858a0a42fdb2048ddaeba228ac',
    '0278345852fe4857a15c28fcdecac0e9',
    'c621e3e3a1c94e1a9119e1f4da2ea244',
    '2e4924e5ab064d5194cef95af9a4d3cb',
    '3bcdbc9649c64128872a4fd959dfcb8e',
    '1b6d489affd546a792f0699f83326950'
])

# Content Manager to call clone_items method on
content_manager = gis.ContentManager(TO_GIS)

# Clone Items
try:
    content_manager.clone_items(items_to_migrate, folder=FOLDER_TO_PUT)
except Exception as error:
    print("Error cloning data: ", error)

Automation with Complex, Pre-Formatted Excel Templates by Robert Walker

I recently had the pleasure of automating a reporting system for a client that required the use of an existing complex and pre-formatted Excel template with specific page breaks, merged cells, borders, and more.  Here I describe how I was able to use Python and Excel together while maintaining the original template structure and formatting.

Read More

ArcGIS Pro Tips Coming from ArcMap by Robert Walker

ArcGIS Pro, the ArcMap replacement desktop product from ESRI, has been out for almost a year and adoption where I work has been slow and steady, but I consistently find the visual and nomenclature shock to be present to first-time users.  Hopefully this post will help some first-time ArcGIS users navigate the interface and get to using ArcGIS Pro more quickly.

Read More

Digitizing Complex Lines Quickly using ArcGIS by Robert Walker

I'm sometimes tasked with digitizing complex lines from historic documents or drawings for use in figures or analysis.  Often these lines are complex or numerous, such as areas of interest that follow a long-gone stream bank or historic contours.  Here I describe a workflow to quickly digitize these types of features using ArcGIS and the Spatial Analyst Extension.

Workflow Requirements:

  • ArcGIS with Spatial Analyst Extension
  • A georeferenced raster of the drawing to be digitized

Workflow:

Step 1 - Find your colors

Digitize_Layers.PNG

Add your georeferenced raster using the underlying color bands.  For example, most scanned documents will contain an R-Band, G-Band, and B-Band (usually Band-1, Band-2, and Band-3 in ArcGIS).  

Using the identify tool on each band, try to find the range of values the represent the features you are trying to digitize.  For example, in the image below, I'd like to digitize the red lines.  Using ArcGIS I figured out that Band-1 represents the color red on a scale of 0-255 (standard RGB scales) where 255 is more red and 0 would be no red.  More complex colors might require combinations of bands and different ranges of values.  Also, the white colors here will have R, G, and B bands at 255 (white is all colors).  Therefore, in later steps, it will be helpful to have the other bands present and filter out white.

20161208160232262_0001.jpg

Step 2 - Raster Calculator

Now that you know which colors/band values you need, we will use the Raster Calculator to create a new raster where the values you want (red in my example) are set to 1 and all values you don't want are set to 0.

Open the Raster Calculator (found in Spatial Analyst Tools > Map Algebra) and write a conditional equation that will perform the above operation.  In my example, the equation to get only red is:

Con("20161208160232262_0001.jpg - Band_1">250,Con("20161208160232262_0001.jpg - Band_2"<200,Con("20161208160232262_0001.jpg - Band_3"<200,1,0),0),0)

The result:
 

As you can see, I now have a raster that has roughly the red lines I need digitized as 1's and the rest as 0's.  Perfect!  

Step 3 - Convert Raster to Points

Now that we have successfully categorized the data into binary, we can easily convert the raster to points which have a value at the point of either 1 or 0.  Use the Conversion Tools within ArcMap to convert the raster to points.  This may take a long time if your raster is large or high resolution (clipping a raster or re-sampling a raster can reduce the time if needed).  Once completed, you will have a point for each cell of the raster.  Use a query definition to query for only the 1's (e.g., grid_code = 1).

Points result from the raster calculation

Points result from the raster calculation

Step 4 - Digitize!

Now that you have a series of points, you can quickly digitize the lines by creating new features with snapping turned on.  If you have very complex or very numerous features, you might also find it worthwhile to add a new field (e.g., "feature") to the points and group your points by feature.  For example, I could select the upper most area and assign all of those points to a "feature" value of 1, the next area to 2 and so on.  Then I can convert the points to lines or polygons using the "feature" field as the identifier.  Caution:  The points from a raster are generally ordered (OID) top to bottom, left to right.  You will need to add some sort of logical order to the points first and use the "Sort Field" option.

Conclusion

There you have it - a quick and easy workflow to get from a paper document to digitized lines quickly! Hopefully you find this workflow helpful and it saves you some time!  Do you have another workflow for digitization of existing features?  Have you used a similar workflow for other analysis?  Let me know in the comments!