Pi-Hole Install, Results, and Resources by Robert Walker

Pi-Hole is a custom DNS server designed specifically to block ads, trackers, and malware. Pi-Hole can run from virtually any always-on hardware, including a Raspberry Pi, old PC, NAS, or Cloud Service.

See https://pi-hole.net/ for more information.

Here I will discuss my installation process, some initial results, and some extra resources.

Read More

React/Front End Bloat (and general update) by Robert Walker

I'm still here!

Wow!  It's been a busy few months as I've started to integrate into my new position!  Thank you everyone that has been reaching out to me with questions or other stuff and for your patience.

When I started my new position I was tasked with being the project manager and principal software engineer for a brand new web application using a React front-end and Django back-end.  After some busy few months we finally pushed a beta app on-time out last week!  It's already impressing and new work rolling in :)  Unfortunately, I can't share specifics or links. But I can share one of the things I have learned - one strategy to optimize your front-end load time.

Read More

Turning the Page by Robert Walker

I'm excited to announce that I've recently changed careers and will be working as a software engineer!  

Currently, I'm writing web applications for internal company use using Django + ReactJS, so expect a blog or two on those in the future.  Also, this career is a knowledge domain change for me, so I'm doing a lot of reading and learning!  Exciting times ahead!

New Demo Page! by Robert Walker

Happy to announce that along I've started a demos page!  I'll be adding new demos occasionally that highlight some neat things you can do with JavaScript. 

Along with this blog, which features many demonstrations and how-to's, I think this demonstration page will help some users thing of new possibilities!

I'll try to keep as many of these demos to this site URL as possible, but may link externally for Node.js, ArcGIS Online, or other reasons I need server-side production. 

Grow with Google Scholarship Award! by Robert Walker

Woo!  I've just been granted a Grow With Google "Google Developer Challenge Scholarship"!  Over 100,000 applications were received, but only 10,000-15,000 applicants were accepted into this program.  You can learn more about this program here.

Over the next several months I will be provided with free Udacity online courses, mentorship, challenges, and review.  If I excel at the program (top 10%), I will have the opportunity to work towards a nano-degree Front-End Web development.

GrowWithGoogleDeveloperChallengeScholarship.png

Spectre Attack by Robert Walker

First - Read about it here: https://spectreattack.com/

Given the seriousness of the attacks capabilities (leaking data from processor cache based on what the process speculated was going to happen), I expect to see lots of attention thrown at this problem.

In the short-term, there appears to be little we can do software side to prevent the exploit, but I did notice that all of the researches required very precise timings to prevent cache time-out and/or overwrites.  They use a variety of pauses (empty loops) and timing checks to make sure they are accessing what they want.  

One of the more serious proof-of-concepts shown uses a JavaScript to execute inside of a Chrome instance.  The researchers even note here that they used HTML5 timings to work around the built-in measures of Chrome which deliberately round performance.now() and event.timeStamp so as to prevent micro-second level performance attacks such as this one.  

Therefore I expect, at least on software side, that many libraries may implement their own versions of time rounding to prevent other programs from knowing the exact execution times.  Keep this in mind if your program requires timing precision!

Bonus:  A JSFiddle to show how timing can be different in Chrome and Edge.  First, load in Edge and see the timing - it should be precise to many decimal places (floating point +/- 5 µS).  In Chrome it is rounded to three decimal places (generally, millisecond accuracy).  Attackers can use these simple functions to time their attacks in Edge, but its more difficult in Chrome.

AGOL Migration Script by Robert Walker

ESRI recently released their ArcGIS Python API v1.3.  In the release post ESRI highlights the clone_items() method of the ContentManager class.  Given that we recently merged several accounts and had a new central ArcGIS Online (AGOL) domain I found this method super helpful.  We have dozens of apps, maps, and feature layers to move.  This single method made it a one-script process!  Just feed it the ID(s) of an App or Map and it will migrate all of the content required to make it work.

Hopefully to save you some time, you can copy the script below.  Just fill in the "blanks" and run it.

"""
Script to migrate AGOL items to another AGOL.  Requires ArcGIS Python API 1.3+

Author: Dylan Walker
Date: 12/25/2017

"""
import json

from arcgis import gis

FOLDER_TO_PUT = 'THE FOLDER TO SAVE ITEMS TO (OPTIONAL)'  # Folder on TO_GIS to put items into, optional
FROM_GIS_URL = 'THE *FROM* AGOL OR PORTAL URL'  # URL of the GIS Portal/AGOL to copy FROM
TO_GIS_URL = 'THE *TO* AGOL OR PORAL URL'  # URL of the GIS Portal/AGOL to copy TO

# Passwords JSON file - make this your own; username is the key, password is the value 
# (optional, you can hard-code the passwords if you want)
# Example: { "username1": "password1", "username2": "password2" }
with open('pw.json') as pwf:
    pws = json.load(pwf)

# Setup Usernames and passwords for the TO/FROM GIS
FROM_UN = 'USER NAME OF THE FROM GIS'
FROM_PW = pws[FROM_UN]
TO_UN = 'USER NAME OF THE TO GIS'
TO_PW = pws[TO_UN]

# GIS Objects with the items to copy and GIS to put items into
try:
    FROM_GIS = gis.GIS(FROM_GIS_URL, FROM_UN, FROM_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)

try:
    TO_GIS = gis.GIS(TO_GIS_URL, TO_UN, TO_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)


def migrate_items(items):
    """
    Builds a list of items to migrate from a list of ids passed in
    :param items: list of item ids that will be migrated
    :return: list of items
    """
    migrate_list = []
    for item in items:
        try:
            migrate_list.append(gis.Item(FROM_GIS, item))
        except RuntimeError as err:
            print("An error occurred: ", err)
    return migrate_list


# Build items to migrate; list of item ids
items_to_migrate = migrate_items([
    # LIST YOUR APP/MAP/LAYER IDs HERE
])

# Content Manager to call clone_items method on
content_manager = gis.ContentManager(TO_GIS)

# Clone Items
try:
    content_manager.clone_items(items_to_migrate, folder=FOLDER_TO_PUT)
except Exception as error:
    print("Error cloning data: ", error)

Example will the "blanks" filled in for my own use (user names removed for privacy) and no pw.json file included (duh).

"""
Script to migrate GeosyntecMD items to Geosyntec AGOL.  Requires ArcGIS Python API 1.3+

Author: Dylan Walker
Date: 12/25/2017

"""
import json

from arcgis import gis

FOLDER_TO_PUT = 'some project folder'  # Folder on TO_GIS to put items into, optional
FROM_GIS_URL = 'https://geosyntecmd.maps.arcgis.com'  # URL of the GIS Portal/AGOL to copy FROM
TO_GIS_URL = 'https://geosyntec.maps.arcgis.com'  # URL of the GIS Portal/AGOL to copy TO

# Passwords JSON file - make this your own; username is the key, password is the value
# Example: { "username1": "password1", "username2": "password2" }
with open('pw.json') as pwf:
    pws = json.load(pwf)

# Setup Usernames and passwords for the TO/FROM GIS
FROM_UN = "some user name here"
FROM_PW = pws[FROM_UN]
TO_UN = "another user name here"
TO_PW = pws[TO_UN]

# GIS Objects with the items to copy and GIS to put items into
try:
    FROM_GIS = gis.GIS(FROM_GIS_URL, FROM_UN, FROM_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)

try:
    TO_GIS = gis.GIS(TO_GIS_URL, TO_UN, TO_PW)
except RuntimeError as gis_err:
    print("An error occurred: ", gis_err)


def migrate_items(items):
    """
    Builds a list of items to migrate from a list of ids passed in
    :param items: list of item ids that will be migrated
    :return: list of items
    """
    migrate_list = []
    for item in items:
        try:
            migrate_list.append(gis.Item(FROM_GIS, item))
        except RuntimeError as err:
            print("An error occurred: ", err)
    return migrate_list


# Build items to migrate; list of item ids
items_to_migrate = migrate_items([
    'f165836f45934d11a94e7550a9ca7c97',    
    '21dc60858a0a42fdb2048ddaeba228ac',
    '0278345852fe4857a15c28fcdecac0e9',
    'c621e3e3a1c94e1a9119e1f4da2ea244',
    '2e4924e5ab064d5194cef95af9a4d3cb',
    '3bcdbc9649c64128872a4fd959dfcb8e',
    '1b6d489affd546a792f0699f83326950'
])

# Content Manager to call clone_items method on
content_manager = gis.ContentManager(TO_GIS)

# Clone Items
try:
    content_manager.clone_items(items_to_migrate, folder=FOLDER_TO_PUT)
except Exception as error:
    print("Error cloning data: ", error)

Automation with Complex, Pre-Formatted Excel Templates by Robert Walker

I recently had the pleasure of automating a reporting system for a client that required the use of an existing complex and pre-formatted Excel template with specific page breaks, merged cells, borders, and more.  Here I describe how I was able to use Python and Excel together while maintaining the original template structure and formatting.

Read More

Upcoming Presentation by Robert Walker

Thank you for taking time to read my blog.  If you want to discuss any of the topics I've blogged about or more I'll be at the ISWA World Congress 2017 in Baltimore, Maryland on September 26 to discuss and demonstrate how the data science and machine learning I do can help landfill operators optimize landfill gas production. 

You can read my abstract and learn more about he conference here.

Removing 'Ad-Block-Blockers' from WebSites by Robert Walker

Many websites have started implementing 'Ad-Block-Blockers'.  In other words, if the site detects that an advertisement did not load on their website, they open up a pop-up or other method that makes reading the website impossible until you allow ads or remove the nuisance.  Personally, I hate ads (especially the annoying ones that are animated or have sound), so I just work around the nuisance - here's how you can too!

Read More

Time-Series PivotCharts with Dynamic Date-Axis by Robert Walker

Using PivotCharts in Excel is a powerful analysis, automation, and visualization tool - however, they are lacking when it comes to Time-Series charts and require a bit of trickery to work as expected.  Here I describe some of those tricks and provide a working example of how to make dynamic time-series PivotCharts.

Read More

How to Quickly Create Time-Series PivotCharts by Robert Walker

Time-Series charts are a staple data visualization tool in many industries.  Creating easy to manipulate time-series charts has been done for a while online (i.e., Plotly.js, Tableau), many clients require their data to stay "local" - either Excel or a secured database - nothing online or 3rd party.  That's where Excel PivotCharts can fill the gap.  This tutorial will show you how to accurately present time-series data using a PivotChart.

Read More