ArcGIS Pro, the ArcMap replacement desktop product from ESRI, has been out for almost a year and adoption where I work has been slow and steady, but I consistently find the visual and nomenclature shock to be present to first-time users. Hopefully this post will help some first-time ArcGIS users navigate the interface and get to using ArcGIS Pro more quickly.Read More
I'm happy to announce that I've completed the Microsoft Professional Program Certificate in Data Science! After six-months of courses capped off by a challenging machine learning problem issued by Microsoft data scientists, I've learned a lot and am excited to use my new skills. As proof of my accomplishment Microsoft has issued this official certificate!
A brief overview of how I see ArcGIS Online (AGOL) and some anecdotes about how I use it.Read More
A short description of how I've begun to use Machine Learning APIs to leverage powerful algorithms to analyze data and derive information quickly.Read More
I'm sometimes tasked with digitizing complex lines from historic documents or drawings for use in figures or analysis. Often these lines are complex or numerous, such as areas of interest that follow a long-gone stream bank or historic contours. Here I describe a workflow to quickly digitize these types of features using ArcGIS and the Spatial Analyst Extension.
- ArcGIS with Spatial Analyst Extension
- A georeferenced raster of the drawing to be digitized
Step 1 - Find your colors
Add your georeferenced raster using the underlying color bands. For example, most scanned documents will contain an R-Band, G-Band, and B-Band (usually Band-1, Band-2, and Band-3 in ArcGIS).
Using the identify tool on each band, try to find the range of values the represent the features you are trying to digitize. For example, in the image below, I'd like to digitize the red lines. Using ArcGIS I figured out that Band-1 represents the color red on a scale of 0-255 (standard RGB scales) where 255 is more red and 0 would be no red. More complex colors might require combinations of bands and different ranges of values. Also, the white colors here will have R, G, and B bands at 255 (white is all colors). Therefore, in later steps, it will be helpful to have the other bands present and filter out white.
Step 2 - Raster Calculator
Now that you know which colors/band values you need, we will use the Raster Calculator to create a new raster where the values you want (red in my example) are set to 1 and all values you don't want are set to 0.
Open the Raster Calculator (found in Spatial Analyst Tools > Map Algebra) and write a conditional equation that will perform the above operation. In my example, the equation to get only red is:
Con("20161208160232262_0001.jpg - Band_1">250,Con("20161208160232262_0001.jpg - Band_2"<200,Con("20161208160232262_0001.jpg - Band_3"<200,1,0),0),0)
As you can see, I now have a raster that has roughly the red lines I need digitized as 1's and the rest as 0's. Perfect!
Step 3 - Convert Raster to Points
Now that we have successfully categorized the data into binary, we can easily convert the raster to points which have a value at the point of either 1 or 0. Use the Conversion Tools within ArcMap to convert the raster to points. This may take a long time if your raster is large or high resolution (clipping a raster or re-sampling a raster can reduce the time if needed). Once completed, you will have a point for each cell of the raster. Use a query definition to query for only the 1's (e.g., grid_code = 1).
Step 4 - Digitize!
Now that you have a series of points, you can quickly digitize the lines by creating new features with snapping turned on. If you have very complex or very numerous features, you might also find it worthwhile to add a new field (e.g., "feature") to the points and group your points by feature. For example, I could select the upper most area and assign all of those points to a "feature" value of 1, the next area to 2 and so on. Then I can convert the points to lines or polygons using the "feature" field as the identifier. Caution: The points from a raster are generally ordered (OID) top to bottom, left to right. You will need to add some sort of logical order to the points first and use the "Sort Field" option.
There you have it - a quick and easy workflow to get from a paper document to digitized lines quickly! Hopefully you find this workflow helpful and it saves you some time! Do you have another workflow for digitization of existing features? Have you used a similar workflow for other analysis? Let me know in the comments!
Part 2 of how to use Power Query in Excel to retrieve data from ArcGIS Online. In this part I cover how to access secured resources.Read More
As part of my job as a data analyst/scientist it often means I need to build models of the data to understand the data, test the data, and visualize the data. Today I'm going to focus on the later - visualization; specifically 3D Printing computer visualizations to help clients, colleagues, and the public understand the complex models and data we use.
Modeling for 3D Printing
When I model data I often want to include all of the variables - depth, concentrations, soil type, geology, surfaces, well screens, water tables, infrastructure, and more. This helps the engineers and scientists I work with see the best picture I can give them and helps me analyze the data to see what makes sense and what doesn't (e.g., a groundwater sample above the water table might trigger an investigation to see if the water table has risen or dropped or if maybe the sample was mis-labled and is really porewater or surface water or maybe there is a perched zone of groundwater not in my model!)
However, when it comes to doing a model that I plan to 3D print I have to make the model much simpler to accommodate the technical limitations of the printer, like size, thickness, lack of transparency, inability to 'zoom', etc. Therefore, I often find these 3D printed models most useful in non-technical discussions, such as for marketing and public relations.
Impact of a 3D Model
Even with a simplified model, the impact can be huge for a client or a consultant. Being able to physically see and hold the site can provide a huge benefit. If nothing else, the models often make huge first impressions and can start a conversation. Many times, especially for public presentation, such as in trials or public proposals, having a simplified 'real' model for a judge, jury, or public to handle and visualize can have lasting impressions on how they perceive the site and understand what will or has happened during an environmental investigation, design, or remediation.
If you are interested in learning more about how to create these models please leave a comment or contact me.
Just getting started! This was our first light show ever during Halloween 2016. I plan to add a little bit each year and grow the show! Expect more "how to" and "behind the scenes" blogs in the future!
A guide to using Power Query to access ArcGIS Online Feature Services using Dynamic Parameters for an easy-to-update tool.Read More
Beginner tips and tricks for quickly navigating data in Excel.Read More
Power Query is a super powerful addition to Microsoft Excel. Learning to harness its power leads to whole new levels of quick and easy data exploration, manipulation, and analysis!Read More
Do you want want to learn Python and how to use the ArcPy library? So do a lot of my co-workers who do ArcGIS and wonder if there is an easier/better way to do routine work or repetitive work.
I often encourage those doing any ArcGIS work to do a few things to document their work:
1) Build a model for any geoprocessing you do; and
2) Learn Python to make it even easier.
Why do I encourage this?
Mostly - documentation, but also to make life easy.
In any science is important to document our methods of doing things and in ArcGIS it's very easy to run a geoprocessing (GP) tool in a non-saved or temporary MXD file (or even in ArcCatalog). Then how does someone else know how you did the work? Or even you after a few days of doing other things?
If you create a model for your GP tool and save it inside of a shared toolbox with a good name, description, and metadata - you're problems are solved. Hard-code the inputs and outputs as parameters and everything from beginning to end is recorded.
So you've made some models -some simple (e.g., one GP tool with an input and output) and some complex (long workflows of chained tools). Great! What if you don't want to constantly have to build the models or want to have some sort of dynamic input? You can export the models to Python and edit them to 1) learn Python and ArcPy and 2) to save the files for documentation/backup.
Of course, all this being said - it's best to save your MXDs (or maybe have one MXD with that you only do GP work in) so that there is a results trail for others to follow.
Tired of paying Dell, Lenovo, Apple, etc. hundreds more for the same PC you could build yourself but don't know where to start? Good news! Here's another (among hundreds) of 'How-To' guides on the internet!
So what is this guide and why is it different? I will try my best to not use any jargon and to explain as much of any jargon I have to use as I can. I won't get hyper-technical or super-detailed nor will I cover the specifications or details of specific hardware. This is written to be as simple and straight-forward as possible. I just want you to be able to build any PC you want with some basics - from here you can research any specifics you need.
Why am I qualified to write this guide? I've been building all types of computers since I was eight years old and I've repaired and touched nearly every type of component since EEPROM.
So where to start... purpose? need? components? None of these... confidence is where we should start!
The single biggest factor in getting started if you've never built a PC before is just mustering the confidence to invest several hours and to lay down hundreds of dollars to buy something you might fuck up! But don't worry... It's really hard to mess up a basic PC (if you're into extreme PCs like liquid cooling, overclocking, case modding, or hardware modding where messing up is entirely possible - this isn't the guide for you). If you don't believe me and you don't have money that might be gone if you do mess up then you can build a low-powered micro or nano PC for <$200 just to get started.
Yes! You can build a PC if you have no experience! Let me show you how!
So what do you need to start?
- A basic vocabulary and a basic idea of each component
- A screwdriver set
Basic Vocab and Component Purpose/Analogy
- Component - A 'piece' of the PC
- OS - Operating System (i.e., Windows, Linux, OSX, etc.); the 'heart' of the PC
- SSD - Solid State Drive; the storage of choice for most contemporary PCs; 'long-term memory' (e.g, "C:") (note: I will not be covering HDDs in this guide as they are quickly being phased out except for servers)
- GPU - Graphics Processing Unit; creates the graphics that get output to your display
- CPU - Central Processing Unit; the 'brain' of the PC
- Motherboard - The component you will plug everything into; the 'nervous system' of the PC
- RAM - Random Access Memory; the 'short-term memory' of the PC
- PSU - Power Supply Unit; plugs into some components on one end and the power receptacle on the other end.
- BIOS - Basic Input/Output System; controls the start-up process of the PC; BIOS is becoming replaced by UEFI, but some cheaper motherboards/older motherboards still use BIOS.
- UEFI - Unified Extensible Firmware Interface; same purpose as BIOS, but better!
Make sure you have a small and medium phillips (+) and flathead screwdrivers. You'll mostly use the medium phillips if building a basic PC.
Choosing your Components
There are a lot of finer details in building a PC that you should investigate if you are planning on getting the 'best bang for your buck' like Front Side Bus (FSB), North Bridge and South Bridge, CPU Pin counts, PCI-E slots, M.2 slots, L2 and L3 Cache, etc. etc. - But as a simple tutorial I won't delve into those. My suggestion? As a first time PC builder is to look at your favorite Dell or pre-made 'custom' PC from boutique sellers (Alienware, Falcon Northwest, etc.) and see what specifications and options they provide. Guides like these from Techspot also offer suggestions for Budget, Midrange, and High-End PCs. Look them over and just order the parts yourself and skip all the math...
Getting Your Components
Browse around online. Search Amazon, NewEgg, TigerDirect, MicroCenter, etc. for the best prices. Often different sites will run different specials. They're all the same parts - get whichever gives the best deal!
Putting it All Together AKA The Fun Part!
This is the process I usually follow when putting together my PCs - after you've done a few PCs, you'll find the process that works best for you too!
1. Unbox the motherboard and place it on top of the box it came in and the static paper it was wrapped in. Find the slot for the CPU - it may have stickers on it. If so, remove them. Release the CPU clamp if it is locked.
2. Unbox the CPU. Carefully remove the chip from its packaging (the pins are very small - don't bend or break them). Try to only touch the CPU along the edges and avoid putting fingerprints on the smooth top or the pins underneath (don't worry if you do touch the top - you can clean it with a little bit of cleaning alcohol and a lint-free cloth). If your CPU came with a fan and/or heatsink place it aside for now.
3.Flip the CPU over and inspect the pins making sure none are bent. Also look for the 'notch' or marker indicating the correct orientation of the CPU on the motherboard.
4. Align the CPU with the motherboard slot (double check!) and then gently place the CPU into place. The CPU should just 'drop' in without any resistance. Do not force it. Lock the CPU into place using the lever or clamp on the motherboard. It may feel like the clamp will break the CPU - but if you've aligned the CPU correctly it will not. The clamp will be tight.
5. Find your heatsink/fan for the CPU. Also locate a fan-port on the motherboard with screen-printed label on the motherboard called "CPU". If you cannot find it, refer to your motherboards manual for where you should plug the CPU fan into.
6. Align the CPU fan and heatsink in such a way that it will be able to plug into the motherboard fan port, but also so that the fan cable will be out of the way of the fan blades and heatsink.
7. Most heatsinks come with thermal paste (usually a gray or white semi-liquid material on the bottom of the heatsink) - make sure yours does. Most heatsinks or fans also have spring-loaded push-locks that hold it against the CPU and onto the motherboard. Align these push-locks with the holes in the motherboard - then gently, but firmly, push the heatsink onto the CPU to create a seal with the thermal paste. Then push the push-locks through the holes until they 'click' through and have a firm hold on the motherboard. If your heatsink is different than described here - consult the CPU manual for installation instructions.
8. Plug in the CPU fan to the motherboard.
9. Unbox your case. Place the motherboard into the case such that the peripheral (keyboard, mouse, USB, sound, etc.) ports go out the back. This should also align holes in the motherboard with offsets in the case 'motherboard tray'. Put some screws in and tighten them down to a firm-hold, but not overly tight. Take care not to push on the screws as you install them lest you slip and dig your screwdriver into the motherboard.
10. Next install your SSD and PSU into the case (don't worry about cables yet). Some cases have 'snaps' for the SSDs, other require screws.
11. Install your RAM into the motherboard. The RAM pieces should snap in with a firm, even push on them, just make sure the RAM locks are open (little plastic clips on the end of the RAM slots).
12. If you have any other components (sound cards, video cards, CD drive, etc.) - install them per the instructions they came with.
Now for the cable work:
- Evaluate where all of the things you need to plug in are: Fans, SSDs, motherboard, GPU, other
- Envision how you will route the cables around the case - try to keep them neat and ordered.
- Plug in your case fans to the motherboard.
- Plug in your case switches to the motherboard (refer to your motherboard manual and case manual for instructions - these vary from manufacturer to manufacturer).
- Plug your SSD into the motherboard internal ports.
- Plug in the PSU to the motherboard and SSD (and GPU if applicable).
- Double check that everything is plugged in and then close up the case!
- Plug in the display and connect the video cable (HDMI, DVI, or VGA)
- Plug the PSU into the wall and make sure the PSU switch is turned on.
First Start AKA Fear Factor!
Double check that you've plugged everything in correctly then press the power button on the case. Hopefully you hear the familiar hum of PC fans! Congratulations! You've built a PC... now the setup and software!
If you don't get the nice hum of PC fans and a BIOS/UEFI POST screen... check over your connections again. Is your PSU on? Is it plugged into the wall? Did you connect the case buttons/switches to the motherboard correctly? Is all the RAM fully-seated in the motherboard? If everything is definitely correct - it's possible you just got unlucky with a bad part. You can request a return and replacement from reputable distributors.
After you have the PC started up, you will need to configure the BIOS/UEFI to boot to your CD or USB with the OS installation files on it. This is where this guide ends....
Perhaps one day I will cover the rest of the process, but nearly everything is very simple and automated at this time (especially for Windows).
Thanks for reading!
Tips and Tricks that make your map layouts easier to work with, including making multiple dataframes align, scales work, and legends snap to grids properly!Read More