Getting started with Raster and PostGIS - I

Steps for loading a raster in PostGIS (PgSql 9.4/ PostGIS 2.2) and pull it from database to view in QGIS 2.14.17.

1. Generate SQL for raster [NOTE: Do not create PostGIS table name starts with underscore, it creates problem while adding raster to QGIS from PostGIS database using DB Manager]

        raster2pgsql -s 4326 -I -C -M -F -t 50x50 -N nan biodiv_ssolnw.tif > biodiv_ssolnw.sql

2. Import generated raster into PostGIS database [Make sure to enable postgis extension in db]

         psql -h localhost -U postgres -d ecolservicedb -f biodiv_ssolnw.sql

the output is:
Processing 1/1: ags_473038164.tif
BEGIN
CREATE TABLE

INSERT 0 1
INSERT 0 1
INSERT 0 1
INSERT 0 1
…
…
….
CREATE INDEX
ANALYZE
NOTICE:  Adding SRID constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding scale-X constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding scale-Y constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding blocksize-X constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding blocksize-Y constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding alignment constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding number of bands constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding pixel type constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding nodata value constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding out-of-database constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
NOTICE:  Adding maximum extent constraint
CONTEXT:  PL/pgSQL function addrasterconstraints(name,name,name,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean,boolean) line 53 at RETURN
 addrasterconstraints
----------------------

Read More

Setup Jenkins Bit-bucket environment in Windows Server 2008

Steps for setting Jenkins/Bitbucket CI environment in Windows - little trickier than Linux

1) Download and Install Jenkins-2.60.1(current as of the day written)

2) Once installed changed the port from C:\Program Files (x86)\Jenkins\jenkins.xml to ‘9090’ and
restart jenkins

3) Set security options to “false” from config.xml

4) Then open localhost:8090 and see if you can see Jenkins Interface

5) Install all default plugins plus following plugins to work Jenkins correctly with Bitbucket and
dev requirements for my workflow
        a. Bitbucket pull request builder plugin
        b. Maven integration plugin
        c. MS build plugin
       d. Next build number plugin
       e. NodeJS plugin
       f. SSH Agent Plugin      
       g. SSh plugin
       h. Self-organizing swarm plugin
       i. Categorized view plugin
       j. Publish-over ssh
       k. Schedule build
       l. Nugget
       m. Vagrant

6) Install git bash in the machine

7) From git bash, generate ssh-keys , ssh-keygen

8) Add the public key(id_rsa.pub) in BitBucket under group account

Read More

Prevent WPF Global Keyboard Hook from stops working after hitting keys a while C# .NET

The article written by Dylan Currier, Low Level Global Keyboard Hook / Sink in C# .NET is one the simple, easily understandable, and and working post on Global Keyboard hook out there in internet. It worked in my WPF application and I added following line in the WPF app to refresh hook after each key press- IDK  if it is write or wrong, more likely to be a wrong practice to hook/unhook every time but it gets the job done. The primary reason to refresh hook is to prevent app hanging (not responding to the key press) after few clicks on the WPF button.

void _listener_OnKeyPressed(object sender, KeyPressedArgs e)
        {
           
          _listener.UnHookKeyboard();
          _listener.HookKeyboard();
        }

Following is the video from Dylan Currier  on Low Level Keyboard Hook in C# / .NET"

Read More

When to use ArcGIS Online to publish geospatial data: CASE STUDY EUROPEAN ENVIRONMENT AGENCY

ArcGIS Online is a workspace and a web platform for creating and sharing maps. It is possible to upload your data, create a layer or a map, define the extent, set some visualization options, assign permissions to your layer or map, share, and create web mapping applications or story maps. For information about the other capabilities please visit http://doc.arcgis.com/en/arcgis-online/.

The ArcGIS online platform is designed to host small datasets, for example demographic datasets. The datasets uploaded to ArcGIS online are relatively small in size, and should not exceed 1-2MB, or around 5000 records. This is to keep the agencies costs down and keep in line with our ArcGIS online pricing plan. It is for this reason that ArcGIS online should not be used to host large datasets, and tiled image services must NOT be uploaded to ArcGIS online. To publish bigger datasets and image services, please refer to the ArcGIS Server.

Source: http://discomap.eea.europa.eu/map/giseionet/Publishing_data_to_the_web_with_ArcGIS_Online.pdf

Read More

Code snippet: Identify to which polygon a point belong to?

Code sample to find point inside a polygon in shape file using GDAL/OGR 2.x and C#.


using OSGeo.OGR;
using OSGeo.OSR;
using OSGeo.GDAL;

1:  public void FindPolygonBelongsToPoint()   
2:      {  
3:        try   
4:        {  
5:          var shapePath = @"C:\Locations_WGS84.shp";  
6:          Ogr.RegisterAll();  
7:          OSGeo.OGR.Driver shapeDriver = Ogr.GetDriverByName("ESRI Shapefile");  
8:          DataSource shape = shapeDriver.Open(shapePath, 0);  
9:          Layer layer = shape.GetLayerByIndex(0);  
10:    
11:          Feature polygonFeature = layer.GetNextFeature();  
12:    
13:          long featureCountInLayer = layer.GetFeatureCount(1);  
14:          for (int i = 1; i <= featureCountInLayer; i++) {  
15:    
16:            Geometry polygonGeom = polygonFeature.GetGeometryRef();  
17:    
18:            string coordinates = "POINT (-100.726 38.995)";  
19:    
20:            SpatialReference spatialRef = layer.GetSpatialRef();  
21:            Geometry point = Ogr.CreateGeometryFromWkt(ref coordinates, spatialRef);  
22:    
23:            bool isInside = point.Within(polygonGeom);  
24:    
25:            if (isInside)  
26:            {  
27:              var val = polygonFeature.GetFieldAsString(4);  
28:              Console.WriteLine("Found one");  
29:            }  
30:    
31:            polygonFeature = layer.GetNextFeature();            
32:          }         
33:            
34:        }  
35:        catch (Exception ex)   
36:        {  
37:    
38:          Console.WriteLine(ex.Message);  
39:        }          
40:        
41:      }  

Read More

Code snippet: To highlight selected feature in ArcMap programmatically using ArcObjects

Code snippet to highlight selected feature in ArcMap programmatically using ArcObjects and return the STGeomFromWKB   string of selected feature.

To return the geometry you should import Microsoft.SqlServer.Types  from Nuget and SqlServerSpatial140.dll  from C:\Windows\System32

 using System;  
 using System.Collections.Generic;  
 using System.Data.SqlTypes;  
 using System.Linq;  
 using System.Runtime.InteropServices;  
 using System.Text;  
 using System.Windows;  
 using ESRI.ArcGIS.ArcMapUI;  
 using ESRI.ArcGIS.Carto;  
 using ESRI.ArcGIS.Catalog;  
 using ESRI.ArcGIS.CatalogUI;  
 using ESRI.ArcGIS.Desktop.AddIns;  
 using ESRI.ArcGIS.Display;  
 using ESRI.ArcGIS.esriSystem;  
 using ESRI.ArcGIS.Framework;  
 using ESRI.ArcGIS.Geodatabase;  
 using ESRI.ArcGIS.Geometry;  
 using Microsoft.SqlServer.Types;  
   
   
 namespace Test.Addin.ArcCommands  
 {  
   class SelectFeatureTool : ESRI.ArcGIS.Desktop.AddIns.Tool  
   {  
     protected override void OnMouseDown(MouseEventArgs arg)  
     {  
       IMxDocument mxDocument = ArcMap.Application.Document as IMxDocument;  
       IActiveView activeView = mxDocument.ActiveView;  
       IMap map = mxDocument.FocusMap;  
       ILayer layer = map.get_Layer(0); //Get 1st Layer  
       IFeatureLayer featureLayer = (IFeatureLayer) layer;  
   
       IPoint identifyPoint = activeView.ScreenDisplay.DisplayTransformation.ToMapPoint(arg.X, arg.Y);  
       ESRI.ArcGIS.Carto.IIdentify identifyLayer = (IIdentify)layer;  
       IArray array = identifyLayer.Identify(identifyPoint);  
         
       int oid = -1;  
       if (array != null)  
       {  
         object obj = array.get_Element(0);  
         IFeatureIdentifyObj fobj = obj as IFeatureIdentifyObj;  
         IRowIdentifyObject irow = fobj as IRowIdentifyObject;  
         IFeature feature = irow.Row as IFeature;  
         oid = feature.OID;  
       }  
       HighlightClickedFeature(featureLayer, oid, activeView);  
     }  
   

Read More

Python code snippet : Merge multiple CSVs (Column Wise) into a large CSV file using ID

Following is the python code snippet to merge multiple CSVs into a large CSV (column wise) using CSV's unique ID field. All input CSV's must have same length.

 import csv

 import itertools

 outCombinedStatTxtName ="FormattedTxtResults.csv"  
 tempCsvFiles = glob.glob(str(TempFolderPath)+'\*.csv')  
 openedFileList = [open(fn, 'rb') for fn in tempCsvFiles]  
 readers = [csv.reader(fn) for fn in openedFileList]  
   
 result = open(outCombinedStatTxtName, 'wb')  
 writer = csv.writer(result,delimiter=',')  
   
 multipleIndices = False  
 for row_chunks in itertools.izip(*readers):  
   tempFormattedRow = list(itertools.chain.from_iterable(row_chunks))  
   fidIndices = [i for i, x in enumerate(tempFormattedRow) if x == "ID"]  
   if(len(fidIndices) > 1):  
     fidIndices.pop(0)  
     redundantIndices = tuple(fidIndices)  
     multipleIndices = True  
       
   if(multipleIndices):  
     tempFormattedRow = [ tempFormattedRow[i] for i in xrange(len(tempFormattedRow)) if i not in set(redundantIndices) ]      
     writer.writerow(tempFormattedRow)  
   else:  
     writer.writerow(tempFormattedRow)  
 result.close()     
 [fn.close() for fn in openedFileList]  

Read More

Code Snippet: Select feature polygon(s) on Mouse Click ArcObjects C#

Code snippet of custom feature(s) selection tool on mouse click and highlight it using ArcObjects and C#.

 class SelectFeatureTool : ESRI.ArcGIS.Desktop.AddIns.Tool  
   {  
     protected override void OnMouseDown(MouseEventArgs arg)  
     {  
       IMxDocument mxDocument = ArcMap.Application.Document as IMxDocument;  
       IActiveView activeView = mxDocument.ActiveView;  
       IMap map = mxDocument.FocusMap;  
       ILayer layer = map.get_Layer(0); //Get 1st Layer  
       IFeatureLayer featureLayer = (IFeatureLayer) layer;  
   
       IPoint identifyPoint = activeView.ScreenDisplay.DisplayTransformation.ToMapPoint(arg.X, arg.Y);  
       ESRI.ArcGIS.Carto.IIdentify identifyLayer = (IIdentify)layer;  
       IArray array = identifyLayer.Identify(identifyPoint);  
         
       int oid = -1;  
       if (array != null)  
       {  
         object obj = array.get_Element(0);  
         IFeatureIdentifyObj fobj = obj as IFeatureIdentifyObj;  
         IRowIdentifyObject irow = fobj as IRowIdentifyObject;  
         IFeature feature = irow.Row as IFeature;  
         oid = feature.OID;  
       }  
       HighlightClickedFeature(featureLayer, oid, activeView);  
     }  
   

Read More

Catch Exception - Attempted to read or write protected memory in ESRI ArcObjects .Net 4.0 Framework

First - Don't write the code to throw "Attempted to read or write protected memory" ..it is bad.

Second - If you are working on legacy codes and COM objects that throws "Attempted to read or write protected memory" while writing ArcMap add-ins, catch exceptions to prevent application crash, in my case ArcMap.


Step #1 - Add following snippet to config file - app.config for Arc-addin

<configuration>
   <runtime>
      <legacyCorruptedStateExceptionsPolicy enabled="true" />
   </runtime>
</configuration>

Step #2

Add -

[HandleProcessCorruptedStateExceptions] 
[SecurityCritical]

on the top of function you are tying catch the exception

[HandleProcessCorruptedStateExceptions]
[SecurityCritical]
public IFeatureLayer GetOrCreateFeatureLayer(string path, esriGeometryType type, ISpatialReference sr)
{

//body

}

Read More

Good compatibility of latest vagrant and virtual box

Continuing from my older post  on Vagrant 1.7.2 / 1.9.2 and Virtualbox 4.3.12  / 5.1.14. I upgraded into Vagrant 1.9.2 and 5.0.x and 5.1.x , because of security vulnerabilities on those Virtualbox versions, but my shell provisions and puppet provision that used to work on old setting suddenly stopped working and tried various workaround to make them working but failed until Virtualbox 5.1.18. I use chocolaty, to upgrade/install vagrant and virtual box in Windows 7 host and able to fire 'vagrant up' to provision my new guest.

Step 1

Install chocolaty in host machine using cmd.exe (Run elevated)

@powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

  choco -v (Just to make sure chocolaty installed correctly)
  
  0.10.3 (as of 3/16/2017)

Step 2
Open Windows' power shell (Run elevated) 
1. Install or upgrade oracle virtual box

choco install virtualbox -version 5.1.18
                                                                                                         5.1.18 r114002
choco upgrade virtualbox

2. Install or upgrade vagrant

choco install vagrant -version 1.9.2

choco upgrade vagrant

3. Restart the machine

4. Vagrant up ....Boom all old scripts starting work again!

Read More