Posts Tagged ‘ArcGIS Server’

Enterprise GIS as Virtual Infrastructure Proving Grounds

June 14, 2011

Beware dork out session below

I’d like to solicit advice from the GIS\VM community regarding our ArcGIS Server 10 upgrade on VMWare ESXi, especially regarding the table below. Are there any major factors which I left out or factors which look like they might be a problem?

We all know the “cloud” is all the rage. Even Apple is doing it. I don’t want to rehash the vague benefits, but rather some of the concrete obstacles.

At the Port of San Diego we have a Virtual Infrastructure leveraging VMWare’s ESXi 3.5 software. About a year ago our Development and Production environments (Linux\Oracle11g\SDE and Win2003\AGS9.3.1) were ported over to this environment. We gave the physical machines over to these environments. For about 6 months everything worked fine. I regularly checked the ArcGIS Server Logs- No errors. Managers, administrators and users were all satisfied with the performance.

On January 26th we had an unplanned server outage. The system has never been the same since. The server and the services were restarted successfully, but our performance dropped to unacceptable levels. I could get the system to error by simulating a few users using different browsers. It certainly didn’t feel as snappy. In some cases layers wouldn’t load and we started getting lovely errors like those below in our ArcGIS Manager log.

Tension between what I refer to as “hardware guys” and “software guys” ensued.  As a software guy my feeling was that something in the VM had changed. At one point the production GIS webserver was placed into a “throttled” state. Our Oracle DBA referred to it as “nice” mode. The hardware team took exception to that categorization. The hardware guys claimed it was an OS issue and the servers weren’t even leveraging all the virtual hardware we had access to. Our Oracle DBA pulled the Linux\Oracle11g\SDE machine out of the Virtual Infrastructure and onto a physical Solaris machine to free up some resources. Performance increased but has never reached an acceptable level for a critical business system.

I have troubleshot this performance issue from every angle I can think of. I worked towards increasing the performance of the services: caching aerials, using optimized symbology, finer scale dependencies, republishing services as MSD based services and fixing every issue associated with the analyze button on the Map Service Publishing toolbar. I did everything I could think of at an OS level, defragmenting virtual and LUN disk, disk cleanup, checkdisk, defragmenting page files. I learned the nitty-gritty details of perfmon counters to ensure the application server was performing properly. I delved into the performance tab available in our VMWare Infrastructure Client and tweaked our VM settings to leverage memory ballooning. We created a new volume specifically for page files. Disabled Acceleration, disconnected virtual floppy and CD drives. If you are still reading you probably understand my frustration.

That brings us to today and our upgrade to ArcGIS 10. Our plan is to do an in-place database upgrade but create new Windows Server 2008 64-bit VMs, thereby overcoming the 32-bit OS memory limitation. According to a white paper from Esri and VMWare on deploying ArcGIS Server in VMWare Infrastructure: “Under average conditions, a CPU in a SOC machine can support about four concurrently active service instances. … If each machine is a dual-CPU system, this configuration can accommodate about 16 users simultaneously performing operations on services.” Under this logic our 4 core machine should be able to handle 64 concurrent users. My goal is to identify the inevitable differences between the white paper and our future environment. This is not an apples-to-apples comparison. Below is a spreadsheet outlining a series of factors which could effect performance.

Click on image to open larger version

As the webserver administrator should I have followed our Oracle DBA’s lead and moved back to physical machines? None of our other mission critical business systems (Email, Document Management, or ERP) are on the Virtual Infrastructure, nor is there any intention of moving them to VMs. Our Enterprise GIS intends to be considered among this group and leveraged in a similar way. If our implementation is successful we hope to prove not only that our GIS implementation ready for prime-time, but also that our Virtual Infrastructure can support other critical business systems. This will pave the way for other systems to move to the Virtual Infrastructure thereby realizing the efficiency, availability, flexibility and financial benefits of managing our own cloud.

ESRI ArcGIS Server 9.3 for VMware Infrastructure Deployment and Technical Considerations Guide White Paper
http://www.vmware.com/files/pdf/ESRI-DeploymentGuide-v1.0.pdf (permalink)

MyNewFavoriteThing Zen

Teaching a Class at on the ArcGIS Viewer for Flex at Mt. San Jacinto College

February 17, 2011

I’ll be teaching an 8 hour seminar style class titled Creating a GIS Website using ArcGIS Viewer for Flex on Friday, May 13th. The class will focus on manipulating XML files to quickly get a GIS web application deployed. We will also discuss differences between GIS for paper and web mediums as well as an introduction to server technologies.

This class is just one of the great opportunities being offered as part of the FREE! Friday GIS Workshops at Mt. San Jacinto College. The classes covers a wide range of topics from introduction and technical classes to industry specific GIS applications and integration with other systems.

Considering it is an advanced class, I have received a few questions about what knowledge is needed to take the class. The ArcGIS Viewer for Flex lowers the barrier to entry to quickly deploy a map to the web. It does not require any programming, but it does mean students should be familiar with GIS concepts so they can be applied in a server environment. Below is a link to a 16 minute video which covers many of the concepts in the class. We will go into more detail, but if you can follow along with the video you should be fine.

ArcGIS Viewer for Flex: A Quick Introduction (16 minute video)
http://resources.arcgis.com/gallery/video/arcgis-server/details?entryID=65D30C0E-1422-2418-7F6F-52EBCEC0E676

Creating a GIS Website using ArcGIS Viewer for Flex flyer
http://evarigisconsulting.com/Documents/FlexClassFlyer.pdf

How to sign up
http://msjcgisprogram.eventbrite.com/

WorldwideDrinkingHabits Zen

Leveraging GIS and Blackberry Investments Using Freeance View

August 16, 2010

Here at the Port of San Diego we have made substantial investments in our Blackberry Enterprise Server (BES), Blackberry devices and ArcGIS Server. Freeance View, the free version of Freeance’s Mobile suite, allows us to leverage these investments. This software enables our Port employees access to our internal GIS resources on their Blackberrys. There are certainly pro and cons to this integration effort.

Pros-

  • Freeance Software is dead easy to implement. No extra classes or programming needed. The server side work took about an hour. There was no special configuration needed on our BES
  • Security and network configuration is already fully configured. There was no talk about punching holes or implementing security for our services.

Cons-

  • Blackberrys deployed at the Port 1) have small screens 2) no touch screen and 3) no integrated GPS. This might be fine for email, but it is a sub-par environment to work with GIS data.

Currently this product is in development; meaning we don’t guarantee it’s full functionality 24×7. Depending upon feedback from our users we might deploy it as a production tool. In the video below we show how to 1) download Freeance View 2) configure the software to use our development GIS resources and 3) navigate the map to show how an easement cuts across a parcel.

See the links below, if you are interested in trying out this product on the Blackberry simulator used in the video.

The Blackberry emulator/simulator used in the video can be downloaded from the link below
http://na.blackberry.com/eng/developers/resources/simulators.jsp

You will also need BlackBerry Email and MDS Services Simulator
http://na.blackberry.com/eng/developers/browserdev/devtoolsdownloads.jsp

MostInterestingArticleIHaveReadInAWhile Zen

Creating a Tree Inventory Using ArcGIS Server and Google Streetview

March 22, 2010

The benefits of trees cannot be overstated. They provide numerous financial, health and quality of life benefits for our community. The well-being of our urban forest at the Port of San Diego is one of the key resources we manage. The PortGIS Program attempts to identify resources which are advantageous to represent spatially. By placing utilities, proposed projects, the Port Master Plan, the 2010 Tidelands Mapbook, aerial photos and our tree locations within one spatial context we empower our Port employees to make  informed decisions by driving an enterprise-wide common operating picture.  Plans for future development will account for our urban forest resources from the outset if they are included in the primary set of guiding factors. It is much more costly and less likely to occur if urban forestry management is left to expensive last-minute change orders.

Much of the Port Tidelands can be seen through Google Streetview. Some find this disconcerting and an invasion of privacy, but Google Streetview is an invaluable and (IMHO) underutilized tool for creating and managing GIS Data. It can save substantial time and monies by moving expensive field work to the desktop. Managers can easily implement quality control measures by reviewing decisions made in the field by lesser experienced personnel in real-time. Real functionality comes from combining  Google Streetview images and GIS data housed in a Relational Database Management System. This PortGIS Tree Inventory tool is a departure from our other web applications in that it empowers end users to change GIS data. Our other tools are read only.

Below is a video showing how to use this tool.

Beware- GIS Dork Out Session below

From a technical GIS/database/developer perspective, this project started out as a proof of concept. Our goal was to enable our non GIS professionals to easily create, edit and manage a discrete set of GIS data through the web. In order to do this the data needed to versioned and housed within an SDE database. We also wanted to enable security, and grant permissions through Active Directory, so only certain Port employees are able to make changes. Our goals are to continue to build on this functionality and to expand the capabilities of the PortGIS Program. We would like to take this data into the field to collect more information, i.e., standard breast height of our trees and/or take pictures of the trees using a connected handheld device. Taking the PortGIS program mobile will enable us to offer real-time GIS data management to our non-GIS professionals where and when it is most convenient. This way they can use GIS as a tool which complements their primary focus of writing leases, creating architectural renderings, fixing electrical conduits, or making our urban forest as healthy, financially viable and beautiful as possible.

MyFavoritePicOfMy3MonthOldDaughter Zen

Accessing Aerial Photos within AutoCAD using ArcGIS for AutoCAD and the mapiinsert Command

October 1, 2009

At the Port of San Diego, our CAD Designers and Mapping Technicians can access our new 2009 4-inch aerials photos within AutoCAD in 2 ways. Each has their own advantages and disadvantages. Understanding these pros and cons empower our CAD designers to be more efficient and effective. I suggest any substantial project would be best approached by utilizing both methods.

ArcGIS for AutoCAD- Great for everyday workflow
Pros
fast
access to content on other ArcGIS Servers, including ArcGIS Online
easy to create image with defined extent
Cons
Best resolution not guaranteed at all scales

mapiinsert command – Printing
Pros
best resolution guaranteed at all scales
CAD users more familiar
Cons
slow

As a footnote, it should also be mentioned that ArcGIS for AutoCAD has another powerful component; the ability to create GIS attribute data within AutoCAD. As the PortGIS Program matures it will require us to gain more information about our features which originate in AutoCAD. What is the material/width/flow capacity of this pipe? Collecting this type of attribute data will allow us to 1) display and symbolize this data across the enterprise and 2) do spatial analysis, such as identifying where weaknesses might be in our utility infrastructure.

ArcGIS for AutoCAD can be downloaded from ESRI at the link below.
http://www.esri.com/software/arcgis/arcgis-for-autocad/download.html

GeographyHuntingBinLaden Zen
and another

GIS Dork-Out Session: ArcGIS Server Pathing and Roundtrips in ArcGIS for AutoCAD

September 2, 2009
I recently received the email below from a fellow GISer.
Hi Ari,
I attended your presentation at the UC last month which I found very interesting, and spoke to you briefly afterwards. I wondered if you’d be able to answer a couple of questions I have about your ArcGIS Server implementation. We are going through the implementation process ourselves at the moment. Our scope is global although right now we’re focused on data that is within the UTM Zone 10 area in southwest BC.
Do you still keep data files on your network? We anticipate that our network will be required for working shapefiles, MXDs, etc. But I’ve also learned that to publish MXDs the data within them has to have a UNC filepath, not a mapped drive. Is that the kind of setup you have? Because the data will be stored in single features classes for multiple projects and clients I anticipate having to create a lot of LYR files.
I am also interested in knowing more about how you deal with CAD data. We have been able to generate a test map service that included some basemap CAD topo, but I’m still not clear on the full power of integrating our CAD data. Our SDE database will have WGS84 projected data, and our CAD is mostly NAD83 UTM (e.g. zone 10). I think you mentioned you can edit CAD data directly within SDE or a map service – is that correct? Did you create any customization to do that?
If you have any time to get back to me with advice on those questions, or any other tips, I’d really appreciate it.
Thanks,
Kieran
……………………………………………….
Kieran Smith, M.Sc.
GIS Analyst
Knight Piésold Ltd.

    

I am so glad that you enjoyed our presentation. There are many GISers out there dealing with many of the same issues. You seem to have two main issues 1) UNC paths and 2) CAD. I will address them each individually.

1) I assume you are referring to the warning 10027 “Layer’s data source is referenced via a UNC path” when you use the Analyze Button ATT26608 on the Map Service Publishing toolbar. ArcGIS Server likes local file system path names, but this can create problems when trying to access data on your network. The best way to get around this is to make the source paths the same whether you open the mxd on your local box or on the server. Mapped drives to shares do not give this warning. The easiest way to solve this is to find a drive letter which is not being used on the server and your personal machine. Share the proper folder and map to the share with the same drive letter on both machines. The mxd will open properly on both machines and will allow you to publish using either ArcGIS Server Manager on the server or ArcCatalog from your personal machine.

Another method is to use UNC paths for your development efforts and file system paths for production maps, data and services. To do this you should install ArcGIS Desktop on your server machine. When you want to create a production service then resource the data in your MXDs to the servers local file system path and republish the service from ArcGIS Server Manager using file system paths. UNC paths will work fine for development efforts. I don’t think there is a performance cost, but it is not quite as stable. This is the method we use at the Port of San Diego. We have a “production” folder which holds MXDs and raster and vector data, and we publish our services to a “production” folder. Everything in these production folders use file system paths and outside these folders we use UNC paths.

2) The product I mentioned in our presentation is ArcGIS for AutoCAD. There are 2 parts to this tool. The first is the ability to use AutoCAD as a client to view ArcGIS Services (including ArcGIS Online data out of the box for free). The second is the ability to give an engineering drawing characteristics which closely resemble GIS data including the ability to define coordinate systems and/or give attributes to features. When this dwg is brought into ArcGIS Desktop and/or published using ArcGIS Server these characteristics shine through. If all your GIS data, services and CAD drawings have defined Coordinate Systems it will all reproject on the fly. Of course, if the CAD designer moves the data 100 feet over this will throw everything off. The CAD designer really needs to follow both CAD and GIS standards.

This functionality creates the ability to do what we have been calling a “roundtrip.” CAD data is edited in AutoCAD, served through ArcGIS Server and then consumed within AutoCAD using ArcGIS for AutoCAD.

Don Kuehne, ESRI’s technical product manager for CAD interoperability, describes it well on his blog at the link below. http://giscadblog.blogspot.com/2009/03/to-serve-cad.html

Below is a screenshot of a “roundtrip” in action. The image and the linework are both coming from ArcGIS Services.

JustCantGetEnoughAwesomeBasketballShots Zen