Archive for the ‘GIS Techy Stuff’ Category

Microsoft’s Obstacles Are Our Obstacles

July 22, 2011

The Director of the Business Information Technology Services Department asked us: Who does what we do well? After thinking about this question, I have tried to refocus the question by thinking about what obstacles we have? Who overcomes those obstacles well? My answer is Microsoft.

Eric Schmidt, Google’s Executive Chairman, does not consider Microsoft as part of the “Gang of Four”; Google, Facebook, Apple and Amazon. These are the tech companies he feels are driving growth and innovation.

Considering the dominance Microsoft had in the previous decade, and in many ways still holds, they have obstacles which their competitors don’t. Microsoft, more than any other technology company, has to manage a wide range of legacy products and they need to maintain compatibility with those products while staying innovative and ahead of the bleeding edge. Microsoft needs to keep one foot solidly planted in their current customers needs, but they cannot allow other firms to dictate their future.

Many of Microsoft’s products date back to an era when being in a connected environment was optional and client-side applications ruled the day. Now virtually all devices are connected all the time. Of the “Gang of Four”, only Apple provides a platform which could be useful in a disconnected world. The “Gang of Four” has a major advantage over Microsoft in that their businesses can be more easily managed server-side. For example, Facebook recently added a program called deals which offers coupons and online offers. No client-side changes needed to be made. End users did not need to purchase or install new software.

While Microsoft had a good jump on web-based email with it’s Hotmail service, it has been criticized for being late to the game, failing to capitalize or loosing ground in the mobile OS, mobile hardware, tablets, online documents, search, browser and virtual environment spaces. Microsoft continues to be successful in desktop operating systems, office software suites, Sharepoint, Exchange, development tools, databases. The only space they have recently made substantial inroads into is with their video gaming system, XBOX. I believe their success with XBOX is because this was a new environment for Microsoft which wasn’t expected to integrate with their legacy products. This is much the same environment in which the “Gang of Four” compete.

At the Port of San Diego we have similar issues. We have a mature IT landscape where we have made significant investments in solutions from Novell, Cisco, SAP, Esri and OpenText Hummingbird. We also have mature coworkers and clients (AKA Port employees) who have a clear and often resolute understanding of how things (should) work. More than a few times I have encountered situations where clients have requested a system only to learn that they will need to be the source of the authoritative data. For example, Joomla separates web design from content. When it is explained that anyone can share information across the Port through Joomla, suddenly the solution isn’t quite as appealing; responsibility which they thought would be provided for them falls into their lap.

On June 1st, 2011 Steven Sinofsky, President of Microsoft’s Windows and Windows Live Division, and Julie Larson-Green, Corporate Vice President of Windows Experience, sat down with Walt Mossberg and Kara Swisher from the Wall Street Journal to discuss the future of Windows, in particular the next incarnation of the OS, currently known as Windows 8. The most visible changes from Windows7 is a reorganization of the graphical user interface (GUI) to leverage the use of tiles, instead of familiar windows. Tiles allow key information from within an application to be displayed outside the application. The other major change is that Windows 8 will be the same across all form factors. Everything from 8 inch displays to wall size displays, touch screen interface to any combination of peripherals and accessories. Click here to watch the full video.

The reason why this conversation is important is not because new features are coming to Windows. The new functionality only provides a context to discuss much more consequential changes in the IT world. This conversation is important because Microsoft needs to engage people to invest in the new technology while providing a succession plan for it’s legacy products which are currently key components to their customers business.

Unexpectedly, this larger and more important concept seemed to be well over the Wall Street Journal host’s head. Throughout the interview Walt Mossberg kept voicing shortsighted comments along the lines of; just make it pretty like Apple does. He didn’t seem to understand basic concepts like the difference between Windows and Office or the role of a desktop PC vs a server in a traditional IT landscape. His amateur perspective is useful because this is how end-users\clients\employees react to our choices. His uneducated, yet bombastic, attitude also resembles personalities and perspectives we encounter. His frustration mirrors a non-technically savvy Port employee’s frustration. Microsoft’s obstacles are our obstacles.

Throughout the conversation Walt Mossberg kept pushing Steven Sinofsky to answer why Windows 8 will be built upon future versions of Windows. Why is Microsoft so committed to bulky and sluggish Windows? Why not start from scratch? Why not expand Windows Phone OS to tablets, as Apple has done, instead of making the desktop OS work on tablets as well? Below are a few key points which show how Microsoft makes the decision to either keep one foot in the legacy world by making their current functionality forward compatible and\or completely transition out of a legacy product and into a new tool.

On the decision to build Windows OS to include tablets
WM: It is not the strategy which has been followed by your two bigger competitors, Google and Apple. Who felt like they needed to have a … well Google didn’t have a legacy computer OS, and Apple did.

SS: Well they [Google] built one.

WM: They built one. But they built it for phones and tablets.

SS: And then they went and built their own legacy OS.

WM: Eric [Schmidt] was explaining. Now you didn’t listen. He said one’s for typing and ones for touching.

SS: And you bought that?

WM: I am just saying what he [Eric Schmidt] said. Being a reporter. Apple took a computer OS and I guess stripped it down to some extent then added more stuff back into it, but tried to customize something. They didn’t move the Mac OS to the iphone and the ipad. You are taking a different approach. Why? You just explained why you could do it. Why would you want to do it? Why is it a better thing for consumers?

SS: It’s better because of all of the things Windows can bring with it.

WM: Here’s were I need to be snarky and say mean things. Like viruses and craplets and all the things people get on their Windows computers.

SS: Or printing, or using solid state storage really well or external hard drives

WM: The other guys use solid state storage pretty well.

SS: Without wierdness and stuff like that. What we tried to do with Windows 8 was reimagine how to work with a PC. Really think from the chip all the way to the user interface what it is we could do different. Not just get touch or a slate or a tablet made but to really think about the broad range of scenarios and really reimagine what we wanted to do with Windows. You could sort of say we colored outside the lines in terms of how we built this release. So we are excited to talk about it because we think it is a different approach. Maybe it is because we are not in the Gang. We chose a different avenue. We have a real customer and engineering scenario things we thought about. We have an approach that is different but also builds on the value of 400 million PC that will be probably be sold in the year we release this product. That is a big number. What will happen is all of a sudden, all of those PCs accrue all the benefits of the work that we are doing.

WM: So both tablets and PCs, laptops, desktops, whatever. All of this work, all of this coloring outside the lines spreads across all the …

SS: Right. What we’ve done is look from the ground up how to rethink how you interact with Windows. The kind of programs you can run. How you get those programs. All of that work and really bring it. A word we used a lot in developing it is “modern.” How to think about it in a different way that really solves a bunch of the things that people see, or say they see, solved in an ipad. I think we can do that and bring with it all of these benefits you have.

WM: Every program that runs on desktop windows will run on ..

SS: It’s Windows. Windows is there. Everything that runs on Windows7, any device you can plug into a Windows machine. Everything just runs.

When pushed about reinventing Microsoft Office
WM: So why didn’t the Office … Seriously. If you’re really going to this bold new, color-outside-the-lines design, and it’s part of your whole design ethos and design family, why didn’t the Office team write or rewrite Office for that kind of approach?

LG: Well. They may do something … in the future. But we don’t think people should have to give up everything they know and love to get to a more mobile form factor.

More pushing about Microsoft Office on Tablets
KS: And if you’re looking for the whole experience, the idea (that) everything’s changed, I was sort of looking at this very beautiful screen and went, uck, the ugly old house if back, you know? Why not switch the whole thing? … I mean, this is a whole different look. Because Bing fits into this nicely. Why not do everything?

LG: There’s a lot of utility in existing Office around running macros and doing things that take keyboard intensive time, and I’m sure that the Office team will look at what we’re doing…

ARM Hardware Support Changes
Question from Audience: Do you have a strategy for existing Intel applications as they move to ARM?

SS: On our ARM machines … we are not going to introduce a virtualization model and a way to run old x86 software. That turns out to be technically really challenging and we decided the experience we could deliver with modern applications all written in HTML5 and Javascript … would probably yield a better experience over time.

Comparison to Mistakes by IBM in 1981
Question from Audience [Tim O’Reily]: I want to return to the question of the Gang of Four in a particular way. First off, I really like the UI that you are showing but it strikes me that one of the big differences between what you are showing and the Gang of Four is that a big part of what they have are applications in which the data is what’s driving things and in particular network effects in data seems to be what makes Google and Facebook and Apple and Amazon such interesting dynamic companies. They are harnessing billions of people to build a service that actually gets literally better the more people use them. Now Microsoft has services of that nature but they don’t seem to be featured in what you are doing. I am wondering if you are making the same mistake that IBM made back in 1981 when they thought hey, this is all about hardware and you guys are thinking oh this is all about software and you are not integrating Microsoft’s data assets into the core of your OS.

I guess the question is,  to what extent are they informing the product design. It would be really fabulous, it seems to me, if you guys actually thought how to show that and make that part of your story so that people understand that that is actually what makes it a modern platform not having a touchscreen.

LG: I showed a few of the elements on how we are rethinking. I didn’t show notifications or how applications talk to each other and get information from each other to make the experience better, but the whole start screen being alive and being connected from the get go without having to go to your desktop then decide where you want to go today. Everything is just there in front of you. When we are done, and we can show the full version of Windows, those applications and things will be part of the experience. All the services behind it. We work very closely with that team.

SS: We are the Windows and Windows live team so we have Hotmail and Messenger all part of one team. We just aren’t on that today.

I want to point out Microsoft did not take the same approach to all topics. They are going to be backward compatible with Office and other applications, but they are not going to implement a virtualization model on ARM processors. At Windows 8 they decided to go distinctly different directions on these 2 topics. I am sure these concepts were explored ad nauseam in Redmond.

Unfortunately, Microsoft is in the uncomfortable position of meeting both the shallow expectations of Walt Mossberg (just give him a nicer monitor and install Rainmeter with the Omnimo 4 skin – he won’t know the difference) and the relevant and profound expectations of Tim O’Reily all while supporting a wide range of mission critical legacy systems. The “Gang of Four’s” advantage is that they simply don’t come with baggage. Is the lesson to leverage our investments or start afresh? The answer for both the Port of San Diego and Microsoft us is clear as mud.

WhatTheHeckIsWrongWithPeople Zen


Enterprise GIS as Virtual Infrastructure Proving Grounds

June 14, 2011

Beware dork out session below

I’d like to solicit advice from the GIS\VM community regarding our ArcGIS Server 10 upgrade on VMWare ESXi, especially regarding the table below. Are there any major factors which I left out or factors which look like they might be a problem?

We all know the “cloud” is all the rage. Even Apple is doing it. I don’t want to rehash the vague benefits, but rather some of the concrete obstacles.

At the Port of San Diego we have a Virtual Infrastructure leveraging VMWare’s ESXi 3.5 software. About a year ago our Development and Production environments (Linux\Oracle11g\SDE and Win2003\AGS9.3.1) were ported over to this environment. We gave the physical machines over to these environments. For about 6 months everything worked fine. I regularly checked the ArcGIS Server Logs- No errors. Managers, administrators and users were all satisfied with the performance.

On January 26th we had an unplanned server outage. The system has never been the same since. The server and the services were restarted successfully, but our performance dropped to unacceptable levels. I could get the system to error by simulating a few users using different browsers. It certainly didn’t feel as snappy. In some cases layers wouldn’t load and we started getting lovely errors like those below in our ArcGIS Manager log.

Tension between what I refer to as “hardware guys” and “software guys” ensued.  As a software guy my feeling was that something in the VM had changed. At one point the production GIS webserver was placed into a “throttled” state. Our Oracle DBA referred to it as “nice” mode. The hardware team took exception to that categorization. The hardware guys claimed it was an OS issue and the servers weren’t even leveraging all the virtual hardware we had access to. Our Oracle DBA pulled the Linux\Oracle11g\SDE machine out of the Virtual Infrastructure and onto a physical Solaris machine to free up some resources. Performance increased but has never reached an acceptable level for a critical business system.

I have troubleshot this performance issue from every angle I can think of. I worked towards increasing the performance of the services: caching aerials, using optimized symbology, finer scale dependencies, republishing services as MSD based services and fixing every issue associated with the analyze button on the Map Service Publishing toolbar. I did everything I could think of at an OS level, defragmenting virtual and LUN disk, disk cleanup, checkdisk, defragmenting page files. I learned the nitty-gritty details of perfmon counters to ensure the application server was performing properly. I delved into the performance tab available in our VMWare Infrastructure Client and tweaked our VM settings to leverage memory ballooning. We created a new volume specifically for page files. Disabled Acceleration, disconnected virtual floppy and CD drives. If you are still reading you probably understand my frustration.

That brings us to today and our upgrade to ArcGIS 10. Our plan is to do an in-place database upgrade but create new Windows Server 2008 64-bit VMs, thereby overcoming the 32-bit OS memory limitation. According to a white paper from Esri and VMWare on deploying ArcGIS Server in VMWare Infrastructure: “Under average conditions, a CPU in a SOC machine can support about four concurrently active service instances. … If each machine is a dual-CPU system, this configuration can accommodate about 16 users simultaneously performing operations on services.” Under this logic our 4 core machine should be able to handle 64 concurrent users. My goal is to identify the inevitable differences between the white paper and our future environment. This is not an apples-to-apples comparison. Below is a spreadsheet outlining a series of factors which could effect performance.

Click on image to open larger version

As the webserver administrator should I have followed our Oracle DBA’s lead and moved back to physical machines? None of our other mission critical business systems (Email, Document Management, or ERP) are on the Virtual Infrastructure, nor is there any intention of moving them to VMs. Our Enterprise GIS intends to be considered among this group and leveraged in a similar way. If our implementation is successful we hope to prove not only that our GIS implementation ready for prime-time, but also that our Virtual Infrastructure can support other critical business systems. This will pave the way for other systems to move to the Virtual Infrastructure thereby realizing the efficiency, availability, flexibility and financial benefits of managing our own cloud.

ESRI ArcGIS Server 9.3 for VMware Infrastructure Deployment and Technical Considerations Guide White Paper (permalink)

MyNewFavoriteThing Zen

Leveraging GIS and Blackberry Investments Using Freeance View

August 16, 2010

Here at the Port of San Diego we have made substantial investments in our Blackberry Enterprise Server (BES), Blackberry devices and ArcGIS Server. Freeance View, the free version of Freeance’s Mobile suite, allows us to leverage these investments. This software enables our Port employees access to our internal GIS resources on their Blackberrys. There are certainly pro and cons to this integration effort.


  • Freeance Software is dead easy to implement. No extra classes or programming needed. The server side work took about an hour. There was no special configuration needed on our BES
  • Security and network configuration is already fully configured. There was no talk about punching holes or implementing security for our services.


  • Blackberrys deployed at the Port 1) have small screens 2) no touch screen and 3) no integrated GPS. This might be fine for email, but it is a sub-par environment to work with GIS data.

Currently this product is in development; meaning we don’t guarantee it’s full functionality 24×7. Depending upon feedback from our users we might deploy it as a production tool. In the video below we show how to 1) download Freeance View 2) configure the software to use our development GIS resources and 3) navigate the map to show how an easement cuts across a parcel.

See the links below, if you are interested in trying out this product on the Blackberry simulator used in the video.

The Blackberry emulator/simulator used in the video can be downloaded from the link below

You will also need BlackBerry Email and MDS Services Simulator

MostInterestingArticleIHaveReadInAWhile Zen

GIS Dork-Out Session: ArcGIS Server Pathing and Roundtrips in ArcGIS for AutoCAD

September 2, 2009
I recently received the email below from a fellow GISer.
Hi Ari,
I attended your presentation at the UC last month which I found very interesting, and spoke to you briefly afterwards. I wondered if you’d be able to answer a couple of questions I have about your ArcGIS Server implementation. We are going through the implementation process ourselves at the moment. Our scope is global although right now we’re focused on data that is within the UTM Zone 10 area in southwest BC.
Do you still keep data files on your network? We anticipate that our network will be required for working shapefiles, MXDs, etc. But I’ve also learned that to publish MXDs the data within them has to have a UNC filepath, not a mapped drive. Is that the kind of setup you have? Because the data will be stored in single features classes for multiple projects and clients I anticipate having to create a lot of LYR files.
I am also interested in knowing more about how you deal with CAD data. We have been able to generate a test map service that included some basemap CAD topo, but I’m still not clear on the full power of integrating our CAD data. Our SDE database will have WGS84 projected data, and our CAD is mostly NAD83 UTM (e.g. zone 10). I think you mentioned you can edit CAD data directly within SDE or a map service – is that correct? Did you create any customization to do that?
If you have any time to get back to me with advice on those questions, or any other tips, I’d really appreciate it.
Kieran Smith, M.Sc.
GIS Analyst
Knight Piésold Ltd.


I am so glad that you enjoyed our presentation. There are many GISers out there dealing with many of the same issues. You seem to have two main issues 1) UNC paths and 2) CAD. I will address them each individually.

1) I assume you are referring to the warning 10027 “Layer’s data source is referenced via a UNC path” when you use the Analyze Button ATT26608 on the Map Service Publishing toolbar. ArcGIS Server likes local file system path names, but this can create problems when trying to access data on your network. The best way to get around this is to make the source paths the same whether you open the mxd on your local box or on the server. Mapped drives to shares do not give this warning. The easiest way to solve this is to find a drive letter which is not being used on the server and your personal machine. Share the proper folder and map to the share with the same drive letter on both machines. The mxd will open properly on both machines and will allow you to publish using either ArcGIS Server Manager on the server or ArcCatalog from your personal machine.

Another method is to use UNC paths for your development efforts and file system paths for production maps, data and services. To do this you should install ArcGIS Desktop on your server machine. When you want to create a production service then resource the data in your MXDs to the servers local file system path and republish the service from ArcGIS Server Manager using file system paths. UNC paths will work fine for development efforts. I don’t think there is a performance cost, but it is not quite as stable. This is the method we use at the Port of San Diego. We have a “production” folder which holds MXDs and raster and vector data, and we publish our services to a “production” folder. Everything in these production folders use file system paths and outside these folders we use UNC paths.

2) The product I mentioned in our presentation is ArcGIS for AutoCAD. There are 2 parts to this tool. The first is the ability to use AutoCAD as a client to view ArcGIS Services (including ArcGIS Online data out of the box for free). The second is the ability to give an engineering drawing characteristics which closely resemble GIS data including the ability to define coordinate systems and/or give attributes to features. When this dwg is brought into ArcGIS Desktop and/or published using ArcGIS Server these characteristics shine through. If all your GIS data, services and CAD drawings have defined Coordinate Systems it will all reproject on the fly. Of course, if the CAD designer moves the data 100 feet over this will throw everything off. The CAD designer really needs to follow both CAD and GIS standards.

This functionality creates the ability to do what we have been calling a “roundtrip.” CAD data is edited in AutoCAD, served through ArcGIS Server and then consumed within AutoCAD using ArcGIS for AutoCAD.

Don Kuehne, ESRI’s technical product manager for CAD interoperability, describes it well on his blog at the link below.

Below is a screenshot of a “roundtrip” in action. The image and the linework are both coming from ArcGIS Services.

JustCantGetEnoughAwesomeBasketballShots Zen