Dark Data & The Data Disconnect

Dark DataI ran across a blog the other data that used the term “dark data” when describing an organization’s data that is collected during standard business activities that never really gets used for any purpose other than its original purpose.  Dark Data is that data the gets created, collected, stored but rarely revisited for any purpose after its initial use.

Before we get into the world of Dark Data, let me take a quick detour to set the stage for this particular topic.  I’ve written about Shadow IT many times in the past and specifically touched on the topic of ‘dark data’ in a post titled Data Disconnect. In that post, I wrote the following about the “data disconnect” problem:

You can do your job. And…you can do your job well.  But…your work is being done in a vacuum.

Your data might remain in that vacuum. But worse…whatever knowledge that data might create (or has created) will remain in that vacuum as well.

The data disconnect problem is real and is one that hasn’t been addressed much in recent years. The same can be said of Shadow IT, although many organizations I speak with tell me they have been able to get in front of Shadow IT recently.

So – back to Dark Data.  Do you think the data disconnect creates dark data?

While it may not be the only reason dark data exists, it is a main driver of the phenomenon in my experience.

Let’s take a look at a scenario taken from a real-world example.

The Scenario

You are the director of marketing for a mid-sized business.

One of your goals for this year is to revamp your web analytics engine to better understand your site visitors and help inform decisions on how well your on-site marketing is working and what types of content work best for your clients.

You reach out to your IT group to ask for help and receive very little in the way of support. The first response you get is a standard “it isn’t in the budget for the year” or “we’ll need to run that through our portfolio management system”.   You talk to the CIO who says budget and process are the drivers for this type of thing and there’s not much he can do.

What do you do?   You do the same thing every marketing person has done for years…you go out and start talking to web analytics software and consulting companies to see how they can help.

After a few months and a selection process, you decide to go with company X for their analytics platform. This platform requires very little in the way of integration with your website so it takes very little to get the few lines of code integrated into the site.  You do need to get the IT group to do add these lines of code (which takes much longer than you thought it should).

Your new analytics engine is in place and you are now able to see start asking and answering better questions about how visitors are consuming your website.

Your goal of implementing a new system has been met and your Marketing VP is ecstatic with your performance and gives you a nice bonus and some well deserved kudos.

Within a few months, other folks within the organization start you questions about the website. You are able to quickly and easily tell them how many visitors you’ve had, where they come from, what they do while there and various other types of questions. You are using your analytics data for its intended purpose.

Then one day you hear of a new project. This project is looking at all areas of customer ‘touch’ and analyzing how well those touch points are working and whether these touch points a can be improved upon.  The CIO asks you how to interface with your analytics data to begin to use it alongside other analytical data to begin to study the organization’s touch points with clients.

You ask your analytics provider how you can start to integrate their data with your local data. The answer “we don’t provide that capability”.

You’ve just realized you are trapped in the data disconnect. You have data that you can’t use anywhere else in the organization. You have data that is useless outside of your web analytics engine.   You have dark data.

Moral of the Story

This is a real story from a real organization.

The outcome of this particular scenario is the marketing department had to backtrack and start using another analytics engine that integrated with the other systems used throughout the organization. This backtrack left about 6 months of web analytics sitting out in the cloud in a provider’s system without any way of analyzing that data outside their tool.

This sort of thing happens every day in business. People go out and purchase their own applications and systems without IT involvement because the cloud makes it easy to do so. This is Shadow IT.

There are approaches to alleviate Shadow IT, but you can never fully stop it today. It is much too easy to plunk down a credit card number and order services via the cloud today.

The IT group can help to alleviate Shadow IT by being a bit more proactive as well as a bit more interactive when dealing with the organization. It’s not longer good enough to say “no” or “the process is…” or “our budget doesn’t allow…”.  The organization is looking for solutions and the IT group needs to start being the group that says ‘yes’ – or at least ‘let’s figure out how to make this work’.

Shadow IT can lead to a data disconnect which can lead to one form of dark data where data lives in your applications but cannot (or at least is not) used for any other analytical purpose.

How is your organizations fighting the data disconnect and dark data?

Image Credit: Data Packets on Flickr

Shadow IT and Enterprise Security – The boogeyman and the shadows

This post sponsored by the Enterprise CIO Forum and HP.

First off…if you aren’t familiar with the boogeyman, its the bad person in stories that are normally told to kids to keep them in line.

The boogeyman is also used by adults to scare other adults to make them stay in line. IT Professionals use the boogeyman to keep the organization in line (e.g., “you can’t do that because it will open a security hole”). Ooooh! The boogeyman!

But…the boogeyman IS real when it comes to Security.   There really are people out there looking for flaws in IT security.  There really are boogeymen out there.  And they lurk in the Shadows.

I’ve written a lot about Shadow IT in the past.  I like the idea of technology ownership outside of IT. It does two things: 1.) forces IT to stop saying ‘no’ to everything and start looking for ways to say ‘yes’ or at least “no…but we can help do that with our ____ system” ; and 2.) shows that technology is much more important to the organization than just a bunch of ‘computers’.

CIO’s should be taking advantage of the rise in Shadow IT.  They should be using this time to push for more budget to help the organization. They should also be building up a ‘consulting’ arm to help the rest of the organization better understand technology and how to select, implement and manage said technology.

Part of this consulting arm will need to focus on education as mentioned by Christian Verstraete in his Enterprise CIO Forum article titled Shadow-IT, it’s forbidden to forbid. This education is important as it helps the organization understand the underlying issues found within Shadow IT (data disconnect, information optimization, etc) but it should also stress the security implications found within Shadow IT.

These security implications are extremely important.  There’s nothing more dangerous to an organization than to have a non-secure system open for infiltration.

Because of the importance, IT must spend time talking about these issues and educating the organization. But..rather than use the old methods of  ‘the boogeyman’ that most people are used to hearing, it needs to be done in such a way to explain the dangers while keeping people interested.

The second you talk about “PCI Security” or “compliance” or one of the other boogeyman avoidance terms that we like to use in IT, people’e eyes glaze over and they immediately go on the defensive. They are reminded of the boogeyman stories of their youth and how those stories weren’t really true…so why should they believe you?

Rather than use the IT terminology to talk about security, we need to build a conversation around real issues that the organization can understand. Rather than use “PCI Compliance”, why not talk about “stolen credit cards”?  Sure they aren’t the same thing exactly…but the main point is conveyed.

Tell someone that they can’t do something because of “PCI Compliance Issues” and they’ll most likely go right ahead and do it.

Tell them they can’t do it because the system they want to use might allow credit card information to be stolen, and they’ll take a step back and ask you for help…or at least you hope they do.

True Story

I worked for an organization that had a HUGE PCI Compliance issue. A group was using a third-party web server to host their website.  Credit Card information  was being submitted via a plain text web form (!) with no security (!!) and stored in a local text file for reference (!!!) and then emailed (!!!!) to a person to run through the credit card processor.  The CIO and many IT professionals spent many hours talking to the group who were collecting the CC info. They talked about ‘PCI Compliance”, “Personally Identifiable Information” and other key terms….but the group kept on doing what they were doing.

Now…there are two things to learn here. The first, the CIO didn’t have the power to force this group to stop doing what they were doing. That’s bad for the CIO and bad for the organization. That’s a bigger issue. The 2nd issue at work here is much easier to resolve. Its the issue of educating the organization on the dangers of collecting data.  Rather than use “PCI Compliance” and the other big / buzz words, the CIO should have taken a step back and said “hey….look…I just downloaded your text file from your server and now have 2million credit card numbers”.    He didn’t do that though….he fought the “PCI Compliance” fight.

It took a boogeyman sneaking into that group’s web server and stealing those credit card numbers (and a lot of other Personal information) and then contacting the company offering to sell the information back to them before the group realized they had a problem. But..it was too late. The boogeyman had come out of the shadows.

That organization spent a good deal of money to fix that problem. The group responsible stuck to their claim that they had never really understood the ‘dangers’ in doing what they were doing. After many months of begging and pleading by the CIO, 10 minutes of hacking by a boogeyman finally told the story of how dangerous their setup was.

Talking about the Boogeyman

These folks were doing everything wrong. The CIO had no ‘power’ to stop them either.  Many things were going wrong in this instance…but one wonders if the approach to education had been a bit different, would things have turned out differently? There’s no way to tell but I would hope the answer is “yes”.

So…we in IT KNOW the boogeyman is out there. We KNOW the damage that can be done. But…we have done a poor job of communicating the danger to the organization. We tell stories of the boogeyman. We use big words and lots of jargon and the eyes of our clients glaze over.

We need to find ways to highlight the dangers of the boogeyman without actually making people think about the boogeyman.  We need to talk about the dangers of Shadow IT while educating our clients to the real dangers that exist using language they can understand.

The boogeyman exists. He lives in the Shadows.  Its our job to help shine the light on him and help them stear clear of him.

Image Credit: Boogeyman By Billie Jane on Flickr

This post sponsored by the Enterprise CIO Forum and HP.

Data Disconnect and Shadow IT

This post sponsored by the Enterprise CIO Forum and HP.

Yes Shadow IT again.

But…rather than rehash the things I’ve talked about before, I wanted to take some time to walk through a few issues that aren’t always discussed when we talk about Shadow IT.

The first is Data Disconnect, which I’ll talk about here. The 2nd is…well…you’ll have to check back later this week for that post.

Data Disconnect is exactly what it sounds like. Your Shadow IT ‘initiatives’ might just have created an environment where you have multiple disparate systems with a data disconnect.  The data in one system isn’t readily available for use in another…nor is it readily available for use in any other system.

Why might this be a problem? Lets look at a few examples.

Project Management

You need a project management tool and don’t want to use the IT provided Microsoft Project platform. You hate Gantt Charts and feel that you really need collaboration tools, which is something that Project doesn’t really do well.  The IT group offers you access to the new SharePoint system for Project Management + collaboration, but you really don’t want to have to go through the hoops necessary to get your team trained up on this new system and, if you are honest with yourself, you really hate SharePoint and will do anything to not use it.

So…you go out and plunck down your credit card number for access to Basecamp for you and your team.  You love basecamp. Your team loves basecamp.   The platform is a great place to collaborate and manage your marketing projects.

But…what do you do when the day arrives where you have to leave Basecamp?  What do you do with your data stored inside the app? What happens to all of the project knowledge from the projects and project teams that have worked inside Basecamp?    Do you just let them go or do you spend a few days downloading everything manually (or perhaps there is an API that lets you do this?).

Regardless…these things happen. While the Basecamp app was great while you used it, what happens when you stop using it (or are told to to stop using it)?  If you think about this issue beforehand, it may not be an issue at all since you can keep ‘local’ copy of data/knowledge…but the collaborative environment inside Basecamp might be gone forever once you stop using it.

Another Example – Web Analytics

You have an analytics tool that you love to use to analyze your websites. Whether its Google Analytics or  some other platform…you use it for years to track and analyze the websites of your organization.

Your CIO takes the initiative to purchase an enterprise license to a different platform to for web analytics. While they come to you for advice on this new platform, they decide to go with a platform different than your favorite one.

You love your analytical platform and have no interest in ever giving it up, but you try out the new system but it just doesn’t do what you think you need it to do.

So…you stick with your analytics platform and IT sticks with theirs. Their platform is used to track analytics of the ecommerce site and other IT driven websites.   You continue to use yours to analyze the sites that you are responsible for.

Then…you and your IT counterparts are asked to start combining your analytics reports.  Rather than push a few buttons to get a combined report, you now have to do a bunch of spreadsheet engineering to combine data…or perhaps you can build some scripts to combine data.  But…either way, you are doing additional work to combine data.

Data Disconnect

In both of the above instances, you are in the Data Disconnect world.

The Data Disconnect isn’t a reason to shy away from the world of Shadow IT, but it is something that everyone needs to be aware of when thinking about “going around” IT to purchase/implement your own systems.

In all of the groups I’ve worked with, there’s been little discussion of the Data Disconnect…but there should be.This Data Disconnect is also one of the issues that the IT group / CIO need to focus on as an educational aspect within their organizations as Christian Verstraete wrote in his Enterprise CIO Forum titled Shadow-IT, it’s forbidden to forbid.

You can do your job. And…you can do your job well.  But…your work is being done in a vacuum.

Your data might remain in that vacuum. But worse….whatever knowledge that data might create (or has created) will remain in that vacuum as well.

Image Credit: Disconnected by By Bee-Brilliant on Flickr

This post sponsored by the Enterprise CIO Forum and HP.

Clouds and Shadows – Managing Shadow IT with the Cloud

This post sponsored by the Enterprise CIO Forum and HP.

Clouds and shadows by ericbrownCloud computing is here to stay.

Oh wait…I hate statements like that.  Sorry about that.

Statements like the above are made by people trying to defend the ‘cloud’ or whatever new or important product/service they are trying to sell today (or one they want to sell tomorrow).

That said…the cloud is an important aspect of technology that every organization should have already adopted into their technology strategy and roadmap. Ignore the option of the cloud and you may find yourself without a job soon.

Many IT professionals look at the cloud through fear-filled eyes. They see the cloud as their competition. They see the cloud as their job.  This fear is natural and understandable…cloud services are a form of ‘outsourcing’.  But…IT Pro’s shouldn’t be fearful and/or ignore the cloud…we should embrace it and plan for widespread adoption.

If the IT professional doesn’t plan for the adoption of cloud-based services, others will.   Much like the world of Shadow IT within organizations…the cloud can enable an even greater Shadow…or it can help the CIO and IT shine some light within these shadows and start to transform Shadow IT into a properly managed technology infrastructure.

Terence Ngai, an HP Blogger and employee, writes about this very topic in an article titled Cloud computing puts an end to shadow IT on the Enterprise CIO Forum.  In the article, Terence suggests that the Cloud will end Shadow IT for good. I disagree with that premise but I do think the cloud is a good start to getting a handle on Shadow IT.

Terence writes:

Cloud computing gives business execs and IT leaders a unique opportunity to work together to develop an IT strategy that really meets the needs of the business. Why? Because more than any other technology, cloud computing offers undeniable benefits that can close the gap between business and IT.  Line of business owners can quantify the business value of needed resources. And IT leaders could use that info to create a winning business case for cloud computing and demonstrate the value of IT.

I don’t disagree with that at all.

Terence implores business leaders to learn about Cloud technology and services and then help guide their IT groups towards those services.  Not a bad approach for selling more services into the enterprise 🙂

One thing that bothers me about Terence’s suggested approach is this: If there’s a CIO or IT group out there that hasn’t already developed a technology strategy that includes the cloud, the leadership of that IT group should be replaced immediately.  That doesn’t mean that their strategy should be to use the cloud…just that they’ve thought about how the cloud can be used when it is needed.

Personally, I don’t think the cloud will end Shadow IT.  The cloud is an enabler of shadow IT…and if a CIO or IT group cannot find a way to insert themselves into the discussions around Shadow IT and moving services to the cloud, they’ll find themselves without jobs in the near future.

The cloud is here to stay (ack!)…and those of us in IT need to find ways to ensure cloud based services are used in a secure and professional manner. We know people within the organization are going to the cloud for services that they can’t get from IT (or at least think IT is too slow to deliver those services in a timely manner) so let’s help them understand the benefits and the challenges of the cloud.

By developing a technology strategy that includes the cloud and cloud based services, maybe…just maybe…shadow IT can be managed. It won’t be completely banished but at least IT professionals can get a better grasp on situation.

The cloud has helped to expand Shadow IT…its time for IT groups to incorporate both into technology strategy and roadmaps.  Without embracing the ‘clouds and shadows’, the organization will continue to move faster than the IT group and continue to make IT and the CIO less relevant and less valuable.

Image Credit: Cloud & Shadows

This post sponsored by the Enterprise CIO Forum and HP.

A story of a CIO, IT and Marketing

Versus 2004 By magnacasta on flickrI heard a pretty good story recently.  It goes something like this:

A new CIO joins the company. He’s not completely new to the organization but he is new to the role and a new employee. Previously, he’d been running a couple of projects for an outsourced ‘partner’ within the organization and had ingratiated himself with the leadership team. They liked what he did and offered him the role of CIO.

The new CIO started reaching out to the organization to see how he could help. One of the first groups he spoke to was the marketing group.

A meeting was set between the CIO and the main people within marketing.  The CIO told the marketing group that he’d loved to work with them and do whatever was needed.  They mentioned the various changes and development efforts they’d been trying to get done within their content management system and were told he’d absolutely support them with as many resources as needed.

In addition to promising support and offering resources,  he cautioned against doing things outside the process.  He cautioned the marketing group on following the proper processes to get new websites and new technology implemented.  He cautioned against trying to lead the way with their own technology.

He used the line that we’ve all heard (and some of us have said)…he said “IT is hear to help, but remember – IT owns the technology, Marketing owns the content“.

The meeting ended well. The marketing group felt like they’d be able to work with this CIO and the CIO felt like he’d made real progress.

Fast forward a few weeks.

The marketing group needed to get a new website live.  They reached out to the IT group to get a resource assigned to build out the necessary containers for the new website in the content management system.

They were told that no resources were available now but that they should be able to get to the project in the next month.   A month delay in the project wasn’t necessarily that bad so they waited.

And waited.

The next month arrived and the marketing team was told that there were still no resources. And there wouldn’t be any for a few months.

So…what did the marketing team do?

The marketing group leadership went to the CIO to ask for help.  He wasn’t available to meet but promised resources asap.

But no resources were assigned.

So…the marketing group did what they had to do.  They had a deadline that they had to meet. They setup a website on their own using an externally hosted web server using open source software. They designed the site, filled it with content and turned it live within 2 weeks.

From all accounts, the new website was a hit…it did what the marketing group (and their target clients) needed it to do. And it was done without the promised help of the CIO & IT.

Fast forward a few weeks.

The CIO called another meeting with the marketing group.

This time the CIO wasn’t cordial. This time he was confrontational.  He was angry that the marketing group ‘went around’ IT. He was angry that they didn’t use the organization’s content management system for the new website.  He was angry that marketing rolled out a new project technology and his IT group wasn’t leading the charge.

The CIO talked about the need to standardize technologies. He talked about the need to follow process. He talked about the need to let IT lead technology initiatives.

The marketing team responded to the angry CIO in a like manner.  They talked about the lack of support and lack of resources. They shared the timelines and requests for IT support and the lack of that support.  They shared their frustration and feelings of no support from IT.

The CIO agreed that things could have been done better and he promised to improve things in the future. He talked about improving the processes in place to ensure marketing is supported.  He mentioned hiring additional resources too.

And the CIO reiterated the need for Marketing to let the IT group run and manage technology.  He reiterated the mantra of “IT owns the technology, Marketing owns the content.”

Fast forward a few more weeks.

The marketing group needed to make some changes to one of their major websites.  They needed some major functional changes and needed IT support.   This new project needed to be a quick turn too as the new functionality needed to be released within a month to coincide with a new marketing campaign.

They went to the IT group and followed the process outlined by the CIO but were told it would be 2 months before IT resources could be freed up for their new project.

The marketing team was furious.  They had to have this new functionality live in a month.  What did they do?

They did what they had to do….they hired an outside consultant to come in and make the necessary changes to the website.  They had to sneak these changes around the IT group’s production code change process – but that wasn’t hard to do.

They got their needed changes made and were happy.

Fast forward to the present

The CIO is still promising to help marketing…but he’s unable to deliver.  He’s unable to deliver the project resources needed by the marketing group.

The marketing group has completely stopped talking to the IT group about any new technology projects. They’ve hired their own agency to act as their development arm.

The CIO and IT group are furious at the marketing group and are trying to get the CEO to ‘force’ marketing to come back in-house and use IT for all technology initiatives.

Both teams are at an impasse. They can’t understand the other’s needs.  They can’t understand how to work together.  They are now fighting against each other for resources and ‘control’.

I’m not sure how the story will play out in the end, but I suspect the marketing team will come out ahead since the future of this organization is squarely on the shoulders of their marketing efforts.

Great story, right?

I thought it was.

Stories like this play out every day between IT and Marketing. There are many reasons for these types of events, but the main reasons can be summed up with the following points.

  1. Committing before understanding.  The CIO completely over-committed to the marketing group without really taking the time to understand their needs.
  2. Failing to plan.  Neither the CIO nor the marketing team did much real planning. The marketing group didn’t provide =enough time for the IT group to bring the marketing projects into the IT project portfolio to ensure proper planning.  The CIO failed to ensure that his promises to the IT group were considered and included in IT project plans.
  3. Failure to communicate. The CIO and Marketing leadership stopped talking.  Communication is key.
  4. I vs You / IT vs Marketing.   The CIO led the relationship off with the “I own the technology, you own the content” mentality.  This immediately puts people into the mode of picking a ‘side’…maybe its subconscious, but it happens.  No longer is there and “I” and a “you”…there’s only room for ‘we’ within today’s organizations.
  5. Confronting rather than Understanding.  Once the CIO heard about the marketing group going ‘rogue’, he confronted them.  He then promised to solve the problem (and failed).   Rather than confronting, try understanding.

Next time you find yourself working with another group, think about this story and the issues I outlined above.  Can you find ways to work better with your coworkers and colleagues?  Can you communicate better? Perhaps understand the needs of the IT group better?

What can IT and Marketing do to work better together?  What changes can IT make to better support marketing? What changes can marketing make to work better with IT?

These are questions that face every company today…what are we doing to answer them?

Image Credit: Versus 2004 By magnacasta on flickr

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.