Eric D. Brown, D.Sc.

Data Science | Entrepreneurship | ..and sometimes Photography

Tag: data (page 6 of 10)

Dark Data & The Data Disconnect

Dark DataI ran across a blog the other data that used the term “dark data” when describing an organization’s data that is collected during standard business activities that never really gets used for any purpose other than its original purpose.  Dark Data is that data the gets created, collected, stored but rarely revisited for any purpose after its initial use.

Before we get into the world of Dark Data, let me take a quick detour to set the stage for this particular topic.  I’ve written about Shadow IT many times in the past and specifically touched on the topic of ‘dark data’ in a post titled Data Disconnect. In that post, I wrote the following about the “data disconnect” problem:

You can do your job. And…you can do your job well.  But…your work is being done in a vacuum.

Your data might remain in that vacuum. But worse…whatever knowledge that data might create (or has created) will remain in that vacuum as well.

The data disconnect problem is real and is one that hasn’t been addressed much in recent years. The same can be said of Shadow IT, although many organizations I speak with tell me they have been able to get in front of Shadow IT recently.

So – back to Dark Data.  Do you think the data disconnect creates dark data?

While it may not be the only reason dark data exists, it is a main driver of the phenomenon in my experience.

Let’s take a look at a scenario taken from a real-world example.

The Scenario

You are the director of marketing for a mid-sized business.

One of your goals for this year is to revamp your web analytics engine to better understand your site visitors and help inform decisions on how well your on-site marketing is working and what types of content work best for your clients.

You reach out to your IT group to ask for help and receive very little in the way of support. The first response you get is a standard “it isn’t in the budget for the year” or “we’ll need to run that through our portfolio management system”.   You talk to the CIO who says budget and process are the drivers for this type of thing and there’s not much he can do.

What do you do?   You do the same thing every marketing person has done for years…you go out and start talking to web analytics software and consulting companies to see how they can help.

After a few months and a selection process, you decide to go with company X for their analytics platform. This platform requires very little in the way of integration with your website so it takes very little to get the few lines of code integrated into the site.  You do need to get the IT group to do add these lines of code (which takes much longer than you thought it should).

Your new analytics engine is in place and you are now able to see start asking and answering better questions about how visitors are consuming your website.

Your goal of implementing a new system has been met and your Marketing VP is ecstatic with your performance and gives you a nice bonus and some well deserved kudos.

Within a few months, other folks within the organization start you questions about the website. You are able to quickly and easily tell them how many visitors you’ve had, where they come from, what they do while there and various other types of questions. You are using your analytics data for its intended purpose.

Then one day you hear of a new project. This project is looking at all areas of customer ‘touch’ and analyzing how well those touch points are working and whether these touch points a can be improved upon.  The CIO asks you how to interface with your analytics data to begin to use it alongside other analytical data to begin to study the organization’s touch points with clients.

You ask your analytics provider how you can start to integrate their data with your local data. The answer “we don’t provide that capability”.

You’ve just realized you are trapped in the data disconnect. You have data that you can’t use anywhere else in the organization. You have data that is useless outside of your web analytics engine.   You have dark data.

Moral of the Story

This is a real story from a real organization.

The outcome of this particular scenario is the marketing department had to backtrack and start using another analytics engine that integrated with the other systems used throughout the organization. This backtrack left about 6 months of web analytics sitting out in the cloud in a provider’s system without any way of analyzing that data outside their tool.

This sort of thing happens every day in business. People go out and purchase their own applications and systems without IT involvement because the cloud makes it easy to do so. This is Shadow IT.

There are approaches to alleviate Shadow IT, but you can never fully stop it today. It is much too easy to plunk down a credit card number and order services via the cloud today.

The IT group can help to alleviate Shadow IT by being a bit more proactive as well as a bit more interactive when dealing with the organization. It’s not longer good enough to say “no” or “the process is…” or “our budget doesn’t allow…”.  The organization is looking for solutions and the IT group needs to start being the group that says ‘yes’ – or at least ‘let’s figure out how to make this work’.

Shadow IT can lead to a data disconnect which can lead to one form of dark data where data lives in your applications but cannot (or at least is not) used for any other analytical purpose.

How is your organizations fighting the data disconnect and dark data?

Image Credit: Data Packets on Flickr

Links for Sept 22 2013

  • The big data Wild West: The good, the bad and the ugly — Tech News and Analysis

    Quote: Ron Bodkin of Think Big Analytics discusses the best and worst practices for adopting big data technologies and actually getting results. Companies must beware of dangerous decisions, charlatans and disastrous missteps.

  • Good news and bad news for CIOs – The CIO Leader

    Quote: So are we seeing the start of a trend in which boards are turning to external consultants instead of their own CIOs? And is this just to validate the CIO’s views or are consultants being used to fill gaps in the CIO’s experience, skills, credibility or capability? Either way it’s a potential worry for CIOs and could be further evidence that CIOs and IT functions are not evolving quickly enough to meet the needs of the business in the digital age.

  • I Don’t Want You to Humanize Your Brand

    Quote: When I talk about humanizing an organization, I mean changing how we lead and manage and operate in ways that tap into the power of being human. Social media is great for that, because it allows us to be authentic, which is very deeply human. It allows us to build relationships, which is very deeply human. When we tap into that human element, we find a whole lot of power. Can this affect your brand? Sure. When I interact with the humans in your company who are supported in being authentic, I’m getting a different feeling. Those people tweeting for you are actually interacting with me. Maybe being funny. Listening to me. Telling the truth. As a human, I really like that. That glow is going to rub off on your brand–that abstraction I hold in my head about what your company/product is and what it means to me. But the abstraction didn’t get more human.

  • Gartner On Big Data: Everyone’s Doing It, No One Knows Why – ReadWrite

    Quote: Big Data is all about asking the right questions, which requires business context, and then iterating on your project as you learn which data sources are valuable, and which questions yield real insights. You don’t have to know the end from the beginning, but you should have a clearer view of what you hope to achieve with Big Data than the Gartner seems to indicate most have.

  • How your company can start using data: Think small and internal first — Tech News and Analysis

    Quote: Companies looking to integrate more data into their business decisions should start small and see what they have already, data scientists from PayPal, MailChimp and Import•io said at Structure:Europe 2013.

  • Data-Driven Marketing Step Two: Tear Down The Silos – Forbes

    Quote: A recent study showed that most marketers believe silos –both internal and external to marketing –prevent them from effectively executing campaigns. Meanwhile, Gartner’s prediction that CMOs will soon be spending more on technology than their counterpart CIOs has IT worried that marketing is going to start driving technology initiatives in a vacuum. How can CMOs build the collaborative relationships needed to drive revenue growth in today’s data-driven marketplace? As we continue to explore my five-step plan around big data marketing, let’s discuss Step Two: Tear Down the Silos. Now more than ever, it’s imperative for marketers to cooperation and communication throughout the enterprise.

  • Hey, CIOs: In a BYOD world, your new job is service provider — Tech News and Analysis

    Quote: As employees bring their own devices to work and departments make their own technology decisions, the role of the CIO has changed, forcing them to give up control. But with loss of some control comes opportunity.

  • Innovate on Purpose: Innovation that’s too big to fail

    Quote: The only way ideas become too big to fail and remain that way is when senior executives all agree to embrace the idea and support it not just at conception, but through development and launch. It’s always been my suspicion that this was behind what Jobs did at Apple, when he drastically cut number and range of products at Apple. When you have very few products or capabilities, they become too big and important to fail. Everyone has a stake in the success of the limited number of big projects, and the decisions of executives and the actions of the culture become much more amenable.

  • The importance of the Voice of the Customer on projects and beyond | Adrian Reed’s blog

    Quote: When considering changes, great questions to ask can include “Who is the customer here?”, What would our customers say about what we’re about to do?” and “Does what we’re about to do mean we’re appealing to new (or different) customers?” These are simple questions that sometimes have extremely far reaching implications, and it is well worth asking them often during change projects. In mid-size and large organisations, it’s easy for the real customer to get lost in the paperwork. In a plethora of product and system specifications, nobody asks “who is this product designed for?” Smart organisations avoid this.

  • The Rise Of The Chief Data Officer Will Boost The Demand For Semantic Tech –

    Quote: Those companies that haven’t yet got their own CDOs onboard, or that seek some aid in running data-driven decision-making experiments before scaling them within their organizations, have another source of help they can turn to. This week, decision sciences and analytics firm Mu Sigma opened an Analytics Center in Austin, Tx. with up to 300 data scientists. It already has a similar center in Bangalore, India with some 2500 data scientists, who will work closely with the Austin staff. The company includes in its decision support stack technologies applied math algorithms in machine learning, natural language processing, and artificial intelligence.

Links for Sept 8 2013

  • Anticipate the Corporate Culture of “No!” – Sales Aerobics for Engineers ® Blog

    Quote: Do you work in a Culture of No? Some of you will be defeated by the situation, shoulders slumped, as you return to your cubicle. The design budget wasn’t approved, again. The sale didn’t close, again. Whatever the situation, the corporate culture of “No!” stops you in your tracks. Having a corporate culture of No wastes time and cuts into your ROI, as well.

  • Data is a Game Changer – OCDQ Blog – Obsessive-Compulsive Data Quality by Jim Harris

    Quote: Properly supported by enabling technologies, businesses of all sizes, and across all industries, can capture and analyze data to uncover hidden patterns and trends that can help them achieve better service, more winners, and fewer unforced errors.

  • Business Analytics Go Great with Your Cleaner, Greener City

    Quote: So take it from the people keeping San Francisco fresh and green: you don’t need to take up weight lifting to get out from under all that data. You just need the right tools to manage it and put it to good use. are exactly what you and your midsize business need.

  • Cloud kudos only grow, as does recognition of its true value | Cloud Computing – InfoWorld

    Quote: The strategic use of data is a side benefit of cloud computing, but it’s a value that goes way beyond any operational cost benefits. Most businesses have yet to understand that fact, but it will become readily apparent as we move into data analytics services that can scale in the cloud.

  • How Social Analytics Can Improve Enterprise IT Efficiency –

    Quote: Using social analytics on previously unused data sets — reading email headers, calendar data, instant messaging logs — can deliver a lot of business value, especially for large, distributed enterprises that aren’t on a first-name basis with most of their employees, he says.

  • Kneale Mann | Leadership: Too Busy to Get it Done

    Quote: Business Week recently published a study stating that only 38% of people can make effective decisions today due to the overwhelming amount of data which is increasing at a rate of 60% a year. It may be time for us to have less meetings, put our phones away, and have more meaningful interaction.

  • The Dreaded “W” of Workflow – Effective CRM

    Quote: A closing note about automating work; don’t speed things up that your customer doesn’t expect to be speeded up. For instance, if you are Jesse James, building killer custom motorcycles, your customer is willing to wait; speeding this process up would inevitably lead to defects, and waste, that they are not willing to tolerate (i.e. Correction). However, that same customer might value regular updates regarding the status of the job (including production stage pictures so they can see the progress). This is a something that automation can certainly help with.

  • QZipcar puts a face on hybrid IT – TotalCIO

    Quote: Figuring out a way to prioritize IT-related tasks can help take some of the pressure off IT departments that need to focus on the demand for agility, Harrington said. “That solution set means there are some things engineering does really well and some things where the needs are better served by SaaS-based tools,” he said. “It’s a happy hybrid between the two that makes it really work.”

Links for August 18 2013

  • Unicorn, Shmunicorn: Be a Pegasus | UX Magazine

    Quote: But who better to take on the product challenges of the future than cream-of-the-crop UX professionals? No one is closer to the intersection of people’s goals and a company’s products than the designers sweating over every detail of the user experience, day in and day out. Rather than re-inventing yourself as a part-time, mediocre coder, consider aiming your trajectory squarely at these product leadership positions.

  • Do you really want your IT department to define your application stack? : CloudAve

    Quote: So before you sign your purchase order for a PaaS, you should wonder if you are not selling your flexibility without any need for it…

  • Michael Fauscette: Innovation Alone Isn’t Enough

    Quote: The path from high flying tech leader to "for sale sign" is a sad one and one that is pretty complex to explain. As with most decline stories there are internal and external factors to blame

  • Go Where the Data Take You, Even If It’s to New Jersey…

    Quote: Keri’s piece makes clear that Trout’s ridiculous abilities were quite evident when he was a high schooler in New Jersey. Yet even in the post-Moneyball era the people who spot baseball talent for a living missed him, and they essentially missed him because of their accumulated expertise, experience, and intuition — not despite these things. When are we going to learn to get ourselves out of the way and listen to what the evidence is telling us?

  • The vanishing cost of guessing – O’Reilly Radar

    Quote: Big data isn’t a magical tool for predicting the future. It’s not a way to peer into someone’s soul or decide what’s going to happen, even though it’s often frighteningly good at guessing. Just because the cost of guessing is dropping quickly to zero doesn’t mean we should treat a guess as the truth. As we become an increasingly data-driven society, it’s critical that we remember we can no more predict tomorrow with today’s data than we can prevent drowning by banning ice cream.

  • What is a Digital Disruptor? — CIO Dashboard

    Quote: In the age of digital disruption, companies need to invest in exploration. They should imagine the possibilities of potential disruptive business designs coming down the pike and they should explore the perpetual parade of emerging technologies. Some companies will have two options: they can disrupt or get disrupted.

  • The Biggest Missed Opportunity in Social Media

    Quote: More and more organizations are spending lots of time and resources on social media every day. Yet they seem oblivious to the key element that can impact their success the most: their employee base! The organizations that figure out how to use this important resource will surely gain a competitive advantage. Is your organization tapping into its employees to boost its social efforts? If so, how? If not, why?

Links for August 4 2013

  • CIOs need to learn to become enablers rather than gatekeepers — Tech News and Analysis

    Quote: CIO bashing has become popular sport, but in reality companies are now finding an edge by focusing less on cost-cutting and more on how IT can help enable better functionality and innovation.

  • Jim’s Notebook: Measuring What Matters

    Quote: Ultimately, the work we do is about improving the customer experience, taking it on faith that it will result in improving the financial performance of our firms – and generally, this turns out to be true. Having a vague sense that you’re doing the right thing is likely better than having no sense at all – and better still in feeling a sense of accomplishment for doing something terrible. It cannot be taken for granted that a general metric is an indicator of success at anything but making a number better.

  • Content Strategists Must Become Engineers of Content-Driven Customer Experiences | The Content Wrangler

    Quote: I believe that all content strategists should be knowledgeable about the entire content ecosystem, the content lifecycle, content tools, technologies, standards, and methodologies — even those that fall outside of their area of specialization, or are tangentially-related to the project on which they are working. Content strategy problems are seldom problems with content alone.

  • DIRECTV CIO On Getting IT To Recover From Fear Of Failure – Forbes

    Quote: Beyond improved trust and motivation, the program has changed how IT solutions are being deployed. “We were always very focused on not putting anything into production until we really had it rock solid,” says Gerjets. But this year the IT team has started to roll out new tools and applications as beta solutions, letting users try them out and give feedback while they are being developed. “This is something, culturally,” says Gerjets, “that we have never been able to do before.”

  • How to Know When You’re Ready for Beta | On Product Management

    Quote: Beta testing is a key part of product development, but it can be difficult to get the timing right. Product teams are often pressured to get new products through beta and out the door as quickly as possible, but if a product goes into beta too early it can defeat the purpose of running a beta test altogether.

  • Why the world’s governments are interested in creating hubs for open data — Tech News and Analysis

    Quote: The Open Data Institute, a non-profit in London that’s working to create demand for open data through research, storytelling, consulting and startup incubation, is starting to get some velocity.

  • The Power of Algorithmic Marketing –

    Quote: There are already some signs that suggest the algorithmic marketing transformation is underway, especially in industries that rank high in terms of big data value potential, such as financial or information services. As algorithmic marketing takes hold, it is likely that big winners – and potentially even bigger losers – will emerge in the near future. In North America, grew 30 to 40%, quarter after quarter, throughout the recession, while other major retailers shrank or went out of business. How Amazon achieved these results may be surprising

  • Knowing Where Our Customers Live | Partners in EXCELLENCE Blog — Making A Difference

    Quote: We must have empathy. We must recognize what our customers are going through. We must see things through their eyes, understand their emotions.

  • Five Steps to Enterprise Gamification | UX Magazine

    Quote: This five-step process to gamification begins with a good understanding of the player and the mission. This is followed by psychological research on motivation. Based on this solid foundation, we advocate a thoughtful application of game mechanics and rules to create a core engagement loop. We recommend you start small, monitor closely for best results. The enterprise context including legal and ethical considerations cannot be ignored. And remember: make it fun!

  • The year after the CIO’s honeymoon with Big Data, regret.

    Quote: Here’s the thing: of course Big Data should be exploited. But like a balance investment portfolio, the factors that drive customer excellence need to be tended to first: no customers means no need for Big Data – except to find your next job.

Links for July 28 2013

  • How Innovations Spread

    Quote: We believe that we can just tell people the facts and ideas will spread. That’s not really how it works. Neil deGrasse Tyson explains that persuading others is about more than just putting the facts out there and letting the chips fall as they may.

  • Nobel-winning economist: end software patents and cut patent protection to 10 years — Tech News and Analysis

    Quote: As the problem of patent trolling grows, calls for reform are coming from all quarters — including from a famous economist who proposes major changes to the system.

  • Sometimes You Need to Make a Deliberate Mistake to Make Progress

    Quote: Many people, after accruing a reasonable amount of experience, often wait out perfection. We think we’re not ready yet, or that we need more time before our skill level or work proficiency can take us somewhere. At those times, when we fail to act because we know we’ll fail, mistakes might just provide the solution. Next time you feel yourself stuck, consider screwing up on purpose. You might not get what you really want, but you’ll probably learn something that’ll bring you a little closer to your goal.

  • Flexibility: get focused | Hobbs on Tech

    Quote: In other words, flexibility is more in grayscale than black and white, and you should attempt to shoot for the middle ground allowing streamlined flexibility.

  • Why we’re doing things that don’t scale by Jason Fried of 37signals

    Quote: So remember, efficiency shouldn’t always be the goal. Especially early on. If you want to learn, you have to struggle. Automating away all the struggle may teach you how to be efficient, but it won’t teach you much about your customers.

  • Returning ‘Intimacy’ to the Retail Customer Experience – Highly Competitive – software industry insights

    Quote: While data and analytics can tell retailers many things, they’re not the only factors for shaping more intimate experiences. Personalization takes a lot of work to truly pique the interest of shoppers / buyers with the right engagement and information. It also requires restraint and good judgment to avoid the “creepiness factor”. The best approach may be less frequent, high quality interactions to avoid the sense of being stalked. Retailers that show they respect the security and privacy of their customers demonstrate the essence of intimate shopping experiences where the happy customer comes first. Paying back value to customers in exchange for using their data and showing that you can be trusted earns more loyalty and business.

  • Long term engagement funnels | Hobbs on Tech

    Quote: Really, you will almost certainly have multiple engagement funnels: one per primary audience. You may want different audiences to take different actions, all anchored by your overall website vision. A professional association may have separate interactions for students, professionals, and potential customers of the professionals. A product company may target consumers and businesses differently. Each particular organization is unique, and there is no plug and play audience list (and associated long term engagement funnels per industry). For example, some think tanks target the general public and others do not, which means that the goals and engagement funnels are very different.

  • “Big Data” Is Not “Big Data” Unless It Gives You Actionable Insight

    Quote: Although big data is a hot topic in marketing circles these days, it is important for marketers to focus on the actionable insight that a “big data” platform can bring rather than focusing on ticking a box on the hot trend of the week. Today, modern Search “big data” platforms can leverage the large quantities of data available across disparate sources and produce insight for those with an actionable mindset.

  • Looking for complexity

    Quote: Starting to see, understand, appreciate, and adjust for the complexity of the people network will help put in place the right technical network and systems and processes. We need to look at each side of the equation and balance them out to achieve a whole solution that can be successful.

  • From Conversation to Content-Delivery: Professors Predict a Change in Twitter

    Quote: “Twitter will become less of a communications vehicle and more of a content-delivery vehicle, much like TV. Peer-to-peer contact is likely to evolve to the next great thing, but with 500 million followers, Twitter isn’t just going to disappear. It’s just going to become a new way to follow celebrities, corporations, and the like,”

« Older posts Newer posts »

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.