Mining for knowledge in a social word

A segment of a social network
Image via Wikipedia

Last week I published a post titled Mining for Knowledge where I discussed some of the research that I’ve been doing in my doctorate program.

One of the favorite lines from the article, and one that resonated with a few others as well. The line was:

…converting tacit (i.e., internal) knowledge to explicit (i.e., external) knowledge is one of the most difficult things to do.

I’ve been thinking about this (and reading A LOT of articles, papers and books on the subject) and have come to the conclusion that trying to force someone to convert tacit knowledge to explicit knowledge is a wasted effort.


Can I truly convert 100% of my knowledge into the written form?  Will the context of my knowledge be converted?  Perhaps a good portion of my knowledge can be converted, but can my experiences, thoughts and believes that shaped that knowledge be converted?  Can I ‘write down’ the knowledge that I have and truly make it meaningful to others?  I don’t think (feel free to disagree here).

Does that mean that an organization should stop trying to gather an individual’s internal knowledge to add to overall organizational knowledge-base?  Nope…. definitely not.

Rather than forcing a conversion from tacit to explicit (which is darn near impossible), are there ways to manage the internal knowledge of people?  Managing that knowledge is a much easier process that converting that knowledge.

Knowledge is best internalized when wrapped in context

Nonaka and Takeuchi, the godfathers of Knowledge Management, argue in their book The Knowledge-Creating Company: How Japanese Companies Create the Dynamics of Innovation (affiliate link) that tacit knowledge can be converted into explicit knowledge only through externalization and describes this process as being one of dialogue, discussion and reflection.

Basically, they’re saying that in order to share internal knowledge, you’ve got to start a dialogue with others.  That’s why activities like storytelling, mentoring and other forms of social interaction can play a huge role in knowledge managment…they help to start and maintain dialogue and discussion on various topics.  These activities help to provide context around knowledge, which helps a person internalize that knowledge and make it their own.

In my previous article I talked about ‘mining for knowledge’. I talked about using web 2.0 platforms to capture knowledge and to share knowledge. All good stuff (and still interesting to me) but I’m looking at other methods to make these platforms more social.  Make dialog and discussion a more active portion of these tools.

If we can find ways to create dialogue and discussion within the enterprise, knowledge sharing would happen much more naturally.    This is why I like the idea of Enterprise 2.0.  While some people hate E2.0, I think there’s some real value there. Of course, E2.0 won’t solve world hunger and probably will never truly win over its detractors, there are many aspects to the idea that make sense.

What would it mean for an organization’s knowledge managements capabilities if a system could be implemented that found indexed the many disparate repositories of structured and unstructured data sources found throughout the enterprise and then provided that information in a socially aware platform that could wrap context around the indexed knowledge as well as provide a mechanism for dialogue, discussion and reflection?   You’d have a platform that could capture and share explicit and tacit knowledge.

Anyone know of any companies with products in this space?  I know SocialText is out there but I don’t think they have a platform as robust as the one above. SharePoint also has some aspects to this but not everything.

Mining for Knowledge

Mining for KnowledgeIn my doctoral research, I’ve been researching ways to improve knowledge capture and sharing methods, specifically within project teams but the ideas can be dissemenated around the organization.

One of the biggest issues I’ve found while working as a consultant is the amount of knowledge that I walk away with after a project is complete.  Sure, I try to share this knowledge in every way possible but converting tacit (i.e., internal) knowledge to explicit (i.e., external) knowledge is one of the most difficult things to do.

Let’s assume though, that some portion of the knowledge that I hold in my head is converted into some form of writing at various periods throughout a consulting project.  Where does that explicit knowledge live?  In an email?  In some document stored on a server?  In a knowledge repository somewhere?

In the past, this problem has been attacked using centralized knowledge repository platforms.  These systems require users to log in and ‘enter’ their knowledge into the system.  Many of these platforms have been well built and some have been successfully used in organizations, but the success stories are far outweighed by the stories of KM repositories sitting idle and unused.

So…how can we get that tidbit of knowledge from my brain into some form of knowledge repository without me logging in and ‘entering’ it into the system?

Web 2.0 as knowledge repository

The use of Web 2.0 tools (blogs, IM, wikis, etc) has become ubiquitous..  If incorporated into a project environment, these tools might allow an easy and efficient method for capturing and sharing knowledge throughout project teams and project organizations.

The key to retrieving knowledge from tools is to make the user experience as seamless as possible. For example, an employee creates a blog on an organization’s intranet and then uses this blog to write different topics, some that pertain to her project and some that don’t.

Perhaps this employee is participating in two projects within the organization and she writes about topics that might be of interest to a portion of the organization and project team members.  While she writes about interesting topics and at times, writes about her experiences on the projects that she’s worked on, perhaps her blog posts aren’t widely read.  This employee has attempted to convert a portion of her tacit knowledge to explicit knowledge but few people on the project team or within the organization find this knowledge because its tucked away in the intranet site (which is rarely used anyway).

In the above scenario, knowledge was converted from tacit to explicit but few people are able to absorb this knowledge and make it their own (i.e., perform the conversion from explicit to tacit knowledge).  What would happen if this knowledge were indexed, searched and shared with the rest of the project team in something akin to a project knowledge ‘journal’?

Since Web 2.0 platforms are ubiqutious, why can’t we use these tools as our knowledge repository?  Employees and project team members are already using them…so can we find a way to ‘mine’ these platforms for knowledge?

Could a system be built that ‘mines’ these web 2.0 platforms along with other unstructured data (documents, email, etc) to ‘build’ a knowledge repository available to the entire organization?

Mining for Knowledge

I’m currently looking at ways to use text mining methods and techniques to mine for knowledge. Text mining looks to be a good approach to solving this problem because it allows for knowledge to be gathered without additional work by project team members.

There are other approaches that could be used for gathering knowledge from project team members, but all require additional work to input information.  For example, a project team using a manual approach could ask team members to regularly update their blog and to ‘tag’ their posts with a special project tag or keyword so that a non-intelligent aggregation system (RSS, etc) could simply pull these tagged posts into a central repository.  While this is a good approach, it relies on the end-user to tag their content correctly, accurately and in a timely manner.  Tagging, and other categorization and taxonomic approaches, require the user to do something to allow their knowledge contribution to be categorized, indexed and found by aggregation systems and other users.

Using text-mining methods against pre-existing tools and platforms takes away the human fallibility issues found in current knowledge management repository platforms or by requiring a user to ‘tag’ a piece of content correctly as described above.

Using text-mining and other data mining approaches, I’m looking at ways to build semi-autonomous systems to index and organize both structured data and unstructured data pulled from blogs, email, IM, social networks, documents, spreadsheets and any other location / data sources. This system could aggregate knowledge found via text mining and social network analysis and build a project knowledge ‘repository’ that will contain all knowledge for any specific project. This repository will be searchable and will contain both manually curated content (e.g., content uploaded by project team members) and automatically curated / generated content based on text-mining and indexing techniques.

There are some major privacy issues here of course. How can you mine a users email and find the relevant knowledge without truly invading their privacy?  Not sure you can but I’m looking at it.

Trust & Mined Knowledge

One key element of this new inter-connected world that we live in is trust.   How can I trust that the information I read on a web page is worthwhile, honest and accurate?   If I want to know something about organizational behavior do I read go read a Wikipedia article on the subject or do I go look through the Harvard Business School’s Organizational Behavior faculty pages and find publications written by the faculty there?

Which of these two sources of knowledge would you trust to be more accurate?

The same can be said of knowledge captured and shared within an organization. How do you know that the white paper on your new API is true?  Is it because it was released? Is it because of the author(s) of the paper?   What if you had a knowledge-base generated by an autonomous agent using text-mining techniques…how would you know to trust the information contained in it?  Who wrote the content?  Were did it come from?

This is where trust comes into play. If you could ‘see’ the qualifications of the author or authors of the knowledge base articles would you trust the content more?  If I knew that the worlds leading authority on organizational behavior wrote the Wikipedia article on the subject, I’d tend to trust that article more.

This is another aspect of my research…building trust into the mined knowledge using social network analysis (SNA) methods & techniques.  Using SNA techniques, can the background, profiles, connections and knowledge of the users within an organization be automatically (or semi-automatically) generated to provide some form for initial trust metric to show that mined knowledge can be trusted?

I don’t know if it can…but I’m looking into it 🙂

Next Steps?

So what are the next steps for me and this research?

I’m working on a research paper now that I hope will outline the research in more detail.

Lots of questions still exist and there is quite a bit of research left to do.  I do believe I’m headed in the right direction as evidenced by an HBR video & Blog tilted How Knowledge Management Is Moving Away From the Repository as Goal which discusses a similar topic.

Look for more on this topic from me in the coming months.

Related articles by Zemanta

Improving Knowledge in Projects

This post is an excerpt of a recent paper titled “Improving Knowledge in Projects.” If you’d like to read the full paper, click here for the PDF version of Improving Knowledge in Projects.

In the project management world, a considerable amount of research exists to describe the reasons behind the success or failure of projects in the information technology space. Most of this research focuses on failures being caused lack of executive sponsorship, lack of project management methods, lack of change management processes, project scope size and/or project duration (Reich, 2007).

While these causes of failure are quite common, a large and overlooked stumbling block that causes failure in IT projects is the lack of proper knowledge management methodologies throughout the project management lifecycle.

This paper provides a brief review of the literature within the knowledge management space, and more specifically, how to manage knowledge transfer, sharing and application in projects. The main question that is initially asked (but not really answered) in this paper is that following: Can a framework containing “best practices” be developed that can be use to improve knowledge transfer and sharing in project based groups and organizations?

Brief Literature Review
Although most project management methodologies claim to be interested in knowledge management, none offer any real guidelines or practices for gathering and maintaining knowledge throughout the project lifecycle.

Disterer (2002) argues that traditional project management methods are overly concerned with efficiency and effectiveness of project team members, which in turn makes the act of capturing knowledge a lower priority during a project (Disterer, 2002). This is compounded by the fact that the knowledge needs of future projects doesn’t lie within the context of the current project requirements therefore project managers and leaders do not focus on these efforts (Disterer, 2002).

Leseure & Brookes (2004) take this claim a step further by stating that knowledge transfer is one of the largest issues in projects today when they write “Knowledge is generated within one project and then lost. Failure to transfer this knowledge. leads to wasted activity and impaired project performance” (Leseure & Brookes, 2004, p. 103). Leseure & Brookes’ designed a research project that would attempt to benchmark knowledge management practices within projects to help provide a broader and more qualitative evidence of knowledge management methods in projects. The results of this research pointed to two main areas that could improve knowledge management in projects: collecting knowledge in projects; and managing tacit knowledge (Leseure & Brookes, 2004, p. 106). By focusing on these two areas, organizations can help to improve project knowledge management.

Kasvi, Vartiainen, & Hailikari (2003) performed research on how knowledge is managed in projects and what knowledge management capabilities are required for proper knowledge management in projects (Kasvi, Vartiainen, & Hailikari, 2003). The researchers used interviews to gather data on knowledge management capabilities and practices in various organizations. The results provided interesting feedback on organizational knowledge practices in projects and led the authors to observer that “knowledge management practices were weak and unsystematic” (Kasvi et al., 2003, p. 578) and that paper documents and interactions with colleagues were the most important sources of knowledge.

Research by Karlsen & Gottschalk (2004) addresses the factors that affect knowledge transfer in projects (Karlsen & Gottschalk, 2004). The authors used surveys to gather information from project managers and organizations on knowledge transfer in projects. The survey results showed that organizational culture plays a key role in knowledge transfer within projects and should be the main area that organizations focus when looking at knowledge transfer methodologies capabilities.

Research by Slaughter & Kirsch (2006) extends the concept of the importance of organizational design and culture on knowledge management with the introduction of Knowledge Transfer Portfolios. This research, which was conducted as a field study in the area of Software process improvements, provides some very interesting ideas on organizational design and knowledge transfer and outlines the following three items as being key for knowledge transfer within organizations: Proximity, Frequency of Interactions and Relationships (Slaughter & Kirsch, 2006).

The Framework

Blaize Horner Reich, an Associate Professor at Simon Fraser University and Visiting Associate at Templeton College, Oxford University has developed a framework for knowledge management within IT Projects which seems promising. This framework, published in the Project Management Institutes’s Project Management Journal in an article titled “Managing Knowledge and Learning in IT Projects: A Conceptual Framework and Guidelines for Practice“, proposes a three level model that addresses the following topics:

  • Describes what “knowledge management in IT projects” is
  • Provides a typology of critical IT project knowledge
  • Identifies the top ten knowledge-based risks found in IT projects. In addition, key principles for knowledge management in IT projects are provided for use in helping build strong knowledge management capabilities within IT projects.

Reich’s framework is a good place to start as it provides a model built upon sound principles and research in the IT project space.

The first ‘level’ in Reich’s framework defines IT Project knowledge management as:

Knowledge management in the context of a project is the application of principles and processes designed to make relevant knowledge available to the project team. Effective knowledge management facilitates the creation and integration of knowledge losses and fills knowledge gaps throughout the duration of the project (Reich, 2007, p. 8).

The second ‘level’ in Reich’s framework consists of a typology of IT Project knowledge. This typology contains four distinct types of knowledge: process, domain, institutional and cultural. A brief definition of these types of knowledge follows:

  • Process Knowledge: knowledge that project team members have regarding the project (tasks, methodologies, timelines, structure, etc) (Chan & Rosemann, 2001; Meehan & Richardson, 2002; Reich, 2007).
  • Domain Knowledge: knowledge that a project team or member has about the industry, technology, processes, current situation, business and products (Chan & Rosemann, 2001; Reich, 2007).
  • Institutional Knowledge: knowledge that a project team or member has about the organization (Reich, 2007).
  • Cultural Knowledge: knowledge about the organizational culture as well as cultural backgrounds of the project team members (Reich, 2007).

The third ‘level’ of Reich’s framework consists of knowledge-based risks in IT projects. The author has listed ten risks that can affect knowledge in IT projects. These risks are:

  • Lessons aren’t learned
  • Flawed team selection
  • Changes in the project leadership team
  • Lack of knowledge of project team roles
  • Poor knowledge integration
  • Poor knowledge transfer within projects
  • Changes in project team
  • Determining “who knows what” (knowledge maps)
  • Project team changes between phases
  • Failure to Learn

Building Upon Reich’s Framework

Using the knowledge contained within Reich’s framework, and knowledge generated during a literature review, the following seven topics must be considered as key pieces of this framework in order for it to address project knowledge management:

  • Continuous Learning / Lessons Learned – Ensures that all “lessons learned” are documented and shared throughout the organization and applied in future projects.
  • Organizational Design – Develops proper project teams to ensure that the necessary knowledge transfer mechanisms can be implemented per Slaughter & Kirsch’s (2006) research
  • Organizational Culture – Works toward building a culture that is pre-disposed to sharing knowledge.
  • Human Capital Management practices – Covers the human capital management aspects to ensure proper motivation for project teams.
  • Project Selection – Covers proper project selection as well as team member selection.
  • Risk Management – Covers the aspects of project risk management as well as knowledge-based risk management as described in Reich (2007).
  • Knowledge Typology Management – Covers the four types of knowledge outlined in Reich (2007) including Domain, Process, Institutional and Cultural Knowledge.

There is a considerable amount of research and thought that needs to be done to further develop this framework but an initial model can be found in Figure 1. This model provides a high-level overview of the areas that must be considered when developing project knowledge management practices.

Click for Figure 1: Project Knowledge Management Model

The model is separated into three “sections” plus feedback loops. The three sections are: Project Leadership, Project Teams and Projects. Each section contains the areas of focus for that particular project entity and outlines the areas of responsibility. A description of each section as well as the feedback loops follows.

  • Project Leadership – the project leadership section covers the higher level “strategic” aspects of project knowledge management and contains Organizational Design, Culture, Human Capital Management Practices, Project Selection and Management, Risk Management and Knowledge Management.
  • Project Teams – the project teams section covers the individualistic aspects of project knowledge management and contains the knowledge types (Domain, Process, Institutional and Cultural) as well as project management methods and processes.
  • Projects – the projects section contains the more “tactical” aspects of project knowledge management and includes Knowledge Transfer, Lessons Learned and Continuous Interaction.
  • Feedback Loops – In addition to the three sections, the model contains feedback loops that are used to ensure that continuous feedback is provided from each layer of the project knowledge management model. For example, project teams will continuously communicate with each other throughout projects regardless of which projects they are working on. Project Leadership will always be “kept in the loop” on all projects.

This framework provides an easy to recognize area of responsibility for the seven key topics that must be considered to ensure proper knowledge management in projects. As an example, let’s consider Slaughter & Kirsch’s (2006) research on Knowledge Transfer Portfolios. One of the key outputs of that research was to show that organizational design plays a key role in knowledge transfer. Using the model shown in figure 1, it is easy to see that organizational design lies solely on the shoulders of the organization’s leadership to consider.

There is still a considerable amount of research that needs to be completed in order to create supporting data to support this framework. Although not complete, the model does show areas in which organizations can begin to consider making changes to improve project knowledge management.

The extent of knowledge management in most project management methodologies begins and ends with the “lessons learned” document that is created after the completion of a project. This document is a good exercise, but doesn’t do much to manage knowledge during the project or ensure that knowledge is transferred between project members because project members must know to read the document to receive any value from it.

It is widely reported that project failure rates are still very high (Ahn, Joo, Cho, & Park, 2005; Owen, Burstein, & Mitchell, 2004; Pawlowski & Robey, 2004; Reich, 2007; Scarbrough, Bresnen, Edelman, Laurent, & et al., 2004). Industry research shows fifty to sixty percent of all projects are considered failures (IT-Cortex, 2007). While most research blames these failures on poor project management and/or lack of executive sponsorship (Reich, 2007), the fact that there is very little knowledge transfer and sharing between project teams has to play a key role in allowing these failures to occur.

By building a framework that can be used to help improve knowledge transfer within project teams, it is hoped that the failure rate due to knowledge-based issues will drop significantly. This framework, which still is the early development stages, should help organizations understand the underlying requirements for project knowledge management, provide best practices for knowledge management in projects and provide a way to build a corporate culture that is focused on sharing knowledge.


  • Ahn, H. J., Joo, L. H., Cho, K., & Park, S. J. (2005). Utilizing knowledge context in virtual collaborative work. Decision Support Systems, 39(4), 563.
  • Chan, R., & Rosemann, M. (2001). Managing knowledge in enterprise systems. Proceedings of the Americas Conference of Information Systems.
  • Disterer, G. (2002). Management of project knowledge and experiences. Journal of Knowledge Management, 6(5), 512.
  • IT-Cortex. (2007). Statistics over IT Failure Rate. Retrieved 3 May 2008, from
  • Karlsen, J. T., & Gottschalk, P. (2004). Factors Affecting Knowledge Transfer in IT Projects. Engineering Management Journal, 16(1), 3.
  • Kasvi, J. J. J., Vartiainen, M., & Hailikari, M. (2003). Managing knowledge and knowledge competences in projects and project organisations. International Journal of Project Management, 21(8), 571.
  • Leseure, M. J., & Brookes, N. J. (2004). Knowledge management benchmarks for project management. Journal of Knowledge Management, 8(1), 103.
  • Lytras, M. D., & Pouloudi, A. (2003). Project management as a knowledge management primer: The learning infrastructure in knowledge-intensive organizations: Projects as knowledge transformations and beyond. The Learning Organization, 10(4/5), 237.
  • Meehan, B., & Richardson, I. (2002). Identification of Software Process Knowledge Management. Software Process: Improvement and Practice, 7(2), 47-55.
  • Mitchell, V. L. (2006). Knowledge Integration and Information Technology Project Performance. MIS Quarterly, 30(4), 919.
  • Owen, J., Burstein, F., & Mitchell, S. (2004). Knowledge Reuse and Transfer in a Project Management Environment. Journal of Information Technology Cases and Applications, 6(4), 21.
  • Pawlowski, S. D., & Robey, D. (2004). Bridging User Organizations: Knowledge Brokering and the Work of Information Technology Professionals1. MIS Quarterly, 28(4), 645.
  • Reich, B. H. (2007). Managing Knowledge and Learning in It Projects: A Conceptual Framework and Guidelines for Practice. Project Management Journal, 38(2), 5.
  • Rice, M. P., Oconnor, G. C., & Pierantozzi, R. (2008). Implementing a Learning Plan to Counter Project Uncertainty. MIT Sloan Management Review, 49(2), 54.
  • Santhanam, R., Seligman, L., & Kang, D. (2007). Postimplementation Knowledge Transfers to Users and Information Technology Professionals. Journal of Management Information Systems, 24(1), 171-199.
  • Scarbrough, H., Bresnen, M., Edelman, L. F., Laurent, S., & et al. (2004). The Processes of Project-based Learning: An Exploratory Study. Management Learning, 35(4), 491.
  • Slaughter, S. A., & Kirsch, L. J. (2006). The Effectiveness of Knowledge Transfer Portfolios in Software Process Improvement: A Field Study. Information Systems Research, 17(3), 301.

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.