I learned Python…and much more

I’ve spent the last two weeks with my head buried in programming languages.

I’ve been needing to re-write some scripts for data analysis for my research. I initially wrote some scripts in R but found that R is particularly slow when it comes to this type of analysis (more accurately I should say that my implementation of these analysis techniques is slow).

So..I started looking for a more economical way to do this analysis.  I’m using PHP to do some of the up-front data collection so my logical choice was to dust off my PHP skills and build some analysis scripts using PHP.

So I got out my PHP books and started coding. After a few days, I had a pretty impressive set of scripts that would take my collected data, run a bayes classification filter on that data for sentiment and then summarize that data.  I was proud of myself…until I realized that the implementation of my classification algorithm would be difficult to justify in an academic setting….or at least that I’d have to spend a lot of time defending and justifying it at a later date. This was also one of the reasons that I wanted to re-write the R scripts.

So…I revisited my approach.  Was there anything written in PHP that was well received in the academic world? Of course not.

One approach that is used by many researchers in  text classification and sentiment analysis is to use the Python language and the Natural Language Toolkit (NLTK) — and there are plenty of academic articles citing the NLTK…so that helps me with defending my algorithms in my dissertation work.

Now…I’ve never looked at Python. I couldn’t have written a “Hello World” program in python.  But…it needed to be done, so I found some resources on the web and dove in.  Over the course of a few hours I wrote my analysis and summary scripts in python….and was absolutely amazed at how quick this language is. My buddy Jeff is probably getting tired of me telling him how great python is … but oh well…he’ll keep hearing it 🙂

I was able to get the time that my analysis takes down from 8 to 9 hours in R to about 1.5 hours in python. Talk about a time saver!  Now…most of that time savings is probably due to new approaches to the analysis rather than just a pure python vs R speed issue….but the re-writing forced me to rethink my approach.

Why tell you about my newfound skillz (I’m told you have to use ‘z’ in this usage of the word)?

Part of me wanted to brag a bit 🙂

But, more importantly, learning a new programming language isn’t necessarily about the language itself…its about the discovery process.   For me, learning Python forced me to rethink my approaches to the data analysis I was working on…and the outcome is a faster analysis with potentially more accurate results as well as a more defensible algorithm. Learning a new language forced me to think through my approach. It forced me to think about the inputs and outputs.

When is the last time to you took a step back and rethought your approach?   You don’t need to learn Python to do it…just take a step back from your day-to-day grind and really look at what you are doing. Is it working for you?  Is it working for your team and/or organization?

If the answer isn’t an unequivocal ‘yes’, then maybe you need to rethink your script(s) and look for a new approach.

Quick WordPress Tip – Building your own shortcode

Wordpress LogoThis is a bit of a break from my normal blogging, but I felt like sharing a quick self-hosted wordpress tip that others might find useful.

In my regular Foto Friday posts, I have a section of content that I copy and paste into each post that points readers to other areas to find my photograph on the web as well as ways to help me with my addiction habit of photography.

The text is:

See more photos in my flickr photostream and/or my 500px portfolio. If you like my photography, feel free to support my addiction habit by visiting Adorama (affiliate link) to purchase new or used photographic gear. Or, if you are interested in renting gear before buying, try out my favorite camera, lens and gear rental site –  Borrowlenses.com (affiliate link) – the folks at Borrowlenses are awesome.  All proceeds from clicking the above affiliate links go to more photographs. 

For each Foto Friday post, I would go back to the previous post and copy the text and then paste it into the new post. While not difficult, this was getting to be a chore for me….and I started wondering how to make it more automated.

I’ve built themes and custom functions for WordPress before so I knew about shortcodes….but hadn’t really used a shortcode for just adding text before…but I figured it would work just fine….and it did.

This is a fairly easy thing to do for those of you who are technically inclined.  All you need to be able to do this yourself is a self-hosted version of WordPress and access to your theme’s “functions.php” file – and some basic HTML (and perhaps PHP) knowledge.

Building a Custom Shortcode

I found my theme’s functions.php file a pulled it off my server to edit it.  Using Notepad++ (create free editor for Windows), I added the following lines:

[sourcecode language=”html”]
/* Add shortcode */
add_shortcode(‘phototext’, ‘addPhotoText’);
The above is the definition of the shortcode text to use. In the add_shortcode function, the first field ‘phototext’ is my actual shortcode text that I want to use in each post. The second field ‘addPhotoText’ is the name of the function that should be run whenever the shortcode ‘phototext’ is found. Pretty simple so far, no?

Now…the code below is the actual ‘addPhotoText’ function. The only thing I want this function to do is insert the HTML code listed below into a post whenever the ‘phototext’ shortcode is found.

[sourcecode language=”html”]
/* Function for added shortcode */
function addPhotoText() {
return ‘
See more photos in <a href="http://www.flickr.com/photos/ericbrown">my flickr photostream</a> and/or my <a href="http://500px.com/ericbrown">500px portfolio</a>. If you like my <a href="https://ericbrown.com/photography">photography</a>, feel free to support my <del>addiction</del> habit by visiting <a href="http://www.adorama.com?kbid=65146">Adorama</a> (affiliate link) to purchase new or used photographic gear. Or, if you are interested in renting gear before buying, try out my favorite camera, lens and gear rental site – <a href="http://www.borrowlenses.com/?blpid=photographyminute">Borrowlenses.com</a> (affiliate link) – the folks at Borrowlenses are awesome. All proceeds from clicking the above affiliate links go to more photographs.

Once the above shortcode and function have been added to the functions.php file, I uploaded the file to my test server to make sure i didn’t fat-finger something. Once the new shortcode was confirmed to work, I moved it over here to my blog.

So now…anytime i type my shortcode phototext wrapped in brackets (a shortcode is wrapped in brakcets – e.g., [shortcode]), I’ll get the following added to any post/page:

See more photos at my dedicated Photography website. If you like my photography, feel free to support my addiction habit by purchasing a copy for your wall and/or visiting Amazon (affiliate link) to purchase new or used photographic gear.

Shortcodes make life much easier for WordPress users. If you have some content that you like to re-use (author bio, etc), take a look at using shortcodes…they may save you time.

BTW – If you have any questions about WordPress, shortcodes or anything else, drop me a line and I’ll see what I can do to help.

Agility and The New CIO

The New CIO is a weekly article about the challenges facing today’s CIO as well as what can be done to prepare for future challenges.

Are you agile?

I’m not talking about the Agile development methodology…I’m talking about BEING agile.  Are you able to deliver what your organization needs (with less resources)?

Build it….or they will

Do you have a Shadow IT group or groups in your organization?  If you said no, I’d bet you’d be wrong.   There are always Shadow groups within organizations….the difference between your organization and others has to do with the governance and network security.  Perhaps you don’t have a full-blown shadow IT group with non-supported applications and and/or hardware, but I’d put money on the fact that there is a person within your company that has purchased access to a software-as-a-service (SaaS) product.

Why would someone go outside your IT organization?  That’s an easy answer….because you can’t get things done for them.

Forget that IT governance, security and processes have to be followed…the end-user can’t get what they need from you so they go out and find something on their own.  If you’re lucky, all they’ve done is purchase a SaaS application. If you’re unlucky they’ve gone out and procured a third-party platform that they want to integrate with the rest of your IT systems.

Why does Shadow IT exist?

There’s a ton of reasons that these types of things occur, but most simply, Shadow IT arises because people don’t feel they are getting what they need from the IT group.

They feel like it’s too difficult to work with IT.  It takes too long to get things done.  There are too many restrictions placed upon them from IT.

You know what? They’re right!  At least from their perspective, they are.  Think about it.  Should it take a year to get a new project management tool setup for your Marketing group?  Should your Finance group really have to care about the technical restrictions?  I say no.

How do you fix it?

While I believe the real answer is a difficult and deep one but the place to start to looking for answers is a simple one: Be agile.

Make it easier for your organization to get things done. Make it easy for your Marketing group to implement a PM tool.  Take the restrictions out of the equation when talking to Finance about a new platform.  Understand their needs first and then you and your IT staff figure out how to deliver the system that they need.

The New CIO will have to be agile and build an agile team. Agility can mean many thinks but it doesn’t mean you have to ignore IT governance, security or your processes…just don’t use them as excuses for not getting things done.

Join me next week for another article in The New CIO series.

Enhanced by Zemanta

Agile Project Management and Product Strategy – A Case Study

Due to the length of this post, I’ve created a PDF version for those that would like to download it and/or print it out. Click here for the PDF version of Agile Project Management & Product Strategy.

Over lunch recently, a good friend and very experienced project manager shared his thoughts on managing software development projects when he said:

“Software developers are so hard to manage. They never do what I tell them, never report status and are always behind schedule and over budget. Most times, they never finish the project!”

I asked him to think about what he just said and to think about whether he is ‘managing’ the project or ‘leading’ the project (a project manager should be a good leader first and foremost). I then jumped into my philosophy on managing projects using a more agile mindset and finished with the following comment:

Perhaps software developers aren’t any harder to manage than any other function…perhaps it’s the system that has been put in place to manage the developers and their activities that causes the problems.

I believe most software developers are bright and hardworking people…but they tend to be put into situations that drain all of the initiative and drive they may have. Traditional software development methods (e.g., Waterfall, etc) can and have been used quite successfully in software development projects but the large up-front requirements gathering, milestones and end-of-project testing usually result in software that is late and buggy. Of course there are successful projects that have used these traditional methods. The majority of successful projects are those that are given the appropriate level of resources, people and time and give the product development team the ability to get their job done without much hassle. Some smaller organizations have a difficult time with these traditional methods because they aren’t able to fully resource the project and aren’t able to be as flexible and responsive with these methods.

I recently encountered one of these smaller organizations who needed some help crafting a product development plan. As you will see, we were able to change the development plan from one that delivered a revenue generating product in 12 months to one that delivered a product with ~85% of the same features in 6 months at less than half the cost. Oh…and we were able to capture $3.5 million in revenue after 6 months.

Case Study – Software Development

I was approached by a small technology company to see if I would be interested in helping them re-design their product development strategy to release a product within 6 to 9 months in order to beat their competition to market and gain an edge over their competition. The product had to be something that clients found useful, valuable and worth paying money for and needed to be able to compete with the ‘big boys’ that were moving into this space. It appeared to be a pretty daunting task: take a product that was planned for release in 9 to 12 months and have it ready for market in 6 months.

The company is a small startup focused on a niche market in the technology field product uses a third-party content management server package to serve up large amounts of data from the enterprise network. They had some very good developers and had hired a consulting company to develop a product development strategy to provide for a product release to their client base in 9 to 12 months. This same consulting company had been planning on actually doing the development work as well.

After reviewing the plan, the senior leadership team and board of directors felt that 12 months was much too long to wait to enter the market. They had heard rumors that some very big names in the industry were starting to make moves that would allow them to enter this niche market within 12 to 18 months and felt that they needed to be first to market in order to gain a competitive advantage in the marketplace.

I agreed to help with the project and joined the company in a consulting role to help them create a new plan that would see a product released within the next 6 months.

The Original Product Development Plan

On my first day with the company, I walked into the office and found myself staring at a very large Gantt chart hanging on a wall. The chart had the words “Product Development Roadmap” written across the top and looked just like every other Gantt chart ever created (e.g., milestones, resources, dates, timelines, critical path outlines, etc etc). My first thought was that least they understood what they were trying to do and had a plan. My second thought was ‘what are they delivering and when are they delivering it?’

I sat down with the team and over the next few days and discussed the product development strategy. I realized that the original development plan created with zero lead time (each task could only start when the previous task finishes). My gut feeling told me that his was done to lengthen the project and create more billable hours for the previous consulting company.

Since the plan was created using zero lead times, it required that a considerable piece of the network architecture design had to be completed before database design could be completed. In addition, a significant portion of the database design had to finish before any work on the CMS Integration could be started. The User Interface design could begin during Database design and was scheduled to begin after network design in conjunction with CMS integration.

As you can see from the high-level overview of the plan shown in Table 1, it would have taken 9 months before a product was available to demo and/or beta release to a client and 12 months for a final release.

Table 1: Original Development Plan

Network Architecture100 Days
Database Design100 days
Content Management System Integration160 days
User Interface120 days
Demo to Clients9 months from start of project
Documentation30 days
Testing30 days
Total11 to 12 months from start of project

After reviewing the plan and talking with the leadership of the company, I started looking at methods to “bring the plan in” so we could provide a demo and beta release much sooner than 9 to 12 months out.

The original resource schedule for the development provided for 1 network architect, 1 database architect, 1 content management expert, 1 user interface ‘guru’, a documentation specialist and 2 QA specialists.

The New Product Development Plan

To create the new development plan, I visited a few key clients to gather requirements for features that they felt must be in the software in order for it to be valuable to them. After these visits, I sat down with the development team and created a “Top Feature List” to help us focus the development efforts on only those features that were the most valuable to our clients.

Interestingly enough, the items that ended up being the most important items for the clients were the last things to be implemented in the original development plan. It’s amazing what a little bit of client communication will do!

Using agile software development methods, common sense and the “Top Feature List” gathered from my discussions with clients, the team and I created a development plan. The new plan would allow for the most important (i.e., valuable) features that our key clients were looking for to be ready for release as a Beta release within 3 months.

Due to the small team size and the inability to bring in additional developers, some of the tasks still needed to be performed in serial but most of the tasks were performed in parallel with each other. Table 2 shows the new product development plan.

Table 2: New Development Plan

Start Iteration 1
Database Design – DBA Resource30 days
Network Architecture – Network Resource45 days
User Interface – GUI Resource45 days
CMS Integration – DBA Resource15 days
Total Time45 days
Demo to Clients1.5 months from start of project
Start Iteration 2
User Interface – GUI Resource45 days
Documentation – Document Resource15 days
CMS Integration – DBA Resource45 days
System Testing- QA Resource30 days
Total Time45 days
Beta Released to clients3 months from start of project
Start Iteration 3
Network Architecture – Network Resource45 days
User Interface – GUI Resource30 days
CMS Integration – CMS Resource15 days
System Testing – QA Resource15 days
Total45 days
Beta2 Released to clients4.5 months from start of project
Start Iteration 4
Network Architecture – Network Resource30 days
User Interface – GUI Resource15 days
CMS Integration – DBA Resource30 days
Documentation – Document Resource15 days
System Testing – QA Resource45 days
Total45 days
Final Release to clients6 months from start of project

The budget for the new plan turned out to be less than half of the original budget, although we weren’t trying to dramatically cut costs, the shortening of the development cycle by half brought us considerable cost savings.

The outcome of the plan thrilled senior executives, investors and clients. Based on the first Beta release, we were able to book about $1.5 million in revenue that may never have been booked if we had used the original plan. The initial Beta release was nothing like the final release from the original plan, but it did provide the key features that the client(s) wanted to see.

The release after the final iteration contained about 85% of the features from the original plan. Think about that…85% of the product features in half the time at less than half the cost! That’s the power of agile thinking!

After the final iteration, the product was released to quite a bit of fanfare. In addition to the revenue we had already gained with the Beta release, we were able capture another $2 million in revenue from new clients and create a great footing for the company in the marketplace. At the time that version 1.0 was released, the competition was still mired in product planning and development and their release dates seemed to be at least another 12 months out.


After releasing version 1.0 and capturing $3.5 million in revenue, the company was able to attract quite a bit of interest from investors and potential buyers. They ended up being gobbled up by a much larger competitor for quite a bit of money (many times multiple earnings). I’d like to say it was all my doing…but it wasn’t…I just helped them to see a new way of looking at the development process.

Using agile methods allowed us to deliver real value to our clients and to our company by create a product in half the time. We were able to deliver $3.5 million in revenue because we focused on the needs of our clients and delivered those features that they thought were the most valuable.

That is what Agile Project Management is all about. Delivering value to clients.

Note to readers:
When I use the word ‘agile’, I don’t want to imply that I am explicitly talking about iterative or agile software development methods such as eXtreme Programming (XP), Scrum, Feature Driven Development (FDD) or other types of ‘Agile’ methods. Agile methods like XP and Scrum are the framework for my project management methods for managing software development projects but I am not a strict proponent of any one method or process over another. By using the word agile, I am trying to capture the frame of mind that someone should have to be able to accurately focus and respond to clients’ needs in today’s competitive world.


Reblog this post [with Zemanta]

Agile & Iterative Development

I just finished the book “Agile & Iterative Development: A Manager’s Guide” by Craig Larman.

This book provides a very good overview for managers of the Iterative & Agile Development methods. I liked this book…it isn’t too in-depth…perfect for someone who needs to know the basics about Scrum, XP, UP and other Iterative/Agile methods.

Two things about the book that keep it out of my ‘recommended book list’ are:

  1. I thought it could have used a bit more editing/revision prior to release as their are some minor errors, but on the whole this is a very good book.
  2. Removal or Revision of Chapter 4. This chapter is an attempt to bring all of the agile/iterative methods together into a ‘story’ but it just doesn’t work that well for me. What might have helped is to move this chapter toward the end of the book after all the methods have been discussed.

Overall…this is a good book and one worth reading if you are interested in learning more about Iterative & Agile development topics. The book really made me think about the ‘tried and true’ PMI methods for managing projects and how those methodologies aren’t really a good fit in the world of software development.

After reading the book (and a few other Agile books) I’ve begun to think about ways to move Agile methods from software/product development to other areas such as IT Management, Service Management and other areas of business.

[tags] Agile, Agile Management, Agile Project Management,Iterative Development, Agile Development, Software Development [/tags]

If you'd like to receive updates when new posts are published, signup for my mailing list. I won't sell or share your email.