CIO vs. CTO

When I’ve been out selling my A Part-Time CTO services, I’ve sometimes run up against a company saying they have a CIO already, so why do they need a CTO? It’s a good question. I’ve been reading a lot about the role of the CIO and it’s constantly evolving responsibilities. A good primer on the subject is a white paper from PWC title “The situational CIO”.

After talking with CEOs and other CTOs & CIOs, I think I’ve come to the conclusion that you’re never really have both a CIO and a CTO. Your company is going to fall into one of two categories:

• Technology is integral to your company. You have a CTO that manages the entire IT organization, including software development and infrastructure. Your CTO reports directly to the CEO and is a peer with the other CxO executives.
• Technology is newer to your company and/or not as integral. In those cases, you probably already have a CIO who overseas several areas. There is a Director or VP of Technology that reports directly to the CIO

This is only my conclusion after a dozen or so conversations over a month, so I could be wrong. However, I think understanding where technology in general and a CTO in particular fits into an organization structure, compared to much older and well defined areas, is very valuable. It allows for technology to be given it’s proper seat at the table, so to speak.

Sr. vs. Jr Engineers

When putting together a team of developers or engineers, there is a grand belief that most fall into one of two buckets: Senior or Junior. That designation never really worked for me because it’s so fuzzy. Unfortunately, it’s here to stay so I’ll be using it for this post.

I’ll be using “Sr.” and “Jr.” to represent the two buckets, with quotes, on purpose. Again, I hate the labels because it makes one seem inferior to the other. Of course, that’s exactly why some people like them, but I digress.

Which begs the question: “How do you know which bucket someone falls in?”

If you find yourself in the position of interviewing and staffing a team, you’ll find out that there’s an easy answer that most people use: years of experience. In fact, I’ve found that the cut off seems to be 5 years. If you’ve got under 5 years of experience, you’re Junior. Over 5? Senior.

For me, however, I believe that nothing could be further from the truth. When I interview people, the whether they fit for the role depends just as much on their thought process and the kind of experience they have then how much of it. I’ve met people with 10+ years of experience that I would have a hard time hiring for “Jr.” positions. I’ve also met incredibly smart people with 2-4 years of experience that I would put in the right “Sr.” role. If someone never evolves their thinking process, I don’t think that makes them “Sr.”, regardless of their years of experience.

If I could distill how I distinguish people to one point it would be: how the person understands and processes tradeoffs.

Software development is a difficult profession. As an engineer progresses in their career, they begin to understand that the decisions that go into shipping software are not black and white. There is a spectrum of gray. “Sr.” resources understand this and can make decisions in direction, architecture and implementation that reflect that mindset. They can analyze and articulate decisions about what will scale and what won’t. For example:

• How will file uploading be handled when there is more than one web server?
• What happens if I leave the team? Will others be able to work with what I’ve built?
• Implementing X works for our needs now, but will need to be changed when Y becomes a requirement.

“Jr.” resources are usually more concerned with the task at hand and getting it done as quickly as possible, regardless of what position that puts them in the future.

To put it another ( albeit over simplified ) way: “Jr.” resources frame all of their decisions in terms of days. What are they implementing today, tomorrow and Friday. “Sr.” resources frame their decisions in terms of weeks & months. How is what they are implementing today going to look next week, next month and 3 months from now.

And don’t fall into the aggregating superstar myth. You need both types of people to ship software. Don’t just think that “Sr.” > “Jr.” and shoot for filling your team with all “Sr.” people.  Depending on the actual people on your team, it could work out.  However, chances are that you can crank out more software with a 1:3 or 1:4 ratio of “Sr.”:”Jr.” people.

But will it scale?

Whenever I’m in a discussion about platforms or language choices, someone will invariability bring up how some language or platform theoretically performs and use that as a reason to use it / not to use it. This is especially true in startups. ‘Will it scale?’ is always a topic of initial conversations and choices.

As an aside, I’ve seen many startups collapse under the complexity of a product they ‘built to scale’. In reality, they should have been more focused on creating an iterative, responsive design and being able to pivot easier. But I digress.

Before discussing what my answer is in that situation, I think it’s important to point out the difference between performance and scalability, since people usually confuse the two:

Web site performance is how fast a web site responds to a request. User clicks button, requests a page or does some otherwise comparable action, the server handles it and returns a result. A sites performance is how fast that happens, whatever it is. Are there a lot of DB calls? Images to load? I/O ? Is the response gzip’ed?

Web site scalability, on the other hand, is how a sites performance is affected under increased load, either user or data or something else. How is performance altered when there’s 10,000 users making requests? 100,000 users? What about the backend? How do the database tables respond with 1,000,000 rows? 100,000,000? That’s scalability. It’s dangerously hard to plan for scalability, especially since so many sites are a large collection of moving parts. Sometimes you won’t know how things interact with each other until they do.

One of my favorite sayings with regards to scalability is ‘Everything is fast for small n’

At a micro level, performance is much more important. Users love fast, responsive pages. At a macro level, scalability is much more important. Users want stable, reliable software. If you have tons of the former, but none of the latter, you’ll be screwed.

Now, back to technology platform and framework choices. My answer to questions of platform scalability is always “With one exception, languages & platforms don’t scale, architectures do.”

That exception is functional languages like Erlang that are based message passing communication as opposed to shared memory, like many OOP languages ( C#, Java, etc.. ) This gives them the ability to support concurrent execution from the start rather than rely on a threading model made up of locks, semaphores and mutexes. This doesn’t mean that they’re scalable out of the box, just that they support more scalable constructs from the start.

The design and architecture of your site will have much more to do with the scalability that any specific technology choice you make in the beginning. Your deployment and operations architecture can make or break your site once it’s actually made public. Spend a lot more time planning your caching strategy, connection pooling and proxy / web server / app server setup than arguing over which language can create 10,000 strings the fastest.

Distributed Version Control Systems

Most development and engineering teams have long since seen the benefits of using a version control system (VCS) to maintain and version their source code. For many years the popular free choice was CVS. That was supplanted several years ago by Subversion. Subversion adoption steadily increased throughout the years until it supplanted CVS as the default choice for companies wanting a free source control product.

Note: For the purpose of this post, I’m concentrating on free source control systems. There are several products, from Perforce to TFS to Clearcase that are widely used, but their price tags and complexity usually keep them out of the reach of smaller companies. (TFS does support Subversion, but it’s still several thousand dollars.)

That is, until a new breed of source control systems, called Distributed Version Control Systems (DVCS), emerged. Offering a more decentralized model than traditional VCS, DVCS promised to remedy some of the more clunky aspects of centralized VCS, especially when working with multiple branches and distributed teams. As such, scores of high profile OSS projects are migrating their code to a DVCS, including Python, Ruby on Rails to Linux.

But does that make it right for you, your team and your company?

It’s important to remember that this is a tool choice for your company as a whole, not just your engineering team. As such it’s important to include several factors when making a decision like this, such as:

• What, if any, are the licensing costs for using the tools client? How about the server?
• How user friendly is the tool? Does it require 3rd party GUIs or addons to work well for your needs?
• What’s the deployment model? What types of servers, security and access will you need? What about backups and redundancy?
• What kind of support do 3rd parties offer in terms of integration with the tool?
• How mature is it? Can it handle what your team is going to throw at it?
• Is it supported? How are security fixes rolled out? How easy are they to apply?
• Who’s going to be using the tool? Just the developers? The engineering team? The entire company?

In the case of a DVCS, the tool choices are all stable and their usage is growing. Developers love them for their support of offline commits and much improved branching algorithms.

Note: Recent versions of Subversion have added the ability to merge branches that comes very close to the functionality offered by a DVCS.

However, rising above pure technical ability, most (maybe all) DVCS have some drawbacks. Three that come to my mind:

First, right now, there’s not a whole lot of integration with 3rd party systems. There is a nice ecosystem of bug trackers, wikis, file comparison tools, system admin tools and IDEs that offer built in support for Subversion. The corresponding ecosystem for DVCS choices is much smaller, though growing every day.

Second, the tools for working with a DVCS are pretty primitive and almost completely command line based. Now, this may not be an issue if developers are the only people working with the DVCS. However, non-developers are working with a VCS with greater frequency as the benefits of versioning artifacts and content become apparent. Trying to make them work with command line tools will not be easy.

Lastly, there is quite a learning curve between tradition VCS like CVS / Subversion and a DVCS. This could result in a gap in productivity. Though, as with any tool, the more experience your team gets with the tool, the better they will be at working with it. However, it’s quite easy to use a DVCS without using some of it’s major benefits ( like ease of branching, etc.. ) Thus without changing your workflow to utilize those benefits, you’ll be missing out on some of the very reasons to use a DVCS.

Note: I’m also ignoring the use case of maintaining mirrored repositories, one DVCS and one Subversion. In my opinion, I can’t see a team gaining enough of a benefit to outweigh the maintenance of two entire server solutions, complete with access control, backups, etc..

While DVCS are increasing in popularity, I’d have a hard time recommending them for a company to use as it’s VCS right now, except in some very specific circumstances. There’s just too much of a ecosystem to throw out by going with a DVCS over Subversion. Subversion is a known entity, it’s fast, it’s stable and it’s widely supported.

I have no doubt this is going to change in the ( possibly very near ) future. One of the nice things about most DVCS solutions is the ease at which you can migrate from a Subversion repository. As such, it’s probably best to wait for the tool support to catch up before taking the plunge.

In-House or Outsource?

Do we hack out a prototype somehow, find a developer or outsource the development?

• Cheap - this only takes up the time of your and your founders.
• Total Control Over Implementation - Since you’re building it, you have the kind of fine grained control over everything.
• Flexibility - You won’t need to commit to something, either cash, equity or involving other people, upfront. You can make decisions as needed.

• Slow - You’re not technical. Depending on the level of complexity and difficulty, it might not be the most efficient use of your time.
• Un-knowledgeable - Since it’s just you and your partners, you might not be making the best choices.

• Implementation Speed - Developers working full time will be much more efficient for you and your product since they’ll be dedicated to your company.
• Technical Expertise - Bringing expertise in-house is the best way to foster communication and feedback when building a product.

• Money - Unless you have cash to pay, the pool of developers willing to work for equity is small.
• Equity - It’s not unheard of for a developer to look at a web based company, see that he’s asked to build the entire product and ask for a large chunk of equity as compensation.
• Access - Good developers are tough to find.
• Communication - Hiring a technical employee early on can sometimes result in a founder developer gap.
• Need - If you just need a prototype or initial version of your product built, hiring a staff might be premature. While your team is trying to iterate and perform Customer Developer, your tech staff will be idle. And there’s nothing idle developers like to do more than think of new features to implement.
• Lack of Knowledge - Building out a technical team when you’re not technical is fairly difficult.

• Capacity - development companies have people waiting to build your product.
• Known Entity - You only need to guide the development of the product, they take care of finding developers, development process, etc..
• Delayed Commitment - You don’t need to worry about hiring on people too early.
• Cost - Outsourcing tends to be cheaper than hiring on developers and building your product internally.

• Cost - Most development shops work on a cash basis. There are a few that take equity, but a ) they’re rare and b ) you’re making a significant decision about your company pretty early on. If the relationship sours for some reason, severing it could become pretty hairy.
• Lack Of Communication - Typically there is a very “over the wall” approach to development where you specify requirements upfront and they give you what you specify. The type of day-to-day communication that takes place when developing a product is limited when the development staff is outside of the actual company.
• Lack of aligned interests - While there is a desire to perform good work, development shops are not necessarily looking at the long-term outlook for your product. They may make certain concessions or decisions that could hinder the flexibility of your application

There is also a hybrid variant of #2 and #3 above that I’ll call a development partner. A development partner is an consulting company that works very closely with you, similar to an internal development staff. They are looking at a long term relationship and are more likely to take a craftsman approach your product development. Thus, they typically take a vested interested in ensuring whatever they build will be able to evolve with the company. Lastly, they are more likely to take reduced rates, be open to a debt-to-equity relationship with you and your company or some other more flexible compensation setup.  

So given all of that, which should you choose? Well, like most decisions, it depends:

• #1, from my point of view, is out. If you’re not someone who can get up to speed VERY quickly and hack out a rough prototype, it’s simply not worth your time and effort to make the attempt. Unless, of course, it’s your very, very last option. Options #2 & #3 would have to not be possible for you to go this route.
• #2 is also difficult, though not impossible. It’s only a viable route if you have to cash to hire someone and you can keep them busy enough with relevant work, not just aimlessly spinning their wheels. Without the cash, you’re very close to being DOA. Most of the technically inclined engineers who would have the itch to work at a start for equity are already working on their own idea.
• #3 is a viable option if the relationship is properly managed and the product itself is fairly well thought out ahead of time. However, it should be known that, especially early in a products history, it will fluctuate. Having a outsourced relationship that can withstand those fluctuations is critical. You should place a premium on outsourced companies where the speed of feedback and communication can be the quickest.

The Ideal Web Architecture for Responsive Development

For years, whenever I’ve been involved in building a new web based architecture, I’ve always advocated that the engineers follow a simple guiding principle:

Design as though your web site is but one possible interface to your overall system.

This is because eventually most systems will need to be accessed in a variety of different ways:

• Web Browsers
• Console Applications ( ETL data loaders, etc.. )
• Fat client apps ( iPhone, Android, etc… )
• 3rd party services ( via an API of sorts )

The pattern of an N-Tier architecture has been a best practice for over a decade. However, that pattern has typically espoused a logical separation while maintaining a single physical deployment. There is a implication that the communication between the tiers takes place in-process. What I’m describing is a more of a physical separation of tiers and/or services.

Why would you do this? Well, the primary reason to separate entities like this is to encapsulate them. This insulates them and minimizes the effects changes that one component has on another. This is true for inter-connected components at most levels: the more two things are decoupled, the more one is resistant to changes in the other. This goes double for software products. This separation and encapsulation also allows you to changes things more easily in case you need to pivot your product to match an evolving business model.

“Isn’t this premature optimization?” No.

I’m not advocating building something you don’t need or trying to make things more complex at the start in the hopes that you’ll need it later on. What I am saying is that there are clear patterns and paths that can be taken to ensure that you’re not impeding your products ability to change and evolve as the business needs. There is a clear distinction between trying to solve problems before you have them and maintaining a flexible and evolutionary design.  Even if you never have to support multiple clients, your overall system will still be more flexible and responsive to change than if it were a more monolithic architecture.

For example, most newer web sites utilize AJAX technologies to make calls asynchronously back to a server, retrieve small data payloads, then process/display the results to the user. This updates the page without requiring a full refresh. Those calls are usually web service based, utilizing a URL to as the endpoint. It doesn’t take a large jump to make sure that each call isn’t specific to a particular client. Rather, it should not care who is calling it ( web page, command line app, python script, etc.. ), should be as RESTful as possible and as simple as possible. Taken in this light, suddenly your own calls become an API of sorts for others to possibly use. You end up building a web site utilizing your own API calls. Eating your own dog food.

Another example comes from the widely popular MVC architecture. It’s the architecture that Ruby on Rails, Django, ASP.NET MVC and a host of other frameworks are based on. The two pieces most closely tied to rendering data from your backend are the views and the controllers. However, if you’re building a web site, intimately tying your controllers with the expectation that the views are HTML based limits the would-be applications for the logic contained in your controllers. It’s better to not make such assumptions and rely on the view to render the data as it sees fit. If you’re using one of the above frameworks, your controllers will be URL based. If that’s the case, then suddenly reusing large parts of your backend becomes easier. For example, swapping out your regular web site for one optimized for mobile devices becomes a job of swapping out a set of views only, not rewriting your backend.

One last benefit of basing your product off of an architecture like this is that it makes it easier to scale individual pieces. However, that’s a large topic and will have to be addressed in another post.

Startup Product Development: Simple Now vs. Complex Later

When starting any software project, there’s an age old argument: should we build something simple that solves our current problem or should we use an existing product that’s more complex, but more feature rich, since we know that’s where we’re going to end up in the future?

This is especially true when starting a company because you don’t want to get into a situation where you have something that won’t scale up to handle your impending traffic and users. (Your company is going to be a huge success right? )

One side of the argument says to start simple and solve the problem(s) at hand. That’s not the same as making short sighted, possibly limiting, decisions. It means trusting that your engineering staff can handle any iterative changes required to support your changing needs. Understand any tradeoffs in performance & scalability that you’re making now in exchange for speed & time to market.

The other side of the argument is to target something that is more complex, but that solves more complex problems. The thinking is that when you eventually reach the point where you have the more complex problems, your solution will be waiting there, dormant, ready to be turned on. The proposed benefit is that you won’t end up with something simple that you need to trash once it outgrows your needs. You can grow into your product as needed.

However, in my experience, product development rarely, if ever, happens that way.

I recently ran into this very situation last week while talking to a startup founder. He had a client that needed some simple web sites built. However, the client believed that his needs would grow to require a more full-featured CMS platform. The founder reasoned that they should just start with the CMS, so when the need arises, it will take a minimal amount of effort to support the additional complexity.

I argued the opposite. Making the “just flip a switch” assumption ignores a large swath of other assumptions, any of which could prove to be a giant roadblock:

FYI: these assumptions are not mutually exclusive. You can ( and probably will ) encounter one or more of them.

• That you’re actually going to need the complexity - Sad to say, but sometimes the need just doesn’t materialize.
• That you understand all of your futures needs enough to select a product that would solve them. - Premature optimization is the root of all evil. Everyone loves to make predictions and be proven right. In reality, it almost never works out that way. You’re not omniscient. If you’re doing Customer Development ( you are, aren’t you? ), you’ll be taking your direction from your eventual market anyway. Why make some of the most critical decisions about your company when you have the least amount of information possible?
• That the product you picked can actually solve your future needs - If you’re like most companies, an exhaustive product evaluation isn’t in your initial project plan.
• That, in the time between implementing the simple solution and needing the more complex solution, nothing changes with the product that you’ve chosen that could impede your transition - Products change & evolve. New versions are introduced, sometimes with backwards incompatible changes. Even understanding any version changes and testing your product for any issues is going to be a non-trivial effort.
• The cost of maintaining and administrating the product you’ve selected is worth the trouble - Maintenance, Backups, database updates, security updates, etc… are all factors to consider when evaluating any product. If you don’t stay on top of updates ( especially security updates ) you can find yourself in a nightmare scenario.
• That the effort it takes implement your product in the short term allows for an easy transition to the more complex solution in the future - There’s a good chance that you’ll be new to whatever product you choose. There’s an even better chance that, however you initially use it, that your usuage won’t be optimal.

Those are just a few to keep in mind. Any one of those assumptions can cost you time and money. Both are precious resources in a startup.

However, an oft neglected repercussion of building too much too quickly is that the extra functionality can calcify your product and make it very rigid. Releases become more complex, new features take longer to implement and bugs take longer to fix. You can find yourself a prisoner of your product, maintaining functionality and features that no one ( or very few ) people use. It can demoralize a engineering team, making them more and more susceptible to the nuclear option: the big rewrite.

I think the tendency to lean towards a more exhaustive solution upfront comes from a time when the effort require to change software was much higher than it is today. When systems were written in C, C++, Perl or even Java, making changes was a large undertaking. The thought of possibly throwing away chunks of code was nerve racking.  It represented a huge investment in time and money. However, with todays rapid development languages and frameworks like Ruby/Rails & Python/Django, the investment required to create something, both in time and money, is rapidly shrinking.

To me, for any new product development effort to succeed, you should do three things:

• Start simply and build only what you need, nothing more.
• Pick a platform / language / framework that allows you to pivot and change direction easily.
• Hire good engineers that are experienced with different technologies and can change and adapt to different technology needs. This will by far the hardest, but it can be done.