How I Proved a Startup Wasn't Worth It - For Only $500 (With Details)

In one week* - and for just $498.38 - I was able to decide that an idea I’d been mulling over for the past 3 years was not a viable business endeavor. Why share a failure? Because success stories abound, but I learn more from what didn’t work than from what did. What follows is a post mortem of SendBetter.Email, a video tutorial series.

The hypothesis

Small business and non-profit marketers will subscribe to a tutorial series that details the proper way to setup and use popular email marketing platforms.

For several years I freelanced providing digital strategy and hands-on support for small businesses and non-profits. As the Digital Swiss Army Knife, I managed and built websites, administered servers, and ran email marketing platforms just to name a few responsibilities. One commonality all new clients shared was that their email marketing platform, usually Mailchimp, was a misconfigured mess.

As a result of this experience, I thought that there was an opportunity to build a subscriber-based website providing this same type of guidance on a larger scale than what I could manage by working with clients directly. Two sites served as inspiration — RailsCasts (which I subscribed to) and GoRails.

The setup

Here’s where I close the loop on the ‘one week*’ above. Setting up the infrastructure took closer to six (6) months, while the test only took one week. Yes, this setup was overkill (not lean) and could have been done with much less effort. I did learn how to build a custom WordPress theme in the process — so not all was wasted.

The infrastructure was made from combining and configuring the following services and platforms.

• WordPress - the self-hosted version
• Restricted Content Pro - A premium WordPress plugin to handle subscriptions and access to content
• A custom (read: home-made) WordPress theme
• Vimeo - video hosting
• Mailchimp - The initial subject matter and email platform
• Stripe - Payments platform
• Lucky Orange - Heat mapping and video recordings of visitors

The initial library of content was eleven (11) episodes mixed between freely available to view (no subscription necessary) and premium episodes (subscription required). My assumption was that visitors would be more likely to subscribe if there were several tutorials available. This assumption turned out to be irrelevant, but did expand the timeline to build the site before launching the tests.

Initial test objectives

Realizing that the likelihood of having a runaway success from the start was slim, I wanted to establish benchmarks and start the process of optimizating the ads and of the site.

For the ads, I wanted to learn if general ads promoting the site would perform better than ads for a specific episode. And, using the Facebook Dynamic Creative feature, I wanted to find a combination of images and copy that converted well.

On the site, I wanted to see if people would watch the videos and/or read the transcripts; learn what rate a visitor would explore other episodes (pages per visit); and, optimistically, what rate people would subscribe.

Ultimately, the subscription/conversion rate would allow me to calculate the cost per acquisition (CPA) — the key metric to determine the viability of this concept. With a CPA established, you can learn if revenue generated from a subscriber is more or less than the cost of the ads to get that subscriber.

Why Facebook first? Or, why Facebook first.

Another assumption I had was that Facebook would be the best platform to market through. And by best, I mean that it should allow me to target a relevant, niche audience at a lower cost than another platform (eg. Google Ads).

Based on my original hypothesis, and using Facebook’s advertising platform that leverages demographic information on all of their users, I was able to create a targeted audience of ~26,000 that “Liked” or have a job title in entrepreneurship, small business marketing, email marketing, and nonprofit marketing.

And to explore which ad composition — both image and copy — would perform better than others, Facebook Dynamic Creative feature was very compelling. Effectively, by uploading up to 10 images, 5 headlines, 5 post descriptions, and a few other items, Facebook will mix-and-match the above and randomly display them until some start to outperform others.

One feature of Facebook that I wasn’t able to benefit from, and might have made a difference in my experience, is to leverage an existing audience — either from a business Page or an existing email list.

Facebook (in)action

My first Facebook frustration was with the infrastructure required to run a single ad. In addition to a Facebook Business Manager account, and an Advertising account, a Facebook Business Page needed to exist. While existing businesses probably already have this, setting these up to simply run some tests was annoying.

The next frustration was figuring out how to navigate through the campaign setup to get to the Dynamic Creative feature. This was less than intuitive. Thanks, Mark!

Facebook ads were launched on a Thursday. By Saturday I knew that the current approach wasn’t going to result in enough traffic fast enough to learn and iterate. The default setting for Facebook ads is to spread out your desired budget over 30 days. On Saturday, I changed this to 7 days but traffic was still slow to come by.

For this type of testing, Facebook’s pricing model isn’t conducive. During setup, the only option provided is impressions — a cost per thousand (aka CPM). In other words, Facebook wants to charge based on the number of times it shows the ad, and not based on the number of times a user clicks on it. Thus, our interests aren’t aligned.

As a result of a CPM pricing model, I received low traffic from Facebook at an effective cost-per-click (CPC) of $6.64. In other words, while Facebook charged me to show the ad, it cost me $6.64 for each actual visitor.

The first iteration — Google Ads

By Sunday I was ready to try something new. My $46.47 paid to Facebook had resulted in 7 visitors.

My first visitor, as a result of Google Ads, came quicker and for less money than Facebook — but still not the volume I was hoping for.

However, I did make a mistake when switching to Google Ads that resulted in lower traffic, too. Google estimated that there are 10,000 relevant searches per month, which breaks down to an average of 333 searches per day. And, at an industry-wide average click-through rate of just under 2%, that’s only an estimated 7 visitors per day. Still not enough site traffic to learn quickly and iterate. To help with this, Google does a nice job providing recommendations on a daily basis to improve your paid search efforts. I accepted almost all of these.

By Thursday, a full week after launching the Facebook ads, I’d spent $83.91 with Google Ads and received 40 visitors to my site — mostly to the specific episode. That’s a CPC of $2.10.

Observing my visitors behavior

Remember that what I wanted to learn was if visitors would watch the full episode videos and/or read the transcripts; learn how curious they would be about other content was available by visiting other episodes; and, if any would subscribe.

To track visitor behavior, I leveraged three services — Google Analytics, Vimeo, and Lucky Orange. Of these three services, Lucky Orange was by far the most useful in helping me determine to pivot to the next concept instead of persevering with SendBetter.Email — and my use of their service only scratched the surface.

What exactly is a Lucky Orange? They bill themselves as “The Original All-in-One Conversion Optimization Suite,” or as “DVR for your site.” If you’re familiar with services like HotJar, Mouseflow, or SessionCam — just to name a few — then you are already familiar with the market Lucky Orange competes in.

Of the services Lucky Orange provides, I relied largely on visitor recordings.

What exactly did I learn? Visitors appeared to be only interested in solving a specific problem and not interested in a source for additional, related content.

• 13%. The number of visitors that visited more than one page.
• 17 seconds. The average time spent on the site per visit.
• Zero (0). No one watched a video, according to Vimeo.

While watching replays of visitors that did stay long enough to register in Lucky Orange I observed that most were on mobile devices, they did scroll deep on the page, but did not appear to be scrolling slow enough to fully digest the content. This leads me to believe that the format (video tutorial) isn’t the right format for this audience.

Next steps

Based on these observations, my next step would be to change the format of an episode. Currently, the content is a video tutorial with a transcript and I would change it to be screenshots with itemized steps to follow. My tests would then focus on increasing the session duration (benchmark: 17 seconds) and decreasing the bounce rate (87%).

By increasing the session duration, the assumption is that visitors would be digesting (and possibly acting upon) the content.

By decreasing the bounce rate, the assumption is that the visitors found the original content valuable enough to explore other content on the site.

While only the session duration is needed to test the change in format, the bounce rate shows if these visitors are also likely to consider subscribing — my original hypothesis.

Next, next steps

If the session duration did improve, but the bounce rate remained high I might try to find ways to better promote other episodes, or buy ads for other episodes that might appeal to a different audience — an audience that might have a greater tendency to subscribe!

If neither the session duration or the bounce rate improves, I would revisit the ads as well as promote other episodes.

In hindsight

Build less. Test sooner. I spent too much time going from concept to testing. Not counting the years I spent thinking about doing this, I spent about 6 months (in my spare time) getting to the place where I could run the tests. This clearly was not a minimally viable product for the purposes of testing the concept.

What I could have — should have — done instead.

• Use a static website with one(ish) post and other “protected” posts that required a subscription. While I did learn how to build a custom WordPress theme as part of this experiment (a plus!), reducing the scope would have sped up the setup time. And, I could have simulated the idea that there were other tutorials available — but only to subscribers — without actually spending the time to make those tutorials.

• Test the post format from the beginning. This also would have been a time saver and validated from the beginning whether or not the video format was the preferred format.

In other words, reducing the scope would have cost less and allowed for quicker hypothesis testing.

Google Ads is better than Facebook ads. Don’t get me wrong, Facebook ads can be very powerful and useful. However, for my scenario, I found Google Ads to be far more effective — cost and garnering traffic.

• Google Ads is contextually relevant, while Facebook ads are out of context for the viewers. By this, I mean that when someone encounters an ad in a Google search result, they are related to the question they’ve turned to Google to answer. Users of Facebook, on the other hand, encounter the ads as they are scrolling through their News Feed and not relevant to the mindset they have at the time.

• CPC is better than CPM. With the goal of getting traffic to the site, Google Ads’ pricing model (cost-per-click) aligns our interest with theirs. By charging based on impressions (CPM), Facebook is systematically not incented to care about relevant clicks to the extent Google Ads is. This difference also makes Google Ads more cost effective than Facebook.

Episode ads performed better than generic ads. On both platforms, the ads for a specific tutorial episode had a much higher click-through rate than the ads for my overall concept.

Final thoughts

Did I give up too soon? Maybe.

Though, after 6 months of prep work, the realization that maintaining this effort was going to require weekly dedication to new content, and the lack of traction of visitors showing broader interest, I decided to re-energize myself by moving on to the next idea I’d been mulling. Namely, sharing my thoughts on product development here.

I’d love to know what questions, thoughts, or critiques you have. Send those to me at evan [at] productownering.com.