Developer.
114 stories
·
54 followers

A Downtime Irony

32 Comments and 46 Shares

So many things can go wrong and often do, but I spend a good third of my time working on infrastructure, monitoring, and analytics so that they don’t.

Here’s what happened: At approximately 4:30pm PT feed fetching ceased. The feed fetchers were still working, which is why my monitors didn’t fire and alert anybody. But I have a second large Mongo database server used exclusively for collecting data about feeds being fetched. There are approximately 75 servers dedicated to feed fetching. These analytics look at average fetch times on a per task server basis. I use these analytics to ensure that my task servers are humming along, as they each use a ton of network, cpu, and memory.

This Mongo analytics servers works in a curious way. If you right-click on a feed and go to Statistics you’ll see the feed fetch history for a feed, stretching back a hundred fetches if the feed has had any issues in fetching. I keep these statistics on an analytics server separate from the regular Mongo server. I do this so that if the mongo analytics server goes down, everything will operate normally.

But the mongo server didn’t go down. It merely gave this error:

OperationError: Could not save document (Can't take a write lock while out of disk space)

Mongo continues serving read queries while not allowing write queries. I didn’t plan for that! And it gets worse. The way MongoDB stores data is that is just keeps growing, even as you delete data. NewsBlur only saves the last few fetches, but deleting old fetches doesn’t give you back any disk space. Every other database server I use has an autovacuum process that takes care of this maintenance work (PostgreSQL, Redis, Elasticsearch, but not MongoDB). It’s unfortunate that this is yet another instance of MongoDB being the cause of downtime, even though the fault lies with me.

The server that is meant to only be used to ensure things are working correctly was itself the culprit for feeds no longer being fetched. This is the ironic part.

NewsBlur’s developer during happier times wearing the 2013 NewsBlur t-shirt in Big Sur

Now comes the painful part. On Wednesday morning (yesterday) I packed my car and headed down to Big Sur to go backpack camping for the first time. I’ve car camped plenty of times, but I felt confident enough to pack my sleeping bag and tent into a big bag and head ten miles into the woods of coastal California.

I headed out, away from cellular service, at 4pm PT, half an hour before the analytics server ran out of disk space. And then returned nearly 24 hours later to a bevy of alarmed tweets, emails, direct messages, and a voicemail letting me know that things were haywire.

But the real problem is that I set a vacation reply on both my personal and work email accounts to say that I’d be out until September 3rd. Now, I hired a firm to watch the servers while I’m at Burning Man starting this Saturday. But I figured I could get away with leaving the servers for twenty four hours. And I neglected to tweet out that I’d be gone for a day, so theories cropped up that I was injured, dead, or worse, ignoring the service.

Brittany, NewsBlur’s developer’s girlfriend, can handle any situation, including driving a hysterical developer three hours back to San Francisco without breaking a sweat.

If you’re wondering, I think about NewsBlur first thing in the morning and last thing at night when I check Twitter for mentions. It’s my life and I would never just give up on it. I just got cocky after a year and a half of nearly uninterrupted service. NewsBlur requires next to no maintenance, apart from handling support requests and building new features (and occasionally fixing old ones). So I figured what harm could 24 hours of away time be? Boy was I wrong.

If you made it this far then you probably care about NewsBlur’s future. I want to not only assure you that I will be building better monitoring to ensure this never happens again, but to also offer anybody who feels that they are not getting their money’s worth a refund. Even if you are months away from payment, if you aren’t completely satisfied and think NewsBlur’s just about the best thing to happen to RSS since Brent Simmons released NetNewsWire back in 2004, then I want to give you your money back and let you keep your premium account until it expires.

I would like to also mention how much I appreciate the more light-hearted tweets that I read while on the frenetic three hour drive back to San Francisco from Big Sur. I do this for all of your happiness. If I did it for the money I’d probably find a way to juice the data so that I could at least afford to hire an employee. This is a labor of love and your payment goes directly into supporting it.

Big Sur is where a good many new ideas are thought.
Read the whole story
rafeco
3 days ago
reply
Don't feel bad, databases gonna database.
acdha
5 days ago
reply
This is such a perfect sysadmin story. Kudos to Samuel for sharing the details.
Washington, DC
samuel
5 days ago
reply
Ugg, I feel so horrible about the downtime, and so soon after feeling so wonderful.
The Haight in San Francisco
sredfern
5 days ago
I'd rather you have a holiday and I deal with a couple of hours of downtime :)
brucealdridge
5 days ago
you deserve a holiday ... and 75 feed servers?!?!
samuel
5 days ago
That's only the feed fetchers. I also have a few dozen apps servers and a dozen different database servers. NewsBlur's a hungry beast!
larand
5 days ago
Stuff happens. Thank you for all the work you've put into NewsBlur, and I have absolutely no intention of asking for a refund. Hell, I'd renew early if need be.
mp4328
5 days ago
have an amazing Burning Man experience!! - don't worry about us while you're out there.
jqlive
5 days ago
No worries man. Stuff happens. Enjoy Burning Man. Thanks for all the hard work.
rikishiama
5 days ago
no worries here too. great honest post that makes me feel good I'm a year-plus paying user.
wreichard
5 days ago
Ditto to all these comments!
acdha
5 days ago
Enjoy a break. The rest of us can use some practice dealing with not clicking refresh like a rat in a behavioural study…
kpjackson
5 days ago
Let me add my note of appreciation for your honesty and dedication to this great product. Downtime happens and we can all learn lessons from our mistakes. It's when we don't learn that we should really feel horrible. Oh, and I plan to be a happily paying customer for years to come. Also, any plans for a 2014 NB Tee?
StunGod
5 days ago
Having been in similar situations in the past, I totally understand. You're doing good work here, and NewsBlur is the only reader I use. I'll be paying for it as long as you're willing to accept payment.
samuel
5 days ago
I would love to get a 2014 NewsBlur t-shirt out there, but I haven't found a designer yet. Most want $1000+ for a design, and I'm telling them there's only about a hundred of these things that are going to sell. Happy to hear ideas!
getwired
4 days ago
So sorry that it had to happen while you were away. Thanks for the status update - but mostly, thank you for the wonderful service you've built. It's invaluable to me (I feel like I owe you more than I pay.) Thanks!!!
Share this story
Delete
28 public comments
laza
4 days ago
reply
Don't feel horrible, this just showed us how valuble Newsblur is for our workflow or procrastination :) , and i'm so jealous that you are going to Burning Man, you should write a blog post about your experience there.
Belgrade, Serbia
emdot
4 days ago
reply
I absolutely love NewsBlur. Thanks to Samuel for the update -- things like this make me even more happy to use the service. Thanks for your hard work.
San Luis Obispo, CA
satadru
4 days ago
reply
I'm considering it a hallmark of the intensity of the academic program I'm in that I didn't much notice the Newsblur downtime.
New York, NY
jbouvier
5 days ago
reply
Any developer knows downtime happens. The only thing you can do is your best to prepare, and when the shit hits the fan, be honest about the cause. This is how all developers & companies should treat these inevitable occurrences.
Chicago, IL
chengjih
5 days ago
reply
Hah, I am so far behind on my feeds I didn't even notice the issue. 6000+ unread articles before the outage, 6000+ after.
llucax
5 days ago
Same here, I don't have as many unread stuff, but this is just a RSS feed reader, come on! Let this guy have a day off!
loic
5 days ago
reply
Murphy + mauvais timing + gestion de crise = super post
France
smadin
5 days ago
reply
Well, here's what happened with Newsblur.
Boston
smilerz
5 days ago
reply
Murphy strikes again.
Chicago or thereabouts
sirshannon
5 days ago
reply
Exactly.
Cafeine
5 days ago
reply
That combo of bad timing and technical glitch is just crazy. -_-;
Paris / France
Berstarke
5 days ago
reply
Meh. No worries here. Murphy happens to everybody, man.
bronzehedwick
5 days ago
reply
No intention of asking for a refund. :)
Brooklyn NY
gradualepiphany
5 days ago
reply
Hopefuly it doesn't put you off going backpacking again. Thanks for being so diligent!
Los Angeles, California, USA
sredfern
5 days ago
reply
Here here.
Sydney Australia
lkraav
5 days ago
reply
What needs to happen in order for Sam to be able to hire an employee? How much would the yearly rate go up?
Tallinn, Estonia
renefischer
5 days ago
Thats a question I also want to raise. There definitely need some kind of business continuity for situations like that (or even worse one). I really appreciate the work Samuel is doing here and I love Newsblur, but I also see that its to heavily depends on one person.
jlvanderzwan
5 days ago
reply
Gotta work on your bus factor, Sam! :)

http://en.wikipedia.org/wiki/Bus_factor
renefischer
5 days ago
reply
I realy like what you are doing Samuel. I hope there is some kind of business continuity just in case something urgent is happening to you and you can't react personaly to help this lovely service. If not, please take that downtime as a signal to think about something to prevent situations like that. Newsblur is IMO too heavily depending just on one (great) person.

I'm going to renew my subscription, which is ending right now to support the further development of this great service.
Pliening
wmorrell
5 days ago
reply
It is honest self-assessments like these, and the overall openness of NewsBlur, that makes it better than GReader for me. I remember a few downtimes there that were real downtimes, not just "it takes slightly longer to view feeds".
lasombra
5 days ago
reply
Samuel, it sucks, but it's part of the job. I love NewsBlur and paid for a premium account after 1 hour of using it. I'm on the second run and all I have to say is "Great job man!"
UK
pablooo
5 days ago
reply
$PWD
DrewCPU
5 days ago
reply
The whole day, I thought "Poor Sam, he's gotta be going nuts!" I hope you enjoyed your much-deserved offline day.

Meanwhile, I was looking at all of my feeds manually and realized that I didn't need some of them and did some cleanup.
New Jersey
kazriko
2 days ago
Hah. I was doing the same, but mostly moving sites that haven't updated in years to my "Dead Sites" folder. I keep them there just so I can see if something suddenly revives on me.
rewingau
5 days ago
reply
The main thing I learnt from this downtime was just how much I value Newsblur. Hint - a lot!

And as for the downtime - the demon Murphy laughs at your camping trip...
Canberra, Australian Capital T
alannashaikh
5 days ago
reply
This is really no big deal. Stuff happens.
rtreborb
5 days ago
reply
Hard to be upset with you, Samuel. Thanks for all you do!
tedder
5 days ago
reply
it's all good. you have a great story to tell at a conference now.
Uranus
DavidForest
5 days ago
reply
Thank you for the explanation. I value it when services I use break explain what occurred. Things like this happen, as anyone in technology based roles would know. I for one still really like using NewsBlur and will continue to do so as a paying customer. Keep up the good work and enjoy Burning Man.
rubin110
5 days ago
reply
Thanks for rocking the thin blurry line between open source and able to financially support you and your work. In all honesty a random day of down time is a blessing with the number of feeds I've got going. Thanks again for all you've done.
San Francisco, CA, USA
kazriko
5 days ago
reply
No chances for requesting a refund here. The service is too good, and has been too stable. This is a really unusual bit of downtime. Thanks for being on top of it for so long. I was still able to fetch and read the important feeds by instant-fetching then looking at each feed separately, so no real issue other than being time consuming.
Colorado Plateau
denubis
5 days ago
I completely agree. Downtime happens, and developers deserve vacations. This is a wonderful service.
g_hoges
5 days ago
+1 from me too. I love the service, and tolerate far more downtime from things like banking platforms. Continue to rock, Samuel
lasombra
5 days ago
Right on man. Downtimes happen and Samuel has been rocking on the downtime arena for a very long time. It wasn't the end of the world. It sucks, but happens.
murrayhenson
5 days ago
Seconded. Also, Samuel took the time to quickly comment on what went wrong and to really accept responsibility. You can't buy that but it's worth a hell of a lot.
JamesDiGioia
5 days ago
And the thing about it is it's not strictly "downtime" - you could access the service, and with some minor hacky workarounds, continue to use it. Definitely not asking for a refund.
JimB
3 days ago
I quite agree with you. Everyone is entitled to a break from the grind, and *sometimes* computers do unexpected things. The amazing thing is that you sorted out so quickly once you found out.
cmarshall
2 days ago
Every system has issues occasionally - the important thing is how you handle them. The two keys are communication and getting it fixed. You done good.

The Hobbit: The Battle of the Five Armies

2 Comments and 3 Shares

Here's the trailer for the third and final movie in Peter Jackson's The Hobbit trilogy:

The Hobbit was initially supposed to be just two films but Jackson decided to split the second film into two. From Wikipedia:

According to Jackson, the third film would contain the Battle of the Five Armies and make extensive use of the appendices that Tolkien wrote to expand the story of Middle-Earth (published in the back of The Return of the King).

The second movie was better than the first so I'm looking forward to this one. But then again, I'm totally in the tank for Jackson's take on Middle Earth (I did the Weta Digital tour when I was in New Zealand) so I would see it even if the first two movies sucked.

Tags: movies   Peter Jackson   The Hobbit   trailers   video
Read the whole story
rafeco
5 days ago
reply
I thought the first Hobbit movie was the most wretched adaptation of a novel ever filmed.
petrilli
5 days ago
As someone who actually enjoyed the extended cuts of the first, these have been terrible. The first one was so horrendous that I didn't even bother to see the 2nd. He needs an editor, stat.
Share this story
Delete
1 public comment
nickoneill
14 days ago
reply
COMING THIS SUMMER: Dwarf Fortress, the Movie
san francisco

Police Impunity in Ferguson

3 Comments and 5 Shares

One more on the police in Ferguson. Matt Yglesias:

The other two men in the photograph, despite presumably being police officers, are not identifiable at this time. Unlike normal police officers, they are not wearing name tags or badges with visible numbers on them. When police arrested the Washington Post’s Wesley Lowery and the Huffington Post’s Ryan Reilly, they weren’t wearing badges or name tags nametags either. Reasonable people can disagree about when, exactly, it’s appropriate for cops to fire tear gas into crowds. But there’s really no room for disagreement about when it’s reasonable for officers of the law to take off their badges and start policing anonymously.

There’s only one reason to do this: to evade accountability for your actions. […]

Policing without a name tag nametag can help you avoid accountability from the press or from citizens, but it can’t possibly help you avoid accountability from the bosses. For that you have to count on an atmosphere of utter impunity. It’s a bet many cops operating in Ferguson are making, and it seems to be a winning bet.

Disgraceful. Every police officer should not only always wear their badge and name tag while on duty, they should be proud to do so. (And in most cases, that’s true.)

Read the whole story
rafeco
7 days ago
reply
Removing one's badge while on duty should be a firing offense for policemen.
Share this story
Delete
1 public comment
JayM
7 days ago
reply
.
Boston Metro Area

Testing Without A Brain

1 Share

tagged testing, rant.

Testing Without A Brain

Testing software is perhaps one of the most controversial topics among programmers. Opinions range from “test first and test everything” to “tests slow us down”. As with so many things, the truth lies somewhere in between. After making my own experiences in a lot of different projects, I have to admit to myself that finding the perfect balance is still a hard a lot of times.

Starting out in a new project, time pressure usually is low and we programmers are highly motivated to get our hands dirty with coding rather than eliciting yet another requirement. I found that people tend to write unit tests more readily at this stage. And arguably the code quality seems to be better than when working under tight deadlines. But that’s not necessarily related to the tests, it could very well be attributed to the relative freshness of the people on the project.
As projects get into a more intense phase the ratio between test-code and production-code usually takes a hit. So is this a bad thing?

How much is too much

Kent Beck, the originator of the term Test Driven Development for sure has a ton of experience in writing software and in testing it. It might come as a surprise for many that he draws a pretty tight line between what is actually too much testing1:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence…

If I don’t typically make a kind of mistake … I don’t test for it.

To me that sounds very reasonable. Blindly striving for a high test coverage just doesn’t cut it. On the contrary: if a programmer is just guided by his coverage report, he will probably write some hard to maintain test code. And hard-to-maintain test code will eventually lead to reduced testing since the damn tests require more and more work to keep them in sync with changes made to the software. Those tests will get kicked out – and that with good reason!

There is another issue I found with some of my unit testing: Overuse of abstractions. While I’d consider abstractions in general a desirable and good thing, they really only make sense when there is the actual need for an abstraction. If you only have one implementation of some functionality in your code, why go through the effort of generalizing? Since by definition there is only one thing to abstract about, your abstraction will probably have to be refined when introducing a second implementation.
But writing unit-tests mandates the use of abstractions since you only want to test some piece of code in isolation of the rest of the system – for which some kind of interface definitions come in handy. So you bite the bullet and create your interfaces, change the code around, build mocks and other scaffolding and hours later start writing your tests…while at the same time you could have cranked out this extra feature or stomped this nasty bug. Sounds like a tough choice to me.

The kind of software

When people show me their incredibly cool new test framework or preach about the absolute imperative of having thoroughly unit-tested code, the examples they come up with usually lend themselves to a unit-test centric approach. But we rarely have to write the next sorting algorithm or code to serialize and de-serialize data.
The day-to-day stuff often involves database access, updating a UI, network code a.s.o. Those things are by far harder to cover by unit-tests and tend to involve some heavy machinery. Time spent with setting up unit-tests might be spent more effectively otherwise.

The kind of testing

I still consider testing my software with due diligence an integral part of the job. For some code this might very well be exhaustive unit testing, in other cases a different strategy might prove the better choice – like moving more into a more global approach where the functionality is tested as part of a bigger use-case. This kind of testing – often called integration-testing – is much harder to automate and thus does not always allow for regression testing, but it can be the right choice and help you to implement and test your code more quickly.
There is another very different flavor of testing I so far omitted: the kind of testing in a dynamically typed language that makes sure that trivial (or not so trivial) changes do not completely mess up your code-base. Those tests – I’d rather call them syntactic checks – are a prerequisite to stay sane. I did a couple of larger ruby programs and it was really painful to come back to the code at some point later: to find out I had a hard time to introduce changes just because I broke the code in unintended ways and my test coverage was not high enough to detect it.

Striking a balance

So what would be good advice for that new guy that started in your C++ department yesterday. Test everything? Strive for high coverage? I’d say yes – but as with every advice this should be taken with a grain of salt. If you end up with double the amount of code and possibly a different design I’d say you clearly overshot. In the end it’s not the unit-tests that are responsible for great software – it’s the talent and the ability of the developer.
On the other hand if the new guy writes your next rails application, the answer is much simpler: test the shit out of the code!


  1. see Kent Beck’s reply on stackoverflow to “How deep are unit tests”

You can contact me via e-mail or send me a tweet @marcontwit

Read the whole story
Share this story
Delete

Sorry about your startup

2 Comments and 3 Shares

So I came across yet another article on Hacker News running a post-mortem on a failed startup. Right off the bat it asserts:

Entrepreneurs often write about what’s going right, but too rarely write about what’s gone wrong.

I’m sure there was a point in tech bubble history when this was true. But that point was a long time ago. Startup guys write about what went wrong all the damn time. I am pretty sure if I started a startup pitched as a platform for other startups to explain why they tanked, I would get VC money for it, especially if I could get a good deal on failr.com.

And I don’t want to pick on the author of this particular article. He’s a solid enough writer, his startup looks genuinely interesting rather than stupid, and hey, he sure went above and beyond in mistake-making. (“And so I learned that we hadn’t been paying payroll taxes for almost three years.” Oopsie!) But these articles are becoming post hoc navel-gazing bemoaning subsets of the same problems, over and over. They’re Chinese-American takeout menus of fail: I’ll have the “hired too many people soon” and “didn’t scale fast enough” from Column A, “poor communication” and “ill-defined cofounder roles” from Column B, and some extra sweet and sour sauce.

And this makes this soul-baring part of the performance, like consciously dressing down and overpaying for loft space. They’re written for potential investors and employees who hang around sites like Hacker News. They’re spin. Bob’s heart-rending tale of how he spent ten months with no income so he could avoid laying off his last three employees as long as possible, eventually living under his office desk and subsisting entirely on chocolate chai and teriyaki jerky, ensures you remember him as a naïve but sincere and selfless CEO who gave his all and shared his mistakes with the world. Without this bracing splash of sincerity aftershave, you might instead remember Bob as a twenty-two year old who burned through $7M of VC money creating an iOS app that sends the word “Meh” to selected friends on your contact list. (“We see an immense upside potential vis a vis the growing Irony as a Service (IaaS) market.”)

Ultimately, most startups in the current tech boom are going to fail for one of three reasons:

  1. The core idea of the company isn’t that good. Maybe it’s Pets.com, or Color. And if you’re pitching your startup as “like X for Y”—like Facebook for geek girls! like Instagram but only for cat pictures!—you have a problem.
  2. The core idea won’t generate more income than outcome before you run out of money. Why, yes, everyone loved StoreYourHugeFilesforFree.com, but it turns out you can’t make it up in volume after all.
  3. Your team is collectively not experienced enough to see mistakes before they kill you.

Your idea wasn’t that good, you didn’t have the capital, or you didn’t have the experience. That’s 99% of why all businesses fail. Yet those reasons are almost never in “Why My Startup Failed” articles except as hidden subtext. The stories Silicon Valley most likes to hear about itself are stories about why outliers aren’t outliers—why anyone can move right out of college into founding the new Facebook or Google. And so when we fail, we tell ourselves stories that don’t disrupt that myth. It’s absolute heresy to suggest that real world experience often outweighs youthful energy and a degree from Stanford, but most of the “mistakes you should never make” aren’t mistakes someone with the appropriate work experience would have made.

If your startup fails and it helps you to write about it, write about it. But don’t write about it because you want to prove to the world and future investors that you’re a cool guy after all. Write about it brutally honestly. Get it out of your system. Then don’t put it online. We love you, but we’ve heard it already. Next time hire an accountant.

Read the whole story
rafeco
30 days ago
reply
Yep.
acdha
29 days ago
The part I'm still surprised by is how similar this was to the .com bubble – I know the tech industry doesn't think long-term but I'd have expected the memory to last at least a decade.
superiphi
29 days ago
As long as there is so much money around with nowhere else to get invested...
acdha
29 days ago
@superiphi: that's one of the best arguments for breaking VC out of its nerdy white guy focus – there's no way the diminshing returns we've seen on startup ideas represents the sum total of things humanity could build…
Share this story
Delete
1 public comment
pfctdayelise
29 days ago
reply
Love this, but I think people should still keep writing about failure, if only to try and counter the shiny continually promised by VCs and other startup wankery.
Melbourne, Australia

The philosophy of great customer service

4 Comments and 20 Shares

I was honestly surprised that my company, CD Baby, was such a runaway success. But I was even more surprised to find out why.

CD Baby had lots of powerful well-funded competitors, but after a few years they were all but gone, and we dominated our niche of selling independent music. 150,000 musicians, 2 million music-buying customers, $139 million in revenue, $83 million paid directly to musicians.

What was the secret to CD Baby’s success? I never did any marketing. Everyone came by word-of-mouth. But why? I honestly didn’t know.

So whenever I was out talking with my musician clients, I’d ask them. For years, I asked hundreds of clients why they chose CD Baby instead of the alternatives. Or I’d just listen as they’d rave to others nearby about why they loved it.

Was it the pricing? The features? Nope. The #1 answer, by far, almost every time someone raved about the company, was this:

“You pick up the phone! I can reach a real person.”

They called and got a real person on the 2nd ring, instead of an automated call-routing system. Or they emailed and got a surprisingly helpful personal reply, instead of an impersonal scripted FAQ response.

And that was it. Who could have guessed? That despite all efforts put into features, pricing, design, partnerships, and more, clients would choose one company over another mainly because they liked their customer service.

I structured the business to match this priority. Out of 85 employees, 28 people were full-time customer service.

Since then, many entrepreneurs and interviewers have asked for my customer service tips and tricks, but I recently realized it’s not something you can add on top, it’s really a philosophy - a mindset that has to come from the core.

I’m no expert on the subject, but I’ve learned a few things from 16 years of experience, so here are the 6 key mindsets that I think guide great customer service:

(Actually, I prefer the term “client care”, since “client” implies a relationship, instead of “customer”, which is transactional. But I'll use the normal term instead of confusing things by using mine.)


#1: You can afford to be generous

The #1 most important mindset to start with, underlying everything, before engaging in communication with a customer or client, is that your business is secure.

Even if it’s not, you have to feel that it is. Money is coming your way. You are doing well. You are one of the lucky ones. Most are not so fortunate. You can afford to be generous.

All great service comes from this feeling of generosity and abundance.

Think of all the examples of great service you’ve encountered. Free refills of coffee. Letting you use the toilets even if you’re not a customer. Extra milk and sugar if you need it. A rep that spends a whole hour with you to help answer all your naive questions.

Contrast it with all of the bad experiences you’ve had. Not letting you use the toilets without making a purchase. Charging an additional 50 cents for extra sauce. Salespeople who don’t give you a minute of their time because you don’t look like big money yet.

All bad service comes from a mindset of scarcity, feeling like they’ll go out of business if they don’t fiercely guard their bottom line.

They say the reason those in poverty so often stay in poverty is that short-term thinking of desperate survival doesn’t leave room to think of long-term solutions.

If you really feel secure, abundant, that you have plenty to share, then this feeling of generosity will flow down into all of your interactions with customers. Share. Be nice. Give refunds. Take a little loss. You can afford it.

Of course it’s also just smart business. Losing 10 cents on extra sauce can mean winning the loyalty of a customer who will spend $1000 with you over the next 10 years, and tell 20 friends that you’re awesome.


#2: The customer is more important than the company

Think of a time where you had to make a big decision. For example, the choice between a job that pays more money versus another that pays less but gives more freedom.

Do you remember how it felt when you were conflicted between these two choices? Weighing pros and cons, going back and forth?

The way you resolved this was to finally decide which value was more important to you. For example: more money or more freedom.

Most of us don’t decide which value is most important to us until we’re forced to make this decision.

But if you want great customer service, you need to make this value choice up-front, and decide that your customer’s happiness is your top priority, above company profitability, then make sure that everyone in the company knows this and acts upon it.

You can’t micro-manage the details of every possible scenario, so make sure everyone in the company knows that whenever they have to make decision about what is the right thing to do, always do what’s best for the customer, what would make them the happiest, and don’t worry about the company. The customer is more important than the company.


#3: Customer service is a profit center

Companies put so much energy into sales - getting people to buy - but they don’t put as much effort into the customer experience after people buy.

Anyone can see the reason to focus on getting customers to buy. It’s obvious profit. But it takes some wisdom, experience, and long-term thinking to understand that keeping your existing customers thrilled is even more profitable.

Customer service is not an expense to be lessened. It’s a core profit center, like sales. It’s something you put the best people on, not the cheapest.

You’ve heard the old business truism that it’s 5 times harder to get a new client than it is to get repeat business from an existing client, so this is where you put it into practice.

Hire the sweetest most charming people and make sure they have all the time in the world to spend with your clients, making sure they’re so heard, and so happy with your service, that they’ll tell everyone they know.

Hire enough people so that they have the time to pick up the phone, instead of routing people into an automated system. If they’re so busy that their communications are getting too succinct, it’s time to hire another. It’s worth it.


#4: Every interaction is your moment to shine

Probably only 1% of your customers or clients ever bother to make a customer service interaction.

So when they do, this is your time to shine. Three minutes spent talking with them is going to shape their impression of your company more than your name, price, design, website, or features all combined. This is your shining moment to be the best you can be, to blow them away with how cool it was to contact you.

If your customer service is taught to be efficient, it sends the message, “I don’t really want to talk with you. Let’s get this over with quick.”

Since that’s what everyone else does, do the opposite. Take a few inefficient minutes to get to know anyone who contacts you.

For example, at CD Baby, if someone would call, saying, “I’d like to talk with someone about selling my music through you,” we’d say, “Sure. I can help. What’s your name? Cool. Got a website? Can I see it? Is that you on the home page there? Very cool. Is that a real Les Paul? Awesome. Here, let me listen to a bit of the music. Nice, I like what you’re doing. Very syncopated. Great groove. Anyway... so... what would you like to know?”

I can tell you from my own experience of being a self-promoting musician for 15 years that it’s SO hard to get anyone to listen to your music. So when someone takes even a couple minutes to listen to you, it’s so touching that you remember it for life.

This isn’t some sales technique, it’s just good human behavior. It makes life better. It makes work more fun. It’s the right thing to do. And it pays off.

When people would call to buy music, we’d ask them where they heard of the artist, not in some monotonous scripted way, but as part of engaging the customer in a little conversation, sincerely interested in the details, maybe asking if they often discover new music that way, or whatever. Then we’d include these details in the order on the backend, so the musician could see it, too. It helped the musician be more connected to their fans, and helped both them and us understand why people were buying music.

Imagine what you’d do if Paul McCartney called. You’d drop everything, gush some praise, be thrilled that he’d contact you at all, and give him all the time in the world for whatever he wants. So that’s how we should treat everyone that contacts us. Why not? You don’t have time? Make time. It’s how everyone deserves to be treated.

You know there’s research that says that we don’t smile because we’re happy. We smile first, and the physical act of smiling makes us happy. So I think the act of acting your best, being sincerely interested in others, taking the time to make each person happy, even if you weren’t in the mood at first, is a great way of actually being your best.


#5: Lose every fight

Customer service often starts when someone has a problem, and is upset.

But kind of like you need to feel secure for your business to be generous, you need to feel secure enough to lose every fight.

Whenever they’re upset, let them know that they were right, and the company was wrong. They win. You lose. And you’re prepared to do whatever it takes to make them happy again.

I’m saying this, but let’s admit that it’s so hard to turn off our human nature to feel things are directed at us - to lash back, and show them they’re wrong, to not lose this fight. Occasionally, still, I start typing a response that’s not so nice, but after years of getting burned for doing that, I catch myself, and replace it with something angelic instead.

But you know that scene in the movies, where someone is saying something nasty or secret, and then realizes their microphone is on, so they immediately straighten up, correct themselves, and say the publicly-acceptable thing instead?

Well, your microphone is on. There is no private communication in customer service. Anything you say is likely to be put onto someone’s blog or Facebook, retweeted, and seen by everyone.

So you must be the best version of yourself. You must let them win every fight. You must humbly bow to your superior, and make them happy. And kinda like I said about how smiling makes you happy, I think the act of doing this every day is very peaceful. It feels like daily empathy practice.

Over the years, my company had some huge evangelists: people who loudly told everyone they met that they absolutely must use CD Baby to sell or buy their music. Funny thing is, when I’d look back through that person’s history of communication, I’d often find out that the first time they contacted us, they were loudly upset about some problem. I think the lesson learned is that loud people are loud people, whether complaining or praising, so when you get some loud complaint, take it as an opportunity to do whatever it takes to make them so happy that they become a loud evangelist.


#6: Rebelliously right the wrongs of the world

You know there’s this little passive-aggressive move we all do, when we don’t like how someone is behaving, we instinctively “take the high road” to show them how to behave?

Like if someone is talking too loudly in a quiet place, you speak extra-quiet to them. If someone is being a complete slob, you clean up your zone before confronting them.

It’s a kind of defiant act that says, “No. You’re doing it wrong. Here. Watch me. I’ll show you how it’s done.”

Well, I think your business is your little part of the world where you can right all the wrongs of the world, and show them all how it’s done.

To do this, you need to be rebellious. Don’t follow norms. Don’t do what the other businesses are doing. Instead, think of the worst you’ve experienced, and do the opposite. Show them how wrong they were. It’s very cathartic.


What do you think? Did I miss any important ones? I’d love to hear any suggestions.

girl on phone by Mark Roy
(Photo by Mark Roy.)
Read the whole story
samuel
30 days ago
reply
Best part of this post: "I think the lesson learned is that loud people are loud people, whether complaining or praising, so when you get some loud complaint, take it as an opportunity to do whatever it takes to make them so happy that they become a loud evangelist."
The Haight in San Francisco
rafeco
35 days ago
reply
Brilliant guide to customer service.
Share this story
Delete
1 public comment
cjhubbs
35 days ago
reply
Principles for great customer service. Awesome.
Iowa
Next Page of Stories