|
|
Cotse - Brief History - Spring 2004
Every year or so I write a brief history of Cotse to date. Considering we are nearing our five year anniversary, it's that time again.
The Old
I guess what lead up to becoming Cotse all started back in '91-'92, I started a site called The Security Garage. It was a simple site (back before frames, I think the blink tag was new) that hosted system and network security information. Nothing really special happened except some other later well known sites websnaked it and used it as a base for their start.
In '95-'96 I moved the security garage to a dedicated domain and changed it's name to Gielda's Garage. Still nothing special, just a collection of reference material. That got popular, well popular enough to go over the cheap base rate of the webhost, and started costing too much. So in '98 I used it as an excuse to get the wife to go for me adding high speed connections to build a home data center.
It wasn't easy convincing her that a $1000 / mo bill was a necessity (In '98 those without early cable in their area were out of luck for broadband because of the price and not only did you get billed for the connection, you got billed by traffic). It was my best performance to date, I successfully argued that remote hosting was more expensive in the long run and that it would be better to pay far more now for our own pipes. I got my broadband.
Something is born
Moving the Garage to a completely new setup somehow lead me to believe that it also needed a new name. I wanted something with a hook to it that would be remembered. There were many stupid ideas being tossed around and eventually I came up with either the stupidest or the smartest name for a website at that time, "The Church of the Swimming Elephant" one night on irc (story here). It was certainly memorable.
[Cotse is the website, we use Packetderm, LLC as our professional name.]
For the domain name we shortened it to Cotse (I later found out the dutch got a kick out of this). March 25, 1999 the line went live, on April 1, 1999 the first web server went up, a P200 with 64 meg running Solaris x86 and apache. I was on a six month contract for BBN Planet/GTE Internetworking at the time, so money wasn't bad. Plus it would give me the opportunity to develop more than just a web site, full control to easily add anything I wanted in terms of equipment was a definite plus.
Using the Garage as a base a group of us put together the Cotse.Com web site as an online reference. Everyone involved was some one I met in my consulting travels. That gave us contacts in most larger tech companies and we built for that audience, the admins. Within three months that server was flattened by traffic, but it did manage to hold on up to 150k hits a day with heavy cgi. We moved to donated Sparcs and Ultra Sparcs. We built a big "flea market" of security information and traffic just kept coming. It was starting to get expensive to support. So when we hit around 1 million hits a day we tried advertising.
Big mistake. First we were at the tail end of the ad rage and second I didn't then know how the advertisers planned for it to work. They basically planned on getting full name recognition off our traffic for free and they worked out their setup so that is what they got. They required control of the amount delivered and they controlled how much they'd pay for that. It was a recipe for disaster for the site.
Advertising for Pennies
We'd deliver 1 million views and we'd get this back from them: "We're sorry, according to our stats you only delivered 10,000 views and of those views only 1000 were unique. We only pay unique views and here is your $45 for those views." Huh? They had thousands of excuses as to how those numbers were true, and some were quite rational, but I had my own logs too and I knew how many times the image actually got pulled.
They were effectively getting their name and product plastered all over in front of peoples eyes and were only willing to pay $45. Right. Then to top it off we were approached by a few bottom feeders who wanted to trade us our site for promises. One company named Netwhirl wanted us to sign over our site and then they'd fill it with advertising and pay us $ridiculouslysmallamount per month. Sure that makes sense, give up everything we built for nothing but a promise. I blew them off and later watched them rename then blow up when chat logs between the president and VPs hit the net showing discussion of them scamming webmasters.
From Advertising to Security
About this time we were drowning, cost of traffic and site support was growing, Cotse was hosted across a global load balance with Genuity, and there was no income. I was draining our bank account and my wife was getting antsy about that. I dropped the advertising completely and removed some of the high traffic pages we had to chop back traffic. We almost drowned, but late summer in 1999 we had some security work fall into our laps.
We were certainly good at security, it was what most of us did in our real jobs anyway. It blossomed into us working out a niche where we partnered with physical security and investigation services. Many of their customers had a need for what we provided and we needed their sales force. It was a good match.
We were getting enough jobs that I could not work and do Packetderm too, both were suffering, so I jumped to Packetderm full time to focus on building it. It worked out well for a little while. From the late summer of 2000 to the spring of 2001 we were earning our keep doing security work. Local and remote penetration testing, system and network forensics (We were quietly pulled into some of the big compromises to reconstruct), and security policy consulting.
A Rude Awakening
During this time, we had a free e-mail service and anon web proxy going. It was nothing special, but it hid IP's, allowed posting to usenet anonymously, provided web surfing and mail, and it had grown to 13,000 users. Being a free privacy service and easy to sign up meant idiots signed up (offer something and someone will always be there to abuse it). One morning after three days straight working and 3 hours sleep the doorbell rang. I got out of bed, trudged out to the door still asleep, and peered out. Well it appeared I was now standing in my underwear in front of the Secret Service, FBI, DEA, and others in suits with badges.
Yeah, this was just what I wanted to wake up to, straight from my dream of being on vacation in Hawaii surrounded by topless hula girls holding plasma displays to the harsh reality and humorless personality of black suits and badges. And I was standing before them hair a mess, unshaven, in my underwear, and looking like I just came off a four day bender. Good god I was going to be on COPS. Fortunately, this was not the case.
Was I aware that someone had used my service to send death threats to the President of the US, numerous FBI agents, a DEA agent and his family, and sent a bomb threat to the Bomb Data Center? Well, no. But I am now. This taught me not to be so stupid and block those addresses. They never did catch him, but I always keep my eye out, I'd still like to return the favor.
Moving On and Up?
Things were moving with Packetderm. We put together a business plan that involved us doing security consulting, releasing some of our home developed tools as commercial offerings, and beginning a subscription privacy service. We had potential, we had a solid history and a good plan. We took that plan and flew to Chicago to attend a garage.com venture capital conference. We polished our presentation skills, chatted with VC houses and there was interest from some well known ones. We flew back on top of the world, things were flowing and it had been lots of fun playing CEO at that conference. This was spring 2001.
Woohoo, buy out by Compaq anyone?
But things took a turn within a month of our return, ironically it was right after we met with Compaq about a potential partnership. It appears that links to our site kept floating around their security services division. Plus we'd covered a few big names in doing security work. They decided to look closer. We met and they discussed potential partnership for a year while they evaluated purchase. The problem we hit was that they wanted us to guarantee a staff of twenty five. We didn't have 25 people, plus the people we did have were remote.
Up until this point we'd been doing a lot of work overseas for big banks, some in the states, but not a lot of it focused in the Boston area. We just didn't have local staff beyond myself and our CFO, and of the two, only one knew security well. We might have been able to get it had we gone for venture capital, but I had second thoughts about that route. As I pitched investors I was seeing a common theme, they required that you basically hand over your company for money to build it. I was not interested in building something just to be booted and replaced in a year or two. This was what most VC houses were doing to founders. Since no financing meant no staff, we couldn't do the compaq partnership.
Which turned out to be for the better, a few months later they merged with HP, everything goes to hell in a handbasket in that merger for a little while. This would have placed us in a funded position beholden to making it with Compaq just when the managers would all be be dropping new projects and playing "cover your butt and hope not to get laid off while updating your resume" games. We would have ended up hanging in the wind.
Oops false alarm, go back to your regularly scheduled poverty
Of course we didn't know that at the time. It was a bit of a blow to morale to have to push back on that. Especially because we were pumped about the Compaq deal, it was the ticket. But, we had to wave bye to it and that hurt. Everything started going south at this point. Our CFO bailed and the majority of the staff left became apathetic.
On top of this the market dove, companies stopped spending, security was overhead, they preferred to just quietly cover up compromises and quickly rebuild the servers, our work dropped to levels where it wasn't going to keep things floating. Seemed like everyone was going bankrupt. This could actually help us, if we could just hold on, maybe all our competition will go bankrupt. Didn't happen, but many did.
So, another change in focus. Cotse.com was now costing real money to run and our income was virtually non-existent again. We were floating on savings from the security work, but they were not going to last long. The security part of the business plan was effectively dead, we were left as a few techs with no sales talent, kind of where we started. That left commercial offerings of our tools and the privacy service.
We had some cool tools that should have done well. Winetd was a full blown inetd with a twist for windows. Prior to honeypots being fashionable it allowed fake services to be run in addition to real ones. We included a number of them in the release as well as the skeleton for people to easily develop others. We set it so you could basically duplicate any OS and any service you wanted plus monitor/log everything the attacker did. There was even the possibility of a virtual file system once an attacker "cracked" through telnet or something else you set. It would even do full information gathering on the attacker while he thought he was exploring some machine. The plan was to add firewall features as well as IDS capability. I think that would have been a great tool, don't you?
We also had other tools, like a fully "administer by mail" web server, where people could send documents to it and have them categorized and placed on the web. Great for nearly every department, it was instant intranet where even a secretary could easily mail a document to the web. I liked that, full layered access control, full template control, and more. We'd also developed a security test management tool. It was good for managing a large security evaluation that had many people working on different tasks. Much of these tools we'd built to help ourselves manage Cotse and the security work. We only had one problem at this stage, we were broke and we did not have coders to change these tools to a commercial focus. Lack of money always kills my good ideas. So it left me with the privacy service.
The Privacy Service
The problem with the privacy service was the fact that my vision for it required a team of programmers, and that I did not have. But everything else faded and I was then unemployed with no income and nothing to do. The job market was drying and no one was hiring, which made matters worse. So having nothing to do but code, I figured a slow building of the privacy service offerings was better than none and started building. I spent April - early July coding and putting together the base for Cotse.Net privacy services. I never left the house, woke to sit in front of the computer and stayed there till bed time.
By June the fact that I was going to run out of money by August was pounding on me, I wasn't finished with the privacy service and it would not be finished by then. I was rapidly approaching screwed. So I just took a risk and put what I had done live and started it as subscription. It went live in its base form on July 4th 2001. That date wasn't planned, I actually didn't really realize it was July 4th till the next day, I just had no life at this stage and was not out at any 4th parties, but instead home on the computer working. Surprisingly, people signed up. I kept coding and building, more sign ups came in. It extended my out of money deadline a little but did not eliminate it. Then a run of bad luck.
Hardware gnomes after we go subscription. A SCSI card goes, but goes in a way where it takes out 5 drives in succession first before it identifies itself as the problem. Ethernet cards die, a router dies, a firewall goes, then a motherboard. Good god, when will it stop. Hardware failures kept taking us out. I checked power, network, everything I could think of that may cause this, nothing. It had to be gnomes. I played loud anti-gnome music when working on the systems and in October 2001 I took a job.
Back to Work
I basically worked two jobs for the following year, commuted 120 miles a day to a Solaris Admin job and then came home to spend every moment I could coding for Cotse. This job's first day consisted of: "Here are 7 netra's, 3 sunfire 280r's, a 420r and assorted network devices, including CSS11000's, SSL Accelerators, and snmp network monitoring tools-including remote cam, This needs to be built and live by Friday (It was Monday). I was the only one there with the exception of the manager of IT that knew Unix at all. This was a MS shop that had just sold a Unix project.
It was to be a licensing setup with load balancing from web, through middleware, to back end database and very high traffic. What a way to start a job. I did get it done. Then I got to watch it sit idle as contract issues came up. What a waste of effort, I hate corporate BS. During this time John was working on documentation and building a helpdesk. We were going to need good support if this was going to go. John turned out to be invaluable. He would always be the first to call if there was a glitch.
During our gnome infestation he had been calling so frequently that he feared my wife, probably because she always answered at every hour and he was concerned she was getting angry with him. I'm not sure, but perhaps my daughter pulled one of her color the dog-walls-self bits and my wife unleashed at her when he was on the phone, that could scare anyone...except my daughter. But even with the help my work started suffering. We had enough users that uptime was critical, support took time, and a few times I had to leave at my job to ride 60 miles to fix something on Cotse.
I was also staying up for days at a time between supporting my job and Cotse. I started falling asleep at work. Companies don't put up with this for long and wouldn't you know, they had to cut back. Surprisingly I managed to stay alive through the first couple cuts. I was doing more network and Exchange work then, managing the PIXs, troubleshooting strange mail problems, put together a filtering mail gate for them, a centralized AV update server, moving them from radius to tac_plus, and other odd jobs when I was awake. But soon enough the contract squables ended and they lost their one Unix customer, the one I was brought in to build the licensing setup. It wasn't writing on a wall, it was skywriting 300 feet high. I knew it was coming, I shifted full focus to Cotse and waited. A couple weeks later it came, I was laid off.
Back to Cotse
I looked for more work, Cotse was only supporting itself and growth. No one was hiring. I kept it going basically on autopilot for a little while, it continued to grow. I later was able to further develop the back end and add services. I put together load balancing as load grew, full automation of administration, and self monitoring and repairing servers. On the administration side I built it so that even if a machine crashed with a hardware failure I only had to watch my pager and 6 - 8 minutes later the service came back on the hot spare machine. Power for each system was set up to be network controllable. I set it up to hot swap providers should a line go down, automatically failing over. (I don't want to be burned again). This gave me time to focus even more on new services.
I rebuilt the proxy and moved from strictly CGI to a full custom set of anon proxy servers. I added usenet and more. Our e-mail was great, but it wasn't exceptional, my goal with Cotse.Net is to offer better services and more features at a comparable or better price. So I pulled Dave in and explained what I needed there. Dave was an expert at text processing and though he didn't understand e-mail at the time it was basically text processing at it's finest. I worked with Dave and he put together exceptional e-mail processing. This made our e-mail second to absolutely none. An average of 10 a day were subscribing. It was moving.
What's life without commotion?
Things were purring and I'd been working constantly for well over a year so I decided to take a vacation with the wife. The family owns a cottage in RI and we headed there for a week. We planned on using it as base for touring the RI seashore on my Valkyrie. I was looking forward to it. It rained. There was only sun two days. Guess what happened those days?
5 am. I'm almost ready to wake up. Kandi (my wife) and I had watched the weather the night before and made plans to take the bike up to Point Judith and then out to Block Island on the ferry. [This would have been redemption for our last trip to Block Island. I didn't have the Valk then, we were going to rent mopeds. Problem was when we got there, she didn't want her own. So we rode double on this poor moped. For note, I'm 6'7" around 300 lbs. and am mainly some strange biker/techie hybrid. I alone can kill a moped. But now I had a passenger too. There was no way around it, we were a funny looking sight. People pointed and laughed. I waved like it was a one man parade. It was all I could do and fortunately I knew no one. Always a plus in situations like this.] Well, this trip with the Valk was to be redemption for the previous trip.
Damn, the phone goes off (ringtone - Flight of the Valkyries). I hate irony. Dave is on the other end, "Hey, I can't get to Cotse". I quickly try to connect, I can't either. I hang up with Dave and call our provider. "No, no trouble in that area that we are aware. We'll check and call back." I call back, "we can trace it to your loop, it's somewhere there." "Crap." I tell my wife I have to go and jump on the Valk. 100 miles later and in a time that I would be ashamed to quote (lets just say I burned an entire tank of gas and was well into reserve in that time) I arrive at the servers. They are all purring, lines are down. I call my provider again, they start on it. No one can see remotely why it would be down. They need to send someone out. They'll see what they can do.
A couple of hours go by, I call again. "Everyone is at a company meeting, we're paging people to see if we can get someone out there. We'll call you back." Few more hours go by, it's afternoon and I take some of the downtime to work on servers. They call back at three. "We talked to a tech, he's seeing what he can do, we'll call you back soon". It's now 4 pm, I call back. "Who? Let me look" I'm on hold, "Yeah, apparently no one can make it today and they scheduled for tomorrow afternoon between noon and four". Uggh.
Nothing I could do. Sitting in front of dead servers with no net access all night did not appeal to me. So I jumped on the Valk and raced 100 mi back to the cottage. We scrapped our Block Island plans, but figured that they'd find the problem and be able to quickly fix it. Perhaps a broken wire, so at worst I should be back for an evening ride up the coast for seafood. Wrong. I get back at the servers at 10 am to be early for the potential noon arrival. Noon comes, no show. Ok, between noon and four they said. I wait till three and call. "Oh, the tech was out there at nine am and didn't see anyone so he left." "WHAT!?!" "YOU scheduled him for between noon and four, I've been here since 10 am" "Oh, yeah, looks like he had an opportunity in the am." "Can you get him back?" "Let me see, I'll call you back."
This is ridiculous, I call around other providers to check to see how fast I can move servers. No one can come up with space and a line same day and I hear a bunch of stuff about it being near 5 pm. Damn nine to fivers, I hate them. I call a buddy of mine, Hoss, he's the one I mostly ride with, and like me, he's someone people are always a bit surprised to find out is in tech. He calls his contacts. Found, someone can provision a T for us and can get it live tonight, but they need confirmation ASAP. I call provider back. "What is the status on the tech?" "We haven't heard back from him yet." Decision time, has to be made now. "Fine, then cancel my service, I'm moving."
I hang up and call Hoss, it's almost like a war call "It's a go.", I shouted into the phone. He's off. He calls right back, it's going in, call X for the IP info and he's heading over. I call, get all settings. I hang up and plug the laptop into the phone line, yuck, dialup. First to the registrars, change root DNS for the nameservers now so their TTL for the change overlaps move time. Then I'd head to scripting. I figured I'd blow reconfigure under pressure if i didn't script it, completely moving a network and servers takes thought. Plus I'd had a number of high speed runs between RI and here and little sleep. Stress was high.
So I map it out and then script it. Everything on all the different servers, a switch, and the firewall that is dependent upon the network IPs. I write one script that just logs into each in succession, changes everything needing changing, then downs the machine. I run it after finishing it, no bug check time and no time to check settings, must get servers in the truck and I'll deal with the problems I know will be there after moving.
Hoss arrives and we tear down the net and carry out the equipment. We drive three miles, carry it in, set it up, plug it in, and turn it on. Total time from pack-move-unpack-go_live, one hour. Everything good. Traffic hitting machines. Looks like script was flawless, lucky break, everything tested fine. I pack up and race back to beach. Get back around 8 pm. Take wife for short ride. Back to cottage. Rain the next days.
Later we found out it wasn't totally flawless, the web site www.cotse.com had problems. First with a duplicate IP because the new provider had created overlaping nets between us and another customer. Then we had a minor net mask problem with www.cotse.com because of a typo in the script for that machine, but the privacy service was a flawless switch.
Love the new provider
I love this service, anyone that can help me pull off what I just did is someone I'll stick with for some time. After all, I had effectively just up and decided to switch providers at 3:30pm, three hours later we were live in a new datacenter. But alas, twas not to be. One week following my arrival home I get woken up to a phone call. Main tech at new service, he's pissed. Screaming about a complaint he received. At the time of the call I know nothing, I had just woken to this screaming.
Later I found out that one of our customers was being harassed by an ex. So they added that ex's address to the deny filter. We allow a custom deny message, so this person wrote "Fuck Off" for the error message. The ex was a local dentist. The ex complained. It is important to note that this ISP was brand new and were billing themselves as the Community ISP. But, I had also explained to them before hand that we were a privacy service that championed freedom of speech. I even used the whole "protecting freedom of speech means protecting speech that though legal, may be offensive to some" bit. He said it would be fine, fine turned out to mean "flying into a rage" when he was actually faced with it.
Hate the new provider
So the tech is screaming. He's going off about swearing and how it won't be tolerated on his service. How he's a good Christian and won't stand for it and he goes on and on getting louder and louder not listening to a thing I try to say shouting over everything I try to say all in a single breath. You get the point.
He's just hung up on how there will be no swearing allowed. It's a subscription e-mail service and I'm not going to post a "No swearing" rule. Anyway I get a pause, it was long enough for me to say "Fuck you, I'm not going to police users language" and hang up. Ok, so it far from the diplomatic approach. But I'm not rational when you first wake me and when I wake to yelling, I'm not going to be in a very diplomatic mood. A rush to move servers again. In the mean time our original lines had been fixed. We move everything back. Total cost, two days downtime, $3500, eating a little crow with a provider we'd canned, and one vacation...just to go back to where we were.
Today
From there things settled down. The systems were purring and self maintaining. Growth was steady. Failover seemed to function properly. I was a happy camper and focused on running Cotse from my bike. I signed up for wireless broadband with Verizon for the bike. I used an iPAQ mounted on the handlebars as a head (Photo: Unit also does GPS and has a 1 gig CF card I share between maps and over 400 mp3s. It plugs into my helmet and I get music, directions spoken to me, and audio/visual alerts of server problems.)
I also built the bike a 802.1x Access Point. This means I can take my laptop, ride to the beach, then as long as I am within wifi range of the bike, connect to the net from the laptop and VPN to servers (or I can just use an access card right in the laptop, but I wasn't going to let that stand in the way of me building a rolling access point. I can stream mp3s to those within wifi range). I rebuilt one mail server from West Dennis Beach on Cape Cod. I handled support issues from the White Mountains in VT. I'm a biker who finally figured out how to combine work with riding. Life is good.
Well today our service is second to none, no one offers what we do. I'm full time Cotse, we are still not rich, but our customers love the service and we are staying alive. We are growing steadily, we're redundant networks, integrated physical security with network, including the sending of real time frame captures from security cams on the servers to cell phone on motion detect to log access. Cotse is right on track to becoming the service I envision.
[On a very loosely related to security cams, but funny, note: One day I was watching a feed from a tripped cam at my house when nobody was home on tape. I watched one of my cats sneak across the pool table and jump on the dog, then hold on and ride it around like a dog rodeo. It was particularly amusing in time lapse. The slow motion effect was great.]
The steady growth allows me to systematically enhance our weak areas, and help to plan the features still not live, such as full vpn services. We're working on becoming a power player in this niche. With luck, we will soon appear on more people's radar as we move to offer a complete solution that really is second to none in features, price, support, and performance, at a very comparable price.
I didn't mean to turn the end of this article into sounding like a sales pitch for us, but I am happy with how far it's come. I also apologize to the individuals helping that I may not have mentioned. Those who help with the document writing, the privacy watch articles, the news, live support, and many dedicated and supporting customers. Thanks to all of you, we are still kicking. Thanks to all of you, we can grow to provide even more services.
/steve
Stephen K. Gielda
Chief Valk pilot, Admin, Mover, and Government Underwear Model
Packetderm, LLC
02/28/2004
|
|
|
 |
|
|
|
Protect yourself from cyberstalkers, identity thieves, and those who would snoop on you.
| |
Stop spam from invading your inbox without losing the mail you want. We give you more control over your e-mail than any other service.
| |
Block popups, ads, and malicious scripts while you surf the net through our anonymous proxies.
| |
Participate in Usenet, host your web files, easily send anonymous messages, and more, much more.
| |
All private, all encrypted, all secure, all in an easy to use service, and all for only $5.95 a month!
|
|
Service Details
|
|
|
|
