
30 years of Internet transfers with curl
Thirty years ago, on November 11, 1996 the young Brazilian student Rafael Sagula posted the source code for his tiny HTTP download tool online. That was httpget 0.1: a little over 100 lines of C code. Bare bones source …
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Thirty years ago, on November 11, 1996 the young Brazilian student Rafael Sagula posted the source code for his tiny HTTP download tool online. That was httpget 0.1: a little over 100 lines of C code. Bare bones source code with almost no error checks and limited functionality.
When I found the source code of that initial httpget release just days after Rafael put it online, I had no idea how HTTP worked. The language to speak over the network to ask for resources. Okay, I already knew we call it protocol but not a lot more. I also had never heard of neither httpget nor Rafael before. I imagine that I found httpget by searching on Altavista since this was before Google, but I can’t recall the details.
Seeing the simplicity of this code it was not difficult to immediately spot the fundamental basics of the protocol. I presume maybe in the same vain how countless others in decades to follow also have thought of it when seeing the details for the first time. Much later I would come to understand that to be an illusion of simplicity, but in those early days I was more amazed and fascinated. I already enjoyed doing network programming and HTTP reasoned with me at once.
httpget 0.1 was almost completely without error checks and it used plain IPv4 addresses; no hostnames or URLs.
I, still a naive and enthusiastic developer with an Open Source mindset (even though that term would not be established until 15-16 months later), addressed some immediate bugs I ran into and added name resolving to the tool. I made diffs and emailed patches over to Rafael; the standard way we exchanged code changes back then. I knew about RCS and CVS for version control, but the use of these tools was spotty. It would actually take until September 1999 until I adopted CVS fully for the curl project. We later switched to using git in 2010.
The average internet connection speed of a normal household in Sweden, where I lived and I still live, in 1996 was 30 kbps. 30 kilobit per second. This was before v90 came and bumped the modems up to 56 kilobits. Ordinary mortals connected to the internet using modems over their single phone lines. Today, the average Swedish connection speed is 200 megabits/second.
Thirty years ago there were just 36 million global Internet users and 250,000 websites. Compared to the 6 or 7 billion users and 1 billion websites of today.
We used simpler protocols and while encrypted protocols like SSL existed, they were not used a lot and only for a few rare sites.
In November 1996, my main home computer was still an Amiga 3000 powered by a 25Mhz 68030 CPU but in early 1997 I upgraded to a faster and more modern machine: an Intel Pentium 120Mhz. I ran Windows 95 on this thing, and its prime use was to run an X server so that I could connect to my work computers and pop up X11 terminals at home. I built the first versions of httpget remotely on SunOS, Solaris and Dell Unix – because they happened to be what we had running at work.
I installed and tested Linux for the first time at work in 1997. I got our first public web servers up and of course I made sure curl worked fine on it. I did not switch to Linux on my home machine until 2000 when I also updated to an Intel Pentium III 733 Mhz box.
In 1996 I was working as a software developer doing embedded systems and Unix development in C, for a small Swedish company called Frontec Railway Systems. While working there, I came to learn about and enjoy IRC, and in my spare time a few friends and I developed an IRC bot we called Dancer. Spare time developing a TCP client written in C.
As the IRC bot slowly grew more capable, I wanted to use its extension capability and provide a currency translation service for the participants in the channel (the IRC term for a chat room). In an international channel like ours, we frequently discussed the price of things and having the bot able to quickly explain how much a hundred German Marks equaled in US Dollars could be a useful service.
My currency service needed to update currency rates maybe once per day so that it could use decently accurate rates. I found an HTTP page listing a bunch of currency rates and the httpget tool for doing the downloads, and then the bot could offer up-to-date currency rate translations. All was good.
I discovered computers and programming as a teenager on a Commodore 64. I got hooked immediately. I first learned Basic and moved on to 6510 assembler pretty quickly. In the late 1980s me and my friends had demo groups where we would write cool demos (and some game attempts) in assembler (under the group name Horizon). I established my habit of spending spare time on programming back then.
In the early 1990s I moved over to Amiga, started writing a text editor, got my first job at IBM where I truly got into a lot of C and Unix for the first time. Together with some friends we ran a BBS (The Holy Grail), for which we eventually would write our own BBS software; me and my brother Björn. I learned Motorola 68000 assembler on the Amiga. I wrote 8086 assembler on PC.
I never attended university. I got my first computer job at IBM when I was twenty. I then thought I would take the uni route later, but I never did. I remain uneducated.
In the early 1990s I discovered both the Internet and the world of freely available source code and I was hooked.
Rafael was not terribly keen on dealing with my patches for httpget. I believe he mostly just put his code out without any particular desire to do anything more with it. Instead I became the httpget maintainer and kept polishing it. My main project was Dancer, the IRC bot. httpget was just a side-project to make sure the currency service kept working fine.
At some point I fell over a gopher site that provided many more currency rates than the HTTP page I was scraping at the time so of course I had to add support for that as well to the download tool. With gopher support added, it was not just http anymore and the name of the tool was no longer accurate. It became urlget. A third listing was found that provided rates over FTP and, yes, I added support for that protocol as well.
Already from the start I made the source code available under a free license, continuing what Rafel kicked off. He hosted the first releases, but during 1997 I took over and hosted the tarballs on the website I set up and administrated for work. I told my friends and the news about the tool slowly spread. I started getting bug reports from users and occasionally someone sent me patches to fix problems or add features.
This was a time of many Unix flavors and as I wanted widest possible use and adoption, I knew that I needed help and assistance from friends to make sure it built and ran fine on all those different systems.
In the beginning of 1998 I added upload support to the urlget tool (both FTP upload and HTTP POST) and again I realized the name of the tool was wrong. I decided I should try a name that contains the word URL, that can be pronounced in English and that is easy (and short) to type on a keyboard. cURL it was.
The leading ‘c’ could be short for client but I also liked that the name could be pronounced as see URL, as the tool would display the resource at the URL. You would see it (in a terminal). Naming things is hard.
curl 4.0 was released on March 20, 1998. By then it had grown to 3,300 lines of code and it featured 24 command line options and it spoke three protocols.
Gradually I lost interest in the IRC bot and the currency service but I stuck with curl. People used it, I got feedback, it improved.
I started and participated in several other Open Source projects, but curl remained my main baby and focus.
The concept of a function library that offers useful functionality to other programs is a concept I learned already on my Amiga days. As an example, I wrote the scripting language (FPL) for our Amiga text editor (FrexxEd) as a library, and we could later use that language for our BBS software, FrexxLink (which ran on OS/2) as well as in Dancer, the IRC bot.
A library is sort of next-level cool: not only is the source code available for everyone, but the functionality is as well; in an easy-to-use way.
The idea of converting curl into a library was already in the back of head from the start and in the summer of 2000 I refactored the internals and in August that year we introduced the first ever libcurl. Maybe someone else wants to do Internet transfers in their services or tools? Turns out some did!
One of the first adopters of libcurl was the PHP language which almost immediately decided that libcurl would become their default internet transfer engine. Being a key component and therefore part of their success journey helped getting users, getting testing, bug reports, feature requests and making the library solid.
curl is a child of the 1990s. Back then there were not a lot of options and language alternatives if you wanted to write a program that would easily build and run on most available operating systems and CPU architectures. Sure, C++ maybe could have been an option but I never liked it – and I still don’t. I’m also a strong believer in continuous and non-stop improvements and iterations; not rewrites. curl was never rewritten and if I get my way, it never will be. We do however repeatedly refactor and rearchitect internals to adapt to a changing world.
I wanted curl to be usable by everyone on every machine so it was designed and written for that purpose from the start. The source code in a portable way and the license in a liberal way.
We did most things wrong at the beginning but we learned, adapted, tried again, iterated and we kept doing release after release. Gradually we learned and did better. Every release with lots of bugs fixed and often a few new features. I kept spending my spare time on curl. Every day, every week, every year. Given enough time, something great can grow.
I did my first twenty-three years of this journey with curl only as my hobby. While curl happened, life also happened in parallel. I have switched jobs many times. I got married, bought a house, got two kids, and saw the kids grow up to be amazing adults. Somewhere in there I also managed to convert my hobby into my full time job. Living the dream.
curl is my life. My life is curl. I have been invited and able to speak to audiences hundreds of times in a dozen countries. I have received multiple awards and even gold medals straight from the hands of royalties for my open source work.
I don’t have an end goal with curl and I never did. I just want it to do internet transfers the best way possible: fast, securely, using the proper protocols in optimal ways. There is no rush towards anything. It is just a non-stop walk forward with never-ending polish and tightening every bolt there is.
Thirty years have flown by and yet the development pace is still intense. We regularly implement new transfer protocol features before anyone else. We have leading implementations for lots of network solutions. We ship code that carries a huge portion of the world’s internet infrastructure on its shoulders without fuss.
I still enjoy building things. Writing code to create something that did not exist before; and maybe even more so to make the program actually do things users want and like. I enjoy fixing bugs partly for that reason, but I also like the puzzle and the challenge of debugging and solving tricky problems. Designing a software architecture, coming up with APIs and figuring out how to provide powerful network transfer functionality to power billions of devices are things that make me eager to get started again every morning – still after thirty years.
It never really gets done. The world of the Internet and its transfers keeps changing. New protocols, new versions, new flavors, new third party libraries, new build tools and environments force us to constantly change and adapt.
I want curl to be a leader in internet transfers, its
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。