Etree.org Server Setup Guide
Tools of the Trade - Things you need to be a siteop
Before you begin life as a siteop, you need to do a little analysis to make sure that you have everything you need to successfully host an FTP site. I am going to assume that you will have read the etree.org Howto by now. If you haven't, go back and do it now. I would summarize these requirements as follows:
You need a computer with sufficient memory and hard drive space to serve out the shows. For memory, probably 96 - 128 MB would be a good number. People have gotten by with 64 MB, but you would probably notice a performance hit in other applications. For hard disk, buy as much as you can. Some people serve with as little as 4 or 5 GB. That is probably a good start. But it will only hold 4 shows in Shorten format. Some of the larger etree.org servers can hold over 50GB. I do not believe it is possible to have too much disk space, but you can try.
You need an upstream link of at least 768 Kb/sec to be successful. This means cable modem, ADSL or better. ISDN is way to small of a pipe, and even some cable modems and ADSL links limit upstream bandwidth well below this number. Remember that you will never reach the posted speed limit on your link, because of the overhead that you have to deal with, and you also need to leave room on your uplink for yourself to be on the 'net. Also make sure that you correctly understand the terminology applied to bandwidth. Kb stands for kilobits, KB stands for kilobytes. Kb usually talks about total bandwidth of a net pipe while KB is the measurement used in most file transfer programs. There are 8Kb in a KB. At 768 Kb/sec it will take about 4 hours for one person to download 1 show. The download time quadruples if you allow four users. It really takes a bunch of bandwidth to host a site. A good place to test your bandwidth and learn more about connection speeds and how it all really works is http://www.dslreports.com/stest.
- FTP Software
Make sure you use good FTP Server software and be sure you know how to correctly configure them. Below are some recommendations:
- Windows 95/98/2000/NT
Two good choices for FTP server programs for Windows 95/98/2000/NT is Serv-U and WAR-FTP
Download: Serv-U v2.5e (direct link - 1.4MB)
Download: WAR-FTP (download locations)
Both websites contain documentation on how to configure. You have the ability to limit the number of connections (both total and from IP) and bandwidth throttling, meaning that you can cap out how much total bandwidth the server will consume.
You wouldn't be running Linux unless you know a bit about what you're doing (one would hope), so here are a few suggestions on what software to use. Many of the etree.org FTP sites running Linux are using proftpd. It has some wonderful features like denying multiple logins and great user/group permissions. You can even run a separate password file if need be. The documentation for it is fantastic and it shows the percent done on a users file with ftpwho.
There is also wu_ftpd. That comes out of the box with most current Linux distributions. There are how-to's on the LDP.
Trey Howland has recently embarked on an incredible project for liunx etree.org users and siteops. He has started the Etree.org Linux Server Configuration Project. There are the beginnings of how-to's and documentation on how to configure a linux box to run a etree.org server. It will include a database of your users, webpage view of ftp contents and proftpd configured how you would like.
Jason Lunz has modified proftpd and created some good scripts to help the process of serving for etree.org under linux. Check out his site, http://falooley.org/linux/
Caleb Epstein had recently revamped the Linux SHN Howto including some great scripts he has written for using shorten and md5 under linux.
Although some etree-ers have successfully hidden an FTP site from an unfriendly ISP, having a good ISP makes things go much smoother. It is even worse if you are "borrowing" bandwidth from your school or company. You don't want to lose your Internet link or your job just to host some files. Ask politely before you throw up a site. If you aren't allowed to host, there are other ways for you to contribute. Seed shows, offer B&P's, spread the tunes. Don't get yourself in trouble just to help the etree.org.
Configuring Your FTP Server
Now that you have decided to make the leap and become a siteop, there are several configuration issues that will either make your life very easy or very hard. Both of the FTP servers I mentioned have pretty good online documentation, so read up on how to make these changes.
- Server Port
The default FTP port is 21. Many firewalls and proxies will not allow FTP traffic on other ports. Every FTP client I know of defaults to this port. "Stupid" users (not techno geeks) may not be able to figure out how to change the port. Also, if you are using another port to hide your FTP site from your ISP, it won't work. I have worked as a network administrator and I can sniff out your FTP site no matter what port you use. You will probably just get in more trouble since by changing the port to hide your activities it tells your ISP that you knew what you were doing was wrong in the first place. Unless you have a really good reason not to use the default port, leave it set to 21.
You should set your to between 3 and 5 minutes. I think this is the best balance. It allows users time to browse your site without being kicked off the server, and also to do things like create or change directories before they do a download. It is also short enough that if you run a popular site and someone does go to sleep on you, the slot is opened up fairly quickly. It also clears out interrupted connections so that machines can reconnect if their connecting craps out and your server thinks that they are still attached, even if they are not.
- Total Connections
You can usually set this per account and per machine. This is how you limit the number of people that get in to your machine. If you don't set this, the whole world will log in and everyone will get shitty rates. A good safe number to start out with is to allow 1 user connection per 100 Kb/sec of upload bandwidth. You can then adjust this from there depending on how hard people hit your machine.
- Connections per Machine
Set it to one. It keeps out the selfish. Every site I know of limits people to one connection at a time.
- Characters per Second / Bandwidth Throttling
This one is entirely up to you. If you have a hugely fat pipe to the net, leave it turned off. If you have a fairly restricted pipe, or you need to have some bandwidth left over to do your job or cruise the net, turn it on. You will have to set the values and play with them to strike the proper balance. On War-FTPD the setting characters per second translates to bytes per second. Watch your performance statistics. In NT I like to look at network utilization. On an Ethernet segment you want to keep it below 30% or you will start having problems with collisions. I am not sure what a good indicator is on a cable modem or ADSL link is.
- Allow Connections by IP
In all honesty this is a bigger headache than it is worth. If you turn it on you have to type in the IP address of every machine you give an account to. Great security, but your fingers will hurt from all of the damn typing. It is also tough to account for DHCP, which most of your clients will use. When their IP changes they may not be able to get in and they will send you mail about it. Better just to keep their IP address in a log file and track it that way.
- Disallow Access by IP
Much more useful. This is how to ban people from your site. When someone is a dick, drop their IP in this list and they don't get in.
Most of your other setting will remain at their default values, and are best not messed with until you figure out what you are doing. It will take a while to work out the bugs from your configuration. Talk to an experienced site admin and get their help.
User Accounts / Permissions
Once you set up your server, you will need to create accounts so that people can log on. These are commonly referred to as L:P's (login : password). You will also need to set permissions on your directories or virtual directories to control what people can do.
A very few sites allow anonymous connections. This sucks. You have no way of controlling who logs in and you have no leverage to collect IP data so that we can track down bad traders, profiteers, and other leaches that need to be banned. My best advice is to turn off anonymous access.
Now on to "real" accounts. For all of these you need to decide what you want the account to have access to. You can control what people can get to by account. Here are the different accounts, and who I suggest you give them to. I also talk about what file permissions you should give each type of account so that these people can do what they need to do to spread the tunes.
- General High Speed Users
Also known as the public account. This is the one that you will give out the most. These are people with high speed access. You can decide how you want to give this one out, but I would NOT post it to any list. Too many people who you don't know would get it. I prefer having them send in an e-mail request with identifying information like e-mail address and IP address so that I can get a hold of them if they are causing a problem on my system, or track them down and ban them if they sell cd's or something. These people need list access to your directories and read access on the files to download them. I also like to turn on the recursive and free switches so that I don't have to set permissions on subdirectories.
- Modem Users
This one is a tough call. Modem users don't use much bandwidth, but they stay logged in forever. Personally, I allow them but most people don't. I would definitely put them in their own user account so that they don't take away a slot from a high speed user. Since these people are always dynamic IP, it is usually worthless to try and collect IP data from them. I did post this l:p to the list. The permissions should be the same as your high speed downloaders. Just a note: Modem users are usually very bitchy, but are usually harmless. It takes so long to get a show by modem that only the most dedicated modem users will ever get a complete show.
You want to create a special account for people to upload stuff to your server. You want to create a directory for uploads and in addition to read and list, these people need permission to create directories and write files in that directory only. Make sure no one else can read from this directory, otherwise you run the risk of someone downloading an incomplete show, or one that fails MD5 checks. Bad shows in circulation is a bad thing. You also have the option to allow your uploaders to delete files and directories. This is a personal choice. If they can delete files, they can remove a file that was interrupted in transfer without having to wait for you to do it, which can make uploads go much faster over shaky connections. But they can also delete files by accident that you may have in that directory from someone else. Your call. This is an l:p that you definitely don't want posted on a list.
- Mirror Site Operators
It is considered common courtesy to give a preferred login to other site operators so that they can mirror shows that get seeded to you without having to wait for one of your "regular" slots to open up. It is common to combine this account with your upload account. Make sure you have at least a few of these available. It will make you a whole lot more popular with other admins on the e-tree, and as a courtesy when you give yours out, you will usually get their mirror login in return. For permissions I like to give site-ops list and read access to everything, including the upload directory. Most site-ops know to check the MD5 file on a show before they host it, so you are fairly safe letting them download from your upload directory. This login can be fairly safely posted to email@example.com without having to worry about them abusing it.
- Special Logins
Most people give out at least a couple individual logins. These are for people that do something really cool for you, like seed an ultra rare show to your site, or just regularly upload cool new material to your machine. To give them back a little something, you create a personal account for them so that they never get closed out of your site. This is totally optional, but if you operate a site for very long, you will probably give out a couple.
etree.org FTP Server Caveats
Being an etree.org siteop is great fun. You will have virtually unlimited access to almost any show that you care to find. You will get these shows first, since these days we are the roots that most trees grow out of. All of this and more can be yours, if only you want it. However, it takes a lot of time, a lot of effort, and a lot of patience to be a good etree.org siteop, but If you stick it out it is very rewarding. Below are a few caveats you should be aware of:
You will get a whole bunch of email. Seriously, a ton of email. There is nothing you can do about this. This is just one of the hazards of hosting up shows that everybody wants. I would suggest getting a good email client and getting really good with filters to cut down on the mess.
There has been a raging debate on whether it is OK to host a show without having listened to it first. I don't think this one will ever be solved. I run dylan.etree.org and I just get too many shows coming in to listen to them all before I host, so I really can't - other than the high demand shows, when quality is of high importance. If you have a smaller site and you have the time to listen first, by all means do so. If you can't, at the very least verify md5sums before you make a show public. If any tracks fail don't post any part of the show. It is just bad form to do so.
People will hammer your site, and there isn't much you can do about this either. Unless you publicly announced your l:p, you should be able to identify the offender and e-mail them to get them to stop. You will probably have to ban a few people at some point to get the point across. You should also ban anyone you see added to our ban list, or anyone you find out something bad about (sold disks, bad trader, or tried to distribute copyrighted material like released CD's). We can have no tolerance for those people.
Tricks of the Trade
Now that you have a configured FTP server ready to go, there are only a couple of things left to do. One of them is getting on the etree.org as a siteop. Please read the Etree.org Server FAQ to find out how to get on the site-ops discussion list. There we talk about siteop stuff: seeds for shows, l:p's for mirrors, shows that are coming down the pipe, bad traders or other leeches, and other stuff like that. Siteops that join and contribute to the community, will be able to get an etree.org dns name directed to their site. After serving for 2-3 weeks, feel free to email firstname.lastname@example.org with your request. Joining the siteops list will also allow you to post to email@example.com, where you tell the world about your site.
You should post the contents of you site to firstname.lastname@example.org at least once every couple weeks, this lets people know what you have available, and how to get to your site. You should probably post the announce message to the site-ops list a day or two before you post it to the public, so that the other mirror operators can grab any new stuff you have before the public makes their mad rush for it.
After doing all of the above stuff, you are pretty much a full blown etree.org siteop.
The process of replicating a collection of files from one FTP server to another is known as "mirroring." When mirroring a show, please observe the following guidelines:
Do not put anything less than a complete, verified disc available for download.
Download the files to a temporary area outside of your 'normal' FTP space (such as an incoming directory), then make the files public after you have downloaded the entire disc and verified them against the md5 signatures. If you cannot host all of the discs for a show, indicate that there are missing discs by creating files where the directories would be, but with the suffix '.absent'. For example, 'ph00-04-20d8.absent'.
VERY SPECIAL THANKS TO DAN GOODMAN FOR HELP WITH THIS SECTION