The VMware Forums have detailed the reason for this problem, but I will summarize.

VMware has a limitation (supposedly due to kernel limitation) in which traffic from host to guest must be put on the wire, when guest to host is not put on the wire. This split causes this situation.

When a guest does not run much traffic outside the host, it’s mac-address-table entry on the connected switch may expire and be removed. Switches handle packets addressed to mac addresses not in it’s table like hubs, broadcast everywhere to find the proper interface. This is normally fine because the next packet from that MAC will cause the table to be updated. However in the special situation of the host <-> guest conversation the packets generated by the guest never get put on the wire hence the mac-address-table never updating. Also causing host generated packets to continue to be broadcast *everywhere*.

Solutions:
* Patch the vmnet kernel module on the host as described in the forum posting. This patch causes guest -> host traffic to be put on the wire as well. Preventing the lopside problem, but causing more traffic on the wire to the switch.
* Statically add the mac-address to the table in the connected switch. This prevents host->guest traffic from moving beyond the connected switch, and doesn’t add additional packets on the wire, but adds additional maintenance.
* Cause the guest to send packets that pass outside the host at least once every 5 minutes (default aging-time). Few additional packets, and no switch maintenance. Install NTPd to synchronize the time.

• Bridged is often unacceptable for experimentation due to it’s unrestricted nature. It also does not pass through the standard netfilter interfaces in the kernel to be filtered.
• NAT is often unacceptable because I cannot control it’s filtering policies. It runs a separate daemon to handle the address translation. This blocks many of my filtering options in iptables.
• Host-only is almost never acceptable since I rarely do any experimentation that doesn’t require a network interface to my network outside my machine.

My favorite solution is to use the host-only networking option, and configure my host machine to NAT and route the traffic. This gives me extreme control over the network policies, addresses, etc. All with a familiar iptables interface.

To accomplish this there are a few steps that need to happen.

• Configure your Virtual Machine to use Host-only networking
• Enable IP forwarding on your host.

  echo 1 > /proc/sys/net/ipv4/ip_forward

• Add the address you want your virtual machine to use on your network as an alias to your real interface.

  ifconfig eth0:0 10.49.220.40 netmask 255.255.252.0

• Add a NAT rule with iptables to translate packets to this new address.

  iptables -t nat -A POSTROUTING -i vmnet1 -o eth0 -j SNAT --to-source 10.49.220.40

• Add any rules you wish to impose to the FORWARD chain in the default filter table. Example here defaults to DROP all packets, but allow DNS to a DNS server, and all traffic to a host for the experiment.

  iptables -P FORWARD DROP
  iptables -A FORWARD -d 10.49.1.25 -p udp --dport 53 -j ACCEPT
  iptables -A FORWARD -d 10.49.1.26 -j ACCEPT

Now your experiment will come from your chosen IP as you would have wanted with bridged mode, but you get the awesome power and flexibility of filtering it via iptables. Great for playing with Windows and it’s included vulnerabilities.

My procmailrc sorts most of my mail into folders for me. When I was writing the script to parse I decided to categorize my folders to make the statistics more meaningful. This leaves me with 4 types of mail: work (automated reports, logs, and such), [tag]spam[/tag] (SpamAssasin, and discarded mail), lists (mailing lists), and Inbox (everything else).

These stats turned out to be quite interesting, at least to me. Since I am the sysadmin for an ISP, I get tons of email. I get the output for any and all cron jobs, interesting snippets of logs, and all mail addressed to common aliases (postmaster, root, webmaster, abuse, daemon, security, etc). This will cause my work category to be quite large. You can see that my work mail accounts for more than half of all deliveries. If you leave out the work category, my spam accounts for about 80% of all of my email, and that doesn’t count all the crap that SpamAssassin or my own filters don’t catch. Holy cow. Spam is a huge problem.

The big dip this week is caused by my experimentation with new anti-spam techniques. I tried out OpenBSD’s spamd. It is amazing. It reduces spam quite a bit, as you can see here. It would show even better results, but I only used it on one of several balanced incoming mail servers. It is a great implementation of [tag]greylisting[/tag]. However, this technique causes some legitimate mail to be delayed by 5min – a few hours. We had a few complaints from customers about delayed mail, so I had to turn it off. I highly recommend this technique for anyone who is battling spam, doesn’t have extremely picky users, and don’t mind slightly delayed mail from time to time.

Last week I used quite a bit of bandwidth on my colocated server. Part of it was due to leaving a bittorrent of a few episodes of “Weeds” running for a few days. Those sure are popular, but I have the max upload rate limited to 800KBps. Not enough to cause problems, but enough to share my generous amount of bandwidth.

While Bittorrent can eat up quite a bit of bandwidth, it wasn’t the culprit. The problem was Tor! I misread the documentation for BandwidthRate and BandwidthBurst. I thought it was in bps (bits), but, in fact, it was in Bps (bytes). Whoops! When I was going through the normal everyday sysadmin duties I noticed that our primary backbone link was a lot closer to being full than it normally should be. A little poking around and I found that my own server was eating up about 14Mbps. I shut it off immediately, and headed for the documentation. To my surprise FAQ 5.17 is very clear about the units for the BandwidthRate. I am a total idiot. Config fixed, tor restarted.

I am not one of those privacy nuts that won’t do anything that isn’t anonymized or encrypted. In fact I tried using tor myself for about 2 minutes. It was painfully slow. I don’t really care who sees most of my traffic, and the traffic I do care about is already either SSL’d or ssh tunneled. However, I understand the occasional need for tor, and it’s obvious benefits. Since I have so much bandwidth available that no one will be using, I thought I would share. Ain’t it nice to be sysadmin of your own ISP.

Interesting note: So far in the first 10 days of this month I have used 981GB on my server. Holy bejeezus