This one image best sums the web server sysadmin part of my 10 years at a ISP.
Yet another (bad) use for the S3 amazon cloud
Found this URL on a website pop-up spam :
http://weeklycontestwinner.s3.amazonaws.com/***********
No I do not recommend clicking on it (but you might get something free …..yeah right). The interesting thing here is the use of the cloud for quick and easy spammer/phishing/whatever-else-malware sites. Futher proof that :
A: Amazon isnt doing good policing of content.
B: The Internet’s underbelly is light years ahead of most IT departments in understanding how to use the cloud.
I could keep going ..just wanted to share something I have seen growing in use over the last year.
You might have a issue if…
# uptime
15:05:59 up 271 days, 22:45, 2 users, load average: 1789.01, 2333.21, 3025.28
This was a managed system where qmail ran away with a high remote concurrency set (5000). System was very responsive despite the load .
How DNS Siezures are like Chaos Theory
I am usually not one to get involved with politics and this matter is no different. Whichever way you lean on this DNS matter I feel it is a good chance to learn more about what options are out there. If you need to catchup here is some reading:
http://torrentfreak.com/bittorrent-based-dns-to-counter-us-domain-seizures-101130/
http://hotair.com/archives/2010/11/27/doj-seizes-domain-names-of-more-than-70-websites-suspected-of-piracy/
http://da.feedsportal.com/c/270/f/470440/s/1023e6ae/l/0Lnews0Btechworld0N0Csme0C32510A10A0Ctorrent0Efinder0Eshut0Edown0Eby0Eus0Eagencies0C0Dolo0Frss/ia1.htm
These show how government is getting more involved in DNS which is the core of how we do things on the net. This opens the door for further understanding/learning of the other options that are available on the net.
One of these is a great project for a open DNS system:
What OpenNIC are doing already and the push to a torrent driven DNS system with the .p2p extension are making the DNS world get interesting!
So what you have here is a change that is causing “ripples” all over the ‘net that could lead to major/minor changes into how DNS is done. Whatever you think will become of this it is going to be interesting to watch play out.
atop with a simulated high ICMP load
PRC | sys 0.22s | user 0.01s | | | #proc 158 | #zombie 0 | clones 0 | | | #exit 0 |
CPU | sys 1% | user 0% | irq 131% | | idle 668% | | wait 0% | | steal 0% | guest 0% |
cpu | sys 0% | user 0% | irq 80% | | idle 20% | | cpu007 w 0% | | steal 0% | guest 0% |
cpu | sys 0% | user 0% | irq 52% | | idle 48% | | cpu006 w 0% | | steal 0% | guest 0% |
cpu | sys 1% | user 0% | irq 0% | | idle 99% | | cpu004 w 0% | | steal 0% | guest 0% |
cpu | sys 0% | user 0% | irq 0% | | idle 100% | | cpu000 w 0% | | steal 0% | guest 0% |
CPL | avg1 0.07 | avg5 0.10 | | avg15 0.07 | | csw 3209 | intr 65623 | | | numcpu 8 |
MEM | tot 5.8G | free 5.4G | cache 185.4M | dirty 0.0M | buff 91.2M | slab 33.8M | | | | |
SWP | tot 2.2G | free 2.2G | | | | | | | vmcom 136.4M | vmlim 5.1G |
NET | transport | tcpi 3 | tcpo 3 | udpi 0 | udpo 0 | tcpao 0 | tcppo 0 | tcprs 0 | tcpie 0 | udpip 0 |
NET | network | ipi 438568 | ipo 438568 | ipfrw 0 | deliv 438569 | | | | icmpi 438565 | icmpo 438565 |
NET | eth0 70% | pcki 438531 | pcko 438566 | si 70 Mbps | so 50 Mbps | coll 0 | erri 0 | erro 0 | drpi 0 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/1
24 root root 1 0.18s 0.00s 0K 0K 0K 0K — – S 7 6% ksoftirqd/7
4314 root root 1 0.01s 0.01s 0K 0K 0K 0K — – S 0 1% atop
3727 root root 1 0.02s 0.00s 0K 0K 0K 0K — – S 7 1% kondemand/7
7825 root root 1 0.01s 0.00s 0K 0K 0K 0K — – R 4 0% atop
7135 root root 1 0.00s 0.00s 0K 0K 0K 0K — – S 7 0% sshd
This was created with several `ping -s 1 -q -f $IP` commands across several systems. The CPU is :
Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
What can I say , I am impressed ! I am going to try and max it out and see what it can do all out! This is a HUGE jump from our old P4 based firewall that barfed at 80-100K pps inbound!
Magento Enterprise 1.9.0.0 Worker vs Prefork MPM
Had to do a quick test of a Magento server getting ready for production recently. Being a big fan of Apaches Worker MPM (even when using PHP with ZTS) I thought i would test their recommended settings for Prefork vs one of my homebrew Worker configs. The results can be found on my wiki at http://misterx.org/wiki/index.php/Worker_vs_prefork_MPM.
What a spoofed DoS attack looks like in atop
Note the packets in/out :
pcki 115264 – pcko 100013
I feel I have a minor ethernet issue as the IRQ load should not be quite that high but that is for another post. This box is a single core P4 so its not too far off.
ATOP – firewall02 2010/10/04 09:34:23 –x— 3s elapsed
PRC | sys 3.02s | | user 0.01s | | | #proc 96 | | #zombie 0 | | clones 0 | | | #exit 0 |
CPU | sys 1% | user 1% | | irq 100% | | | idle 99% | wait 0% | | | steal 0% | | guest 0% |
cpu | sys 0% | user 0% | | irq 100% | | | idle 0% | cpu000 w 0% | | | steal 0% | | guest 0% |
cpu | sys 1% | user 0% | | irq 0% | | | idle 98% | cpu001 w 0% | | | steal 0% | | guest 0% |
CPL | avg1 1.01 | | avg5 1.05 | avg15 1.11 | | | | csw 187 | | intr 4963 | | | numcpu 2 |
MEM | tot 2.0G | free 617.6M | | cache 904.1M | dirty 0.0M | buff 127.0M | | slab 304.8M | | | | | |
SWP | tot 4.0G | free 4.0G | | | | | | | | | | vmcom 123.9M | vmlim 5.0G |
MDD | md1 | busy 0% | | read 0 | write 47 | KiB/r 0 | | KiB/w 4 | MBr/s 0.00 | MBw/s 0.06 | | avq 0.00 | avio 0.00 ms |
MDD | md3 | busy 0% | | read 0 | write 20 | KiB/r 0 | | KiB/w 4 | MBr/s 0.00 | MBw/s 0.03 | | avq 0.00 | avio 0.00 ms |
DSK | sdb | busy 5% | | read 0 | write 57 | KiB/r 0 | | KiB/w 5 | MBr/s 0.00 | MBw/s 0.10 | | avq 4.87 | avio 2.84 ms |
DSK | sda | busy 4% | | read 0 | write 57 | KiB/r 0 | | KiB/w 5 | MBr/s 0.00 | MBw/s 0.10 | | avq 6.40 | avio 2.09 ms |
NET | transport | tcpi 7 | tcpo 4 | udpi 0 | udpo 0 | tcpao 0 | | tcppo 0 | tcprs 0 | tcpie 0 | tcpor 2 | udpnp 3 | udpip 0 |
NET | network | ipi 118305 | | ipo 102606 | ipfrw 3072 | deliv 27 | | | | | | icmpi 16 | icmpo 99524 |
NET | eth2 2% | pcki 3030 | pcko 100013 | | si 881 Kbps | so 24 Mbps | coll 0 | mlti 0 | erri 0 | | erro 0 | drpi 0 | drpo 0 |
NET | eth3 1% | pcki 115264 | pcko 2578 | | si 19 Mbps | so 787 Kbps | coll 0 | mlti 2 | erri 0 | | erro 0 | drpi 208056 | drpo 0 |
PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK WRDSK ST EXC S CPUNR CPU CMD 1/1
3 root root 1 2.89s 0.00s 0K 0K 0K 0K — – R 0 96% ksoftirqd/0
Found the offender via tcpdump:
10:01:51.488936 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 104, id 4711, offset 0, flags [DF], proto: TCP (6), length: 48) 118.110.xx.xx.6697 > 173.201.xx.xx.http: P, cksum 0x6196 (correct), 2735265098:2735265098(0) ack 4261832542 win 63809 <mss 1460,nop,nop,sackOK>
10:01:51.488998 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 45, id 24124, offset 0, flags [DF], proto: TCP (6), length: 48) 97.17.xx.xx.11383 > 173.201.xx.xx.h
ttp: P, cksum 0x34dd (correct), 487590775:487590775(0) ack 1325631541 win 61462 <mss 1460,nop,nop,sackOK>
None of the IPs listed above (even though they have been edited to protect all parties) are ours. I did a `ip route add blackhole` till I could turn off the switch port on the offender.
Useful sysctl commands if you want to stop this (spoofed attacks):
net.ipv4.conf.all.rp_filter = 1
Stops spoofed packets dead in their tracks ! Then you can focus on re-balancing your interrupts or better yet turning off their switch port!
PHP goes boom!
[Fri Oct 01 12:37:39 2010] [error] [client *.*.*.*] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 46912496530832 bytes) in Unknown on line 0
Personally I think thats a little greedy with the RAM but thats just me ;)
Quick IPTABLES Connections Hack
Did a quick hack to see whats going on with a Linux IPTABLES firewall connection wise.
iptstate -s | awk ‘{print $3,$2,$1}’| cut -d”:” -f1-2 | uniq -c | sort -g
This shows you
# number of connections | Protocol | Dest_IP:PORT | Source IP
I will add this to my bash wiki section in case it can help anyone.