Archive for the ‘Linux’ Category.

Yet another (bad) use for the S3 amazon cloud

Found this URL on a website pop-up spam :

http://weeklycontestwinner.s3.amazonaws.com/***********

No I do not recommend clicking on it (but you might get something free …..yeah right). The interesting thing here is the use of the cloud for quick and easy  spammer/phishing/whatever-else-malware sites. Futher proof that :

A: Amazon isnt doing  good policing of content.
B: The Internet’s underbelly is light years ahead of most IT departments in understanding how to use the cloud.

I could keep going ..just wanted to share something I have seen growing in use over the last year.

You might have a issue if…

# uptime
15:05:59 up 271 days, 22:45,  2 users,  load average: 1789.01, 2333.21, 3025.28

This was a managed system where qmail ran away with a high remote concurrency set (5000). System was very responsive despite the load .

How DNS Siezures are like Chaos Theory

I am usually not one to get involved with politics and this matter is no different. Whichever way you lean on this DNS matter I feel it is a good chance to learn more about what options are out there. If you need to catchup here is some reading:

http://torrentfreak.com/bittorrent-based-dns-to-counter-us-domain-seizures-101130/
http://hotair.com/archives/2010/11/27/doj-seizes-domain-names-of-more-than-70-websites-suspected-of-piracy/

http://da.feedsportal.com/c/270/f/470440/s/1023e6ae
/l/0Lnews0Btechworld0N0Csme0C32510A10A0Ctorrent0Efinder0Eshut0Edown0Eby0Eus0Eagencies0C0Dolo0Frss/ia1.htm

These show how government is getting more involved in DNS which is the core of how we do things on the net. This opens the door for further understanding/learning of the other options that are available  on the net.

One of these is a great project for a open DNS system:

http://www.opennicproject.org

What OpenNIC are doing already and the push to a torrent driven DNS system with the .p2p extension are making the DNS world get interesting!

So what you have here is a change that is causing “ripples” all over the ‘net that could lead to major/minor changes into how DNS is done. Whatever you think will become of this it is going to be interesting to watch play out.

atop with a simulated high ICMP load

PRC | sys    0.22s  | user   0.01s  |               |               | #proc    158  | #zombie    0  | clones     0  |               |               | #exit      0  |
CPU | sys       1%  | user      0%  | irq     131%  |               | idle    668%  |               | wait      0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq      80%  |               | idle     20%  |               | cpu007 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq      52%  |               | idle     48%  |               | cpu006 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       1%  | user      0%  | irq       0%  |               | idle     99%  |               | cpu004 w  0%  |               | steal     0%  | guest     0%  |
cpu | sys       0%  | user      0%  | irq       0%  |               | idle    100%  |               | cpu000 w  0%  |               | steal     0%  | guest     0%  |
CPL | avg1    0.07  | avg5    0.10  |               | avg15   0.07  |               | csw     3209  | intr   65623  |               |               | numcpu     8  |
MEM | tot     5.8G  | free    5.4G  | cache 185.4M  | dirty   0.0M  | buff   91.2M  | slab   33.8M  |               |               |               |               |
SWP | tot     2.2G  | free    2.2G  |               |               |               |               |               |               | vmcom 136.4M  | vmlim   5.1G  |
NET | transport     | tcpi       3  | tcpo       3  | udpi       0  | udpo       0  | tcpao      0  | tcppo      0  | tcprs      0  | tcpie      0  | udpip      0  |
NET | network       | ipi   438568  | ipo   438568  | ipfrw      0  | deliv 438569  |               |               |               | icmpi 438565  | icmpo 438565  |
NET | eth0     70%  | pcki  438531  | pcko  438566  | si   70 Mbps  | so   50 Mbps  | coll       0  | erri       0  | erro       0  | drpi       0  | drpo       0  |

PID    RUID         EUID          THR      SYSCPU      USRCPU      VGROW      RGROW     RDDSK      WRDSK     ST     EXC     S     CPUNR      CPU    CMD         1/1
24    root         root            1       0.18s       0.00s         0K         0K        0K         0K     —       –     S         7       6%    ksoftirqd/7
4314    root         root            1       0.01s       0.01s         0K         0K        0K         0K     —       –     S         0       1%    atop
3727    root         root            1       0.02s       0.00s         0K         0K        0K         0K     —       –     S         7       1%    kondemand/7
7825    root         root            1       0.01s       0.00s         0K         0K        0K         0K     —       –     R         4       0%    atop
7135    root         root            1       0.00s       0.00s         0K         0K        0K         0K     —       –     S         7       0%    sshd

This was created with several `ping -s 1 -q -f $IP` commands across several systems. The CPU is :

Intel(R) Xeon(R) CPU W3530  @ 2.80GHz

What can I say , I am impressed ! I am going to try and max it out and see what it can do all out! This is a HUGE jump from our old P4 based firewall that barfed at 80-100K pps inbound!

Magento Enterprise 1.9.0.0 Worker vs Prefork MPM

Had to do a quick test of a Magento server getting ready for production recently. Being a big fan of Apaches Worker MPM (even when using PHP with ZTS) I thought i would test their recommended settings for Prefork vs one of my homebrew Worker configs. The results can be found on my wiki at  http://misterx.org/wiki/index.php/Worker_vs_prefork_MPM.

What a spoofed DoS attack looks like in atop

Note the packets in/out :
pcki  115264 – pcko  100013

I feel I have a minor ethernet issue as the IRQ load should not be quite that high but that is for another post. This box is a single core P4 so its not too far off.

ATOP – firewall02                                                   2010/10/04  09:34:23                                                   –x—                                                    3s elapsed
PRC | sys    3.02s  |              | user   0.01s  |              |               | #proc     96 |               | #zombie    0 |               | clones     0 |               |              |  #exit      0 |
CPU | sys       1%  | user      1% |               | irq     100% |               |              | idle     99%  | wait      0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       0%  | user      0% |               | irq     100% |               |              | idle      0%  | cpu000 w  0% |               |              |  steal     0% |              |  guest     0% |
cpu | sys       1%  | user      0% |               | irq       0% |               |              | idle     98%  | cpu001 w  0% |               |              |  steal     0% |              |  guest     0% |
CPL | avg1    1.01  |              | avg5    1.05  | avg15   1.11 |               |              |               | csw      187 |               | intr    4963 |               |              |  numcpu     2 |
MEM | tot     2.0G  | free  617.6M |               | cache 904.1M | dirty   0.0M  | buff  127.0M |               | slab  304.8M |               |              |               |              |               |
SWP | tot     4.0G  | free    4.0G |               |              |               |              |               |              |               |              |               | vmcom 123.9M |  vmlim   5.0G |
MDD |          md1  | busy      0% |               | read       0 | write     47  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.06 |               | avq     0.00 |  avio 0.00 ms |
MDD |          md3  | busy      0% |               | read       0 | write     20  | KiB/r      0 |               | KiB/w      4 |  MBr/s   0.00 | MBw/s   0.03 |               | avq     0.00 |  avio 0.00 ms |
DSK |          sdb  | busy      5% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     4.87 |  avio 2.84 ms |
DSK |          sda  | busy      4% |               | read       0 | write     57  | KiB/r      0 |               | KiB/w      5 |  MBr/s   0.00 | MBw/s   0.10 |               | avq     6.40 |  avio 2.09 ms |
NET | transport     | tcpi       7 | tcpo       4  | udpi       0 | udpo       0  | tcpao      0 |               | tcppo      0 |  tcprs      0 | tcpie      0 |  tcpor      2 | udpnp      3 |  udpip      0 |
NET | network       | ipi   118305 |               | ipo   102606 | ipfrw   3072  | deliv     27 |               |              |               |              |               | icmpi     16 |  icmpo  99524 |
NET | eth2      2%  | pcki    3030 | pcko  100013  |              | si  881 Kbps  | so   24 Mbps | coll       0  | mlti       0 |  erri       0 |              |  erro       0 | drpi       0 |  drpo       0 |
NET | eth3      1%  | pcki  115264 | pcko    2578  |              | si   19 Mbps  | so  787 Kbps | coll       0  | mlti       2 |  erri       0 |              |  erro       0 | drpi  208056 |  drpo       0 |

PID       RUID            EUID             THR        SYSCPU         USRCPU         VGROW        RGROW         RDDSK         WRDSK       ST        EXC        S       CPUNR         CPU        CMD        1/1
3       root            root               1         2.89s          0.00s            0K           0K            0K            0K       —          –        R           0         96%        ksoftirqd/0
Found the offender via tcpdump:

10:01:51.488936 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 104, id 4711, offset 0, flags [DF], proto: TCP (6), length: 48) 118.110.xx.xx.6697 > 173.201.xx.xx.http: P, cksum 0x6196 (correct), 2735265098:2735265098(0) ack 4261832542 win 63809 <mss 1460,nop,nop,sackOK>

10:01:51.488998 00:0b:cd:3e:c6:93 > 00:30:48:94:94:5f, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl  45, id 24124, offset 0, flags [DF], proto: TCP (6), length: 48) 97.17.xx.xx.11383 > 173.201.xx.xx.h
ttp: P, cksum 0x34dd (correct), 487590775:487590775(0) ack 1325631541 win 61462 <mss 1460,nop,nop,sackOK>

None of the IPs listed above (even though they have been edited to protect all parties) are ours.  I did a `ip route add blackhole` till I could turn off the switch port on the offender.

Useful sysctl commands if you want to stop this (spoofed attacks):

net.ipv4.conf.all.rp_filter = 1

Stops spoofed packets dead in their tracks ! Then you can focus on re-balancing your interrupts or better yet turning off their switch port!

PHP goes boom!

[Fri Oct 01 12:37:39 2010] [error] [client *.*.*.*] PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 46912496530832 bytes) in Unknown on line 0

Personally I think thats a little greedy with the RAM but thats just me ;)

Quick IPTABLES Connections Hack

Did a quick hack to see whats going on with a Linux IPTABLES firewall connection wise.

iptstate -s | awk ‘{print $3,$2,$1}’| cut -d”:” -f1-2 | uniq -c | sort -g

This shows you

# number of connections | Protocol |  Dest_IP:PORT | Source IP

I will add this to my bash wiki section in case it can help anyone.

Why MySQL (and other DBs as well) indexes/keys are good!

I was working with a managed customer who handles their own DBA tasks on their servers when I noticed a recent change had created a major slow down in the speed of their front page. Here is the Apache Bench of their main page (actual URL removed to protect the guilty).

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   46.619197 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    2.15 [#/sec] (mean)
Time per request:       4661.920 [ms] (mean)
Time per request:       466.192 [ms] (mean, across all concurrent requests)
Transfer rate:          218.97 [Kbytes/sec] received

The power of a simple index:
CREATE INDEX phpsession_speed on session_data(phpsessionid);

Server Software:        Apache/2.2.0
Server Hostname:        <REMOVED>
Server Port:            80
Document Path:          /index.php
Document Length:        103866 bytes
Concurrency Level:      10
Time taken for tests:   3.396604 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      10454000 bytes
HTML transferred:       10386600 bytes
Requests per second:    29.44 [#/sec] (mean)
Time per request:       339.660 [ms] (mean)
Time per request:       33.966 [ms] (mean, across all concurrent requests)
Transfer rate:          3005.35 [Kbytes/sec] received

There are MANY more optimizations I need to work with their DBA/Webdev because now their SQL server is saturating a gigabit connection with the webserver at the 30-40 RPS mark but this fixed their immediate problem.

Yet another cool use for SSDs

I did a test for a SSD as a external journal for a ext3 based single SCSI drive with some mildly interesting results ! I was going to rerun with a larger test size but ran out of time. If anyone re-runs this test and can send me the results I would like to see them.

(remove old journal)
tune2fs -O ^has_journal /dev/sdc1
(add new journal a 400M parition on a SSD)
mke2fs -b 4096 -O journal_dev -J device=/dev/sdk1,size=400 /dev/sdk1
tune2fs -j -J device=/dev/sdk1,size=400 /dev/sdc1

All tests done on a Dual Xeon 2.8Ghz (HT enabled) 800FSB 1M cache
Benchmarks used :

tiobench (from RPMForge) run from CLI with defaults `tiobench`
dd : ` dd if=/dev/zero of=/mnt/test.tmp bs=256k count=10000`

Results

400 MB SSD journal
DD: (2.6 GB) copied, 51.3763 seconds, 51.0 MB/s
Sequential Reads
2.6.18-194.el5                2000  4096    1  ###### 99.94%     0.003        0.13   0.00000  0.00000  1211
2.6.18-194.el5                2000  4096    2  ###### 399.0%     0.004        0.25   0.00000  0.00000   518
2.6.18-194.el5                2000  4096    4  ###### 1545.%     0.006       34.99   0.00000  0.00000   151
2.6.18-194.el5                2000  4096    8  ###### 3038.%     0.012      140.04   0.00000  0.00000    77
Random Reads
2.6.18-194.el5                2000  4096    1  836.90 96.41%     0.004        0.03   0.00000  0.00000   868
2.6.18-194.el5                2000  4096    2  ###### 370.3%     0.004        0.03   0.00000  0.00000   434
2.6.18-194.el5                2000  4096    4  ###### 1405.%     0.006        0.41   0.00000  0.00000   140
2.6.18-194.el5                2000  4096    8  ###### 779.5%     0.006        0.05   0.00000  0.00000   208
Sequential Writes
2.6.18-194.el5                2000  4096    1   43.03 23.59%     0.083     2692.94   0.00039  0.00000   182
2.6.18-194.el5                2000  4096    2   41.65 64.84%     0.170     2302.42   0.00078  0.00000    64
2.6.18-194.el5                2000  4096    4   41.87 219.0%     0.322     2970.18   0.00176  0.00000    19
2.6.18-194.el5                2000  4096    8   40.39 422.4%     0.667     3098.38   0.01211  0.00000    10
Random Writes
2.6.18-194.el5                2000  4096    1   14.72 5.746%     0.008        0.03   0.00000  0.00000   256
2.6.18-194.el5                2000  4096    2   14.63 16.48%     0.011        0.05   0.00000  0.00000    89
2.6.18-194.el5                2000  4096    4   56.14 103.4%     0.019       16.04   0.00000  0.00000    54
2.6.18-194.el5                2000  4096    8   40.74 161.6%     0.023       11.21   0.00000  0.00000    25
Default Journal on same disk
DD: (2.6 GB) copied, 100.868 seconds, 26.0 MB/s
Sequential Reads
2.6.18-194.el5                2000  4096    1  ###### 99.85%     0.003        1.80   0.00000  0.00000  1184
2.6.18-194.el5                2000  4096    2  ###### 397.9%     0.004        1.64   0.00000  0.00000   522
2.6.18-194.el5                2000  4096    4  ###### 1590.%     0.006        0.22   0.00000  0.00000   147
2.6.18-194.el5                2000  4096    8  ###### 2807.%     0.011      430.45   0.00000  0.00000    84
Random Reads
2.6.18-194.el5                2000  4096    1  840.96 96.87%     0.004        0.02   0.00000  0.00000   868
2.6.18-194.el5                2000  4096    2  ###### 370.3%     0.004        0.03   0.00000  0.00000   434
2.6.18-194.el5                2000  4096    4  ###### 1382.%     0.006        0.69   0.00000  0.00000   138
2.6.18-194.el5                2000  4096    8  ###### 1339.%     0.007        4.06   0.00000  0.00000   145
Sequential Writes
2.6.18-194.el5                2000  4096    1   23.28 13.37%     0.164     1449.18   0.00000  0.00000   174
2.6.18-194.el5                2000  4096    2   23.16 37.65%     0.330     1522.12   0.00000  0.00000    61
2.6.18-194.el5                2000  4096    4   21.45 125.1%     0.706     1493.51   0.00000  0.00000    17
2.6.18-194.el5                2000  4096    8   21.07 236.6%     1.419     1943.53   0.00000  0.00000     9
Random Writes
2.6.18-194.el5                2000  4096    1   15.15 4.749%     0.008        0.03   0.00000  0.00000   319
2.6.18-194.el5                2000  4096    2   15.22 14.60%     0.011        0.06   0.00000  0.00000   104
2.6.18-194.el5                2000  4096    4   44.56 101.8%     0.020        0.08   0.00000  0.00000    44
2.6.18-194.el5                2000  4096    8   21.06 73.30%     0.018        0.28   0.00000  0.00000    29