DEV VS OPS ROUND 1, FIGHT!

That moment you realize dev is winning the dev ops battle

> pidof /usr/bin/cinder-api
> pidof python
78276 78275 78274 78273 78272 78270 78269 78268 78267 78266 78265 78264 78263 78262 78261 78260 78259 78258 78257 78256 78255 78254 78253 78252 78251 78250 78249 78248 78247 78246 78245 78244 78243 78242 78241 78240 78239 78238 78237 78236 78235 78234 78233 78232 78231 78230 78229 78228 78227 78226 78225 78224 78223 78222 78221 78220 78219 78218 78217 78216 78215 78214 78213 78212 78123 77957 77956 77955 77954 77953 77952 77951 77950 77949 77948 77947 77946 77945 77944 77943 77942 77941 77940 77939 77938 77937 77936 77935 77934 77933 77932 77931 77930 77929 77928 77927 77926 77925 77924 77923 77922 77921 77920 77919 77918 77917 77916 77915 77914 77913 77912 77911 77910 77909 77908 77907 77906 77905 77904 77903 77902 77901 77900 77899 77898 77897 77896 77895 77894 77729 77525 77523 77520 77518 77517 77514 77512 77511 77509 77507 77506 77504 77502 77501 77499 77498 77496 77494 77493 77491 77489 77487 77485 77484 77482 77480 77479 77477 77475 77473 77472 77470 77468 77466 77464 77462 77461 77459 77457 77455 77453 77451 77449 77447 77445 77443 77441 77434 77431 77430 77429 77428 77426 77421 77420 77419 77418 77417 77416 77415 77414 77413 77412 77411 77400 77399 77398 77397 77396 77395 77394 77393 77392 77391 77390 77389 77388 77387 77386 77385 77384 77383 77382 77381 77380 77379 77378 77377 77376 77375 77374 77373 77372 77371 77370 77369 77367 77366 77364 77362 77360 77359 77358 77357 77356 77355 77353 77351 77349 77347 77330 77329 77328 77327 77326 77323 77322 77321 77320 77319 77318 77317 77316 77315 77314 77313 77312 77311 77279 77136 3367

Yes..I know cinder-api is NOT a real process ;) There for effect only…

Fix for OVIRT/RHEV “LSB Network Not Starting” and “Login Service Not Starting” after migrating VM into OVIRT 4.0

Yet another scribble for the google traveler looking for a quick help fix.

OS : CentOS 7

Ran into this issue

Error :

These Services fail in console logs and no login prompt appears:

logind.service
dbus.service

Fix:

As seen via code …

[root@ns03 ~]# cat /usr/lib/systemd/system/rngd.service
[Unit]
Description=Hardware RNG Entropy Gatherer Daemon


[Service]
# Orginal Line commented out ..line with fix replaces it.
#ExecStart=/sbin/rngd -f
ExecStart=/sbin/rngd -f -r /dev/urandom


[Install]
WantedBy=multi-user.target

There are likely other options here …so choose the one that makes the most sense to you.

Fix for Ansible “detected unhandled Python exception” via “‘module object has no attribute DEFAULT_LOCAL_TMP”

This one stumped me for a few minutes when all of a sudden when Ansible couldn’t run on this system. After a quick strace pointed to the fact that Ansible was using :

/usr/lib/python2.7/site-packages/ansible-2.0.0-py2.7.egg/

which was not from the 2.1 version I had installed from a EPEL RPM (the 2.0 was built locally on this system prior to the 2.1 update) . There are lots of ways to clean this up properly  but since this was a non-prod system and I was in a hurry I just :

rm -rf /usr/lib/python2.7/site-packages/ansible-2.0.0-py2.7.egg/ (after testing its removal didn’t break other things too badly of course…you can always just MV to a .old or whatever your preference is)

and recorded my findings for the next google traveler to find and perhaps cleanup properly.

Fail2ban filters added to my Gitlab

I needed a place to drop my few fail2ban filters as I am starting to grow them for some new personal projects so I thought I would share in the event that someone else finds benefits. These are currently on my gitlab server at http://gitlab.misterx.org/Fail2ban/config. Enjoy!

nbd-client CLI Easter Egg

In trying to uncover the version of a CLI tool I uncovered this small Easter Egg.


# nbd-client -v
nbd-client: unrecognized option '-v'
E: option eaten by 42 mice

Just a quick note for others to enjoy…

Learning to love my I/O , a Raspi NBD retrospective.

Overview

After struggling with IO issues on my Raspi I decided I would try something a little less conventional. This was to use a network block device (NBD) served from a CentOS 7 box to my Pidora21 Raspi2 system.

I tested the NBD with FIO and found this

CPU (more through testing coming soon)

  • 100% = 100% of all cores not 100% per core
  • FIO run from 3ed server with FIO listening in server mode on the client. There was some overhead running FIO on the client which was about 5-8%

CPU Load with SDCard IO

  • Server Load : N/A
  • Client Load : 30-40%

NBD Load

  • Server Load : (J1900 Celeron) Approx 3% (nbd-server 1025 /mnt/$FILE.NBD)
  • Client Load : (Raspi 2 – ARMv7 Processor rev 5) Approx 45-55%  (nbd-client $NBD_SERVER 1025 /dev/nbd0) a decent part of this was IRQ load which is harder to balance on a Pi

 

FIO Config

http://gitlab.misterx.org:81/benchmarking/fio-examples/blob/master/tiobench.ini

RESULTS

I mostly focus on Max Bytes (maxb) but the other results do have some minor significance so I left them.

Micro-SDCard on Pi (had to space tests far apart to let the card “catch up”)

 #SEQ Write
 WRITE: io=61956KB, aggrb=1031KB/s, minb=1031KB/s, maxb=1031KB/s, mint=60070msec, maxt=60070msec
#Random Write
 WRITE: io=140372KB, aggrb=2333KB/s, minb=2333KB/s, maxb=2333KB/s, mint=60148msec, maxt=60148msec
#SEQ Read
 READ: io=967888KB, aggrb=16106KB/s, minb=16106KB/s, maxb=16106KB/s, mint=60093msec, maxt=60093msec
#Random Read
 READ: io=315660KB, aggrb=5260KB/s, minb=5260KB/s, maxb=5260KB/s, mint=60002msec, maxt=60002msec

Pi 100Mb Ethernet

#SEQ Write
READ: io=262144KB, aggrb=9798KB/s, minb=9798KB/s, maxb=9798KB/s, mint=26753msec, maxt=26753msec
#Random Write
READ: io=262144KB, aggrb=3258KB/s, minb=3258KB/s, maxb=3258KB/s, mint=80437msec, maxt=80437msec
#SEQ Read
WRITE: io=262144KB, aggrb=19156KB/s, minb=19156KB/s, maxb=19156KB/s, mint=13684msec, maxt=13684msec
#Random Read
WRITE: io=262144KB, aggrb=12867KB/s, minb=12867KB/s, maxb=12867KB/s, mint=20372msec, maxt=20372msec

As you can see aside from Sequential Read the NBD kills the MicroSD Card in the Pi. As with any test your results WILL vary. 

Steps to set this up

  1. (Both) Make sure both boxes have NBD kernel modules built and the tools installed (CentOS dropped support for NBD a while back, but Fedora still has it)
    1. (Build module walkthrough here ) http://www.misterx.org/2013/03/05/getting-nbd-network-block-device-back-in-rhel-6-x-and-centos-6-x/
    2. (Tools) yum install nbd
      1.  RH/CENT7  EPEL
      2. Fedora its in default repos
    3. (Verify Module is loaded)
      modprobe nbd && lsmod | grep nbd
      nbd 9421 1 
  2. (Server) Create a file using ‘dd’ via these steps :
    1. (Make File) : dd if=/dev/zero of=$FILE bs=1G count=$NUMBER_of_Gigs_You_Want (you can also argue you can use fallocate here if you understand what that changes)
    2. (Make FS on File) : ‘(mkfs.xfs | mkfs.ext4 | $WHATEVER_YOU_PREFER) $FILE’
    3. (OPTIONAL – Verify FS) file $FILE (would return something like “$FILE_PATH_AND_NAME: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)” )
  3. (Server) Create a script to start this at boot ‘nbd-server $PORT /path/to/file/to/export’ or use ‘/etc/sysconfig/nbd-server’ and start nbd-server however you prefer.
  4. (Client) Start a nbd-client similar with syntax similar to this ‘nbd-client $NBD_SERVER_HOST  $PORT_OF_NBD_SERVER /dev/nbd0’ example ‘nbd-client NBDSERVER.EXAMPLE.COM 2929 /dev/nbd0’  UPDATE:

    For newer versions of NBD you can use this to use a default port of  10809 

    1. nbd-server -C /path//to/nbd-server/config
    2. nbd-client $NBD_SERVER  /dev/nbd0 -N $SECTION_NAME_IN_SERVER_CONFIG
  5. (Client) Mount NBD locally via ‘mount /dev/nbd$NUMBER_FROM_NBDCLIENT /PATH/TO/MOUNT’ example ‘mount /dev/nbd0 /usr/local/NBD/’

Considerations

  •  If you use your Pi’s NIC for other network related tasks this could cause issues if your NBD is under high load. I used a USB ethernet device on my Pi for other basic network needs so the ethernet is mostly allocated for the NBD and management.
  • This is not perfect and should not be used for mission critical data ! This was used to get space/speed in a pinch for a project. If used in anything resembling production a lot more work would need to be done or more robust solutions should be considered !!
  • The results off the SDcard varied wildly when compared to the NBD but it was never close in any one area outside of SEQ Read.
  • NBD performance will vary based on what the server is using for storage/network and the load it is under. Your results WILL vary !

Adventures with DD

In my humble opinion when it comes to CLI tools `dd` ranks pretty high in my toolkit. Right there with `nc` and a few other ‘ole favorites.

Today I wanted to show a quick hack on how to determine information about a block device using dd. This goes along the same path as  previous post about using `dd` to get LVM configuration off devices.

Well enough talk lets get down to the CLI:

DD IS DESTRUCTIVE USE IT WITH CAUTION AND UNDERSTANDING OF WHAT YOU ARE RUNNING!…ok now that is out of the way…

# Thanks to file(1) magic(5)
# Example 1 - CentOS 6 LVM2
>dd if=/dev/sda3 of=/tmp/info bs=512 count=1000
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 0.0190352 s, 26.9 MB/s
You have new mail in /var/spool/mail/root
> file /tmp/info
/tmp/info: LVM2 (Linux Logical Volume Manager) , UUID: M7pUi7daFBsEQ95UNJUd5pN604qSa0Z


#Example 2 - CentOS 7 LVM2
> dd if=/dev/sda2 of=/tmp/info bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.00487443 s, 84.0 MB/s
> file /tmp/info
/tmp/info: LVM2 PV (Linux Logical Volume Manager), UUID: fYcfT0-3Oyk-J87h-A6SV-rYCc-q1of-zhx7Fd, size: 119508303872


#Example 3 - CentOS 7 XFS
> dd if=/dev/sda1 of=/tmp/info bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.00990939 s, 41.3 MB/s
> file /tmp/info
/tmp/info: SGI XFS filesystem data (blksz 4096, inosz 256, v2 dirs)


#Example 4 - CentOS 7 BTRFS
> dd if=/dev/vdb2 of=/tmp/info bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.00324626 s, 126 MB/s
> file /tmp/info
/tmp/info: BTRFS Filesystem (label "----_fs", sectorsize 4096, nodesize 16384, leafsize 16384)


#Example 5 - CentOS 6 - EXT4
> dd if=/dev/sda1 of=/tmp/info bs=4k count=100
100+0 records in
100+0 records out
409600 bytes (410 kB) copied, 0.0132642 s, 30.9 MB/s
> file /tmp/info
/tmp/info: Linux rev 1.0 ext4 filesystem data (needs journal recovery) (extents) (huge files)

I always forget to test this when I have more “esoteric” filesystems in my home lab. If anyone can and post a comment with the results from other filesystems’s or distros I would appreciate it!

Yum has error on primary.xml.gz or similar when behind squid. (clearing and reloading a URL in Squid)

Verify the entry is getting hit in your cache by watching your logs

TCP_MEM_HIT/200 1597 GET http://linuxdownload.adobe.com/linux/x86_64/repodata/primary.xml.gz – HIER_NONE/- application/x-gzip (or similar)

To reload this entry run

squidclient -r http://linuxdownload.adobe.com/linux/x86_64/repodata/primary.xml.gz

OR

To clear this entry

squidclient -m PURGE http://linuxdownload.adobe.com/linux/x86_64/repodata/primary.xml.gz

You can also whitelist this in  several ways if you do not want this to be cached or do not want it cached for long (ie 24 hours) but I will save that for another post.

Caching packages and yum related information in squid can be helpful if you have a large number of machines behind a company/personal proxy and have limited (either in speed or monthly amounts) bandwidth at your location. While running a local mirror is preferred for many reasons this could fill that role till such time as you feel you need the complexity of local mirror.

Linux Brain Teaser – Fall 2014

This was a fun problem exchanged with a coworker. The problem is simple run this :

ssh $USER@$HOST history 

and have it return the remote history. Lots of ways to solve this one. If you want to confirm you solved it without giving it away just email/PM me. Otherwise show your work in the comments section. Since I have such ridiculously low traffic I will post the answer in 2015!

Quick Hack to use DD to get LVM configs

This is more of a exercise than a real world use example. I have used something similar in a disater recovery situation so it does have some merit ..but good planning can prevent needing hacks like this.

DD IS DESTRUCTIVE USE IT WITH CAUTION AND UNDERSTANDING OF WHAT YOU ARE RUNNING!…ok now that is out of the way…

for i in $(pvdisplay | awk '/PV Name/ {print $3}'); do
dd if=$i of=/tmp/lvm_config_for_$(echo $i |sed 's/\//_/g') bs=$(stat -f -c %s $i) count=10;
done