Recently we had to recover data from a 8-drive soft-RAID5 on Linux. It had 3 devices failed out of 8. 2 of the drives just had some bad sectors on them. The problem is, the Linux kernel reports errors on the device as ata channel numbers, for example as “ata1.00″. All 8 drives were the same model, the serial number is not reported via dmesg, so how to know the device name (like /dev/sda) that is causing these error messages?

The command

grep '[0-9]' /sys/class/scsi_host/host{0..9}/unique_id

will provide output like this:

/sys/class/scsi_host/host0/unique_id:1
/sys/class/scsi_host/host1/unique_id:2
/sys/class/scsi_host/host2/unique_id:0
/sys/class/scsi_host/host3/unique_id:0
/sys/class/scsi_host/host4/unique_id:3
/sys/class/scsi_host/host5/unique_id:4
/sys/class/scsi_host/host6/unique_id:5
/sys/class/scsi_host/host7/unique_id:6

so we can match the unique id used in kernel error messages to the host number. Then the command:

ls -l /sys/block/sd*

Will show us which device name belongs to which host number:

/sys/block/sda -> ../devices/pci0000:00/0000:00:13.2/usb1/1-6/1-6:1.0/host2/target2:0:0/2:0:0:0/block/sda
/sys/block/sdb -> ../devices/pci0000:00/0000:00:13.2/usb1/1-8/1-8:1.0/host3/target3:0:0/3:0:0:0/block/sdb
/sys/block/sdc -> ../devices/pci0000:00/0000:00:12.0/host6/target6:0:0/6:0:0:0/block/sdc
/sys/block/sdd -> ../devices/pci0000:00/0000:00:13.2/usb1/1-8/1-8:1.0/host3/target3:0:0/3:0:0:1/block/sdd
/sys/block/sde -> ../devices/pci0000:00/0000:00:13.2/usb1/1-8/1-8:1.0/host3/target3:0:0/3:0:0:2/block/sde
/sys/block/sdf -> ../devices/pci0000:00/0000:00:13.2/usb1/1-8/1-8:1.0/host3/target3:0:0/3:0:0:3/block/sdf
/sys/block/sdg -> ../devices/pci0000:00/0000:00:12.0/host7/target7:0:0/7:0:0:0/block/sdg

From these two outputs we can see that the unique id 6 maps to host7, and host7 maps to /dev/sdg. And finally, with the command:

hdparm -i /dev/sdg

/dev/sdg:
 Model=ST3500418AS, FwRev=CC34, SerialNo=6VM2KSFD

we can find the serial number of the drive.

deb package for the Linux WMI client (wmic) for Ubuntu 10.10 can be downloaded here.

We use this utility together with collectd, it’s Exec plug-in and rrdgraph.

For example with this script we read the Transactions performance counter for MS SQL Server:

#!/bin/bash
INTERVAL="${COLLECTD_INTERVAL:-60}"

while sleep "$INTERVAL"
do
        TRANSPS=$((wmic -U user%password //server "SELECT TransactionsPersec FROM Win32_PerfRawData_MSSQLSERVER_SQLServerDatabases WHERE Name='_Total'" | grep "_Total" | awk -F "|" '{ print $2; }') 2>/dev/null);
        if [ $? -ne 0 ]
        then
              TRANSPS="U"
        fi
        echo "PUTVAL sqlserver/exec-mssql-wmi/operations interval=$INTERVAL N:$TRANSPS"
done

Note what the rrd file name (in this case operations) should start from the defined word in types.db

Both modems should be set to Bridged Mode

Packages involved: pppd, shorewall, iproute2

/etc/ppp/peers/cat, configuration file for CAT Telecom provider:

noipdefault
#defaultroute
#replacedefaultroute
#usepeerdns
hide-password
debug
noauth
persist
maxfail 0
lcp-echo-interval 30
lcp-echo-failure 5
mtu 1492
plugin rp-pppoe.so eth0
rp_pppoe_service 'cat'
user "lettersandnumbers@hinet.p"
linkname cat
unit 0

/etc/ppp/peers/tot, configuration file for TOT provider:

noipdefault
#defaultroute
#replacedefaultroute
#usepeerdns
hide-password
debug
noauth
persist
maxfail 0
lcp-echo-interval 30
lcp-echo-failure 5
mtu 1492
plugin rp-pppoe.so eth1
rp_pppoe_service 'tot'
user "phoneno@totgoldbiz"
linkname tot
unit 1

I will explain the options.

#defaultroute
#replacedefaultroute
#usepeerdns

These we comment out, we will control default routes in the /etc/ppp/ip-up.local and /etc/ppp/ip-down.local scripts; we are not going to use DNS servers offered by the provider and will use fixed DNS (like Google DNS) instead.

persist
maxfail 0
lcp-echo-interval 30
lcp-echo-failure 5

Persist sets pppd to keep the connection; maxfail 0 prevents pppd from giving up after 5 unsuccessful connections; we are sending line control protocol “pings” every 30 seconds and going to disconnect and try to connect back again if 5 of these “pings” are lost in a row.

unit 0
linkname cat

unit is a very important option to nail the number (ppp0, ppp1 etc.) of the connection. If this option is not specified, the connections will be numbered in the order they are established and the interface number would change all the time.  linkname defines the name of .pid created in /var/run, I check for the existence of these files from PHP script to report in the web interface if the provider connection is on or off.

/etc/network/interfaces (fragment)

auto cat
iface cat inet ppp
pre-up /sbin/ifconfig eth0 up # line maintained by pppoeconf
provider cat

auto tot
iface cat inet ppp
pre-up /sbin/ifconfig eth1 up # line maintained by pppoeconf
provider tot

/etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
100     ppp0
200     ppp1

/etc/ppp/ip-up.local

#!/bin/bash
if [[ "$PPP_IFACE" == "ppp0" ]] ; then
        METRIC=1
else
        METRIC=2
fi

ip route add default dev $PPP_IFACE table $PPP_IFACE
ip route add default dev $PPP_IFACE metric $METRIC
ip rule add from $PPP_LOCAL lookup $PPP_IFACE
shorewall refresh
/usr/sbin/ddclient -syslog -file /etc/ddclient-$PPP_IFACE.conf

pppd does not have option to specify interface metric. There is a proposed patch for it, but it is not in the mainline yet. That’s why we move routing setup to this file. First line adds a default route to the routing table named ppp0 or ppp1. Third line adds a rule: if the source address of a packet is a local address of ppp interface, route it through this specific table (ppp0 or ppp1).

I am not sure if refreshing shorewall config is really needed, but won’t do any harm. ddclient updates DynDNS records using two different config files for two interfaces.

/etc/ppp/ip-down.local

#!/bin/sh
ip route del default table $PPP_IFACE
ip route del default dev $PPP_IFACE
ip rule del lookup $PPP_IFACE
shorewall refresh

/etc/shorewall/interfaces (fragment)

net     ppp0            detect          tcpflags,routefilter,nosmurfs,mss=1452
net     ppp1            detect          tcpflags,routefilter,nosmurfs,mss=1452

Important to clamp MSS to 1452 bytes here

“ip route show” output when both connections are on:

default dev ppp0  scope link  metric 1
default dev ppp1  scope link  metric 2

Primary interface ppp0, with lowest metric is used first. If it will go down, the second default route will be used automatically.

To be continued – from failover to load-balancing…

Fiber: single fiber, dual wavelength with an SC connector.

Optical Network Unit (ONU): Mitsubishi AS-1000GN2-1GC.

Router: Welltech Computer WG-332S (not even listed on Welltech website). I think I never saw the router without any authentication before! It is a VoIP router with two FXS ports, also one LAN port is labelled (by TOT) as IPTV.

We took the Welltech router out and set up a PPPoE connection from Ubuntu Linux server through Mitsubishi modem.

The CPU feqenuce set to 384 MHz
CPU revision is: 0001964c
Determined physical RAM map:
memory: 02000000 @ 00000000 (usable)
On node 0 totalpages: 8192
DMA zone: 64 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 8128 pages, LIFO batch:0
Normal zone: 0 pages used for memmap
Built 1 zonelists.  Total pages: 8128
Kernel command line: console=ttyS1,57600n8 root=/dev/mtdblock4
Primary instruction cache 32kB, physically tagged, 4-way, linesize 32 bytes.
Primary data cache 16kB, 4-way, linesize 32 bytes.
Synthesized TLB refill handler (20 instructions).
Synthesized TLB load handler fastpath (32 instructions).
Synthesized TLB store handler fastpath (32 instructions).
Synthesized TLB modify handler fastpath (31 instructions).
Cache parity protection disabled
cause = 1080002c, status = 1100ff00
PID hash table entries: 128 (order: 7, 512 bytes)
calculating r4koff… 00177000(1536000)
CPU frequency 384.00 MHz
Using 192.000 MHz high precision timer.
Console: colour dummy device 80×25
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 28804k/32768k available (2802k kernel code, 3964k reserved, 634k data, 124k init, 0k highmem)
Calibrating delay loop… 255.48 BogoMIPS (lpj=510976)
Mount-cache hash table entries: 512
NET: Registered protocol family 16
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
Time: MIPS clocksource has been installed.
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
squashfs: version 3.2-r2 (2007/01/15) Phillip Lougher
squashfs: LZMA suppport for slax.org by jro
JFFS2 version 2.2. (NAND) (C) 2001-2006 Red Hat, Inc.
fuse init (API version 7.8)
io scheduler noop registered (default)
Ralink gpio driver initialized
HDLC line discipline: version $Revision: 1.1.1.1 $, maxframe=4096
N_HDLC line discipline registered.
Serial: 8250/16550 driver $Revision: 1.3 $ 2 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0xb0000500 (irq = 37) is a 16550A
serial8250: ttyS1 at I/O 0xb0000c00 (irq = 12) is a 16550A
loop: loaded (max 8 devices)
rdm_major = 254
GDMA1_MAC_ADRH — : 0×00000000
GDMA1_MAC_ADRL — : 0×00000000
Ralink APSoC Ethernet Driver Initilization. v1.60  256 rx/tx descriptors allocated, mtu = 1500!
GDMA1_MAC_ADRH — : 0×00000030
GDMA1_MAC_ADRL — : 0x0af505b6
PROC INIT OK!
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
NET: Registered protocol family 24
ralink flash device: 0×1000000 at 0xbf000000
Ralink SoC physically mapped flash: Found 1 x16 devices at 0×0 in 16-bit bank
Ralink SoC physically mapped flash: Found an alias at 0×800000 for the chip at 0×0
Amd/Fujitsu Extended Query Table at 0×0040
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
Creating 7 MTD partitions on “Ralink SoC physically mapped flash”:
0×00000000-0×00030000 : “Bootloader”
0×00030000-0×00040000 : “Config”
0×00040000-0×00050000 : “Factory”
0×00050000-0x001d0000 : “Kernel”
0x001d0000-0×00750000 : “RootFS”
0×00750000-0×00800000 : “aztechjffs2″
0×00000000-0×00800000 : “wholeImage”
block2mtd: version $Revision: 1.1.1.1 $
usbcore: registered new interface driver cdc_acm
drivers/usb/class/cdc-acm.c: v0.25:USB Abstract Control Model driver for USB modems and ISDN adapters
usbcore: registered new interface driver cdc_wdm
usbcore: registered new interface driver usblp
drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
Initializing USB Mass Storage driver…
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
usbcore: registered new interface driver cdc_ether
nf_conntrack version 0.5.0 (256 buckets, 2048 max)
ip_tables: (C) 2000-2006 Netfilter Core Team, Type=Restricted Cone
arp_tables: (C) 2002 David S. Miller
TCP cubic registered
NET: Registered protocol family 1
NET: Registered protocol family 10
NET: Registered protocol family 17
802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
All bugs added by David S. Miller <davem@redhat.com>
VFS: Mounted root (squashfs filesystem) readonly.
Freeing unused kernel memory: 124k freed
Algorithmics/MIPS FPU Emulator v1.5
devpts: called with bogus options
lm: no version for “struct_module” found: kernel tainted.
dwc_otg: version 2.72a 24-JUN-2008
DWC_otg: Core Release: 2.66a
DWC_otg: Periodic Transfer Interrupt Enhancement – disabled
DWC_otg: Multiprocessor Interrupt Enhancement – disabled
DWC_otg: Using DMA mode
DWC_otg: Device using Buffer DMA mode
dwc_otg lm0: DWC OTG Controller
drivers/usb/core/inode.c: creating file ‘devices’
drivers/usb/core/inode.c: creating file ’001′
dwc_otg lm0: new USB bus registered, assigned bus number 1
dwc_otg lm0: irq 18, io mem 0×00000000
DWC_otg: Init: Port Power? op_state=1
DWC_otg: Init: Power Port (0)
usb usb1: default language 0×0409
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: DWC OTG Controller
usb usb1: Manufacturer: Linux 2.6.21 dwc_otg_hcd
usb usb1: SerialNumber: lm0
usb usb1: uevent
usb usb1: usb_probe_device
usb usb1: configuration #1 chosen from 1 choice
usb usb1: adding 1-0:1.0 (config #1, interface 0)
usb 1-0:1.0: uevent
hub 1-0:1.0: usb_probe_interface
hub 1-0:1.0: usb_probe_interface – got id
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
hub 1-0:1.0: standalone hub
hub 1-0:1.0: ganged power switching
hub 1-0:1.0: individual port over-current protection
hub 1-0:1.0: Single TT
hub 1-0:1.0: TT requires at most 8 FS bit times (666 ns)
hub 1-0:1.0: power on to power good time: 2ms
hub 1-0:1.0: local power source is good
hub 1-0:1.0: enabling power on all ports
hub 1-0:1.0: state 7 ports 1 chg 0000 evt 0000
drivers/usb/core/inode.c: creating file ’001′
usb usb1: HUB Connect Vendor ID: 0×0000
usb usb1: HUB Connect Product ID: 0×0000

phy_tx_ring = 0x018c6000, tx_ring = 0xa18c6000, size: 16 bytes

phy_rx_ring = 0x018c7000, rx_ring = 0xa18c7000, size: 16 bytes
GDMA1_FWD_CFG = 10000
rt2860v2_ap: module license ‘unspecified’ taints kernel.
2860 version : 2.0.0.0 (Aug 19 2009)

=== pAd = c1097000, size = 485360 ===

RTMPAllocAdapterBlock, Status=0
RX DESC a1ec2000  size = 2048
RTMPAllocTxRxRingMemory, Status=0
1. Phy Mode = 9
2. Phy Mode = 9
3. Phy Mode = 9
MCS Set = ff ff 00 00 01
Main bssid = 00:30:0a:f5:05:b7
The UUID Hex string is:2880288028801880a88000300af505b7
The UUID ASCII string is:28802880-2880-1880-a880-00300af505b7!
== RTMPInitialize, Status=0
0×1300 = 00064380
ra2880stop()…Done
Free TX/RX Ring Memory!
eth2.2: Setting MAC address to  00 30 0a f5 05 b8.
device eth2 entered promiscuous mode
VLAN (eth2.2):  Setting underlying device (eth2) to promiscious mode.

phy_tx_ring = 0x01a43000, tx_ring = 0xa1a43000, size: 16 bytes

phy_rx_ring = 0x01a44000, rx_ring = 0xa1a44000, size: 16 bytes
GDMA1_FWD_CFG = 10000
eth2.1: add 33:33:00:00:00:01 mcast address to master interface
eth2.1: add 01:00:5e:00:00:01 mcast address to master interface
eth2.1: add 33:33:ff:f5:05:b6 mcast address to master interface
eth2.2: add 33:33:00:00:00:01 mcast address to master interface
eth2.2: add 01:00:5e:00:00:01 mcast address to master interface
eth2.2: add 33:33:ff:f5:05:b8 mcast address to master interface
device ra0 entered promiscuous mode
eth2.1: dev_set_promiscuity(master, 1)
device eth2.1 entered promiscuous mode
br0: port 2(eth2.1) entering learning state
br0: port 1(ra0) entering learning state
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ra0: no IPv6 routers present
eth2: no IPv6 routers present
eth2.2: no IPv6 routers present
eth2.1: no IPv6 routers present
br0: no IPv6 routers present
br0: topology change detected, propagating
br0: port 2(eth2.1) entering forwarding state
br0: topology change detected, propagating
br0: port 1(ra0) entering forwarding state
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0
ip_table: set wan_name=ppp0