united states of america usa  linkedinfacebookgoogle
Software RAID check - slow system issues

If you are unlucky enough to have a software raid running on your Linux server, you've probably noticed the occasional slowing down of the system. This is because of periodic redundancy checks of MD devices.

It is a cronjob that runs weekly and scans the MD devices for consistency. This can have a serious effect on the system, especially if you are running IO intensive virtual machines. (Such as Microsoft Exchange or SQL). 

One way to solve this problem is to lower the dev.raid.speed_limit_max. By default, it is set to this:

 

dev.raid.speed_limit_min = 1000
dev.raid.speed_limit_max = 200000

(Speed is no less than 1000K/sec and no more than 200000K/sec) 

The dev.raid.speed_limit_max is what you are interested in. This is the value that should be adjusted on a per-system basis. If you set it too low, your system will constantly be checking the raid (2052 minutes until done, given that this runs once per week, 2 of those days it will be checking the raid):

cat /proc/mdstat

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 sda2[0] sdb2[1]
1945121728 blocks [2/2] [UU]
[======>..............] check = 30.1% (586144768/1945121728) finish=2052.8min speed=11032K/sec
bitmap: 4/15 pages [16KB], 65536KB chunk
unused devices: <none>

If you set it too high, it will finish faster, but will cause some of your processes to hang or crash. 

To make the changes permanently, set them up in /etc/sysctl.conf (add these variables). We've set ours to max 10000. Yes, it will take 2 days to run, but at least the perfomance impact is not significant. 

dev.raid.speed_limit_min = 1000
dev.raid.speed_limit_max = 10000

To make the changes without rebooting:

sysctl -w dev.raid.speed_limit_max=10000

Test them out and set it to whatever works for you. You can also change these during runtime, so if it the server load is not high and you want the check to finish sooner, you can set it to a higher number

ALT Consulting is an IT Consulting and Support company, with offices in the US and Canada, providing Linux and Windows support for businesses and organizations. Contact us, if you are looking for reliable and experienced system support specialists.

 

Tel: (786) 646-6461   |   Toll-Free: (877) 454-0758
1990 NE 163rd St, Suite 215, North Miami Beach, FL 33162, USA
Privacy Policy