Knowledge Base & Discussion Forum

UBS server becomes unresponsive

Discuss technical questions on AhsayUBS

Moderator: Support Team

UBS server becomes unresponsive

Postby Rylos » Tue Aug 11, 2015 7:23 pm

I have 3 UBS servers currently, each with 27 clients on them. I have one machine that randomly stops responding to web requests and backups begin to fail to run. All 3 are setup with exactly the same specifications, aside from the amount of disks configured. All have disks located on datastores in the following configuration:
1. Main disk - 40GB - Datastore1 which is the local raid 10 array in the VM host server - OS only. No client data
2. Main client files - 750GB - Datastore2 which is a DELL DAS attached to the VM Host server
3. Large data client files - 2TB - Each UBS has it's own VM Host datastore for these with separate iSCSI luns. These are connected over 10Gbit directly connected to a Synology NAS.

The UBS failing has an additional 75GB disk connected via the VM host iSCSI datastore located on our network which is using the Starwind iSCSI application. I tend to lean at this being the culprit.

When I connected to the machine today via SSH I went to the ubs mnt for the unique datastore and did an ls. The machine froze and would not list any files. I knew then it was completely hosed and did a restart. When that happens the obs process fails to completely stop. It does the final syncs then stalls and never shuts down, forcing me to do a reset from the VM host. What files can I look in on the UBS to try to get an idea of where the problem exists?
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Wed Aug 12, 2015 10:14 am

Hello,

We require some further information about your AhsayUBS setup and the processes that were active at the time when it froze to help us narrow down the cause of the problem.

-Can tell us which AhsayUBS version is installed on the problem server?
-Is it a 32 or 64 bit machine?
-How much RAM is installed your AhsayUBS servers?
-Are your AhsayUBS user home drives configured using ZFS file system?
-The AhsayUBS server is configured with 2TB + 750GB of user home storage. How much of the disk space is utilised?
-You mentioned the other two AhsayUBS server have the same specification apart from disk setup. How large are the user home drives? How much of the disk space is utilised?
-Is replication is enabled?
-When the machine froze where there any disk I/O intensive operations running either by FreeBSD or AhsayOBS,i.e. ZFS scrub, bi-monthly CRC check, weekly/single user rebuild, retention policy, or copying of large seed load to the user home drive?

Can you check the AhsayUBS server for possible disk related issues by either:

1. Logging to your AhsayUBS web admin console under [Information] > [System log] for any errors logs when the machine froze? Do you mind posting the error/warning messages on this thread?

or

2. Log via ssh as root and run the "dmesg" command. Do you mind posting the results on this thread?
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Wed Aug 12, 2015 8:27 pm

I'll post what I can now. Some information may have to wait until another instance of this happens.
System Status
Name Ahsay Universal Backup System
Version 2.23.2.0 (built on Mon Feb 16 12:52:09 EST 2015)
OS Version FreeBSD 8.3-RELEASE (revision 199506)
Platform amd64 on Intel(R) Xeon(R) CPU E5405 @ 2.00GHz x 2 (1996MHz)

4096MB used of 4096MB4096MB available of 4096MB 0% of 4096MB
Device: /dev/label/B3625F3Axd00p03 | Total: 4096M | Used: 4096M | Free: 4096M
Disk space usage
/ubs/mnt/eslsfw
1.5GB used of 33GB32GB available of 33GB 4% of 33GB
Total: 33G | Used: 1.5G | Free: 32G
/ubs/mnt/esms00
63GB used of 736GB672GB available of 736GB 9% of 736GB
Total: 736G | Used: 63G | Free: 672G
/ubs/mnt/esms01
40GB used of 70GB30GB available of 70GB 57% of 70GB
Total: 70G | Used: 40G | Free: 30G
/ubs/mnt/esms02
425GB used of 2TB1.6TB available of 2TB 21% of 2TB
Total: 2T | Used: 425G | Free: 1.6T

64bit
2GB RAM

File systems. I let it do the default setup. All of my user homes are zfs.
xasdasd:~# df -T
Filesystem Type 1K-blocks Used Avail Capacity Mounted on
/dev/md0 ufs 126702 98634 28068 78% /
devfs devfs 1 1 0 100% /dev
/dev/mirror/B3625F3Axesosfw ufs 190252 85202 89830 49% /ubs/mnt/esosfw
/dev/mirror/B3625F3Axesfmfw ufs 756748 330008 366202 47% /ubs/mnt/esfmfw
eslsfwxB3625F3A zfs 35348703 1540970 33807733 4% /ubs/mnt/eslsfw
/dev/md1 ufs 15598 720 13632 5% /var
esms00xB3625F3A zfs 772014258 67010920 705003338 9% /ubs/mnt/esms00
esms01xB3625F3A zfs 74314335 42406192 31908142 57% /ubs/mnt/esms01
esms02xB3625F3A zfs 2097335079 445805636 1651529442 21% /ubs/mnt/esms02


Disk usage on the other servers:
Disk space usage
/ubs/mnt/eslsfw
3.6GB used of 33GB30GB available of 33GB 11% of 33GB
Total: 33G | Used: 3.6G | Free: 30G
/ubs/mnt/esms00
54GB used of 736GB682GB available of 736GB 7% of 736GB
Total: 736G | Used: 54G | Free: 682G
/ubs/mnt/esms01
631GB used of 2TB1.3TB available of 2TB 32% of 2TB

and

sk space usage
/ubs/mnt/eslsfw
2GB used of 72GB70GB available of 72GB 3% of 72GB
Total: 72G | Used: 2G | Free: 70G
/ubs/mnt/esms00
101GB used of 736GB634GB available of 736GB 14% of 736GB
Total: 736G | Used: 101G | Free: 634G
/ubs/mnt/esms01
137MB used of 4.9TB4.9TB available of 4.9TB 0% of 4.9TB
Total: 4.9T | Used: 137M | Free: 4.9T


Replication is enabled but has been suspended as of yesterday to facilitate faster restores. The replication is very bandwidth intensive unlike our previous backup solution.

current dmesg output. I will run this again if the problem happens again.

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE #1: Wed Mar 13 14:39:42 HKT 2013
root@ubs-deployment:/ubs/obj/amd64/ubs/src/amd64/sys/UBS amd64
module_register: module vesa already exists!
Module vesa failed to register: 17
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (1996.22-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Family = 6 Model = 17 Stepping = 6
Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Features2=0x80082201<SSE3,SSSE3,CX16,SSE4.1,HV>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 2147483648 (2048 MB)
avail memory = 1917972480 (1829 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <INTEL 440BX> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata0: [ITHREAD]
ata1: <ATA channel> at channel 1 on atapci0
ata1: [ITHREAD]
pci0: <bridge> at device 7.3 (no driver attached)
pci0: <base peripheral> at device 7.7 (no driver attached)
vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xec000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
mpt0: [ITHREAD]
mpt0: MPI Version=1.2.0.0
pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.0.4> port 0x2000-0x203f mem 0xfd5c0000-0xfd5dffff,0xfdff0000-0xfdffffff irq 18 at device 0.0 on pci2
em0: Memory Access and/or Bus Master bits were not set!
em0: [FILTER]
em0: Ethernet address: 00:0c:29:16:23:3d
pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 22.1 on pci0
pci12: <ACPI PCI bus> on pcib12
pcib13: <ACPI PCI-PCI bridge> at device 22.2 on pci0
pci13: <ACPI PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 22.3 on pci0
pci14: <ACPI PCI bus> on pcib14
pcib15: <ACPI PCI-PCI bridge> at device 22.4 on pci0
pci15: <ACPI PCI bus> on pcib15
pcib16: <ACPI PCI-PCI bridge> at device 22.5 on pci0
pci16: <ACPI PCI bus> on pcib16
pcib17: <ACPI PCI-PCI bridge> at device 22.6 on pci0
pci17: <ACPI PCI bus> on pcib17
pcib18: <ACPI PCI-PCI bridge> at device 22.7 on pci0
pci18: <ACPI PCI bus> on pcib18
pcib19: <ACPI PCI-PCI bridge> at device 23.0 on pci0
pci19: <ACPI PCI bus> on pcib19
pcib20: <ACPI PCI-PCI bridge> at device 23.1 on pci0
pci20: <ACPI PCI bus> on pcib20
pcib21: <ACPI PCI-PCI bridge> at device 23.2 on pci0
pci21: <ACPI PCI bus> on pcib21
pcib22: <ACPI PCI-PCI bridge> at device 23.3 on pci0
pci22: <ACPI PCI bus> on pcib22
pcib23: <ACPI PCI-PCI bridge> at device 23.4 on pci0
pci23: <ACPI PCI bus> on pcib23
pcib24: <ACPI PCI-PCI bridge> at device 23.5 on pci0
pci24: <ACPI PCI bus> on pcib24
pcib25: <ACPI PCI-PCI bridge> at device 23.6 on pci0
pci25: <ACPI PCI bus> on pcib25
pcib26: <ACPI PCI-PCI bridge> at device 23.7 on pci0
pci26: <ACPI PCI bus> on pcib26
pcib27: <ACPI PCI-PCI bridge> at device 24.0 on pci0
pci27: <ACPI PCI bus> on pcib27
pcib28: <ACPI PCI-PCI bridge> at device 24.1 on pci0
pci28: <ACPI PCI bus> on pcib28
pcib29: <ACPI PCI-PCI bridge> at device 24.2 on pci0
pci29: <ACPI PCI bus> on pcib29
pcib30: <ACPI PCI-PCI bridge> at device 24.3 on pci0
pci30: <ACPI PCI bus> on pcib30
pcib31: <ACPI PCI-PCI bridge> at device 24.4 on pci0
pci31: <ACPI PCI bus> on pcib31
pcib32: <ACPI PCI-PCI bridge> at device 24.5 on pci0
pci32: <ACPI PCI bus> on pcib32
pcib33: <ACPI PCI-PCI bridge> at device 24.6 on pci0
pci33: <ACPI PCI bus> on pcib33
pcib34: <ACPI PCI-PCI bridge> at device 24.7 on pci0
pci34: <ACPI PCI bus> on pcib34
acpi_acad0: <AC Adapter> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
Timecounters tick every 1.000 msec
GEOM_RAID5: registered shutdown event handler.
md0: Preloaded image </boot/mfsroot> 134217728 bytes at 0xffffffff80fcf078
acd0: DVDR <VMware Virtual IDE CDROM Drive/00000001> at ata1-master UDMA33
da0 at mpt0 bus 0 scbus0 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da0: Command Queueing enabled
da0: 40960MB (83886080 512 byte sectors: 255H 63S/T 5221C)
da1 at mpt0 bus 0 scbus0 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da1: Command Queueing enabled
da1: 768000MB (1572864000 512 byte sectors: 255H 63S/T 97906C)
da2 at mpt0 bus 0 scbus0 target 2 lun 0
da2: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da2: Command Queueing enabled
da2: 74752MB (153092096 512 byte sectors: 255H 63S/T 9529C)
da3 at mpt0 bus 0 scbus0 target 3 lun 0
da3: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
da3: Command Queueing enabled
da3: 2095104MB (4290772992 512 byte sectors: 255H 63S/T 267088C)
SMP: AP CPU #1 Launched!
GEOM_MIRROR: Device mirror/B3625F3Axesgpbt launched (1/1).
GEOM_MIRROR: Device mirror/B3625F3Axesosfw launched (1/1).
GEOM_MIRROR: Device mirror/B3625F3Axesfmfw launched (1/1).
Trying to mount root from ufs:/dev/md0
ZFS filesystem version 5
ZFS storage pool version 28
iscsi: version 2.2.4.2
VMware memory control driver initialized
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Thu Aug 13, 2015 10:05 am

Hello

Thanks for getting back to us.

The output from the dmesg does not indicate any disk related issues but you may want to check the logs on the Synology NAS web admin console just be on the safe side.

You mentioned your AhsayUBS server is installed with 2GB RAM, the recommended requirement is at least 4GB RAM especially if your are using ZFS file system.

As a general rule the amount memory required by the ZFS file system increases in proportion to the number of files and disk I/O activity on the AhsayUBS server. As replication was also enabled on this machine this will add extra disk I/O activity on top of existing backup/restore/AhsayOBs system jobs such as CRC check, weekly/single rebuild, retention policy, and delta merge etc.

If your AhsayUBS server has insufficient memory this will cause FreeBSD to freeze when there is high disk I/O activity, which is consistent with what you have reported. To prevent this problem from re-occurring we recommend you increase the RAM on the server to 4GB or above. (More memory will also improve the overall performance of ZFS file system)
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Thu Aug 13, 2015 6:57 pm

Thanks, I had wondered if this may be the issue. Early on in our implementation I had asked for a recommended specifications whitepaper but wasn't given anything. Is there something in the manual I missed that lays out things such as this?
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Thu Aug 13, 2015 7:08 pm

Hello

The AhsayUBS system requirements is mentioned in Chapter 4.1 Backup System Requirements of our Ahsay Universal Backup System Administrator’s Guide https://forum.ahsay.com/viewtopic.php?f=94&t=9791
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Thu Aug 13, 2015 7:13 pm

Added memory to one server without issue. Went to the problem server to shut it down and add memory. It stalled shutting down the OBSR process just as if it were in the broken state I initially asked about. In the end, I left it in the state of the screenshot I've included and then powered it down, added the extra memory, then started it again.

https://www.dropbox.com/s/hrpm6hr5q9sxon7/all.jpg?dl=0
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Fri Aug 14, 2015 2:00 pm

Hello

One reason why your AhsayOBS service is not responding within a timely manner to a service shutdown request could be due to the presence of active backup jobs which are still uploading data. AhsayOBS will need to commit any backup data currently in memory to disk before the service finally shuts down.
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Fri Aug 14, 2015 6:46 pm

First thing I did before shutting down was verify there were no active backups under "Manage Log" ==> "Backup Log". I don't have any jobs on that server that would be running during that time of day on a weekday. This server is the only one of the three I have that gives me problems.
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Rylos » Fri Aug 14, 2015 11:04 pm

System failed to send the daily usage report and I noticed this in the logs:

475 09:51:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Starting system cleanup
476 09:51:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Start: System Cleanup
477 09:51:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]Start: System Cleanup: Removing obsolete pending backup job report
478 09:51:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]End: System Cleanup: Removing obsolete pending backup job report
479 09:51:00 AM -- -- [info][system][Thread][Job][SystemCleanup]End: System Cleanup
480 09:55:00 AM -- -- [info][system][Thread][Job][UsageReport]Starting usage report
482 09:55:52 AM -- -- [error][system]Failed to obtain amount of committed virtual memory
483 09:55:52 AM -- -- errno: 2 error: Unable to open /proc/curproc/map
484 09:56:00 AM -- -- [info][system][Thread][Job][BackupJobReminderReport]Starting missed backup reminder report
485 10:02:47 AM -- -- [error][system]Failed to obtain amount of committed virtual memory
486 10:02:47 AM -- -- errno: 2 error: Unable to open /proc/curproc/map
487 10:04:25 AM -- -- [error][system]Failed to obtain amount of committed virtual memory
488 10:04:25 AM -- -- errno: 2 error: Unable to open /proc/curproc/map
489 10:06:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Starting system cleanup
490 10:06:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Start: System Cleanup
491 10:06:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]Start: System Cleanup: Removing obsolete pending backup job report
492 10:06:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]End: System Cleanup: Removing obsolete pending backup job report
493 10:06:00 AM -- -- [info][system][Thread][Job][SystemCleanup]End: System Cleanup
494 10:08:10 AM -- -- [error][system]Failed to obtain amount of committed virtual memory
495 10:08:10 AM -- -- errno: 2 error: Unable to open /proc/curproc/map
496 10:21:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Starting system cleanup
497 10:21:00 AM -- -- [info][system][Thread][Job][SystemCleanup]Start: System Cleanup
498 10:21:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]Start: System Cleanup: Removing obsolete pending backup job report
499 10:21:00 AM -- -- [info][system][Thread][Job][SystemCleanup][PendingEmailCacheCleaning]End: System Cleanup: Removing obsolete pending backup job report
500 10:21:00 AM -- -- [info][system][Thread][Job][SystemCleanup]End: System Cleanup
Last edited by Rylos on Fri Aug 14, 2015 11:28 pm, edited 2 times in total.
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Rylos » Fri Aug 14, 2015 11:13 pm

Trying to login to the admin page on port 8080 and it's stalling as well. This is exactly what I see with this system. The main interface works mostly, I can manage jobs etc, but it will slowly decline until it is unusable. Right now it stalls on software license screen when I click on that option. As I was typing this, the manage log => backup log is now stalling as well. There has to be some way to determine what is going on.
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Mon Aug 17, 2015 12:22 pm

Hello,

Thanks for getting back to us.

You can safely ignore the following errors on your AhsayOBS system log. They are known issues related to the [Manage System]>[System Diagnosis] page on your AhsayOBS web console and not related to your current problem.

[error][system]Failed to obtain amount of committed virtual memory
errno: 2 error: Unable to open /proc/curproc/map


-You mentioned that extra RAM has been added can you tell us the total amount of RAM installed on this machine?
-Did you get an opportunity to check the logs on the Synology NAS web admin console for any disk related errors as suggested previously?
-Is the Synology NAS a dedicated storage device for that particular AhsayUBS server or it is shared amongst other AhsayUBS devices?
-Do mind providing us with the Synology NAS model number?
-At the time you mentioned the AhsayUBS/AhsayOBS web console was "stalling". How many backup/restore jobs were active? Did the backup/restore jobs "stall" as well? Were there any routine system jobs running on AhsayOBS,i.e. retention policy, single/weekly user rebuild, delta merge job, deleting of a large number of files from a backup set etc

The easiest way to determine if the problem is related to AhsayUBS setup is to either:

1. Login to your AhsayUBS web admin console under [Information] > [System log] for any errors logs when the machine froze?
or
2. Log via ssh as root and run the "dmesg" command.
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Mon Aug 17, 2015 7:34 pm

-This system now has 4GB of ram.
-Synology shows no storage errors at all.
-The Synology is shared among the UBS systems but all have their own dedicated datastores in VMware using their own iSCSI LUNS. The disks in the synology are all in one large array though, so the disks are shared between datastores.
-The synology is a RS3412xs
-When this was stalling there were no backups or restores running. I had just rebooted it since it was failing to send the status emails and it froze again within 5 minutes of boot up.

The dmesg output was the same as before. I've included a dropbox download of the entire days worth of system logs from the 14th when this happened along with some of the output from the 13th to view events leading up to the freeze.
https://www.dropbox.com/s/00du0nf7hwec5 ... g.txt?dl=0

The system is running properly at the moment so this is definitely a random issue.
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Re: UBS server becomes unresponsive

Postby Support2 » Tue Aug 18, 2015 3:49 pm

Hello,

The dmesg output does not show any possible errors which may cause your AhsayUBS server to stall.

-To rule our problems on AhsayOBS service do you mind uploading your OBS_context logs for August 14th to Dropbox?

As the Synology NAS is shared by other AhsayUBS servers, at the time the AhsayUBS/AhsayOBS web console was "stalling". Where there a many backup/restore system jobs running on there other AhsayUBS servers connected to the Synology NAS?

You previously mentioned there is a 75GB disk connect via iSCSI using Starwind iSCSI application. You suspected this was the culprit.

-Are the other AhsayUBS server connecting to the Starwind iSCSI application?
-Is the 75GB drive still in use?
-If it is not in use can you disconnect it from this machine?
-If it is in use and there is client data on the drive. Is it possible to migrate it to the Synology NAS?
User avatar
Support2
 
Posts: 390
Joined: Thu Oct 18, 2007 5:53 pm

Re: UBS server becomes unresponsive

Postby Rylos » Tue Aug 18, 2015 8:31 pm

-I will include a link at the bottom to the context logs. It is an encrypted zip file. The password is AhsayBackup12#*

-There were not any active backups or restores on any of the systems. Our busy time starts at 8PM and ends before 5AM the next morning.

-No other servers are using this disk
-Yes, it's still in use
-I'm going to suggest to the customer who uses this drive that we need to move it off of this drive and onto the synology.


https://www.dropbox.com/s/s0oxc7djovts1 ... 4.zip?dl=0
Rylos
 
Posts: 36
Joined: Tue Dec 30, 2014 8:34 pm

Next

Return to AhsayUBS

Who is online

Users browsing this forum: No registered users and 1 guest

Looking for Rbackup Alternative | Vembu Alternative | Novastor Alternative | Asigra Alternative | BackupAgent Alternative? Try our product.


A wholly owned subsidiary of Ahsay Backup Software Development Company Limited  [HKEx Stock Code: 8290]